Evaluating Mixed-Initiative Systems: An Experimental Approach

Mixed-Initiative approaches to Planning and Scheduling are being applied in different real world domains. While several recent successful examples of such tools encourage a wider use of this solving paradigm, research in mixed-initiative interaction is still at an early stage and many important issues need to be addressed. In particular, most of the work has been dedicated to designing working prototypes and identifying relevant features of the mixed-initiative interaction, while much less attention has been given to the problem of evaluating the approach as a whole. This article is aimed at addressing some of the many diverse aspects involved in Mixed-Initiative Planning and Scheduling system evaluation, highlighting the need for a methodology to provide effective evaluation studies for this class of tools. In this paper we consider an established research methodology in experimental psychology, and adopt it to investigate specific aspects of mixed-initiative interaction. Specifically, the experiments described in this article shed some light on three aspects: (a) understanding users' attitude when choosing between automated and mixed-initiative problem solving, (b) investigating recourse to explanation services as a means to foster the users' involvement in the solving process, and (c) investigating possible individual differences (e.g., experts vs. non-experts) in the choice of resolution strategy or access to explanation.