End-user automation authoring in VR and AR through conversations: patterns, errors and user recovery strategies

Automations defined for XR allow end-users to customise environments, but current menu-based interfaces pose challenges for end-user developers. Chatbots powered by Large Language Models (LLMs) could simplify the automation creation process, yet how users articulate their intentions remains underexplored. This work examines how end-users express event-condition-action (ECA) automations in both a Virtual Reality (VR) museum and an Augmented Reality (AR) smart home using a consistent set of tasks. Initially, we used a Wizard of Oz technique in a Wizard of Oz study to determine how users would communicate with a hypothetical agent, identifying dialogue patterns. Building on this, we tested a prototype LLM-based chatbot, revealing that conversational automation authoring exhibits a consistent, high-level dialogue structure across XR settings. Interactions with the LLM lead to more goal-oriented behaviours compared to dialogues with human agents. An analysis of errors shows that system errors were more likely to lead to cascading effects while users showed limited awareness of errors. These findings inform the design of chatbot support for expressing automations in XR environments.