Designing adaptive AI assistance for block-based modelling: a Wizard of Oz study with domain experts

The increasing pervasiveness of software-intensive systems requires involving domain experts more directly in technological development. Visual models, expressed in semi-formal notations, can act as shared artefacts that support communication and collaboration between developers and domain experts. However, modelling with semi-formal notations can be challenging for novice modellers. This study presents an AI-infused, web-based modelling tool designed to support users in formalising domain knowledge without requiring advanced modelling skills. The tool features a block-based, domain-specific language that automatically transforms user-generated structures into semi-formal diagrams. AI-based functionalities include a diagram reader, contextual hints, natural-language instructions, and interaction logging. We evaluated the tool through a Wizard of Oz experiment with agronomists in digital agriculture, where participants completed an exploratory modelling task while interacting with AI assistance. Results reveal three key design implications: (i) adaptive AI support accommodating diverse modelling strategies, (ii) concise, actionable guidance delivered at moments of difficulty, and (iii) practice-oriented assistance that preserves user agency and supports learning-by-doing.