Automatic prompt engineering: the case of requirements classification

Context and motivation: Large language models (LLMs) are increasingly used to address requirements engineering (RE) tasks, including trace-link recovery, legal compliance, model generation, and others. Question/problem: Most of the existing studies rely on static, non-adaptive prompting strategies that do not fully harness the models’ capabilities. Specifically, these studies overlook the potential of automatic prompting engineering (APE), a technique that allows LLMs to self-generate and fine-tune prompts to improve task performance. Principal ideas/results: This research preview aims to study the effectiveness of APE techniques in LLM-powered RE tasks. As a preliminary step, we perform a benchmarking study in which we compare APE techniques with more traditional prompting solutions for the task of requirements classification. Our results show that, on average and with some exceptions, APE outperforms the baselines. We outline research avenues, including the evaluation and tailoring of APE for other RE tasks, and considering the human-in-the-loop. Contribution: To the best of our knowledge, this is the first study to introduce APE in RE, paving the way for a deeper exploration of LLMs’ potential in this field.