Eliciting Metaknowledge in Large Language Models

Advances in Natural Language Processing led to the introduction of Large Language Models (LLMs), that have been found endowed of enriched capabilities and improved performance results when increased in size. Researcher from various disciples argue about whether or not, among the capabilities of LLMs, there is the one of using knowledge about knowledge - usually considered one of the antechambers of meta-cognition in cognitive agents - about a particular task in order to improve or self-correct previous errors. In this work we propose a novel fine-tuning approach, named EXAR, based on a multi-stage LLM fine-tuning leveraging past predictions from an early version of the same, aimed to inject metacognitive features for the task of Question-Answering. The conducted experiments on LLAMA-2-7B-CHAT showed promising improvements on the quality of the outcomes, due to LLM acquired ability to detect its own wrong predictions and forcing itself to repeat submissions, thorough a prompt designed to fix inadmissible predictions whenever are detected. Such detection is achieved by enquiring the same LLM acting as meta-validator, through another prompt specifically designed for such purpose.