2026-06-16T03:40:39Zhttps://iris.cnr.it/oai/request

oai:iris.cnr.it:20.500.14243/5707622026-03-04T01:28:06Zcom_20.500.14243_46com_20.500.14243_21col_20.500.14243_47ou_ou239

Doctor, Is That You? Evaluating Large Language Models on Italy’s Medical School Entrance Exams Piperno, Ruben Bonfigli, Agnese Dell'Orletta, Felice Pecchia, Leandro Merone, Mario Bacco Luca Piperno, Ruben Bonfigli, Agnese Dell'Orletta, Felice Pecchia, Leandro Merone, Mario Bacco, Luca Large Language Models (LLMs) Italian Medical Admission Test NLP in healthcare In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of linguistic and cognitive tasks. This study investigates whether such models can succeed in one of Europe’s most selective academic assessments: the Italian medical school entrance exam. We evaluate a wide selection of open-weights LLMs, ranging from natively Italian-pretrained models to multilingual and Italian-specialised variants, on a benchmark dataset comprising over 3,300 real-world exam questions across five knowledge domains. Our experiments systematically explore the impact of language-specific pretraining, model size, prompt formulation and instruction tuning on exam performance. Results show that large multilingual models, particularly the Gemma-2-9B family, consistently outperform all other systems, surpassing the official admission threshold under all prompting settings. In contrast, models trained exclusively on Italian data fail to reach this threshold, even with larger architectures or instruction tuning. Additional analyses reveal that high-performing models display lower positional bias and greater inter-model consistency. These findings suggest that cross-domain reasoning and multilingual pretraining are key to handling multi-disciplinary educational tasks. 2025 info:eu-repo/semantics/conferenceObject https://hdl.handle.net/20.500.14243/570762 eng ispartofbook:Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025) info:eu-repo/semantics/openAccess license:Creative commons license uri:http://creativecommons.org/licenses/by-nc-nd/4.0/