2026-06-11T07:48:44Zhttps://iris.cnr.it/oai/request

oai:iris.cnr.it:20.500.14243/5704622026-03-04T01:27:55Zcom_20.500.14243_46com_20.500.14243_21col_20.500.14243_47ou_ou239

Evaluating Lexical Proficiency in Neural Language Models Ciaccio C. Miaschi A. Dell'Orletta F. Ciaccio, C. Miaschi, A. Dell'Orletta, F. Large Language Models (LLMs) Interpretability We present a novel evaluation framework designed to assess the lexical proficiency and linguistic creativity of Transformer-based Language Models (LMs). We validate the framework by analyzing the performance of a set of LMs of different sizes, in both mono- and multilingual configuration, across tasks involving the generation, definition, and contextual usage of lexicalized words, neologisms, and nonce words. To support these evaluations, we developed a novel dataset of lexical entries for the Italian language, including curated definitions and usage examples sourced from various online platforms. The results highlight the robustness and effectiveness of our framework in evaluating multiple dimensions of LMs' linguistic understanding and offer an insight, through the assessment of their linguistic creativity, on the lexical generalization abilities of LMs. 2025 info:eu-repo/semantics/conferenceObject https://hdl.handle.net/20.500.14243/570462 10.18653/v1/2025.acl-long.64 info:eu-repo/semantics/altIdentifier/scopus/2-s2.0-105021058451 eng ispartofbook:Proceedings of the Annual Meeting of the Association for Computational Linguistics 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 volume:1 firstpage:1267 lastpage:1286 numberofpages:20 serie:PROCEEDINGS OF THE CONFERENCE - ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. MEETING info:eu-repo/semantics/openAccess Association for Computational Linguistics (ACL) license:Creative commons license uri:http://creativecommons.org/licenses/by-nc-nd/4.0/