2026-06-14T07:47:25Zhttps://iris.cnr.it/oai/request

oai:iris.cnr.it:20.500.14243/5705212026-03-04T01:27:43Zcom_20.500.14243_46com_20.500.14243_21col_20.500.14243_47ou_ou239

TEXT-CAKE: Challenging Language Models on Local Text Coherence Dini L. Brunato D. Dell'Orletta F. Caselli T. Dini, L. Brunato, D. Dell'Orletta, F. Caselli, T. Large Language Models (LLMs) Text Coherence We present a deep investigation of encoder-based Language Models (LMs) on their abilities to detect text coherence across four languages and four text genres using a new evaluation benchmark, TEXT-CAKE. We analyze both multilingual and monolingual LMs with varying architectures and parameters in different finetuning settings. Our findings demonstrate that identifying subtle perturbations that disrupt local coherence is still a challenging task. Furthermore, our results underline the importance of using diverse text genres during pre-training and of an optimal pre-traning objective and large vocabulary size. When controlling for other parameters, deep LMs (i.e., higher number of layers) have an advantage over shallow ones, even when the total number of parameters is smaller. 2025 info:eu-repo/semantics/conferenceObject https://hdl.handle.net/20.500.14243/570521 info:eu-repo/semantics/altIdentifier/scopus/2-s2.0-85218500743 eng ispartofbook:Proceedings - International Conference on Computational Linguistics, COLING 31st International Conference on Computational Linguistics, COLING 2025 firstpage:4384 lastpage:4398 numberofpages:15 serie:INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS info:eu-repo/semantics/openAccess Association for Computational Linguistics (ACL) license:Creative commons license uri:http://creativecommons.org/licenses/by-nc-nd/4.0/