This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence-level likelihood computed with BERT and the GPT-2's perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs.
What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity
Miaschi;Alessio;Brunato;Dominique;Dell'Orletta;Felice;Venturi;Giulia
2021
Abstract
This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence-level likelihood computed with BERT and the GPT-2's perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs.| Campo DC | Valore | Lingua |
|---|---|---|
| dc.authority.orgunit | Istituto di linguistica computazionale "Antonio Zampolli" - ILC | en |
| dc.authority.people | Miaschi | en |
| dc.authority.people | Alessio | en |
| dc.authority.people | Brunato | en |
| dc.authority.people | Dominique | en |
| dc.authority.people | Dell'Orletta | en |
| dc.authority.people | Felice | en |
| dc.authority.people | Venturi | en |
| dc.authority.people | Giulia | en |
| dc.collection.id.s | 71c7200a-7c5f-4e83-8d57-d3d2ba88f40d | * |
| dc.collection.name | 04.01 Contributo in Atti di convegno | * |
| dc.contributor.appartenenza | Istituto di linguistica computazionale "Antonio Zampolli" - ILC | * |
| dc.contributor.appartenenza.mi | 918 | * |
| dc.date.accessioned | 2024/02/20 15:08:20 | - |
| dc.date.available | 2024/02/20 15:08:20 | - |
| dc.date.firstsubmission | 2024/12/18 16:58:48 | * |
| dc.date.issued | 2021 | - |
| dc.date.submission | 2024/12/18 16:58:48 | * |
| dc.description.abstracteng | This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence-level likelihood computed with BERT and the GPT-2's perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs. | - |
| dc.description.affiliations | Università di Pisa; Istituto di Linguistica Computazionale (ILC-CNR) | - |
| dc.description.allpeople | Miaschi, Alessio; Miaschi, Alessio; Brunato, DOMINIQUE PIERINA; Brunato, DOMINIQUE PIERINA; Dell'Orletta, Felice; Dell'Orletta, Felice; Venturi, Giulia; Venturi, Giulia | - |
| dc.description.allpeopleoriginal | Miaschi, Alessio and Brunato, Dominique and Dell'Orletta, Felice and Venturi, Giulia | en |
| dc.description.fulltext | open | en |
| dc.description.numberofauthors | 8 | - |
| dc.identifier.isbn | 978-1-954085-30-5 | en |
| dc.identifier.uri | https://hdl.handle.net/20.500.14243/400474 | - |
| dc.identifier.url | https://www.aclweb.org/anthology/2021.deelio-1.5 | en |
| dc.language.iso | eng | en |
| dc.miur.last.status.update | 2024-12-18T16:10:15Z | * |
| dc.relation.conferencedate | 10/06/2021 | en |
| dc.relation.conferencename | 2nd Workshop on Knowledge Extraction and Integrationfor Deep Learning Architectures | en |
| dc.relation.firstpage | 40 | en |
| dc.relation.ispartofbook | Proceedings of the 2nd Workshop on Knowledge Extraction and Integrationfor Deep Learning Architectures | en |
| dc.relation.lastpage | 47 | en |
| dc.relation.numberofpages | 8 | en |
| dc.subject.keywords | nlp | - |
| dc.subject.keywords | interpretability | - |
| dc.subject.keywords | deep learning | - |
| dc.subject.singlekeyword | nlp | * |
| dc.subject.singlekeyword | interpretability | * |
| dc.subject.singlekeyword | deep learning | * |
| dc.title | What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity | en |
| dc.type.driver | info:eu-repo/semantics/conferenceObject | - |
| dc.type.full | 04 Contributo in convegno::04.01 Contributo in Atti di convegno | it |
| dc.type.miur | 273 | - |
| dc.ugov.descaux1 | 454441 | - |
| iris.mediafilter.data | 2025/04/15 04:23:28 | * |
| iris.orcid.lastModifiedDate | 2024/12/19 16:47:25 | * |
| iris.orcid.lastModifiedMillisecond | 1734623245281 | * |
| iris.scopus.extIssued | 2021 | - |
| iris.scopus.extTitle | What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity | - |
| iris.sitodocente.maxattempts | 1 | - |
| Appare nelle tipologie: | 04.01 Contributo in Atti di convegno | |
File in questo prodotto:
| File | Dimensione | Formato | |
|---|---|---|---|
|
2021.deelio-1.5.pdf
accesso aperto
Licenza:
Creative commons
Dimensione
619.04 kB
Formato
Adobe PDF
|
619.04 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
