We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation.The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systemsreached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.
SemEval-2022 Task 3: PreTENS - Evaluating Neural Networks on Presuppositional Semantic Knowledge
Dominique Brunato;Felice Dell'Orletta;Giulia Venturi
2022
Abstract
We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation.The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systemsreached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.| Campo DC | Valore | Lingua |
|---|---|---|
| dc.authority.orgunit | Istituto di linguistica computazionale "Antonio Zampolli" - ILC | en |
| dc.authority.people | Roberto Zamparelli | en |
| dc.authority.people | Shammur A Chowdhury | en |
| dc.authority.people | Dominique Brunato | en |
| dc.authority.people | Cristiano Chesi | en |
| dc.authority.people | Felice Dell'Orletta | en |
| dc.authority.people | Arid Hasan | en |
| dc.authority.people | Giulia Venturi | en |
| dc.collection.id.s | 71c7200a-7c5f-4e83-8d57-d3d2ba88f40d | * |
| dc.collection.name | 04.01 Contributo in Atti di convegno | * |
| dc.contributor.appartenenza | Istituto di linguistica computazionale "Antonio Zampolli" - ILC | * |
| dc.contributor.appartenenza.mi | 918 | * |
| dc.date.accessioned | 2024/02/19 17:55:59 | - |
| dc.date.available | 2024/02/19 17:55:59 | - |
| dc.date.firstsubmission | 2024/12/16 13:05:08 | * |
| dc.date.issued | 2022 | - |
| dc.date.submission | 2024/12/16 13:05:08 | * |
| dc.description.abstracteng | We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation.The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systemsreached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages. | - |
| dc.description.affiliations | CIMeC, University of Trento, Rovereto Italy; Qatar Computing Research Institute, HBKU, Qatar; ILC-CNR, Pisa, Italy; NETS-IUSS, Pavia, Italy; 5Daffodil International University, Dhaka, Bangladesh | - |
| dc.description.allpeople | Zamparelli, Roberto; A Chowdhury, Shammur; Brunato, Dominique; Chesi, Cristiano; Dell'Orletta, Felice; Hasan, Arid; Venturi, Giulia | - |
| dc.description.allpeopleoriginal | Roberto Zamparelli, Shammur A Chowdhury, Dominique Brunato, Cristiano Chesi, Felice Dell'Orletta, Arid Hasan, Giulia Venturi | en |
| dc.description.fulltext | open | en |
| dc.description.numberofauthors | 7 | - |
| dc.identifier.uri | https://hdl.handle.net/20.500.14243/448002 | - |
| dc.identifier.url | https://aclanthology.org/2022.semeval-1.29.pdf | en |
| dc.language.iso | eng | en |
| dc.miur.last.status.update | 2024-07-22T14:33:13Z | * |
| dc.relation.conferencedate | 14-15/07/2022 | en |
| dc.relation.conferencename | 16th International Workshop on Semantic Evaluation (SemEval-2022) | en |
| dc.relation.conferenceplace | Seattle | en |
| dc.relation.firstpage | 228 | en |
| dc.relation.ispartofbook | 16th International Workshop on Semantic Evaluation (SemEval-2022) | en |
| dc.relation.lastpage | 238 | en |
| dc.relation.numberofpages | 10 | en |
| dc.subject.keywords | Neural Networks | - |
| dc.subject.keywords | Presuppositional Knowledge | - |
| dc.subject.keywords | Evaluation | - |
| dc.subject.singlekeyword | Neural Networks | * |
| dc.subject.singlekeyword | Presuppositional Knowledge | * |
| dc.subject.singlekeyword | Evaluation | * |
| dc.title | SemEval-2022 Task 3: PreTENS - Evaluating Neural Networks on Presuppositional Semantic Knowledge | en |
| dc.type.driver | info:eu-repo/semantics/conferenceObject | - |
| dc.type.full | 04 Contributo in convegno::04.01 Contributo in Atti di convegno | it |
| dc.type.miur | 273 | - |
| dc.type.referee | Sì, ma tipo non specificato | en |
| dc.ugov.descaux1 | 470081 | - |
| iris.mediafilter.data | 2025/04/16 03:33:10 | * |
| iris.orcid.lastModifiedDate | 2024/12/16 17:03:17 | * |
| iris.orcid.lastModifiedMillisecond | 1734364997383 | * |
| iris.scopus.extIssued | 2022 | - |
| iris.scopus.extTitle | SemEval-2022 Task 3: PreTENS - Evaluating Neural Networks on Presuppositional Semantic Knowledge | - |
| iris.sitodocente.maxattempts | 1 | - |
| Appare nelle tipologie: | 04.01 Contributo in Atti di convegno | |
File in questo prodotto:
| File | Dimensione | Formato | |
|---|---|---|---|
|
2022.semeval-1.29.pdf
accesso aperto
Licenza:
Creative commons
Dimensione
360.78 kB
Formato
Adobe PDF
|
360.78 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
