Designing an effective human-machine multimodal interaction environment requires addressing crucial issues such as the correct interpretation of complex user's input from different modal channels. In this context the use of corpora of multimodal sentences is very important because they allow integrating properties and linguistic knowledge which are not formalised in the grammar. This paper provides framework for dynamic multimodal corpora building that semi-automates the extraction of syntactic and semantic information from multimodal dialogues using both grammar inference and interpretation methodologies based on HMM. This method is based on a Multimodal Attribute Grammar and on an HMM-based approach to syntactically and semantically annotate new multimodal sentences. It allows for improving human-computer dialogue because the multimodal corpus evolves by adapting itself to the dynamic change of the human-computer interaction.
MCBF: Multimodal Corpora Building Framework
Caschera Maria Chiara;D'Ulizia Arianna;Ferri Fernando;Grifoni Patrizia
2016
Abstract
Designing an effective human-machine multimodal interaction environment requires addressing crucial issues such as the correct interpretation of complex user's input from different modal channels. In this context the use of corpora of multimodal sentences is very important because they allow integrating properties and linguistic knowledge which are not formalised in the grammar. This paper provides framework for dynamic multimodal corpora building that semi-automates the extraction of syntactic and semantic information from multimodal dialogues using both grammar inference and interpretation methodologies based on HMM. This method is based on a Multimodal Attribute Grammar and on an HMM-based approach to syntactically and semantically annotate new multimodal sentences. It allows for improving human-computer dialogue because the multimodal corpus evolves by adapting itself to the dynamic change of the human-computer interaction.File | Dimensione | Formato | |
---|---|---|---|
prod_367353-doc_157098.pdf
solo utenti autorizzati
Descrizione: MCBF: Multimodal Corpora Building Framework
Tipologia:
Versione Editoriale (PDF)
Dimensione
953.45 kB
Formato
Adobe PDF
|
953.45 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.