Sentiment analysis from textual to multimodal features in digital environments

When social networks actors are involved in the production, consumption and exchange of content and information by texts, images, audios, videos, they act in a shared digital environment that can be considered as a digital ecosystem. On the increasing size of produced data, an open issue is the understanding of the real sentiment and emotion from texts, but also from images, audios and videos. This issue is particularly relevant for monitoring and identifying critical situations and suspicious behaviours. This paper is an attempt to review and evaluate the various techniques used for sentiment and emotion analysis from text, audio and video, and to discuss the main challenges addressed in extracting sentiment from multimodal data. The paper concludes the discussion by proposing a method that combines a machine learning approach with a language-based formalization in order to extract sentiment from multimodal data formalized through a multimodal language.