Using facial emojis and affective vocal bursts in emotional communication: an experimental study

In this paper, we present a study whose goal is to verify the effectiveness of 12 popular facial emojis as a means for self-assessment, and to consider whether vocal bursts can be used to reduce emojis’ ambiguities. To this extent, we developed a set of 12 multimedia emojis, in which pictograms were enriched with coherent vocalizations. We designed an online experiment based on images from the OASIS dataset, in which participants were asked to express their emotions using the multimedia emojis and rate their accuracy through a 5-point Likert scale. To evaluate the effects produced by the audio cues, each multimedia emoji was presented in both audio and silent versions. We gathered 65 participants from across Europe, thus collecting 1560 associations between images and emojis, and the corresponding accuracy ratings, equally divided between silent and sound mode. Analysis of the results showed that vocal bursts did not affect the accuracy of multimedia emojis, thus suggesting that the visual channel prevails over the audio one. Facial emojis proved an effective means of self-assessment, albeit ambiguities did persist, especially related to the arousal dimension, thus confirming previous findings from the literature.