Automatic detection of potentially ineffective verbal communication for training through simulation in neonatology

Training through simulation in neonatology relies on sophisticated simulation devices that give realistic feedback to trainees during simulated scenarios. It aims at training highly specialised medical teams in established operational skills, timely clinical manoeuvres, and successful synergy with other professionals. For effective teaching, it is essential to tailor simulation to trainees' emotional status and communication abilities (human factors), which in turn affect their interaction with the equipment, the environment, and the rest of the team. These factors are crucial to achieving optimal timing and cooperation during a clinical intervention, to the point that they can determine the success of a complex operation such as neonatal resuscitation. Ineffective teams perform in a slow and/or poorly coordinated way and therefore jeopardise positive outcomes. Expert trainers consider human factors as crucial as technical skills. In this context, new technology can help measure learning improvement by quantitatively analysing verbal communication within a medical team. For example, Artificial Intelligence models can work on audio recordings, and draw from extensive historical archives, to extract useful human-factor related information for the trainers. In this study, we present an automatic workflow that supports training through simulation in neonatology by automatically detecting dialogue segments of a simulation session with potentially ineffective communication between team members due to anger, stress, fear, or misunderstandings. Rather than working on audio transcriptions, the workflow analyses syllabic-scale (100-200 ms) spoken dialogue energy and intonation. It uses cluster analysis to identify potentially ineffective communication and extracts the most important related words after audio transcription. Performance is measured against a gold standard containing annotations of 79 minutes of audio recordings from neonatal simulations, in Italian, under different noise conditions (from 4.63 to 14.17 SNR). Our workflow achieves a detection accuracy of 64% and a fair agreement with the gold standard in a challenging context for a speech-processing system, where a commercial automatic speech recogniser reaches just a 9.37% sentence accuracy. The workflow also identifies viable words for trainers to conduct the debriefing session, and can be easily extended to other languages and applications in healthcare. We consider it a promising first step towards introducing new technology to support training through simulation centred on human factors.