Multimodal data extraction and analysis for the implementation of Temporal Action Segmentation models in Manufacturing

With Industry 5.0, operators' physical and cognitive behavior is crucial in any field, particularly manufacturing and production lines. To this end, the need to monitor humans performing specific tasks when working alongside robotic systems has grown further. Guaranteeing the well-being of operators sharing a workspace with industrial robots can drastically reduce risky and harmful situations for the operators while leading the robot to adapt to humans fully. For this purpose, monitoring systems can be very helpful in studying the most suitable deep learning methodologies to obtain information from the movements of the individual operator performing a specific task. Therefore, action segmentation can be fundamental to establishing a new and safe way of communication among operators and robots. In this work, a system for segmenting actions performed by operators assembling an industrial object is developed. The public HA4M dataset has been used to train and test temporal action segmentation models by using MS-TCN++ architecture. Highly discriminant features have been extracted from the dataset, and different training approaches based on multimodal data, including RGB and skeletal joints in Depth and RGB resolutions, have been considered. Results show the effectiveness of the proposed system, laying the foundation for further studies for detecting the operators' actions in the challenging context of Human-Robot Interaction and Collaboration.