Blendshape features meet action units: a clinical mapping for enhancing facial expression analysis

Facial Expression Analysis (FEA) is a computational approach, crucial for understanding human emotions and mental states, with applications in healthcare, social robotics, and driver-state monitoring. Traditional FEA techniques often rely on Action Units (AUs) and the Facial Action Coding System (FACS), which provide a structured framework but may miss subtle micro-expressions and can become ambiguous in complex or rapidly changing emotional states. Blendshape Features (BFs), originally developed for computer graphics, offer a continuous, high-resolution alternative. This study offers two key contributions: (i) a systematic review of AUs and BFs in FEA research, with a focus on their roles in human-centered applications, and (ii) an expert-validated mapping procedure that links BFs to AUs, integrating the descriptive precision of BFs and the interpretive structure of AUs. The mapping was developed through an independent annotation and consensus process involving ten licensed clinical psychologists and psychotherapists with expertise in analyzing nonverbal behavior. Overall, 88% of mappings reached unanimous agreement among the experts, while 98% were supported by a majority (i.e., 6/10). To our knowledge, this represents one of the first publicly documented efforts toward a standardized mapping between MediaPipe’s 52 blendshape coefficients and AUs. This resource provides a milestone for advancing FEA toward greater expressivity, scalability, and psychological interpretability, with direct implications for behavioral science, clinical diagnostics, humancomputer interaction, and mental health assessment.