Recently, a joint research team led by Professor Xiaolan Fu and Research Assistant Professor Feng Liu at the School of Psychology, Shanghai Jiao Tong University, collaborated with the team of Associate Professor Xuezhong Qian at the School of Artificial Intelligence and Computer Science, Jiangnan University. Their study, "Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognition," was formally accepted by IEEE Transactions on Affective Computing, a leading journal in affective computing.
Research at a Glance
For the first time, this study provides a quality benchmark for micro-expression datasets in affective computing, enabling quantitative assessment of human annotation bias introduced during dataset construction.
Research Overview
Micro-expressions are involuntary facial responses that can occur when people attempt to suppress or conceal their true emotions. They are extremely brief, typically lasting about 1/25 to 1/5 of a second, low in intensity, and localized to specific facial regions. These characteristics give micro-expressions important potential applications in clinical psychology, national security, and forensic investigation. Existing micro-expression datasets, however, can be affected by annotators' subjective judgments. The problem is particularly pronounced in cross-cultural settings, where annotation bias in keyframes such as Onset, Apex, and Offset can substantially limit further improvements in model performance.
Figure 1. Difference curves for individual samples obtained after computing frame-level differences in three datasets. Specifically, the L2 norm of inter-frame pixel differences is calculated frame by frame to quantify motion intensity. Its peak serves as a key objective indicator of changes in expression intensity. The manually annotated peak frame in each dataset is marked by a red dashed line, and the maximum difference identified by the differential calculation is highlighted by a dashed box.
Research Motivation
Current micro-expression recognition research faces two major challenges. Traditional manual annotation requires substantial expertise, and manual detection accuracy rarely exceeds 50% even after systematic training. Cross-cultural datasets also exhibit systematic differences in facial muscle movement patterns and expression habits, which can produce significant temporal offsets between annotated keyframes and the actual motion peak, introducing annotation noise that interferes with model learning. Effectively correcting this type of subjective annotation bias may help overcome performance bottlenecks in micro-expression recognition at the data level.
Research Contributions
The study proposes the Global Anti-Monotonic Differential Selection Strategy (GAMDSS) framework. Its main contributions are described below.
1. A pioneering annotation-bias correction paradigm. For the first time, the study systematically analyzes the sources and effects of subjective human annotation errors in micro-expression datasets. It introduces a new approach that improves system performance by refining annotation boundaries without modifying the model architecture, providing a general solution to temporal annotation alignment.
2. Dynamic frame reselection. Within the neighborhood of the original manually annotated keyframes, differential calculations dynamically search for the three frames with the most pronounced motion changes, corresponding to Onset, Apex, and Offset. This constructs a complete rising-and-falling spatio-temporal dynamic representation and effectively reduces subjective errors introduced by manual frame-by-frame comparison.
3. Parameter-efficient two-branch spatio-temporal modeling. A dual spatio-temporal unit with shared parameters extracts fine-grained features from the rising and falling phases of a micro-expression, while a knowledge-injected auxiliary loss strengthens the model's understanding of the complete evolution of the facial movement.
4. Cross-cultural dataset validation. Experiments on seven widely used micro-expression datasets, including CASME II, SAMM, 4DME, and CAS(ME)³, show that GAMDSS delivers particularly pronounced performance gains on cross-cultural datasets such as SAMM and 4DME. Quantitative analysis further confirms that Onset and Apex alone can capture the main variation in single-culture datasets, whereas cross-cultural datasets require modeling all three frames to correct systematic annotation bias.
Figure 2. Overall architecture of GAMDSS. (a) GAMDSS pipeline. The dynamic frame reselection mechanism reselects the three keyframes with the richest motion changes for each dataset. A two-branch spatio-temporal unit with shared parameters extracts features, and an auxiliary loss models the complete evolution of the micro-expression. (b) Spatio-temporal feature extraction and fusion. Swish activation layers are employed to enhance feature nonlinearity and improve optimization stability.
Research Innovations
The study achieves three major theoretical and technical breakthroughs.
1. Quantitative analysis of annotation bias. For the first time, the study establishes a quantitative evaluation framework for subjective errors in micro-expression keyframe annotation. By dynamically adjusting the search-range parameter λ, it systematically compares annotation quality between single-culture and cross-cultural datasets, providing a theoretical basis for dataset standardization.
2. Lightweight plug-and-play design. GAMDSS adds no model parameters and can be integrated into existing micro-expression recognition frameworks with only a few lines of code. It yields consistent improvements across backbones including ResNet, ConvNeXt, Swin Transformer, and RMT.
3. A new spatio-temporal dynamic modeling paradigm. The framework moves beyond the traditional focus on the Onset-to-Apex phase and, for the first time, incorporates falling-phase dynamics from Apex to Offset into the modeling framework, significantly improving the model's representation of the complete evolution of micro-expressions.
Research Assistant Professor Feng Liu is the first and corresponding author of the paper, and Professor Xiaolan Fu is a co-corresponding author. Bingyu Nan and Associate Professor Xuezhong Qian are co-authors. Bingyu Nan is a master's student jointly supervised by Feng Liu and Xuezhong Qian. The study was supported by the Key Special Project on "Proactive Health and Technological Responses to Population Aging" under the National Key Research and Development Program of China (No. 2024YFC3606801) and the Startup Fund for Young Faculty at SJTU (No. 25X010506040).
About IEEE Transactions on Affective Computing
IEEE Transactions on Affective Computing is a Tier 1 TOP journal in the CAS Journal Ranking. It ranks first by Journal Citation Indicator in its Web of Science field, with an impact factor of 9.8 and a 5-year impact factor of 10.2. It is also listed as a Class A international journal recommended by the Chinese Association for Artificial Intelligence (CAAI) and is a leading international journal in affective computing.
arXiv link: https://doi.org/10.48550/arXiv.2603.04766
IEEE link: https://ieeexplore.ieee.org/document/11424978
Code: https://github.com/Cross-Innovation-Lab/GAMDSS
Publication date: March 2026
Authors: Feng Liu*, Bingyu Nan, Xuezhong Qian, Xiaolan Fu*
Affiliations: School of Psychology, Shanghai Jiao Tong University*; School of Artificial Intelligence and Computer Science, Jiangnan University
Citation
F. Liu, B. Nan, X. Qian, and X. Fu, "Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognition," IEEE Transactions on Affective Computing, 2026, doi:10.1109/TAFFC.2026.3671731.
Copyright © Shanghai Jiao Tong University School of Psychology.222
