Speech Emotion Recognition (SER) systems are increasingly deployed in voice-centric applications, yet often suffer from fairness concerns due to speaker-induced variability. In particular, speaker-gender bias can cause systematic disparities in performance across demographic groups (group fairness), while emotionally expressive or acoustically unique speakers may be treated inconsistently despite similar input (individual fairness). Although prior work suggests that group and individual fairness objectives may be inherently incompatible, our proposed two-stage debiasing framework aims to address both: an in-processing approach first mitigates speaker-gender bias (group fairness), followed by a post-processing calibration step that improves consistency across similar instances (individual fairness). While most samples benefit from this dual intervention, we identify a small subset of speech samples that remain difficult to classify fairly. This work focuses on systematically analyzing these fairness-ambiguous samples to understand what makes them challenging. We examine this question from two perspectives: emotion perception and acoustic expressivity. Our analyses on these subsets indicate that: (1) exhibit extreme or atypical emotional ratings, (2) show high acoustic variability, and (3) tend to come disproportionately from specific individuals. These findings suggest that some speakers inherently present greater challenges to fairness optimization, due to the uniqueness of their emotional or acoustic expression. By characterizing these subsets , our work contributes to a deeper understanding of fairness conflicts in SER and offers new directions for developing more robust and inclusive emotion recognition systems.
- Only Subsets Matters: The Effect of Dual Fairness Constraints in Speech Emotion Recognition
- Woan-Shiuan Chien - National Tsing Hua UniversityChi-Chun Lee - National Tsing Hua University, College of Semiconductor Research
- IEEE
- 14
- 112-2628-E-007-012-MY3 / NSTC
- Journal article
- 01/04/2026
- IEEE transactions on affective computing, Vol.17(2), pp.2340-2353
- English