Abstract
Understanding the underlying neuro-perceptual mechanism of humans’ ability to decode emotional content in vocal signal is an important research direction. However, it is well know that obtaining valence from speech features is much difficult than arousal. Arousal can be accurately identified, also automatically-recognized, using speech signal without context. On the other hand, it is much more difficult to recognize valence if speech does not contain context. In this paper, we obtain the fMRI-derived features from blood oxygen level-dependent (BOLD) signals when subjects are exposed to various vocal emotion stimuli. We observe that by using the fMRI-derived feature to predict valence is beneficial to speech-based emotion recognition system. Furthermore, due to the fact that fMRI scanning is costly and time-consuming, we integrate audio features and fMRI-derived features to learn a joint representation by using Gaussian mixture regression (GMR). Finally, the proposed framework demonstrates that we are capable of obtaining an improved categorical emotion recognition using audio features fused with the stimulated vocal-induced fMRI-derived features, which generated from the GMR model.