Abstract
Continuous emotion recognition aims to recognize human emotion from audio-visual sequences. Human emotion can be described as four dimensions including arousal, valence, power and expectation. Previous work mostly uses hand-crafted features, which are not strongly related to emotion. We propose to learn emotion-related features that reflect how much of a person’s emotion changes. In addition, in order to handle the inconsistency problem between facial appearances and dimensional labels, we use residual images which are defined as the difference between two adjacent video frames to capture the relative change between facial appearances. When learning the emotion-related features, we propose an efficient loss, joint ranking and regression loss, to obtain more discriminative features. However, misalignment of adjacent frames due to pose variation degrades the effectiveness of residual images. We use part-based facial landmarks to deal with the misalignment problem. Finally, we propose a fusion temporal network to combine visual and audio cues and model long-term emotional evolution through LSTM. Our experiments demonstrate that our method achieves comparable results with previous work on the AVEC 2012 dataset and the RECOLA dataset