Abstract
In recent years, improving external behaviors’ emotion recognition system by fusing internal behaviors is an important research direction for exports in many science. In this paper, our initial research is modeling an external behavior (audio signal) categorical emotion recognition system by fusing internal behavior (blood oxygen level-dependent signal, BOLD signal). Next, we focus on using convolutional neural network feature extraction method to improve multi-variate pattern analysis (MVPA) in fMRI research. Convolutional neural network (CNN) is a type of deep learning which method bases on computer vision, and achieves locally feature extraction via convolution’s properties: local associativity. Concurrently, CNN takes fMRI to multiple linear and non-linear transform into high-level hierarchical feature. Furthermore, we join majority vote method for different subject of affective decision, in order to ignore a few of subject with more different decision and get a result about an analysis of groups. In addition, we base on the concept of region-based CNN, using CNN feature learning on individual lobe system to achieve more locally feature extraction and try to find the importance lobe about affective computing. Finally, we effectively improve multi-model categorical emotion recognition system by fusing Temporal, Frontal and Parietal Lobe of CNN hierarchical feature and joining majority vote. All experiments in this paper will demonstrate that fMRI’s CNN hierarchical features indeed has capability in emotion recognition and be devoted to increase technique’s emotion recognition performance.