Abstract
Audio and video are highly correlated, so that it is possible to map acoustic features into visual features. The audio-visual conversion not only provides “visible speech” in noisy circumstances, it also provides a better visual experience. Hence it can be applied to a great deal of applications, such as multimedia telephony for hearing-impaired people, human-computer interfaces, lip synchronization of cartoon animation, and low bit rate video conferencing. In this thesis, we propose a real-time audio-to-visual conversion system based on Gaussian mixture model. With the GMM, we can predict visual parameters from corresponding audio parameters. We use this method to map continuous speech into synchronized lip movements. This system utilizes LSPs as the audio features, hence it can be easily integrated with LSP- based speech coders like G.723.1 or MPEG-4 CELP & HVXC speech coding without increasing computation complexity too much. The LSPs are utilized as audio features initially for it’s immediately available during speech decoding process. But the experimental results show that the estimated visual features mapped from LSPs are much better than those mapped from traditional cepstrals. Furthermore, we also propose three model adaptation algorithms to reduce the heavy computation requirement in the training phase. These model adaptation algorithms are tried to examine if a GMM can be trained with fewer computational operations. With model adaptation, the audio-visual conversion system can be easier to be applied in related applications.