Abstract
There are convenient devices capturing multimedia content in modern times. Some professional people, such as moviemaker and video editor, produce attractive and novel multimedia content to invoke emotions of audience with these recorded materials and works about video editing. There is influential power within motion and music. Specially, music has more influential power in the performance at the theater. The examples are the music performed by the orchestra and passionate motion of the conductor. Motion of the character is exaggerated or restrained with music about different rhythm and emotions.While acquiring the videos with wonderful music is now easy, editing them in terms of visual and musical contents is not trivial. While tremendous efforts have been made in editing visual or musical content along, few works address the scenario of manipulating both contents simultaneously. In this work, we present a fully automatic system that aims at manipulating the character motion in a pre-captured video by simply changing its background music. The source video is captured by a static camera at the theater. Although there are several pioneers that devoted themselves to manipulate motions of characters, the restriction is different from the environment at the theater. There are multiple viewpoints videos for generating visual 3D model and they can manipulate motion with the 3D model in previous works. According to the environment at the theater, we propose the method between full 3D reconstruction and 2D deformation with the depth and video capture device. We analyze occlusion relation, and create 2.5D model with approximate 3D model. Lastly, we synthesize the manipulated motion of the character by applying image-based deformation. Then, we show videos that there are motions of conductors according to manipulated skeletons, even if there are highly dynamic motions with occlusion conditions.