Abstract
Motivated by the need to investigate the association between gesture use and language use in cross-lingual remote communication, and for the purpose of making gesture research more efficient and easily accessible, in this thesis we present a multidimensional codified gesture dataset. As part of this project, we also present tools and describe the complete methodology for coding and organizing gestures captured as video data. Our dataset consists of data of communication behaviors in multiple dimensions (time duration, amount, speech content) of four categories of gestures (iconic, metaphoric, pointing, non-iconic) generated by 36 participants in two language groups (group EL1: native vs non-native speaker and group EL2: both non-native speakers) from a lab study. This dataset has been collected with a Kinect-taping tool and processed by an automatic data processing workflow that facilitates gesture categorizing. We accompany it by using methods of human coding for precisely aligning language data and non-verbal gesture data along the timeline of interpersonal communication. Finally, we conduct a series of sample data analyses with statistical visualizations and exploratory analyses that aim to provide insights around the interaction processes of computer-mediated cross-lingual communication. We explore how gesture use associates with individuals’ comprehension and ideation. The dataset provides researchers with an open, archived dataset for exploring the gesture usage in cross-lingual communication and different communication media. The goal is to facilitate the procedure of gesture coding and lower the human cost of it, reducing barriers in research on multimodal interpersonal communication.