Abstract
Crowdsourcing annotation applications have increasingly moved from PCs to smartphones as the smartphone and mobile network are widely used. The most common ways to annotate things in crowdsourcing applications are “Like/unlike”, “Five-star” ratings, “Emoticons” and “Text” tags or comments. However, the annotating method used on phone calls is limited to “Text” in existing system. In this study, we compared the four annotating methods and tried to figure out the most suitable method in crowdsourcing annotation system on phone calls for smartphone users. There are two main goals in this study, the first goal is to find the usage patterns of annotating and investigate the feelings on different kind of calls by analyzing the massive call annotations being collected. The second goal is to establish a predictive model on call feelings. We developed a crowdsourcing annotation system on Android, and collected more than 3,000 user profiles and 100,000 annotations. We compared the four methods on this system and studied the primary audience, user behavior on annotating and efficiency of collecting annotations. In addition, the predictive models on call feeling achieved 73.6% accuracy on average, which is higher than that of the other studies. Overall, this study provides a foundation for follow-up studies in the field of crowdsourcing annotation on phone calls, and the established predictive model can be widely used in estimating customer satisfaction on calls and improving phone-call-related applications.