Abstract
We revisit the study of a wrist-mounted camera system (referred to as HandCam) for recognizing activities of hands. HandCam has two unique properties as compared to egocentric systems [2, 3] (referred to as EgoCam): (1) it avoids the need to detect hands; (2) it more consistently observes the activities of hands. By taking advantage of these properties, we propose a deep-learning-based method to recognize hand states (free vs. active hands, hand gestures, object categories), and discover object categories. Moreover, we propose a novel two-streams deep network to further take advantage of both HandCam and EgoCam. We have collected a new synchronized HandCam and EgoCam dataset with 20 videos captured in three scenes for hand states recognition. Experiments show that our HandCam system consistently outperforms a deep-learning-based EgoCam method (with estimated manipulation regions) and a dense-trajectory-based [4] EgoCam method in all tasks. We also show that HandCam videos captured by different users can be easily aligned to improve free vs. active recognition accuracy (3.3% improvement) in across-scenes use case. Next, we apply finetuning on Convolutional Neural Network [5], and it consistently improves accuracy. More important, our novel two-streams deep network combining HandCam and EgoCam features achieves the best performance in four out of five tasks. Finally, we want to apply our system into daily life. Based on the basic system, we add several steps to upgrade our system to be suitable for real-time application. With more data, we believe the new system with joint HandCam and EgoCam can robustly log hand states in daily life.