Recognition from Hand Cameras

We propose HandCam (Fig. 1), a novel wearable camera capturing activities of hands, for recognizing human behaviors. HandCam has two main advantages over egocentric systems [10, 5, 23]: (1) it avoids the need to detect hands and manipulation regions; (2) it observes the activities of hands almost at all time. These nice properties enable HandCam to recognize hand states (free v.s. active hands, hand gestures, object categories), and discover object categories through interaction with hands. We propose a new method to accomplish these tasks by fusing per-frame state prediction with state changes estimation from a pair of frames. We have collected one of the first HandCam dataset with 20 videos captured in three scenes. Experiments show that HandCam system significantly outperforms a state-ofthe-art HeadCam system (frame-based re-implementation similar to [17]) in all tasks by at most 10.8% improvement in accuracy. We also show that HandCam videos captured by different users can be easily aligned to improve free v.s. active recognition accuracy (3.3% improvement) in crossscenes use case. Moreover, we observe that finetuning Convolutional Neural Network [14] consistently improves accuracy (at most 4.9% better). Finally, a method combining HandCam and HeadCam features achieves the best performance. With more data, we believe a joint HandCam and HeadCam system can robustly log hand states in daily life.