Hierarchical I3D for Sign Spotting

Most of the vision-based sign language research to date has focused on Isolated Sign Language Recognition (ISLR), where objective is predict a single class given short video clip. Although there been significant progress in ISLR, its real-life applications are limited. In this paper, we focus challenging task Spotting instead, goal simultaneously identify and localise signs continuous co-articulated videos. To address limitations current ISLR-based models, propose hierarchical spotting approach which learns coarse-to-fine spatio-temporal features take advantage representations at various temporal levels provide more precise localisation. Specifically, develop Hierarchical I3D model (HS-I3D) consists network head that attached existing exploit different layers network. We evaluate HS-I3D ChaLearn 2022 Challenge - MSSL track achieve state-of-the-art 0.607 F1 score, was top-1 winning solution competition.