Garbage Model Formulation with Conditional Random Fields for Sign Language Spotting

Sign language spotting is the task of detecting and recognizing the signs in a signed utterance, from a set vocabulary. The difficulty of sign language spotting is that the instances of signs vary in both motion and appearance. Moreover, the signs appear within a continuous gesture stream, interspersed with transitional movements between signs in a vocabulary and non-sign patterns (out-of-vocabulary signs and other movements that do not correspond to signs). In this paper, a novel method for designing garbage models in a conditional random field (CRF) model is proposed, which performs an adaptive threshold for distinguishing between signs in the vocabulary and non-sign patterns. A short-sign detector, a hand appearance-based sign verification method, and a subsign reasoning method are included to further improve sign language spotting accuracy. Experimental results show that our system can detect signs from continuous data with an 88% spotting rate and can recognize signs from isolated data with a 94% recognition rate, versus 74% and 90% respectively for CRFs without a garbage model, short-sign detector, subsign reasoning, and hand appearance-based sign verification.