Learning pullback manifolds of generative dynamical models for action recognition

The most recent test-beds for action and identity recognition have exposed the limitations of the, otherwise successful, methods which directly classify features extracted from the spatio-temporal volumes representing motions. In opposition, encoding the actions’ dynamics via generative dynamical models has a number of desirable features when it comes to unsupervised learning of plots, crowd monitoring, and description of more complex activities. However, using fixed, all-purpose distances to classify dynamical models does not necessarily deliver good classification results. In this paper we propose a general framework for learning Riemannian metrics or distance functions for dynamical models, given a training set which can be either labeled or unlabeled. The optimal distance function is selected among a family of pullback ones, induced by a parameterized automorphism of the space of models. In virtue of their relevance to action and gait recognition we consider here the classes of multidimensional autoregressive models of order 2 and hidden Markov models. We study their manifolds and design automorphisms there which allow to build parametric families of metrics we can optimize upon. Experimental results concerning action and identity recognition are presented, which show how pullback learning greatly improve classification performances w.r.t. base distances. Natural extensions of this methodology to hierarchical models for complex activity recognition are envisaged.