Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion

One significant factor we expect the video representation learning to capture, especially in contrast with image learning, is object motion. However, found that current mainstream datasets, some action categories are highly related scene where happens, making model tend degrade a solution only information encoded. For example, trained may predict as playing football simply because it sees field, neglecting subject dancing cheerleader on field. This against our original intention towards and bring bias different dataset can not be ignored. In order tackle this problem, propose decouple motion (DSM) two simple operations, so attention better paid. Specifically, construct positive clip negative for each video. Compared video, positive/negative motion-untouched/broken but scene-broken/untouched by Spatial Local Disturbance Temporal Disturbance. Our objective pull closer while pushing farther latent space. way, impact of weakened temporal sensitivity network further enhanced. We conduct experiments tasks various backbones pre-training find method surpass SOTA methods remarkable 8.1% 8.8% improvement recognition task UCF101 HMDB51 datasets respectively using same backbone.