Extending Momentum Contrast With Cross Similarity Consistency Regularization

Contrastive self-supervised representation learning methods maximize the similarity between positive pairs, and at same time tend to minimize negative pairs. However, in general interplay pairs is ignored as they do not put place special mechanisms treat differently according their specific differences similarities. In this paper, we present Extended Momentum Contrast (XMoCo), a method founded upon legacy of momentum-encoder unit proposed MoCo family configurations. To end, introduce cross consistency regularization loss, with which extend transformation dissimilar images (negative pairs). Under rule, argue that semantic representations associated any pair (positive or negative) should preserve cross-similarity under pretext transformations. Moreover, further regularize training loss by enforcing uniform distribution over across batch. The can easily be added existing algorithms plug-and-play fashion. Empirically, report competitive performance on standard Imagenet-1K linear head classification benchmark. addition, transferring learned common downstream tasks, show using XMoCo prevalently utilized augmentations lead improvements such tasks. We hope findings paper serve motivation for researchers take into consideration important among examples learning.