CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets

Current RGB-D scene recognition approaches often train two standalone backbones for RGB and depth modalities with the same Places or ImageNet pre-training. However, pre-trained network is still biased by RGB-based models which may result in a suboptimal solution. In this paper, we present single-model self-supervised hybrid pre-training framework modalities, termed as CoMAE. Our CoMAE presents curriculum learning strategy to unify popular representation algorithms: contrastive masked image modeling. Specifically, first build patch-level alignment task pre-train single encoder shared via cross-modal learning. Then, passed multi-modal autoencoder capture finer context features from generative perspective. addition, our design without requirement of fusion module very flexible robust generalize unimodal scenario both training testing phases. Extensive experiments on SUN NYUDv2 datasets demonstrate effectiveness experiment results reveal that data-efficient learner. Although only use small-scale unlabeled set pre-training, are competitive state-of-the-art methods extra large-scale supervised dataset Code will be released at https://github.com/MCG-NJU/CoMAE.