Contrastive Deep Supervision

AbstractThe success of deep learning is usually accompanied by the growth in neural network depth. However, traditional training method only supervises at its last layer and propagates supervision layer-by-layer, which leads to hardship optimizing intermediate layers. Recently, has been proposed add auxiliary classifiers layers networks. By these with supervised task loss, can be applied shallow directly. conflicts well-known observation that learn low-level features instead task-biased high-level semantic features. To address this issue, paper proposes a novel framework named Contrastive Deep Supervision, augmentation-based contrastive learning. Experimental results on nine popular datasets eleven models demonstrate effects general image classification, fine-grained classification object detection learning, semi-supervised knowledge distillation. Codes have released .