Quantized Feature Distillation for Network Quantization

Neural network quantization aims to accelerate and trim full-precision neural models by using low bit approximations. Methods adopting the aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes novel highly effective QAT method, quantized feature distillation (QFD). QFD first trains (or binarized) representation as teacher, then quantize knowledge (KD). Quantitative results show that is more flexible (i.e., friendly) than previous methods. surpasses existing methods noticeable margin on not only image classification also object detection, albeit being much simpler. Furthermore, quantizes ViT Swin-Transformer MS-COCO detection segmentation, which verifies its potential in real world deployment. To best of our knowledge, this time vision transformers been segmentation tasks.