Binarized Neural Network With Parameterized Weight Clipping and Quantization Gap Minimization for Online Knowledge Distillation

As the applications for artificial intelligence are growing rapidly, numerous network compression algorithms have been developed to restrict computing resources such as smartphones, edge, and IoT devices. Knowledge distillation (KD) leverages soft labels derived from a teacher model less parameterized achieving high accuracy with reduced computational burden. Moreover, online KD provides parallel through collaborative learning between student networks, thus enhancing training speed. A binarized neural (BNN) offers an intriguing opportunity facilitate aggressive at expense of drastically degraded accuracy. In this study, two performance improvements proposed when BNN is applied network: 1) weight clipping (PWC) reduce dead weights in 2) quantization gap-aware adaptive temperature scheduling networks. contrast constant (CWC), PWC demonstrates 3.78% top-1 test enhancement trainable by decreasing gradient mismatch CIFAR-10 dataset. Furthermore, increases 0.08% over temperature. By aggregating both methodologies, dataset was 94.60%, that Tiny-ImageNet comparable 32-bit full-precision network.