Up to 100x Faster Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) has recently been attracting increasing attention from research communities, attributed to its capability compress a model only using synthetic data. Despite the encouraging results achieved, state-of-the-art DFKD methods still suffer inefficiency of data synthesis, making data-free training process extremely time-consuming and thus inapplicable for large-scale tasks. In this work, we introduce an efficacious scheme, termed as FastDFKD, that allows us accelerate by factor orders magnitude. At heart our approach is novel strategy reuse shared common features in so synthesize different instances. Unlike prior optimize set independently, propose learn meta-synthesizer seeks initialization fast synthesis. As result, FastDFKD achieves synthesis within few steps, significantly enhancing efficiency training. Experiments over CIFAR, NYUv2, ImageNet demonstrate proposed 10x even 100x acceleration while preserving performances on par with state art. Code available at https://github.com/zju-vipa/Fast-Datafree.