Progressive Network Grafting for Few-Shot Knowledge Distillation

Knowledge distillation has demonstrated encouraging performances in deep model compression. Most existing approaches, however, require massive labeled data to accomplish the knowledge transfer, making compression a cumbersome and costly process. In this paper, we investigate practical few-shot scenario, where assume only few samples without human annotations are available for each category. To end, introduce principled dual-stage scheme tailored data. first step, graft student blocks one by onto teacher, learn parameters of grafted block intertwined with those other teacher blocks. second trained progressively connected then together network, allowing learned adapt themselves eventually replace network. Experiments demonstrate that our approach, unlabeled samples, achieves gratifying results on CIFAR10,CIFAR100, ILSVRC-2012. On CIFAR10 CIFAR100, even par schemes utilize full datasets. The source code is at https://github.com/zju-vipa/NetGraft.