Multi-Scale Distillation from Multiple Graph Neural Networks

Knowledge Distillation (KD), which is an effective model compression and acceleration technique, has been successfully applied to graph neural networks (GNNs) recently. Existing approaches utilize a single GNN as the teacher distill knowledge. However, we notice that models with different number of layers demonstrate classification abilities on nodes degrees. On one hand, for high degrees, their local structures are dense complex, hence more message passing needed. Therefore, perform better. other low whose relatively sparse simple, repeated can easily lead over-smoothing. Thus, less suitable. existing single-teacher knowledge distillation based model, sub-optimal. To this end, propose novel approach multi-scale knowledge, learns from multiple capture topological semantic at scales. Instead learning equally, proposed method automatically assigns proper weights each via attention mechanism enables student select teachers structures. Extensive experiments conducted evaluate four public datasets. The experimental results superiority our over state-of-the-art methods. Our code publicly available https://github.com/NKU-IIPLab/MSKD.