# A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-
Distillation

Lirong Wu, Haitao Lin, Zhangyang Gao, Guojiang Zhao, and Stan Z. Li† Lirong
Wu, Haitao Lin, Zhangyang Gao, Guojiang Zhao, and Stan Z. Li are with the AI
Lab, Research Center for Industries of the Future, Westlake University,
Hangzhou 310000, China. E-mail: {wulirong, linhaitao, gaozhangyang,
zhaoguojiang<EMAIL_ADDRESS>
$\dagger$ Corresponding Author.

###### Abstract

Recent years have witnessed great success in handling graph-related tasks with
Graph Neural Networks (GNNs). Despite their great _academic_ success, Multi-
Layer Perceptrons (MLPs) remain the primary workhorse for practical
_industrial_ applications. One reason for such an academic-industry gap is the
neighborhood-fetching latency incurred by data dependency in GNNs. To reduce
their gaps, Graph Knowledge Distillation (GKD) is proposed, usually based on a
standard teacher-student architecture, to distill knowledge from a large
teacher GNN into a lightweight student GNN or MLP. However, we found in this
paper that neither teachers nor GNNs are necessary for graph knowledge
distillation. We propose a _T eacher-Free Graph Self-Distillation_ (TGS)
framework that does not require any teacher model or GNNs during both training
and inference. More importantly, the proposed TGS framework is purely based on
MLPs, where structural information is only implicitly used to guide _dual
knowledge self-distillation_ between the target node and its neighborhood. As
a result, TGS enjoys the benefits of graph topology awareness in training but
is free from data dependency in inference. Extensive experiments have shown
that the performance of vanilla MLPs can be greatly improved with dual self-
distillation, e.g., TGS improves over vanilla MLPs by 15.54% on average and
outperforms state-of-the-art GKD algorithms on six real-world datasets. In
terms of inference speed, TGS infers 75$\times$-89$\times$ faster than
existing GNNs and 16$\times$-25$\times$ faster than classical inference
acceleration methods.

###### Index Terms:

Graph Neural Networks, Graph Knowledge Distillation, Inference Acceleration.

##  1 Introduction

Recently, Graph Neural Networks (GNNs) [1, 2, 3, 4, 5] have demonstrated their
powerful capability to handle graph-structured data in a variety of
applications, including meteorology [6, 7], life sciences [8, 9], and
transportation [10, 11]. Despite their great academic success, practical
deployments of GNNs in the industry are still less popular. One reason for
their gaps is the neighborhood-fetching inference latency incurred by data
dependency in GNNs [12]. Most existing GNNs [13, 14, 15] rely on message
passing to aggregate neighborhood features for capturing data dependency
between nodes within a local subgraph. As a result, neighborhood fetching
caused by data dependency is one of the major sources of GNN latency during
inference. For example, to infer a single node with a $L$-layer GNN on a graph
with average node degree $R$, it requires fetching and aggregating
$\mathcal{O}(R^{L})$ fetched nodes. However, $R$ can be large for real-world
graphs, e.g., 19 for the Amazon-com dataset, and $L$ is getting deeper for the
latest GNN architectures, e.g., $L$=1001 layers for RevGNN-Deep [16]. Compared
with GNNs, MLPs do not suffer from the data dependency problem and infer much
faster than GNNs. However, due to the lack of modeling data dependencies, MLPs
may fail to take full advantage of the graph topological information, which
limits their performance on various downstream tasks.

(a) Four different types of GKD algorithms

(b) Inference Accuracy vs. Inference Time

Figure 1: (a): Illustration of four different types of graph knowledge
distillation algorithms, depending on whether teacher models and GNNs/MLPs are
included in training and inference. (b): Inference accuracy (%) vs. inference
time (ms) on the Cora dataset. If not specifically mentioned, all GKD
algorithms adopt GCN as the backbone by default.

To connect the two worlds of topology-aware GNNs and inference-efficient MLPs,
graph knowledge distillation [17, 18, 19, 20, 21] is proposed to distill
knowledge from large teacher GNNs to small student GNNs or MLPs, yielding two
branches, namely GNN-to-GNN [22, 23, 24] and GNN-to-MLP [25, 26, 27, 28, 29],
as shown in Fig. 1(a). Different from the standard teacher-student KD, there
is a similar technique called GNN Self-Distillation [30, 31], which
regularizes a GNN model by _distilling its own knowledge within GNNs (e.g.,
across different layers or nodes) without other teacher models_. Similarly,
MLP Self-Distillation [32, 33] is proposed to implicitly self-distill
knowledge from structural information, but it does not involve any teachers or
GNNs and is purely based on MLPs.

Considering both inference accuracy and inference time, the above four types
of graph knowledge distillation methods are two completely different worlds.
As shown in Fig. 1(b), GNN-to-GNN KD and GNN Self-Distillation directly deploy
well-distilled GNNs for inference, which helps to improve the inference
accuracy but does not help much in inference acceleration with the data
dependency in GNNs unresolved. On the other hand, GNN-to-MLP KD and MLP Self-
Distillation require only MLPs for inference, which greatly improves their
inference speed, but brings limited performance improvement. In particular,
MLP Self-Distillation completely removes GNNs, which disables it from being
topology-aware, making its inference fastest, but at the cost of undesired
performance drops.

In this paper, we found that neither teachers nor GNNs are necessary to
achieve high-accuracy and high-efficiency inference on graphs. Therefore, we
propose a simple _T eacher-Free Graph Self-Distillation_ (TGS) framework that
does not require any teacher model or GNNs during both training and inference.
The proposed TGS framework is purely based on MLPs, where structural
information is only implicitly used to guide _dual knowledge self-
distillation_ between the target node and its neighborhood, substituting the
explicit information propagation as in GNNs. As a result, the resulting model
enjoys the benefits of graph topology-awareness in training but reduces time
overhead in inference. Moreover, we propose an edge sampling strategy for
batch-style self-distillation instead of feeding the entire graph into memory
to reduce memory usage when scaling to large-scale graphs. Extensive
experiments show that TGS infers as fast as MLPs, but its inference accuracy
is comparable to or even better than state-of-the-art GKD algorithms. We
believe that this work will inspire researchers to rethink the necessity of
teachers and GNNs for graph knowledge distillation. Codes will be public at
https://github.com/LirongWu/TGS.

##  2 Related Work

###  2.1 Graph Neural Networks

Graph Neural Networks (GNNs) can be mainly divided into two categories, i.e.,
spectral-based GNNs and spatial-based GNNs. The spectral-based GNNs, such as
ChebyNet [34] and GCN [35], define graph convolution kernels in the spectral
domain based on the graph signal processing theory. Instead, the spatial-based
GNNs, such as GraphSAGE [15] and GAT [35], directly define updating rules in
the spatial space and focus on the design of neighborhood aggregation
functions. The closest work to ours is APPNP [36], which shares some similar
design insights with TGS in that both first use graph-independent models to
extract node embeddings. However, their main difference is that APPNP applies
Personalized PageRank to explicitly encode the adjacency matrix via message
passing, while TGS uses Mixup and two inference layers to implicitly exploit
the adjacency matrix as supervision signals. We refer interested readers to
the recent survey [1, 37] for more GNN architectures. Despite their great
progress, the above GNNs all share the de facto design that structural
information is explicitly used for message passing, which leaves neighborhood
fetching still one major source of GNN inference latency. To reduce
multiplication and accumulation operations, many inference acceleration
technologies have been applied, including Pruning [38], Quantizing [39], and
Neighborhood Sampling [40].

###  2.2 Graph Knowledge Distillation

Several previous works on graph distillation try to distill knowledge from
large teacher GNNs to smaller student GNNs, termed as _GNN-to-GNN KD_ [41].
The student model in FreeKD [42] and TinyGNN [17] is a GNN with fewer
parameters, but not necessarily fewer layers, which makes them still suffer
from the neighborhood-fetching latency. The other branch of graph knowledge
distillation is to distill from large teacher GNNs to lightweight student
MLPs, termed as _GNN-to-MLP KD_. For example, CPF [26] proposes to distill
knowledge from teacher GNNs to student MLPs, but it takes advantage of label
propagation [43] in MLPs to improve performance and thus remains heavily data-
dependent. Besides, GLNN [25] directly distills knowledge from GNNs to student
MLPs, which has a great advantage in inference speed, but its performance
gains are limited.

Different from the standard teacher-student KD architecture, there is a
similar technique known as _GNN Self-Distillation_ [44], which regularizes a
GNN model by distilling its own knowledge without other teacher models. For
example, GNN-SD [30] directly self-distills knowledge across different GNN
layers. Besides, RDD [31] builds an ensemble teacher using multiple versions
of the model itself without introducing an external teacher model. Compared to
GNNs, MLPs have no data dependency and infer much faster. There are some
attempts on _MLP Self-Distillation_ , such as Graph-MLP [32] and LinkDist
[33], which are purely based on MLP and self-distill knowledge from structural
information by contrastive or consistency constraints. Despite the great
advantage of MLP Self-Distillation in inference speed, their inference
accuracy still cannot match the state-of-the-art competitors. More related
work on graph knowledge distillation can be found in a recent survey [45]. The
closest work to ours is Graph-MLP, but the essential difference between Graph-
MLP and TGS is that the former is a contrastive learning method, while the
latter is based on knowledge distillation. Their main differences are in the
following four aspects: (1) TGS utilizes mixup as data augmentation, while
Graph-MLP [32], as a contrastive learning method, instead discards this; (2)
Graph-MLP treats all nodes within an $r$-hop neighborhood as positive samples
with $\mathcal{O}(R^{r})$ complexity, whereas TGS performs distillation only
within a 1-hop neighborhood with a complexity of $\mathcal{O}(R)$, where $R$
is the average node degree; (3) Graph-MLP treats all nodes within a batch as
negative samples, while TGS only takes one randomly selected node as negative
sample; and (4) as a contrastive learning method, Graph-MLP relies on InfoNCE
to perform contrasting, whereas TGS directly minimizes the MSE losses between
samples.

###  2.3 Graph Contrastive Learning (GCL)

Recent years have witnessed the great success of graph contrastive learning in
learning graph representation [46]. Graph contrastive learning generates
multiple views for each instance through various data augmentation methods and
maximizes the agreement between two positive samples (as measured by mutual
information) against a large number of negative samples. For example, Deep
Graph Infomax (DGI) [47] is proposed to contrast the patch representations and
corresponding high-level summary of graphs. GraphCL [48] is one of the
pioneering works that extends contrastive learning to graphs, and it defines
four graph augmentation methods that achieve good results on the molecular
property prediction task. Its follow-up, GCA [49], combines adaptive
augmentation, which further improves the performance. Furthermore, InfoGCL
[50] investigates how information is transformed and transferred during
contrastive learning based on the information bottleneck, aimed at minimizing
the loss of task-relevant information. There has been some recent work
exploring the necessity of negative samples for graph contrastive learning.
Inspired by BYOL [51], BGRL [52] proposes to perform the self-supervised
learning that _does not require negative samples_ , thus getting rid of the
potentially quadratic bottleneck. Moreover, in addition to the commonly used
InfoNCE [53] as a loss function for measuring agreement, MSE is also widely
used in GCL, such as SelfGNN [54], DMGI [55], etc.

##  3 Preliminaries

Notations and Problem Statement. Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$
denote a graph, where $\mathcal{V}$ is the set of $|\mathcal{V}|=N$ nodes with
features
$\mathbf{X}=\left[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N}\right]\in\mathbb{R}^{N\times
d}$ and $\mathcal{E}$ is the set of edges between nodes. Each node
$v_{i}\in\mathcal{V}$ is associated with a $d$-dimensional features vector
$\mathbf{x}_{i}$. Following the common semi-supervised node classification
setting, only a subset of node
$\mathcal{V}_{L}=\\{v_{1},v_{2},\cdots,v_{L}\\}$ with corresponding labels
$\mathcal{Y}_{L}=\\{y_{1},y_{2},\cdots,y_{L}\\}$ are known, and we denote the
labeled set as $\mathcal{D}_{L}=(\mathcal{V}_{L},\mathcal{Y}_{L})$ and
unlabeled set as $\mathcal{D}_{U}=(\mathcal{V}_{U},\mathcal{Y}_{U})$, where
$\mathcal{V}_{U}=\mathcal{V}\backslash\mathcal{V}_{L}$. The task of node
classification aims to learn a mapping
$\Phi:\mathcal{V}\\!\rightarrow\\!\mathcal{Y}$ on labeled data
$\mathcal{D}_{L}$, so that it can be used to infer the labels
$\mathcal{Y}_{U}$ [56].

Graph Neural Networks. A general GNN framework consists of two key
computations for each node $v_{i}$ at every layer: (1)
$\operatorname{AGGREGATE}$ operation: aggregating messages from neighborhood
$\mathcal{N}_{i}$; (2) $\operatorname{UPDATE}$ operation: updating node
representation from its representation in the previous layer and aggregated
messages. Considering a $L$-layer GNN, the formulation of the $l$-th layer is
as follows,

$\displaystyle\mathbf{m}_{i}^{(l)}$
$\displaystyle=\operatorname{AGGREGATE}^{(l)}\left(\left\\{\mathbf{h}_{j}^{(l-1)}:v_{j}\in\mathcal{N}_{i}\right\\}\right)$
(1) $\displaystyle\mathbf{h}_{i}^{(l)}$
$\displaystyle=\operatorname{UPDATE}^{(l)}\left(\mathbf{h}_{i}^{(l-1)},\mathbf{m}_{i}^{(l)}\right)$

where $0\leq l\leq L-1$, $\mathbf{h}_{i}^{(l)}$ is the embedding of node
$v_{i}$ in the $l$-th layer, and $\mathbf{h}_{i}^{(0)}\\!=\\!\mathbf{x}_{i}$
is the input feature. After $L$ message-passing layers, the final node
embeddings $\mathbf{h}_{i}^{(L)}$ can be passed through an additional linear
inference layer $\mathbf{y}_{i}=f_{\theta}(\mathbf{h}_{i}^{(L)})$ for node
classification on the target node $v_{i}$.

##  4 Methodology

Motivated by the complementary strengths and weaknesses of GNNs and MLPs, we
propose in this paper a simple but effective _T eacher-Free Graph Self-
Distillation_ (TGS) framework, as illustrated in Fig. 2. The proposed TGS
framework enjoys the benefits of GNN-like topology-awareness in training but
keeps the inference-efficiency of MLPs in inference. The following subsections
focus on three key aspects: (1) backbone architecture, how to construct a
“boosted” MLP as backbone; (2) dual self-distillation, how to self-distill
knowledge between the target node and its neighborhood; (3) training and
inference, how to solve training difficulties and make inference with the
trained model.

Figure 2: Illustration of the proposed _Teacher-Free Graph Self-Distillation_
(TGS) framework. In the training stage, the MLP and two inference layers
$f_{\theta}(\cdot)$, $g_{\gamma}(\cdot)$ are jointly trained by the proposed
dual feature-level and label-level self-distillation.

###  4.1 Backbone Architecture

The TGS framework is based on pure MLPs, with each layer composed of a linear
transformation, an activation function, a batch normalization, and a dropout
function, defined as:

$\mathbf{H}^{(l+1)}=\operatorname{Dropout}\big{(}BN\big{(}\sigma\big{(}\mathbf{H}^{(l)}\mathbf{W}^{(l)}\big{)}\big{)}\big{)},\quad\mathbf{H}^{(0)}=\mathbf{X}$
(2)

where $0\leq l\leq L-1$, $\sigma=\mathrm{ReLu}(\cdot)$ denotes an activation
function, $BN(\cdot)$ denotes the batch normalization, and
$\operatorname{Dropout}(\cdot)$ is the dropout function. In addition,
$\mathbf{W}^{(0)}\in\mathbb{R}^{d\times F}$ and
$\mathbf{W}^{(l)}\in\mathbb{R}^{F\times F}$ $(1\leq l\leq L-1)$ are layer-
specific parameter matrices with the hidden dimension $F$.

Given a target node $v_{i}$ and its neighboring nodes $\mathcal{N}_{i}$, we
first feed their node features $\mathbf{x}_{i}$ and
$\\{\mathbf{x}_{j}\\}_{j\in\mathcal{N}_{i}}$ into MLPs and encode them as
hidden representations $\mathbf{h}^{(L)}_{i}$ and
$\\{\mathbf{h}^{(L)}_{j}\\}_{j\in\mathcal{N}_{i}}$. Then, we define two
parameter-independent inference layers for label prediction, respectively,
i.e., $\mathbf{y}_{i}=f_{\theta}(\mathbf{h}_{i}^{(L)})\in\mathbb{R}^{C}$ and
$\mathbf{z}_{j}=g_{\gamma}(\mathbf{h}_{j}^{(L)})\in\mathbb{R}^{C}$, where $C$
is the number of category. Both two inference layers are implemented with one
layer of linear transformation by default in this paper. In the next
subsections, we will discuss in detail how to implicitly self-distill graph
knowledge between the target node $v_{i}$ and its neighborhood nodes
$\mathcal{N}_{i}$.

###  4.2 Dual Knowledge Self-Distillation

In this subsection, we propose dual knowledge self-distillation to learn node
features $\mathbf{X}$ and labels $\mathcal{Y}_{L}$ guided by structural
information $\mathcal{E}$. It consists of two bidirectional modules, one
_Feature-level Self-Distillation_ that distills feature information from the
neighborhood into the target node, and one _Label-level Self-Distillation_
that distills label information from the target node into the neighborhood.

####  4.2.1 Feature-level Self-Distillation

The graphs can be categorized into homophily and heterophily graphs, where the
former fulfills the smoothness assumption while the latter does not.
Considering the importance and prevalence of homophily graphs, for which a
variety of classical GNNs including GCN, GAT, SAGE, and APPNP have been
previously developed, we focus mainly on learning homophily graphs in this
paper. The smoothness assumption indicates that neighboring nodes in a graph
tend to share similar features and labels, while non-neighboring nodes should
be far away. With such motivation, we perform feature-level self-distillation
on the neighborhood by regularizing the consistency of the label distributions
between the target node and its neighboring nodes. Along with connectivity,
disconnectivity between nodes also carries important information that reveals
the node’s dissimilarity. However, the number of neighboring nodes is much
smaller compared with those non-neighboring nodes, which renders the model
overemphasize the differences between the target and non-neighboring nodes,
possibly leading to imprecise class boundaries. To solve this problem, we
modify _Mixup_ [57], an effective data augmentation that performs
interpolation between samples to generate new samples, to augment neighboring
nodes. Specifically, we perform learnable interpolation between the target
node $v_{i}$ and its neighboring node $v_{j}\in\mathcal{N}_{i}$ to generate a
new node, with its node representation defined as

$\displaystyle\mathbf{z}^{\prime}_{i,j}=$ $\displaystyle
g_{\gamma}\big{(}\beta_{i,j}\mathbf{h}_{j}^{(L)}+(1-\beta_{i,j})\mathbf{h}_{i}^{(L)}\big{)},$
(3) where $\displaystyle\ \ \
\beta_{i,j}=\mathrm{sigmoid}\big{(}\mathbf{a}^{T}\big{[}\mathbf{x}_{i}\mathbf{W}_{m}\|\mathbf{x}_{j}\mathbf{W}_{m}\big{]}\big{)}$

where $\mathbf{W}_{m}\in\mathbb{R}^{d\times F}$ and $\beta_{i,j}$ is defined
as learnable interpolation coefficients with the shared attention weight
$\mathbf{a}$. Then, we take augmented neighboring nodes as positive samples
and other non-neighboring nodes as negative samples to simultaneously model
the connectivity and disconnectivity between nodes. Specifically, the learning
objective of feature-level self-distillation is defined as follows,

$\displaystyle\mathcal{L}_{\text{feat}}=\frac{1}{N}\sum_{i=1}^{N}\Big{(}\frac{1}{|\mathcal{N}_{i}|}\sum_{j\in\mathcal{N}_{i}}\big{\|}\mathbf{y}_{i}-\mathbf{z}^{\prime}_{i,j}\big{\|}_{2}^{2}-\frac{1}{M_{i}}\sum_{e_{i,k}\notin\mathcal{E}}\big{\|}\widehat{\mathbf{y}}_{i}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}\Big{)}$
(4)

where $M_{i}\\!=\\!|\mathcal{E}|\\!-\\!|\mathcal{N}_{i}|\\!-\\!1$ is the
number of negative samples of node $v_{i}$. Besides,
$\widehat{\mathbf{y}}_{i}=\textrm{softmax}(\mathbf{y}_{i})\in\mathbb{R}^{C}$
and
$\widehat{\mathbf{z}}_{k}=\textrm{softmax}(\mathbf{z}_{k})\in\mathbb{R}^{C}$.
The loss function $\mathcal{L}_{\text{feat}}$ defined in Eq. (4) essentially
encourages positive neighboring nodes to be closer and pushes negative non-
neighboring nodes away.

####  4.2.2 Label-level Self-Distillation

Thus far, we have only discussed how to use node features $\mathbf{X}$ and
structural information $\mathcal{E}$ for feature-level self-distillation, but
have not explored how to leverage node labels $\mathcal{Y}_{L}$. A widely used
solution to leverage labels is to optimize the objective on the labeled data
$\mathcal{V}_{L}$ as follows

$\min_{\theta,\mathbf{W}^{(0)},\cdots,\mathbf{W}^{(L-1)}}\frac{1}{|\mathcal{V}_{L}|}\sum_{i\in\mathcal{V}_{L}}\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{y}}_{i}),$
(5)

where $\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{y}}_{i})$ denotes the cross-
entropy loss between $\widehat{\mathbf{y}}_{i}$ and ground-truth label
$y_{i}$. However, Eq. (5) only considers node labels $\mathcal{Y}_{L}$, but
completely ignores structural information $\mathcal{E}$. In practice, label
propagation [58] is widely used as an effective trick to simultaneously model
label and structural information and achieves promising results for various
GNNs. However, label propagation involves the explicit coupling of labels with
the structure, so it is heavily data-dependent with the same inference-latency
problem as message passing. Earlier, we have proposed feature-level self-
distillation of Eq. (4) to substitute explicit message passing in Eq. (1), and
next we introduce implicit label-level self-distillation of Eq. (6) to
substitute explicit label propagation to jointly exploit both label and
structural information. The objective of label-level self-distillation is
defined as

$\mathcal{L}_{\text{label}}=\frac{1}{|\mathcal{V}_{L}|}\sum_{i\in\mathcal{V}_{L}}\Big{(}\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{y}}_{i})+\sum_{j\in\mathcal{N}_{i}}\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{z}}_{j})\Big{)}.$
(6)

###  4.3 Training and Inferring

####  4.3.1 Model Training

In practice, directly optimizing Eq. (4)(6) faces two tricky challenges: (1)
it treats all non-neighboring nodes as negative samples, which suffers from a
huge computational burden; and (2) it performs the summation over the entire
set of nodes, i.e, requiring a large memory space for keeping the entire
graph. To address these two problems, we adopt the edge sampling strategy [59,
60] instead of feeding the entire graph into the memory for _batch-style
training_. More specifically, we first sample mini-batch edges
$\mathcal{E}_{b}\in\mathcal{E}$ from the entire edge set. Then we randomly
sample negative nodes by a pre-defined negative distribution $P_{k}(v)$ for
each edge $e_{i,j}\in\mathcal{E}_{b}$ instead of enumerating all non-
neighboring nodes as negative samples. Finally, we can rewrite Eq. (4) as

$\displaystyle\mathcal{L}_{\text{feat}}\\!=\\!\frac{1}{B}\sum_{b=1}^{B}$
$\displaystyle\sum_{e_{i,j}\in\mathcal{E}_{b}}\bigg{(}\big{\|}\mathbf{y}_{i}-\mathbf{z}^{\prime}_{i,j}\big{\|}_{2}^{2}+\big{\|}\mathbf{y}_{j}-\mathbf{z}^{\prime}_{j,i}\big{\|}_{2}^{2}$
(7) $\displaystyle-\mathbb{E}_{{v_{k}}\sim
P_{k}(v)}\Big{(}\big{\|}\widehat{\mathbf{y}}_{i}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}+\big{\|}\widehat{\mathbf{y}}_{j}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}\Big{)}\bigg{)},$

where $B$ is the batch size, and $P_{k}(v)$ adopts the uniform distribution by
default, that is $P_{k}(v_{i})=\frac{1}{N}$ for each node $v_{i}$. $P_{k}(v)$
can also be pre-defined based on prior knowledge, e.g., degree distribution,
but in practice, we find from the experimental results in Sec. 5.5 that
uniform distribution is a reasonable choice that can yield fairly good
performance across various datasets. Similarly, we can rewrite the label-level
self-distillation in Eq. (6) as a batch-style formulation,

$\displaystyle\mathcal{L}_{\text{label}}=\frac{1}{B}\sum_{b=1}^{B}\frac{1}{|\mathcal{V}_{b}|}\sum_{i\in\mathcal{V}_{b}}\Big{(}\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{y}}_{i})+\sum_{e_{i,j}\in\mathcal{E}_{b}}\mathcal{L}_{CE}(y_{i},\widehat{\mathbf{z}}_{j})\Big{)},$
(8)

where
$\mathcal{V}_{b}=\\{v_{i},v_{j}|e_{i,j}\in\mathcal{E}_{b}\\}\cap\mathcal{V}_{L}$
is all the sampled nodes in $\mathcal{E}_{b}$. Finally, the total training
loss can be defined as,

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{label}}+\alpha\mathcal{L}_{\text{feat}},$
(9)

where $\alpha$ is a trade-off hyperparameter.

####  4.3.2 Model Inferring

Once the model training is completed, we can directly omit the inference layer
$g_{\gamma}(\cdot)$ and retain the backbone MLP architecture and the inference
layer $f_{\theta}(\cdot)$ for label prediction. At this time, there is no data
dependency for model inference, and this is attributed to the fact that we
have shifted a considerable amount of work from the latency-sensitive
inference stage to the latency-insensitive training stage. The pseudo-code of
TGS is summarized in Algorithm. 1.

Algorithm 1 Algorithm for the proposed TGS framework

1:Features: $\mathbf{X}$; Edge Set: $\mathcal{E}$; # Batch: $B$; # Epoch: $E$.

2:Initialize parameters $\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$,
$f_{\theta}(\cdot)$, and $g_{\gamma}(\cdot)$.

3:for $epoch$ $\in$ {0,1,$\cdots$,$E-1$} do

4: for $b$ $\in$ {0,1,$\cdots$,$B-1$} do

5: Sample a mini-batch of edges $\mathcal{E}_{b}$ from $\mathcal{E}$;

6: Compute losses $\mathcal{L}_{\text{feat}}$ and $\mathcal{L}_{\text{label}}$
by Eq. (7)(8);

7: Sum up $\mathcal{L}_{\text{feat}}$ and $\mathcal{L}_{\text{label}}$ as
total loss $L_{\text{total}}$;

8: Update parameters by back-propagation of $L_{\text{total}}$.

9: end for

10:end for

11:Predict labels $\mathcal{Y}_{U}$ for those unlabeled nodes
$\mathcal{V}_{U}$.

12:return Predictions $\mathcal{Y}_{U}$, parameters
$\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$ and $f_{\theta}(\cdot)$.

###  4.4 Discussion and Comparison

Comparison with Message and Label Propagation. The core of GNNs is the use of
structural information, where _message passing_ and _label propagation_ are
the two dominant schemes. The message passing models the data dependency
within a local subgraph through neighborhood feature aggregation, while label
propagation focuses on diffusing label information to the neighborhood, and
_they are complementary to each other_. However, both of them involve explicit
coupling of features/labels with structures, leading to data dependency and
inference latency. Different from the explicit message passing and label
propagation, we use structural information as prior, as shown in Fig. 1, to
implicitly guide bidirectional dual knowledge self-distillation: (1) feature-
level self-distillation from neighborhood to the target node as in Eq. (4) and
(2) label-level self-distillation from the target node to the neighborhood as
in Eq. (6), where structural information is never explicitly involved in the
forward propagation. Furthermore, while Eq. (4)(6) is defined on the 1-hop
neighborhood, it can aggregate messages from multi-hops away by multiple
cascades of self-distillation. For example, if there is knowledge self-
distillation between the target node and its 1-hop neighbor and between its
1-hop neighbor and its 2-hop neighbor, then as the training proceeds, the
messages from the 2-hop neighbor will first be distilled to the 1-hop
neighbor, and then progressively propagated to the target node in such a
”cascading” manner.

Comparison with Graph Contrastive Learning. Another research topic that is
close to ours is graph contrastive learning, but TGS differs from it in the
following two aspects: (1) learning objective, graph contrastive learning
mainly aims to learn transferable knowledge from abundant unlabeled data in an
_unsupervised_ setting and then generalize the learned knowledge to downstream
tasks. Instead, TGS works in a _semi-supervised_ setting, i.e., the label
information is available during training. (2) augmentation, graph contrastive
learning usually requires multiple types of sophisticated augmentation to
obtain different views for contrasting. However, TGS augments only positive
samples by simple linear interpolation (mixup). Overall, we believe that TGS
is more highly relevant to knowledge distillation than to graph contrastive
learning, although we are aware that the two research topics have some aspects
in common.

Time Complexity Analysis. The training time complexity of the proposed TGS
framework is $\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$, which is linear
with respect to the number of nodes $|\mathcal{V}|$ and edges $|\mathcal{E}|$,
and is in the same order of magnitude as GCNs. However, with the removal of
neighborhood fetching, the inference time complexity can be reduced from
$\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ to
$\mathcal{O}(|\mathcal{V}|dF)$ of MLPs. A comparison of the inference speeds
of the various methods can be found in Sec. 5.4 and Table. IV.

TABLE I: An overview summary of the statistical characteristics of datasets.

Dataset | Cora | Citeseer | Amazon-Photo | Coauthor-CS | Coauthor-Phy | Amazon-Com |
---|---|---|---|---|---|---|---
$\\#$ Nodes | 2708 | 3327 | 7650 | 18333 | 34493 | 13752 |
$\\#$ Edges | 5278 | 4614 | 119081 | 81894 | 247962 | 245861 |
$\\#$ Features | 1433 | 3703 | 745 | 6805 | 8415 | 767 |
$\\#$ Classes | 7 | 6 | 8 | 15 | 5 | 10 |
Label Rate | 5.2% | 3.6% | 2.1% | 1.6% | 0.3% | 1.5% |

TABLE II: Classification accuracy $\pm$ std (%) on six real-world datasets,
where the best and second results marked by bold and underline, respectively.
If not specifically mentioned, all relevant models adopt GCN as the backbone
by default.

Type | Method | Cora | Citeseer | Coauthor-CS | Coauthor-Phy | Amazon-Com | Amazon-Photo | Avg. Rank
---|---|---|---|---|---|---|---|---
MLPs / GNNs | MLP | $61.86_{\pm 0.43}$ | $59.76_{\pm 0.51}$ | $83.34_{\pm 0.64}$ | $86.24_{\pm 0.66}$ | $66.85_{\pm 1.94}$ | $78.18_{\pm 1.25}$ | 16.00
GCN | $81.28_{\pm 0.42}$ | $71.06_{\pm 0.44}$ | $87.76_{\pm 0.43}$ | $91.89_{\pm 0.42}$ | $77.45_{\pm 1.71}$ | $87.53_{\pm 1.64}$ | 13.33
GAT | $83.02_{\pm 0.45}$ | $72.56_{\pm 0.51}$ | $88.55_{\pm 0.56}$ | $92.36_{\pm 0.47}$ | $82.78_{\pm 1.89}$ | $90.19_{\pm 1.35}$ | 6.33
GraphSAGE | $82.22_{\pm 0.80}$ | $71.22_{\pm 0.58}$ | $88.40_{\pm 0.48}$ | $91.88_{\pm 0.53}$ | $79.23_{\pm 1.63}$ | $88.63_{\pm 1.17}$ | 10.67
APPNP | $83.28_{\pm 0.33}$ | $71.74_{\pm 0.27}$ | $88.74_{\pm 0.62}$ | $92.75_{\pm 0.60}$ | $81.28_{\pm 1.90}$ | $89.49_{\pm 1.28}$ | 6.50
DAGNN | $\underline{84.30}_{\pm 0.51}$ | $73.14_{\pm 0.62}$ | $89.32_{\pm 0.55}$ | $93.10_{\pm 0.67}$ | $80.32_{\pm 1.57}$ | $\textbf{90.72}_{\pm 1.45}$ | 3.67
GNN-to-GNN | TinyGNN | $82.79_{\pm 0.57}$ | $72.67_{\pm 0.72}$ | $88.72_{\pm 0.42}$ | $92.20_{\pm 0.67}$ | $79.22_{\pm 1.69}$ | $89.24_{\pm 1.24}$ | 8.67
FreeKD | $83.80_{\pm 0.53}$ | $73.76_{\pm 0.60}$ | $89.14_{\pm 0.61}$ | $92.63_{\pm 0.71}$ | $80.92_{\pm 1.75}$ | $89.46_{\pm 1.31}$ | 5.17
GNN Self-Distillation | GNN-SD | $81.85_{\pm 0.55}$ | $71.69_{\pm 0.61}$ | $87.80_{\pm 0.50}$ | $92.07_{\pm 0.48}$ | $77.66_{\pm 1.85}$ | $87.80_{\pm 1.52}$ | 11.67
RDD | $83.68_{\pm 0.40}$ | $73.63_{\pm 0.50}$ | $\underline{89.38}_{\pm 0.44}$ | $92.74_{\pm 0.78}$ | $81.84_{\pm 1.48}$ | $89.70_{\pm 0.93}$ | 3.50
GNN-to-MLP | GNN-MLP | $64.53_{\pm 0.42}$ | $62.26_{\pm 0.48}$ | $81.26_{\pm 0.87}$ | $86.47_{\pm 0.71}$ | $69.25_{\pm 1.75}$ | $76.34_{\pm 1.30}$ | -
GLNN | $80.85_{\pm 0.60}$ | $71.21_{\pm 0.80}$ | $87.81_{\pm 0.53}$ | $91.83_{\pm 0.60}$ | $77.96_{\pm 1.70}$ | $87.98_{\pm 1.36}$ | 11.29
CPF | $83.65_{\pm 0.49}$ | $72.98_{\pm 0.47}$ | $89.10_{\pm 0.50}$ | $92.36_{\pm 0.63}$ | $80.90_{\pm 1.52}$ | $89.03_{\pm 1.29}$ | 6.67
FF-G2M | $84.06_{\pm 0.43}$ | $\underline{73.85}_{\pm 0.51}$ | $88.96_{\pm 0.45}$ | $92.83_{\pm 0.58}$ | $81.92_{\pm 1.54}$ | $89.58_{\pm 1.43}$ | 3.83
MLP Self-Distillation | DGI-MLP | $76.39_{\pm 0.46}$ | $67.83_{\pm 0.51}$ | $85.13_{\pm 0.49}$ | $89.41_{\pm 0.45}$ | $73.25_{\pm 1.64}$ | $84.60_{\pm 1.60}$ | -
Graph-MLP | $81.45_{\pm 0.52}$ | $72.87_{\pm 0.70}$ | $88.16_{\pm 0.70}$ | $91.85_{\pm 0.49}$ | $77.23_{\pm 1.76}$ | $87.64_{\pm 1.37}$ | 11.83
LinkDist | $76.70_{\pm 0.47}$ | $65.19_{\pm 0.55}$ | $87.89_{\pm 0.58}$ | $92.16_{\pm 0.70}$ | $76.93_{\pm 1.83}$ | $87.26_{\pm 1.42}$ | 13.67
| TGS (ours) | $\textbf{84.90}_{\pm 0.44}$ | $\textbf{74.08}_{\pm 0.69}$ | $\textbf{89.62}_{\pm 0.40}$ | $\textbf{94.96}_{\pm 0.41}$ | $\underline{83.44}_{\pm 2.09}$ | $90.34_{\pm 0.85}$ | 1.17
|  w/o mixup | $83.18_{\pm 0.41}$ | $72.96_{\pm 0.58}$ | $89.18_{\pm 0.61}$ | $93.24_{\pm 0.56}$ | $82.94_{\pm 1.90}$ | $89.85_{\pm 0.81}$ | -
|  w/ triplet loss | $84.10_{\pm 0.54}$ | $73.60_{\pm 0.63}$ | $89.12_{\pm 0.45}$ | $\underline{93.86}_{\pm 0.48}$ | $\textbf{83.52}_{\pm 2.10}$ | $\underline{90.57}_{\pm 0.78}$ | -

##  5 Experiments

In this section, we evaluate TGS on six real-world datasets by answering five
questions. Q1: How does TGS compare with SOTA graph knowledge distillation
methods? Q2: Is TGS robust under limited labeled data and label noise? Q3: How
does TGS compare with other general GNN models and inference acceleration
methods in terms of inference speed? Q4: How does TGS benefit from negative
samples, mixup-like augmentation, and feature/label self-distillation? Q5: How
do two key hyperparameters, loss weight $\alpha$ and batch size $B$, influence
the performance?

###  5.1 Experimental Setup

####  5.1.1 Datasets

The experiments are conducted on six widely used real-world datasets,
including Cora [61], Citeseer [62], Coauthor-CS, Coauthor-Physics, Amazon-Com,
and Amazon-Photo [63]. For the two small-scale datasets, Cora and Citeseer, we
follow the data splitting strategy in [35]. For the four large-scale datasets,
Coauthor-CS, Coauthor-Physics, Amazon-Photo, and Amazon-Computers, we follow
[25, 33] to select 20 nodes per class to construct a training set, 500 nodes
for validation, and 1000 nodes for testing. A summary of the statistical
characteristics of datasets is given in Table. I.

####  5.1.2 Baselines

In this paper, we consider the following six classical baselines: Vanilla MLP,
GCN [35], GAT [14], GraphSAGE [15], APPNP [36], and DAGNN [64]. In addition,
we compare TGS with four types of graph knowledge distillation methods,
including (1) GNN-to-GNN KD: FreeKD [42] and TinyGNN [17], (2) GNN-to-MLP KD:
CPF [26], GLNN [25], and FF-G2M [27], (3) GNN Self-Distillation: RDD [31] and
GNN-SD [30], and (3) MLP Self-Distillation: Graph-MLP [32] and LinkDist [33].
Moreover, with GCN as the base architecture, three classical inference
acceleration methods are compared, including pruning with 50% weights (P-GCN)
[38], quantization from FP32 to INT8 (Q-GCN) [39], and neighborhood sampling
with fan-out 15 (NS-GCN) [40].

####  5.1.3 Hyperparameter

The hyperparameters are set the same for all datasets: Adam optimizer with
learning rate $lr$ = 0.01 and weight decay $decay$ = 5e-4; Epoch $E$ = 200;
Layer number $L$ = 2. The other dataset-specific hyperparameters are
determined by an AutoML toolkit NNI with the hyperparameter search spaces as
hidden dimension $F=\\{256,512,1024\\}$; batch size
$B=\\{256,512,1024,4096\\}$, trade-off weight $\alpha=\\{0.5,0.8,1.0\\}$. Each
set of experiments is run five times with different random seeds, and the
average accuracy and standard deviation are reported as metrics. Moreover, the
experiments are implemented based on the standard implementation in PyTorch
1.6.0 library with Intel(R) Xeon(R) Gold 6240R @ 2.40GHz CPU and NVIDIA V100
GPU.

TABLE III: Classification accuracy $\pm$ std (%) with limited node labels,
where the best results marked by bold.

Dataset | GCN | ANNPN | DAGNN | GNN-SD | RDD | GLNN | CPF | FF-G2M | Graph-MLP | TGS (ours)
---|---|---|---|---|---|---|---|---|---|---
Cora | 5 labels | $73.10_{\pm 0.87}$ | $76.39_{\pm 0.95}$ | $79.02_{\pm 0.94}$ | $74.32_{\pm 0.85}$ | $76.11_{\pm 0.81}$ | $73.85_{\pm 0.92}$ | $75.92_{\pm 0.90}$ | $77.62_{\pm 0.67}$ | $78.43_{\pm 0.78}$ | $\textbf{80.22}_{\pm 0.87}$
10 labels | $77.52_{\pm 0.63}$ | $79.99_{\pm 0.72}$ | $81.99_{\pm 0.62}$ | $78.55_{\pm 0.64}$ | $79.68_{\pm 0.54}$ | $77.94_{\pm 0.52}$ | $79.20_{\pm 0.64}$ | $79.20_{\pm 0.44}$ | $79.60_{\pm 0.49}$ | $\textbf{82.80}_{\pm 0.45}$
15 labels | $79.47_{\pm 0.56}$ | $80.70_{\pm 0.44}$ | $82.72_{\pm 0.50}$ | $79.89_{\pm 0.55}$ | $80.45_{\pm 0.46}$ | $79.16_{\pm 0.48}$ | $80.12_{\pm 0.51}$ | $80.25_{\pm 0.52}$ | $80.32_{\pm 0.57}$ | $\textbf{83.40}_{\pm 0.53}$
Citeseer | 5 labels | $63.23_{\pm 1.04}$ | $66.28_{\pm 0.88}$ | $69.13_{\pm 0.68}$ | $63.93_{\pm 0.74}$ | $66.06_{\pm 0.57}$ | $63.10_{\pm 0.70}$ | $65.40_{\pm 0.62}$ | $68.12_{\pm 0.59}$ | $\textbf{69.64}_{\pm 0.64}$ | $68.76_{\pm 1.16}$
10 labels | $67.55_{\pm 0.50}$ | $69.23_{\pm 0.64}$ | $71.74_{\pm 0.71}$ | $68.20_{\pm 0.57}$ | $69.68_{\pm 0.66}$ | $67.65_{\pm 0.62}$ | $69.14_{\pm 0.73}$ | $70.76_{\pm 0.53}$ | $70.56_{\pm 0.51}$ | $\textbf{72.66}_{\pm 0.43}$
15 labels | $69.64_{\pm 0.58}$ | $70.17_{\pm 0.44}$ | $72.26_{\pm 0.53}$ | $69.86_{\pm 0.48}$ | $70.32_{\pm 0.57}$ | $69.52_{\pm 0.52}$ | $70.76_{\pm 0.46}$ | $71.59_{\pm 0.47}$ | $71.80_{\pm 0.63}$ | $\textbf{73.10}_{\pm 0.53}$

(a) Robustness on the Cora dataset

(b) Robustness on the Citeseer dataset

(c) Inference time with different layers

Figure 3: (a)(b) Classification accuracy (%) under different label noise
ratios on the Cora and Citeseer datasets, respectively. (c) Inference time
(ms) with different layers on the Coauthor-CS dataset.

###  5.2 Performance Comparison (Q1)

To answer Q1, we conducted experiments on six real-world datasets with
comparison with state-of-the-art baselines. From the results reported in
Table. II, we can observe that: (1) There are some GNN-to-MLP KD and GNN Self-
Distillation methods, such as RDD and FF-G2M, that can achieve comparable or
even better performance than GNN-to-GNN KD, suggesting that both GNNs and
teachers may not be necessary for graph knowledge distillation. (2)
Considering both teacher-free and GNN-less designs, existing MLP Self-
Distillation methods, such as Graph-MLP and LinkDist, lag far behind GNN-to-
GNN KD and cannot even match the performance of vanilla GCNs on some datasets.
(3) Regarding classification accuracy, TGS consistently achieves the best
overall performance on six datasets. For example, TGS obtains the best
performance on the Coauthor-Phy dataset, and more notably, our classification
accuracy is 3.13% and 2.89% higher than that of GLNN and GNN-SD. (4) We
additionally consider two vanilla baselines: (i) GNN-MLP, which trains a GNN
and then directly copies its weights to an MLP for inference, and its
performance is only close to or even poorer than that of vanilla MLP; and (ii)
DGI-MLP, which performs contrastive learning with DGI but uses MLP as the
backbone, and its performance is far superior to that of MLP (suggesting the
potential of the MLP architecture), but still inferior to that of vanilla GCN.
(5) We conduct ablation studies on two important modules, namely mixup
augmentation and loss function. It can be seen that mixup augmentation plays
an important role in the performance. However, even with the removal of the
mixup, TGS still outperforms GCN, GAT, Graph-MLP, and LinkDist, on all six
datasets. However, when mixup is considered, the performance of TGS is even
comparable to those state-of-the-art graph KD methods. Besides, we find that
using triple loss as the objective function slightly degrades performance on
some datasets, but helps improve performance on the Amazon-Com and Amazon-
Photo datasets.

###  5.3 Evaluation on Robustness (Q2)

There has been some work pointing out that the performance of graph learning
algorithms depends heavily on the quality and quantity of the labels. To
evaluate the robustness of the TGS framework, we evaluate it with extremely
limited label data and label noise on the Cora and Citeseer datasets.

####  5.3.1 Performance with Limited Labels

To evaluate the effectiveness of TGS when labeled data is limited, we randomly
select 5, 10, and 15 labeled samples per class for training, and the rest of
the training set is considered unlabeled. From the experimental results
reported in Table. III, we can make three important observations (1) The
performance of all methods drops as the number of labeled data is reduced, but
that of TGS drops more slightly. (2) While GNN-to-MLP KD and GNN Self-
Distillation methods perform well on clean data, as shown in Table. II, their
performance gains are reduced when labeled data is extremely limited. In
contrast, the two MLP Self-Distillation methods, Graph-MLP and TGS, show a
great advantage under the label-limited setting. (3) When only a limited
number of labels are provided, TGS outperforms all other baselines at most
label rates. For example, when trained with 5 labels per class, TGS
outperforms GCN by 7.12% and 5.53% on the Cora and Citeseer datasets,
respectively.

####  5.3.2 Performance with Noisy Labels

We evaluate the robustness of TGS against label noise by injecting asymmetric
noise into class labels, where the label $i$ $(1\leq i\leq C)$ of each
training sample flips independently with probability $r$ to another class, but
with probability $1-r$ preserved as label $i$ [65, 66]. The performance is
reported in Fig. 3(a) and Fig. 3(b) at various noise ratios
$r\in\\{0\%,10\%,20\%,\cdots 60\%\\}$. It can be seen that as noise ratio $r$
increases, the accuracy of TGS drops more slowly than other baselines,
suggesting that TGS is more robust than other baselines under various noise
ratios, especially under extremely high noise ratios. For example, with
$r=60\%$ label noise ratio, TGS outperforms RDD and FF-G2M by 6.92% and 6.41%
on the Citeseer dataset, respectively.

###  5.4 Evaluation on Inference Speed (Q3)

Commonly used inference acceleration techniques on GNNs include Pruning [38],
Quantizing [39], and Neighborhood Sampling [40]. With GCN as the base
architecture, we consider its three variants: P-GCN, Q-GCN, and NS-GCN.

####  5.4.1 Comparison on Different Datasets

With the removal of neighborhood fetching, the inference time of TGS can be
reduced from $\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ to
$\mathcal{O}(|\mathcal{V}|dF)$. The inference time ($ms$) averaged over 30
sets of runs on four datasets is reported in Table. IV with the acceleration
multiple w.r.t the vanilla GCN marked as ${\color[rgb]{0.4,0.71,0.376}green}$,
where all methods use $L=2$ layers and dimension $F=16$. From Table. IV, we
can observe that: (1) While APPNP and DAGNN improve a lot over GCN in
classification accuracy, as shown in Table. III, they suffer from more severe
inference latency. (2) Pruning and quantization are not very effective on
GNNs, given that data dependency in GNNs has not been well resolved. Besides,
the neighborhood sampling considers but does not completely eliminate the
neighborhood-fetching latency, so it infers faster than pruning and
quantization, yet still lags far behind TGS which is based on MLPs. (3) TGS
infers fastest across four datasets.

TABLE IV: Inference time ($ms$) on four datasets, where three commonly used
inference acceleration methods help to speed up GCN, but still considerably
slower than GSDN. Note the acceleration multiple w.r.t the vanilla GCN is
marked as ${\color[rgb]{0.4,0.71,0.376}green}$.

Method | GCN | APPNP | DAGNN | P-GCN | Q-GCN | NS-GCN | TGS (ours)
---|---|---|---|---|---|---|---
Cora | 402.7 | 731.8 | 647.6 | 372.9 (${\color[rgb]{0.4,0.71,0.376}1.08\times}$) | 383.5 (${\color[rgb]{0.4,0.71,0.376}1.05\times}$) | 97.7 (${\color[rgb]{0.4,0.71,0.376}4.12\times}$) | 4.6 (${\color[rgb]{0.4,0.71,0.376}87.54\times}$)
Citeseer | 383.4 | 700.5 | 727.5 | 316.9 (${\color[rgb]{0.4,0.71,0.376}1.21\times}$) | 361.4 (${\color[rgb]{0.4,0.71,0.376}1.06\times}$) | 105.6 (${\color[rgb]{0.4,0.71,0.376}3.63\times}$) | 4.3 (${\color[rgb]{0.4,0.71,0.376}89.16\times}$)
Coauthor-CS | 593.6 | 1099.1 | 672.4 | 498.8 (${\color[rgb]{0.4,0.71,0.376}1.19\times}$) | 566.3 (${\color[rgb]{0.4,0.71,0.376}1.05\times}$) | 129.2 (${\color[rgb]{0.4,0.71,0.376}4.59\times}$) | 7.9 (${\color[rgb]{0.4,0.71,0.376}75.14\times}$)
Coauthor-Phy | 1067.6 | 1467.4 | 708.6 | 922.3 (${\color[rgb]{0.4,0.71,0.376}1.16\times}$) | 997.8 (${\color[rgb]{0.4,0.71,0.376}1.07\times}$) | 269.3 (${\color[rgb]{0.4,0.71,0.376}3.96\times}$) | 12.8 (${\color[rgb]{0.4,0.71,0.376}83.41\times}$)

####  5.4.2 Comparison with Different Layers

To infer a single node with a $L$-layer GNN on a graph with average node
degree $R$, it requires fetching $\mathcal{O}(R^{L})$ nodes. To compare the
sensitivity of TGS to layer depth with other inference acceleration methods,
we report their inference time ($ms$ in log-scale) at different layer depths
on the Coauthor-CS dataset in Fig. 3(c). The inference time of TGS only
increases linearly with the layer depth, but that of other baselines increases
exponentially. Moreover, the speed gains of GNN pruning and quantization are
reduced as the layer depth increases, and they approach the vanilla GCN when
the layer depth is 4. In contrast, at larger layer depths, the speed gain of
the neighbor sampling gets enlarged compared to the vanilla GCN. This
demonstrates that neighborhood fetching is one major source of inference
latency in GNNs, and the linear complexity of TGS has a great advantage,
especially when GNNs become deeper.

(a) Ablation Study

(b) Learning curves

(c) Mean cosine similarity curves

Figure 4: (a) Ablation study on four key model components. (b) Learning curves
of MLPs and TGS on the Cora dataset, showing that self-distillation helps to
regularize the training. The _logarithmized_ vertical coordinate is the cross-
entropy loss between the predicted and ground-truth labels on the training or
validation set, respectively. (c) Mean cosine similarity curves of MLPs, GCNs,
and TGS between the target node with 1-hop and 2-hop neighbors on the Cora
dataset.

###  5.5 Ablation Study (Q4)

####  5.5.1 Component Analysis

To evaluate the effectiveness of negative samples in Eq. (4), mixup-like
augmentation in Eq. (3), and label self-distillation in Eq. (8), we conducted
four sets of experiments: the model without (A) Negative Samples (w/o NS); (B)
mixup-like augmentation (w/o augment); (C) Label Self-Distillation (w/o LSD);
and (D) the full model. Besides, to evaluate the impact of the negative
distribution, we take the nodal degree $d_{i}$ as a prior and preset
$P_{k}(v_{i})=\frac{d_{i}}{|\mathcal{E}|}$ in place of the default uniform
distribution in this paper, denoted as (E) w/o uniform. After analyzing the
results in Fig. 4(a), we can conclude that: (1) Negative Samples and mixup-
like augmentation contribute to improving classification performance. More
importantly, applying them together can further improve performance on top of
each. (2) Label self-distillation helps to improve performance on top of the
stand-alone feature self-distillation. (3) Even without considering any graph
prior, presetting $P_{k}(\cdot)$ as uniform distribution is sufficient to
achieve comparable performance, so this paper defaults to the simplest uniform
distribution without considering complex prior-based distributions.

####  5.5.2 How TGS Benefit from Self-Distillation

Previous attempts have shown that there do exist optimal MLP parameters
enabling its performance to be competitive with GNNs, but it is hard to learn
such parameters by only cross-entropy loss [32, 33]. The proposed TGS helps to
solve this problem with two potential advantages: (1) alleviating overfitting,
and (2) introducing graph topology [25].

Firstly, we plot the training curves (with log-scale vertical coordinate) of
TGS and MLPs on the Cora dataset in Fig. 4(b), from which we observe that the
gap between training and validation loss is smaller for TGS than MLPs, which
indicates that TGS helps to alleviate the overfitting trend of MLPs. Secondly,
we conjecture that the absence of inductive bias, e.g., graph topology, is one
of the major reasons why MLP is inferior to GNN in inference accuracy. To
illustrate it, we plot in Fig. 4(c) the average cosine similarity of nodes
with their 1-hop and 2-hop neighbors for MLPs, GCNs, and TGS on the Cora
dataset. It can be seen that the average similarity with 1-hop neighbors is
always higher than that with 2-hop neighbors throughout the training process
for MLPs, GCNs, and TGS. More importantly, the average similarity of GCNs and
TGS gradually increases with training, while that of MLPs gradually decreases,
which indicates that TGS has introduced graph topology as an inductive bias
(as GCN has done), while MLP does not. As a result, our TGS enjoys the
benefits of topology-awareness in training but without neighborhood-fetching
latency in inference.

###  5.6 Hyperparameter Sensitivity Analysis (Q5)

To answer Q5, we evaluate the hyperparameter sensitivity w.r.t two key
hyperparameters: trade-off weight $\alpha\in\\{0.0,0.1,0.3,0.5,0.8,1.0\\}$ and
batch size $B\in\\{256,512,1024,2048,4096\\}$ in Fig. 5, from which we can
make two key observations that (1) batch size $B$ is a dataset-specific
hyperparameter. For simple graphs with few nodes and edges, such as Cora, a
small batch size, $B=256$, can yield fairly good performance. However, for
large-scale graphs with more nodes and edges, such as Coauthor-Phy, the model
performance usually improves with the increase of batch size $B$. (2) When
$\alpha$ is set to 0, i.e., the feature-level self-distillation is completely
removed, the performance of TGS degrades to be close to that of MLPs. In
contrast, when $\alpha$ takes a non-zero value, the performance of TGS
improves as $\alpha$ increases. However, when $\alpha$ becomes too large, it
weakens the benefit of label information, yielding lower performance
improvements. In practice, we can usually determine $B$ and $\alpha$ by
selecting the model with the highest accuracy on the validation set through
the grid search.

Figure 5: Hyperparameter sensitivity analysis on the batch size $B$ (Left) and
trade-off weight $\alpha$ (Right) on four datasets.

##  6 Conclusion

Motivated by the complementary strengths and weaknesses of GNNs and MLPs, we
propose _T eacher-Free Graph Self-Distillation_ (TGS) framework that does not
require both teacher models and GNNs during training and inference. The
proposed TGS framework is purely based on MLPs, where structural information
is only implicitly used to guide dual knowledge self-distillation between the
target node and its neighborhood. More importantly, we study TGS
comprehensively by investigating how they benefit from neighborhood self-
distillation and how they are different from existing works. Extensive
experiments have shown the advantages of TGS over existing methods in terms of
both inference accuracy and inference efficiency. Despite the great progress,
limitations still exist and a major concern is that TGS is based on the
neighborhood smoothing assumption (a common assumption adopted by various
models such as GCN, GAT, etc.), so how to extend TGS to heterophily graphs may
be a promising direction for future work.

##  7 Acknowledgement

This work was supported by National Key R&D Program of China (No.
2022ZD0115100), National Natural Science Foundation of China Project (No.
U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology
and Integrated Bioengineering of Westlake University.

## References

  * [1] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,” _IEEE transactions on neural networks and learning systems_ , 2020.
  * [2] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” _AI Open_ , vol. 1, pp. 57–81, 2020.
  * [3] L. Wu, H. Lin, Z. Liu, Z. Liu, Y. Huang, and S. Z. Li, “Homophily-enhanced self-supervision for graph structure learning: Insights and directions,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [4] L. Wu, J. Xia, Z. Gao, H. Lin, C. Tan, and S. Z. Li, “Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2022, pp. 519–535.
  * [5] L. Wu, H. Lin, B. Hu, C. Tan, Z. Gao, Z. Liu, and S. Z. Li, “Beyond homophily and homogeneity assumption: Relation-based frequency adaptive graph neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [6] H. Lin, Z. Gao, Y. Xu, L. Wu, L. Li, and S. Z. Li, “Conditional local convolution for spatio-temporal meteorological forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 7, 2022, pp. 7470–7478.
  * [7] M. Ma, P. Xie, F. Teng, B. Wang, S. Ji, J. Zhang, and T. Li, “Histgnn: Hierarchical spatio-temporal graph neural network for weather forecasting,” _Information Sciences_ , vol. 648, p. 119580, 2023.
  * [8] L. Wu, Y. Tian, Y. Huang, S. Li, H. Lin, N. V. Chawla, and S. Li, “MAPE-PPI: Towards effective and efficient protein-protein interaction prediction via microenvironment-aware protein embedding,” in _The Twelfth International Conference on Learning Representations_ , 2024. [Online]. Available: https://openreview.net/forum?id=itGkF993gz
  * [9] L. Wu, Y. Huang, C. Tan, Z. Gao, H. Lin, B. Hu, Z. Liu, and S. Z. Li, “Psc-cpi: Multi-scale protein sequence-structure contrasting for efficient and generalizable compound-protein interaction prediction,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2024.
  * [10] F. Zhou, Q. Yang, T. Zhong, D. Chen, and N. Zhang, “Variational graph neural networks for road traffic prediction in intelligent transportation systems,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 4, pp. 2802–2812, 2020.
  * [11] W. Jiang and J. Luo, “Graph neural network for traffic forecasting: A survey,” _Expert Systems with Applications_ , vol. 207, p. 117921, 2022.
  * [12] Z. Jia, S. Lin, R. Ying, J. You, J. Leskovec, and A. Aiken, “Redundancy-free computation for graph neural networks,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 997–1005.
  * [13] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _arXiv preprint arXiv:1611.07308_ , 2016.
  * [14] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” _arXiv preprint arXiv:1710.10903_ , 2017.
  * [15] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in _Advances in neural information processing systems_ , 2017, pp. 1024–1034.
  * [16] G. Li, M. Müller, B. Ghanem, and V. Koltun, “Training graph neural networks with 1000 layers,” _arXiv preprint arXiv:2106.07476_ , 2021.
  * [17] B. Yan, C. Wang, G. Guo, and Y. Lou, “Tinygnn: Learning efficient graph neural networks,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1848–1856.
  * [18] H. He, J. Wang, Z. Zhang, and F. Wu, “Compressing deep graph neural networks via adversarial knowledge distillation,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 534–544.
  * [19] H. Zhang, S. Lin, W. Liu, P. Zhou, J. Tang, X. Liang, and E. P. Xing, “Iterative graph self-distillation,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.
  * [20] F. M. Nardini, C. Rulli, S. Trani, and R. Venturini, “Distilled neural networks for efficient learning to rank,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 5, pp. 4695–4712, 2022.
  * [21] W. Liu, D. Gong, M. Tan, J. Q. Shi, Y. Yang, and A. G. Hauptmann, “Learning distilled graph for large-scale social network data clustering,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 32, no. 7, pp. 1393–1404, 2019.
  * [22] Y. Yang, J. Qiu, M. Song, D. Tao, and X. Wang, “Distilling knowledge from graph convolutional networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7074–7083.
  * [23] Y. Ren, J. Ji, L. Niu, and M. Lei, “Multi-task self-distillation for graph-based semi-supervised learning,” _arXiv preprint arXiv:2112.01174_ , 2021.
  * [24] L. Wu, H. Lin, Y. Huang, and S. Z. Li, “Knowledge distillation improves graph structure augmentation for graph neural networks,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [25] S. Zhang, Y. Liu, Y. Sun, and N. Shah, “Graph-less neural networks: Teaching old mlps new tricks via distillation,” _arXiv preprint arXiv:2110.08727_ , 2021.
  * [26] C. Yang, J. Liu, and C. Shi, “Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 1227–1237.
  * [27] L. Wu, H. Lin, Y. Huang, T. Fan, and S. Z. Li, “Extracting low-/high- frequency knowledge from graph neural networks and injecting it into mlps: An effective gnn-to-mlp distillation framework,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * [28] Anonymous, “Double wins: Boosting accuracy and efficiency of graph neural networks by reliable knowledge distillation,” in _Submitted to The Eleventh International Conference on Learning Representations_ , 2023, under review. [Online]. Available: https://openreview.net/forum?id=NGIFt6BNvLe
  * [29] L. Wu, H. Lin, Y. Huang, and S. Z. Li, “Quantifying the knowledge in gnns for reliable distillation into mlps,” _arXiv preprint arXiv:2306.05628_ , 2023.
  * [30] Y. Chen, Y. Bian, X. Xiao, Y. Rong, T. Xu, and J. Huang, “On self-distilling graph neural network,” _arXiv preprint arXiv:2011.02255_ , 2020.
  * [31] W. Zhang, X. Miao, Y. Shao, J. Jiang, L. Chen, O. Ruas, and B. Cui, “Reliable data distillation on graph convolutional network,” in _Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data_ , 2020, pp. 1399–1414.
  * [32] Y. Hu, H. You, Z. Wang, Z. Wang, E. Zhou, and Y. Gao, “Graph-mlp: Node classification without message passing in graph,” _arXiv preprint arXiv:2106.04051_ , 2021.
  * [33] Y. Luo, A. Chen, K. Yan, and L. Tian, “Distilling self-knowledge from contrastive links to classify graph nodes without passing messages,” _arXiv preprint arXiv:2106.08541_ , 2021.
  * [34] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” _arXiv preprint arXiv:1606.09375_ , 2016.
  * [35] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [36] J. Klicpera, A. Bojchevski, and S. Günnemann, “Predict then propagate: Graph neural networks meet personalized pagerank,” _arXiv preprint arXiv:1810.05997_ , 2018.
  * [37] Z. Zhang, P. Cui, and W. Zhu, “Deep learning on graphs: A survey,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 34, no. 1, pp. 249–270, 2020.
  * [38] S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” _arXiv preprint arXiv:1506.02626_ , 2015.
  * [39] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in _International conference on machine learning_. PMLR, 2015, pp. 1737–1746.
  * [40] J. Chen, T. Ma, and C. Xiao, “Fastgcn: fast learning with graph convolutional networks via importance sampling,” _arXiv preprint arXiv:1801.10247_ , 2018.
  * [41] X. Xu, F. Zhou, K. Zhang, and S. Liu, “Ccgl: Contrastive cascade graph learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 35, no. 5, pp. 4539–4554, 2022.
  * [42] K. Feng, C. Li, Y. Yuan, and G. Wang, “Freekd: Free-direction knowledge distillation for graph neural networks,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 357–366.
  * [43] A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Label propagation for deep semi-supervised learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5070–5079.
  * [44] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3713–3722.
  * [45] Y. Tian, S. Pei, X. Zhang, C. Zhang, and N. V. Chawla, “Knowledge distillation on graphs: A survey,” _arXiv preprint arXiv:2302.00219_ , 2023.
  * [46] L. Wu, H. Lin, C. Tan, Z. Gao, and S. Z. Li, “Self-supervised learning on graphs: Contrastive, generative, or predictive,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [47] P. Velickovic, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, “Deep graph infomax.” in _ICLR (Poster)_ , 2019.
  * [48] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” _Advances in neural information processing systems_ , vol. 33, pp. 5812–5823, 2020.
  * [49] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Graph contrastive learning with adaptive augmentation,” _arXiv preprint arXiv:2010.14945_ , 2020.
  * [50] D. Xu, W. Cheng, D. Luo, H. Chen, and X. Zhang, “Infogcl: Information-aware graph contrastive learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 30 414–30 425, 2021.
  * [51] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [52] S. Thakoor, C. Tallec, M. G. Azar, R. Munos, P. Veličković, and M. Valko, “Bootstrapped representation learning on graphs,” in _ICLR 2021 Workshop on Geometrical and Topological Representation Learning_ , 2021.
  * [53] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_. JMLR Workshop and Conference Proceedings, 2010, pp. 297–304.
  * [54] Z. T. Kefato and S. Girdzijauskas, “Self-supervised graph neural networks without explicit negative sampling,” _arXiv preprint arXiv:2103.14958_ , 2021.
  * [55] C. Park, D. Kim, J. Han, and H. Yu, “Unsupervised attributed multiplex network embedding,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 5371–5378.
  * [56] X. Yang, Z. Song, I. King, and Z. Xu, “A survey on deep semi-supervised learning,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [57] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [58] X. Zhu and Z. Ghahramani, “Learning from labeled and unlabeled data with label propagation,” 2002.
  * [59] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in neural information processing systems_ , 2013, pp. 3111–3119.
  * [60] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “Line: Large-scale information network embedding,” in _Proceedings of the 24th international conference on world wide web_ , 2015, pp. 1067–1077.
  * [61] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, “Collective classification in network data,” _AI magazine_ , vol. 29, no. 3, pp. 93–93, 2008.
  * [62] C. L. Giles, K. D. Bollacker, and S. Lawrence, “Citeseer: An automatic citation indexing system,” in _Proceedings of the third ACM conference on Digital libraries_ , 1998, pp. 89–98.
  * [63] O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, “Pitfalls of graph neural network evaluation,” _arXiv preprint arXiv:1811.05868_ , 2018.
  * [64] M. Liu, H. Gao, and S. Ji, “Towards deeper graph neural networks,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 338–348.
  * [65] N. Yin, L. Shen, M. Wang, X. Luo, Z. Luo, and D. Tao, “Omg: Towards effective graph classification against label noise,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.
  * [66] J. Xia, H. Lin, Y. Xu, C. Tan, L. Wu, S. Li, and S. Z. Li, “Gnn cleaner: Label cleaner for graph structured data,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.