# Teaching Yourself: Graph Self-Distillation on Neighborhood for Node
Classification

Lirong Wu, Jun Xia, Haitao Lin, Zhangyang Gao, Zicheng Liu, Guojiang Zhao, and
Stan Z. Li
AI Lab, School of Engineering, Westlake University
<EMAIL_ADDRESS>

###### Abstract

Recent years have witnessed great success in handling graph-related tasks with
Graph Neural Networks (GNNs). Despite their great _academic success_ , Multi-
Layer Perceptrons (MLPs) remain the primary workhorse for practical
_industrial applications_. One reason for this academic-industrial gap is the
neighborhood-fetching latency incurred by data dependency in GNNs, which make
it hard to deploy for latency-sensitive applications that require fast
inference. Conversely, without involving any feature aggregation, MLPs have no
data dependency and infer much faster than GNNs, but their performance is less
competitive. Motivated by these complementary strengths and weaknesses, we
propose a Graph Self-Distillation on Neighborhood (GSDN) framework to reduce
the gap between GNNs and MLPs. Specifically, the GSDN framework is based
purely on MLPs, where structural information is only implicitly used as prior
to guide knowledge self-distillation between the neighborhood and the target,
substituting the explicit neighborhood information propagation as in GNNs. As
a result, GSDN enjoys the benefits of graph topology-awareness in training but
has no data dependency in inference. Extensive experiments have shown that the
performance of vanilla MLPs can be greatly improved with self-distillation,
e.g., GSDN improves over stand-alone MLPs by 15.54% on average and outperforms
the state-of-the-art GNNs on six datasets. Regarding inference speed, GSDN
infers 75$\times$-89$\times$ faster than existing GNNs and
16$\times$-25$\times$ faster than other inference acceleration methods.

##  1 Introduction

Recently, Graph Neural Networks (GNNs) (Wu et al., 2020; Zhou et al., 2020)
have demonstrated their powerful capability to handle graph-related tasks.
Despite their great _academic success_ , practical deployments of GNNs in the
industry are still less popular, and MLPs remain the primary workhorse for
practical _industrial applications_. One reason for this academic-industrial
gap is the neighborhood-fetching inference latency incurred by data dependency
in GNNs (Jia et al., 2020).

Most existing GNNs rely on message passing to aggregate neighborhood features
for capturing long-range data dependency between nodes. As a result,
neighborhood fetching caused by data dependency is one of the major sources of
GNN latency during inference. For example, to infer a single node with a
$L$-layer GNN on a graph with average node degree $R$ as shown in Fig. 1(a),
it requires fetching and aggregating $\mathcal{O}(R^{L})$ nodes. However, $R$
can be large for real-world graphs, e.g., 19 for Amazon-com dataset, and $L$
is getting deeper for the latest GNN architectures, e.g., $L$=1001 layers for
RevGNN-Deep (Li et al., 2021). Moreover, the total latency explodes quickly as
$L$ increases, as fetching for successive layers must be done sequentially.
Taking Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016) and MLPs as
examples, Fig. 1(b) illustrates the relationship between the layer number $L$
and the number of fetched nodes. Intuitively, it can be seen that the fetch
number of GCNs is orders of magnitude more than MLPs and grows exponentially
with the layer number. Compared to GNNs, MLPs are free from the data
dependency problem and are easier to deploy. However, due to the lack of
modeling data dependency, MLPs fail to take full advantage of the graph
topological information, which greatly limits their performance on downstream
tasks.

(a) A two-layer GNN

(b) # Fetched Nodes vs. Layers

(c) Accuracy vs. Inference Time

Figure 1: (a): The one-hop and two-hop neighboring nodes to be fetched by a
two-layer GNN. (b): Number of nodes to be fetched for inference with the
number of layers, with 20 nodes randomly picked from the Coauthor-CS dataset.
(c): Inference accuracy vs. inference time on the Cora dataset. If not
specifically mentioned, all relevant models adopt GCN as the backbone by
default.

Considering both inference accuracy and inference time, general GNNs and MLP-
based models are two completely different worlds. As illustrated in Fig. 1(c),
the MLP-based models, such as Graph-MLP (Hu et al., 2021) and LinkDist (Luo et
al., 2021), are faster in inference but with much poorer performance compared
to GNNs. There are two main branches of existing approaches to connect these
two worlds. The first branch is inference acceleration, such as Pruning (Han
et al., 2015), Quantization (Gupta et al., 2015), and Sampling (Chen et al.,
2018), but their improvements in the inference speed are often limited and
come at the cost of undesired performance drops. The other branch is graph
distillation methods, but they either do not help much in inference
acceleration with the data dependency unresolved, such as CPF (Yang et al.,
2021), RDD (Zhang et al., 2020), and TinyGNN (Yan et al., 2020), or they are
not competitive with the state-of-the-art GNNs in terms of inference accuracy,
such as GLNN (Zhang et al., 2021). Therefore, “how can we bridge the two
worlds, enjoy the low-latency of MLPs and high-accuracy of GNNs?” is still a
tricky problem.

In this paper, we propose a simple yet effective Graph Self-Distillation on
Neighborhood (GSDN) framework to reduce the gap between GNNs and MLPs. The
proposed GSDN framework is based purely on MLPs, where structural information
is only implicitly used as prior to self-distill knowledge between the
neighborhood and the target, substituting the explicit information propagation
as in GNNs. More importantly, self-distillation can be done in an offline
manner during training, and then the trained MLP model can be directly
deployed for online inference. In other words, we can shift considerable work
from the latency-sensitive inference stage, where time reduction in
milliseconds makes a huge difference, to the less latency-insensitive training
stage, where time cost in hours is often tolerable. Finally, the resulting
model enjoys the benefits of graph topology-awareness in training but reduces
time overhead in inference. Extensive experiments have been provided to (1)
demonstrate the advantages of GSDN over existing methods in terms of inference
accuracy and inference speed and (2) explore _how GSDN can benefit from
neighborhood self-distillation_.

##  2 Related Work

Graph Neural Networks. GNNs can be mainly divided into two categories, i.e.,
spectral-based GNNs and spatial-based GNNs. The spectral-based GNNs, such as
ChebyNet (Defferrard et al., 2016) and GCN (Kipf & Welling, 2016), define
graph convolution kernels in the spectral domain based on the graph signal
processing theory. Instead, the spatial-based GNNs, such as GraphSAGE
(Hamilton et al., 2017) and GAT (Kipf & Welling, 2016), directly define
updating rules in the spatial space and focus on the design of neighborhood
aggregation functions. We refer interested readers to the recent survey (Wu et
al., 2020; Zhou et al., 2020) for more GNN architectures.

Despite their great progress, the above GNNs all share the de facto design
that structural information is explicitly used for message passing, which
leaves neighborhood fetching still one major source of GNN inference latency.
To solve this problem, a large number of inference acceleration methods have
been proposed to reduce multiplication and accumulation operations.
Specifically for GNNs, pruning (Han et al., 2015) and quantizing (Gupta et
al., 2015) GNN parameters or Neighborhood Sampling (Chen et al., 2018) can
speed up GNN inference, but they have not completely eliminated the
neighborhood-fetching latency, and it often comes at the cost of undesired
performance drops.

Multi-Layer Perceptron. Compared to GNNs, MLPs have no data dependency and are
easier to deploy. There have been some attempts to combine pure MLPs with
sophisticated techniques, such as contrastive learning and knowledge
distillation, which yielded promising results on many graph-related tasks. For
example, Graph-MLP (Hu et al., 2021) designs a neighborhood contrastive loss
to bridge the gap between GNNs and MLPs by implicitly utilizing the adjacency
information. Instead, LinkDist (Luo et al., 2021) directly distills knowledge
from connected node pairs into MLPs without message passing. Despite their
great progress, these methods still cannot match the state-of-the-art GNNs in
terms of classification performance due to the lack of modeling the graph
topology.

Graph Knowledge Distillation. Several previous works on GNN distillation try
to distill knowledge from large teacher GNNs to smaller student GNNs, termed
as GNN-to-GNN. The student model in RDD (Zhang et al., 2020) and TinyGNN (Yan
et al., 2020) is a GNN with fewer parameters, but not necessarily fewer layers
than the teacher GNN, which makes both designs still suffer from the
neighborhood-fetching latency. The other branch of graph knowledge
distillation is to distill from large teacher GNNs to lightweight student
MLPs, termed as GNN-to-MLP. For example, CPF (Yang et al., 2021) proposes to
distill knowledge from GNNs to a student MLP, but it takes advantage of Label
Propagation (LP) (Iscen et al., 2019) to improve performance and thus remains
heavily data-dependent. Besides, GLNN (Zhang et al., 2021) proposes to distill
knowledge directly from GNNs to MLPs, which has a great advantage in inference
speed, but its inference accuracy cannot even match the vanilla teacher GNNs
on a few datasets. Different from the above teacher-student knowledge
distillation framework, there is a similar technique known as Knowledge Self-
Distillation (Zhang et al., 2019), which regularizes a model by _distilling
its own knowledge without other teacher models_. An effective way to achieve
this is to regularize the consistency of the predictions from the “relevant”
data, e.g., the label distributions of same-label nodes. For example, GNN-SD
(Chen et al., 2020) directly distills knowledge across different GNN layers,
aiming to solve the over-smoothing problem, with unobvious performance
improvement at shallow layers. Based on the smoothness assumption, this paper
assumes that neighboring nodes in the graph share similar features and labels,
i.e., they can be considered as “relevant” data. Therefore, GSDN implicitly
self-distills knowledge between the neighborhood and the target by
regularizing the consistency of their label distributions.

##  3 Preliminaries

Notations and Problem Statement. Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$
denote a graph, where $\mathcal{V}$ is the set of $|\mathcal{V}|=N$ nodes with
features
$\mathbf{X}=\left[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N}\right]\in\mathbb{R}^{N\times
d}$ and $\mathcal{E}$ is the set of edges between nodes. Each node
$v_{i}\in\mathcal{V}$ is associated with a $d$-dimensional features vector
$\mathbf{x}_{i}$. Following the common semi-supervised node classification
setting, only a subset of node
$\mathcal{V}_{L}=\\{v_{1},v_{2},\cdots,v_{L}\\}$ with corresponding labels
$\mathcal{Y}_{L}=\\{y_{1},y_{2},\cdots,y_{L}\\}$ are known, and we denote the
labeled set as $\mathcal{D}_{L}=(\mathcal{V}_{L},\mathcal{Y}_{L})$ and
unlabeled set as $\mathcal{D}_{U}=(\mathcal{V}_{U},\mathcal{Y}_{U})$, where
$\mathcal{V}_{U}=\mathcal{V}\backslash\mathcal{V}_{L}$. The task of node
classification aims to learn a mapping
$\Phi:\mathcal{V}\\!\rightarrow\\!\mathcal{Y}$ on labeled data
$\mathcal{D}_{L}$, so that it can be used to infer the labels
$\mathcal{Y}_{U}$.

Message Propagation. A general GNN framework consists of two key computations
for each node $v_{i}$ at every layer: (1) $\operatorname{AGGREGATE}$
operation: aggregating messages from neighborhood $\mathcal{N}_{i}$; (2)
$\operatorname{UPDATE}$ operation: updating node representation from its
representation in the previous layer and aggregated messages. Considering a
$L$-layer GNN, the formulation of the $l$-th layer is as follows

$\mathbf{m}_{i}^{(l+1)}=\operatorname{AGGREGATE}^{(l)}\big{(}\big{\\{}\mathbf{h}_{j}^{(l)}:v_{j}\in\mathcal{N}_{i}\big{\\}}\big{)},\mathbf{h}_{i}^{(l+1)}=\operatorname{UPDATE}^{(l)}\big{(}\mathbf{h}_{i}^{(l)},\mathbf{m}_{i}^{(l+1)}\big{)}$
(1)

where $0\leq l\leq L-1$, $\mathbf{h}_{i}^{(l)}$ is the embedding of node
$v_{i}$ in the $l$-th layer, and $\mathbf{h}_{i}^{(0)}\\!=\\!\mathbf{x}_{i}$
is the input feature. After $L$ message-passing layers, the final node
embeddings $\mathbf{h}_{i}^{(L)}$ can be passed through an additional linear
inference layer $\mathbf{y}_{i}=f_{\theta}(\mathbf{h}_{i}^{(L)})$ for node
classification on the target node $v_{i}$.

##  4 Methodology

Motivated by the complementary strengths and weaknesses of GNNs and MLPs, we
propose a simple but effective Graph Self-Distillation on Neighborhood (GSDN)
framework to reduce their gaps, with an illustration shown in Fig. 2. The
proposed GSDN framework enjoys the benefits of graph topology-awareness in
training but has no graph dependency in inference. In essence, we need to
address the following three important issues: (1) backbone architecture
design, how to construct a “boosted” MLP as backbone architecture; (2)
objective function design, how to self-distill knowledge between neighboring
nodes and the target node; (3) training and inference, how to solve
optimization difficulties in training and conduct predictions with the trained
model in inference.

Figure 2: Illustration of the proposed GSDN framework. In the training stage,
the MLP and two inference layers $f_{\theta}(\cdot)$, $g_{\gamma}(\cdot)$ are
jointly trained by neighborhood feature-level and label-level self-distillatio
defined in Eq. (7) and Eq. (8). Then, the trained MLP is deployed for
inference, where the inference speed of GSDN is much faster than GNNs with
data dependency completely removed.

###  4.1 Backbone Architecture

The GSDN framework is based on a pure MLP architecture, with each layer
composed of a linear transformation, an activation function, a batch
normalization, and a dropout function, defined as:

$\mathbf{H}^{(l+1)}=\operatorname{Dropout}\big{(}BN\big{(}\sigma\big{(}\mathbf{H}^{(l)}\mathbf{W}^{(l)}\big{)}\big{)}\big{)},\quad\mathbf{H}^{(0)}=\mathbf{X}$
(2)

where $0\leq l\leq L-1$, $\sigma=\mathrm{ReLu}(\cdot)$ denotes an activation
function, $BN(\cdot)$ denotes the batch normalization, and
$\operatorname{Dropout}(\cdot)$ is the dropout function.
$\mathbf{W}^{(0)}\in\mathbb{R}^{d\times F}$ and
$\mathbf{W}^{(l)}\in\mathbb{R}^{F\times F}$ $(1\leq l\leq L-1)$ are layer-
specific weight matrices with the hidden dimension $F$.

Given a target node $v_{i}$ and its neighboring nodes $\mathcal{N}_{i}$, we
first feed their features $\mathbf{x}_{i}$ and
$\\{\mathbf{x}_{j}\\}_{j\in\mathcal{N}_{i}}$ into the forked MLP and encode
them into hidden representations $\mathbf{h}^{(L)}_{i}$ and
$\\{\mathbf{h}^{(L)}_{j}\\}_{j\in\mathcal{N}_{i}}$. Then, we define two
inference layers:
$\mathbf{y}_{i}=f_{\theta}(\mathbf{h}_{i}^{(L)})\in\mathbb{R}^{C}$ and
$\mathbf{z}_{j}=g_{\gamma}(\mathbf{h}_{j}^{(L)})\in\mathbb{R}^{C}$ for label
prediction on the target node $v_{i}$ and neighboring node
$j\in\mathcal{N}_{i}$, where $C$ is the category number. Both inference layers
are implemented with linear transformations by default in this paper. Next, we
discuss how to implicitly self-distill knowledge between the neighborhood
$\mathcal{N}_{i}$ and the target node $v_{i}$.

###  4.2 Self-Distillation on Neighborhood

Feature-level Self-Distillation. The smoothness assumption indicates that
neighboring nodes in a graph tend to share similar features and labels, while
non-neighboring nodes should be far away. With such an motivation, we perform
feature-level self-distillation on neighborhood by regularizing the
consistency of the label distributions between the target node and its
neighboring nodes. Along with connectivity, disconnectivity between nodes also
carries important information that reveals the node dissimilarity. However,
the number of neighboring nodes is much smaller compared with those non-
neighboring nodes, which renders the model overemphasize the differences
between the target and non-neighboring nodes, possibly leading to imprecise
class boundaries. To solve this data-imbalance problem, we modify _Mixup_
(Zhang et al., 2017), an effective data augmentation that performs
interpolation between samples to generate new training samples, to augment
neighboring nodes. Specifically, we performe learnable interpolation between
the target node $v_{i}$ and its neighboring node $v_{j}\in\mathcal{N}_{i}$ to
generate a new node, with its node representation defined as

$\mathbf{z}^{\prime}_{i,j}=g_{\gamma}\big{(}\beta_{i,j}\mathbf{h}_{j}^{(L)}+(1-\beta_{i,j})\mathbf{h}_{i}^{(L)}\big{)},\quad\text{where}\
\ \
\beta_{i,j}=\mathrm{sigmoid}\big{(}\mathbf{a}^{T}\big{[}\mathbf{x}_{i}\mathbf{W}_{m}\|\mathbf{x}_{j}\mathbf{W}_{m}\big{]}\big{)}$
(3)

where $\mathbf{W}_{m}\in\mathbb{R}^{d\times F}$ and $\beta_{i,j}$ is defined
as learnable interpolation coefficients with the shared attention weight
$\mathbf{a}$. Then, we take augmented neighboring nodes as positive samples
and other non-neighboring nodes as negative samples to simultaneously model
the connectivity and disconnectivity between nodes. Specifically, the learning
objective of feature-level self-distillation is defined as

$\mathcal{L}_{feat}=\frac{1}{N}\sum_{i=1}^{N}\Big{(}\frac{1}{|\mathcal{N}_{i}|}\sum_{j\in\mathcal{N}_{i}}\big{\|}\mathbf{y}_{i}-\mathbf{z}^{\prime}_{i,j}\big{\|}_{2}^{2}-\frac{1}{M_{i}}\sum_{e_{i,k}\notin\mathcal{E}}\big{\|}\widehat{\mathbf{y}}_{i}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}\Big{)}$
(4)

where $\mathbf{z}_{k}=g_{\gamma}(\mathbf{h}_{k}^{(L)})$, and
$M_{i}=|\mathcal{E}|-|\mathcal{N}_{i}|-1$ is the number of negative samples
(non-neighborhood nodes) of the target node $v_{i}$. The objective
$\mathcal{L}_{feat}$ essentially encourages positive neighboring nodes to be
closer and pushes negative non-neighboring nodes away. Moreover, we have
demonstrated the benefits of mixup-like augmentation and negative samples by
the ablation study in Sec. 5.4.

Label-level Self-Distillation. Thus far, we have only discussed how to exploit
structural information for feature-level self-distillation, but have not
explored how to leverage label information. A widely used solution to leverage
label information is to optimize the objective on the labeled data
$\mathcal{V}_{L}$ as

$\min_{\theta,\mathbf{W}^{(0)},\cdots,\mathbf{W}^{(L-1)}}\sum_{i\in\mathcal{V}_{L}}\mathcal{L}_{CE}(s_{i},\widehat{\mathbf{y}}_{i}),\quad\text{where}\
\ \
\widehat{\mathbf{y}}_{i}=\textrm{softmax}(\mathbf{y}_{i})\in\mathbb{R}^{C}$
(5)

where $\mathcal{L}_{CE}(\cdot)$ denotes the cross-entropy loss between
$\widehat{\mathbf{y}}_{i}$ and ground-truth label $s_{i}$ of node $v_{i}$.
However, Eq. (5) only considers node labels, but completely ignores the graph
structure. In practice, label propagation has been widely used as an effective
trick to simultaneously model label and structural information and achieved
promising results for various GNNs. However, label propagation involves
explicit coupling of labels with the structure, so it is heavily data-
dependent with the same inference-latency problem as message passing. Earlier,
we have proposed feature-level self-distillation in Eq. (4) to substitute
message passing in Eq. (1), and next we introduce implicit label-level self-
distillation of Eq. (6) to substitute explicit label propagation to fully
exploit both label and structural information. The objective of label-level
self-distillation can be defined as follows

$\mathcal{L}_{label}=\sum_{i\in\mathcal{V}_{L}}\Big{(}\mathcal{L}_{CE}(s_{i},\widehat{\mathbf{y}}_{i})+\sum_{j\in\mathcal{N}_{i}}\mathcal{L}_{CE}(s_{i},\widehat{\mathbf{z}}_{j})\Big{)}$
(6)

where
$\widehat{\mathbf{y}}_{i}\\!=\\!\textrm{softmax}(\mathbf{y}_{i})\in\mathbb{R}^{C}$,
$\widehat{\mathbf{z}}_{j}\\!=\\!\textrm{softmax}(\mathbf{z}_{j})\in\mathbb{R}^{C}$,
and $s_{i}$ is the ground-truth label of node $v_{i}$.

###  4.3 Training and Inferring

Model Training. After introducing two important objective functions in Eq.
(4)(6), we start to consider the optimization difficulties and strategies in
training. In practice, directly optimizing Eq. (4)(6) faces two tricky
challenges: (1) it treats all non-neighboring nodes as negative samples, which
suffers from a huge computational burden; and (2) it performs the summation
over the entire set of nodes, i.e, requiring a large memory space for keeping
the entire graph. To address these two problems, we adopt the edge sampling
strategy (Mikolov et al., 2013; Tang et al., 2015) instead of feeding the
entire graph into the memory for _batch-style training_. More specifically, we
first sample a mini-batch edges from the entire edge set
$\mathcal{E}_{b}\in\mathcal{E}$. Then we randomly sample negative nodes from a
pre-defined negative distribution $P_{k}(v)$ for each edge
$e_{i,j}\in\mathcal{E}_{b}$ instead of enumerating all non-neighboring nodes
as negative samples. Finally, we can rewrite Eq. (4) as follows

$\displaystyle\mathcal{L}_{feat}\\!=\\!\frac{1}{B}\sum_{b=1}^{B}\sum_{e_{i,j}\in\mathcal{E}_{b}}\bigg{(}\big{\|}\mathbf{y}_{i}-\mathbf{z}^{\prime}_{i,j}\big{\|}_{2}^{2}+\big{\|}\mathbf{y}_{j}-\mathbf{z}^{\prime}_{j,i}\big{\|}_{2}^{2}-\mathbb{E}_{{v_{k}}\sim
P_{k}(v)}\Big{(}\big{\|}\widehat{\mathbf{y}}_{i}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}\big{)}+\big{\|}\widehat{\mathbf{y}}_{j}-\widehat{\mathbf{z}}_{k}\big{\|}_{2}^{2}\big{)}\Big{)}\bigg{)}$
(7)

where $B$ is the batch size, and $P_{k}(v)$ adopts the uniform distribution by
default, that is $P_{k}(v_{i})=\frac{1}{N}$ for each node $v_{i}$. $P_{k}(v)$
can also be pre-defined based on prior knowledge, e.g., degree distribution,
edge number, but in practice we find from the experimental results in Section.
5.4 that uniform distribution is a reasonable choice that can yield fairly
good performance across various datasets. Similarly, we can rewrite the label-
level self-distillation of Eq. (6) as a batch-style form, as follows

$\mathcal{L}_{label}=\frac{1}{B}\sum_{b=1}^{B}\sum_{i\in\mathcal{V}_{L}\cap\mathcal{V}_{b}}\Big{(}\mathcal{L}_{CE}(s_{i},\widehat{\mathbf{y}}_{i})+\sum_{e_{i,j}\in\mathcal{E}_{b}}\mathcal{L}_{CE}(s_{i},\widehat{\mathbf{z}}_{j})\Big{)}$
(8)

where $\mathcal{V}_{b}=\\{v_{i},v_{j}|e_{i,j}\in\mathcal{E}_{b}\\}$ is all the
sampled nodes in $\mathcal{E}_{b}$. Finally, the total loss for training the
model can be defined as
$\mathcal{L}_{total}=\mathcal{L}_{label}+\lambda\mathcal{L}_{feat}$, where
$\lambda$ is a trade-off hyperparameter.

Model Inferring. Once the model training is completed, we can directly omit
the inference layer $g_{\gamma}(\cdot)$ and retain the backbone MLP
architecture and the inference layer $f_{\theta}(\cdot)$ for label prediction.
At this time, there is no data dependency for model inference, and it is
attributed to the fact that we have shifted a considerable amount of work from
the latency-sensitive inference stage to the latency-insensitive training
stage. The pseudo-code is summarized in Algorithm 1 in Appendix A.

###  4.4 Discussion and Comparison

In this subsection, we would like to compare the proposed GSDN framework with
some existing related work in order to make a better distinction and highlight
our contributions.

Comparison with message and label propagation. The core of GNNs is the use of
structural information, where _message passing_ and _label propagation_ are
the two dominant schemes. The message passing models the long-range data
dependency between nodes through neighborhood feature aggregation, while label
propagation focuses on diffusing label information to the neighborhood, and
_they are complementary to each other_. However, both of them involve explicit
coupling of features/labels with structures, leading to data dependency and
inference latency. Different from the explicit message passing and label
propagation, we use structural information as prior, as shown in Fig. 2, to
implicitly guide neighborhood feature-level self-distillation (from
neighborhood to the target node) as in Eq. (4), and label-level self-
distillation (from the target node to the neighborhood) as in Eq. (6), where
structural information is never explicitly involved in the forward
propagation. Furthermore, we have rethinked the differences between GNNs and
GSDN from the perspective of information flow, explaining how GSDN defined on
the 1-hop neighborhood influence nodes that are multi-hops away; however, due
to space limitations, we place this part of the discussion in Appendix B.

Figure 3: A comparison of three distillation methods.

Discussion on knowledge distillation. Several previous works on GNN
distillation seek to distill knowledge from large teacher GNN to more
lightweight student models. Potential lightweight student models that could be
employed could be a GNN with fewer parameters, termed GNN-to-GNN distillation
(Yang et al., 2020; Yan et al., 2020); or it could be a pure MLP, called GNN-
to-MLP distillation (Yang et al., 2021; Zhang et al., 2021). In contrast to
the teacher-student knowledge distillation schemes, this paper adopts
knowledge self-distillation, which regularizes a model by distilling its own
knowledge without other teacher models. In the proposed framework, we extract
representations with a shared MLP backbone, then use $g_{\gamma}(\cdot)$ and
$f_{\theta}(\cdot)$ for label prediction on neighboring nodes and target node,
and finally train the model by regularizing their label distribution
consistencies, which can be considered as a special version of MLP-to-MLP
distillation. A comparison of these three graph knowledge distillation methods
is shown in Fig. 3.

##  5 Experiments

In this section, we evaluate GSDN on six real-world datasets by answering the
following five questions. Q1: How does GSDN compare with GNN- and MLP-based
models? Q2: Is GSDN robust under limited labeled data and label noise? Q3: How
do GSDN compare with other general GNN models and inference acceleration
methods in terms of inference time? Q4: How does GSDN benefit from negative
samples, mixup-like augmentation, and feature/label self-distillation? Q5: How
do the two key hyperparameters, trade-off weight $\lambda$ and batch size $B$,
influence the model performance?

Datasets. The experiments are conducted on six widely used real-world
datasets, including Cora (Sen et al., 2008), Citeseer (Giles et al., 1998),
Coauthor-CS, Coauthor-Physics, Amazon-Com, and Amazon-Photo (Shchur et al.,
2018). For each dataset, following the data splitting settings of (Kipf &
Welling, 2016), we select 20 nodes per class to construct a training set, 500
nodes for validation, and 1000 nodes for testing. A statistical overview of
these six datasets is placed in Appendix C.

Baseline. We consider the following six classical baselines: Vanilla MLP, GCN
(Kipf & Welling, 2016), GAT (Veličković et al., 2017), GraphSAGE (Hamilton et
al., 2017), APPNP (Klicpera et al., 2018), and DAGNN (Liu et al., 2020). In
addition, we compare the proposed framework with two pure MLP-based
architectures, including Graph-MLP (Hu et al., 2021) and LinkDist. Besides,
five graph distillation methods are considered as baselines: RDD (Zhang et
al., 2020), TinyGNN (Yan et al., 2020), CPR (Yang et al., 2021), GLNN (Zhang
et al., 2021), and GNN-SD (Chen et al., 2020). Moreover, with GCN as the base
architecture, three inference acceleration methods are compared, including
pruning with 50% weights (P-GCN) (Han et al., 2015), quantization from FP32 to
INT8 (Q-GCN) (Gupta et al., 2015), neighbor sampling with fan-out 15 (NS-GCN)
(Chen et al., 2018). Note that there is so much work related to this paper
that it would be impractical to compare with all of them, so we have selected
only those classical works from various fields for comparison.

Hyperparameter. The hyperparameters are set the same for all datasets: Adam
optimizer with learning rate $\alpha$ = 0.01 and weight decay $decay$ = 5e-4;
Epoch $E$ = 200; Layer number $L$ = 2. The other dataset-specific
hyperparameters are determined by an AutoML toolkit NNI with the
hyperparameter search spaces as: hidden dimension $F=\\{256,512,1024\\}$;
batch size $B=\\{256,512,1024,4096\\}$, trade-off weight
$\lambda=\\{0.5,0.8,1.0\\}$. Each set of experiments is run five times with
different random seeds, and the average accuracy and standard deviation are
reported as metrics. Moreover, the experiments of both baselines and our
approach are implemented based on the standard implementation in the DGL
library (Wang et al., 2019) using the PyTorch 1.6.0 library with Intel(R)
Xeon(R) Gold 6240R @ 2.40GHz CPU and NVIDIA V100 GPU. The hyperparameter
sensitivity analysis (Q5) for the trade-off weight $\lambda$ and batch size
$B$ is available in Appendix D.

###  5.1 Performance Comparison (Q1)

To answer Q1, we conduct experiments on six real-world datasets with
comparison to state-of-the-art methods, three types of which are included:
general GNNs, GNN distillation, and MLP-based models. Table. 1 reports the
mean classification accuracy with the standard deviation on the test nodes,
from which we can observe that: (1) While existing MLP-based models, such as
Graph-MLP and LinkDist, can achieve comparable performance to GCN on a few
datasets, they still lag far behind state-of-the-art GNNs, such as APPNP and
DAGNN, and cannot even match the performance of GraphSAGE and GAT on some
datasets. (2) Regarding the graph distillation methods, while the performance
of the distilled student model is significantly improved over the teacher
model on some datasets, such improvement becomes less significant on a few
large-scale datasets, such as Coauthor-Phy and Amazon-Photo. (3) In terms of
classification accuracy, GSDN consistently achieves the best overall
performance on six datasets, even better than the state-of-the-art GNN models,
such as APPNP and DAGNN. For example, GSDN obtains the best performance on the
Coauthor-Physics dataset, and more notably, our accuracy outperforms DAGNN by
1.86%, which once again demonstrates the effectiveness of the GSDN framework
for the node classification task.

Table 1: Classification accuracy $\pm$ std (%), with the best and second
results marked by bold and underline. If not specifically mentioned, all
relevant models adopt GCN as the backbone by default.

Type | Method | Cora | Citeseer | Coauthor-CS | Coauthor-Phy | Amazon-Com | Amazon-Photo
---|---|---|---|---|---|---|---
General GNNs | GCN | 81.28$\pm$0.42 | 71.06$\pm$0.44 | 87.76$\pm$0.43 | 91.89$\pm$0.42 | 77.45$\pm$1.71 | 87.53$\pm$1.64
GAT | 83.02$\pm$0.45 | 72.56$\pm$0.51 | 88.55$\pm$0.56 | 92.36$\pm$0.47 | 82.78$\pm$1.89 | 90.19$\pm$1.35
GraphSAGE | 82.22$\pm$0.80 | 71.22$\pm$0.58 | 88.40$\pm$0.48 | 91.88$\pm$0.53 | 79.23$\pm$1.63 | 88.63$\pm$1.17
APPNP | 83.28$\pm$0.33 | 71.74$\pm$0.27 | 88.74$\pm$0.62 | 92.75$\pm$0.60 | 81.28$\pm$1.90 | 89.49$\pm$1.28
DAGNN | 84.30$\pm$0.51 | 73.14$\pm$0.62 | 89.32$\pm$0.55 | 93.10$\pm$0.67 | 80.32$\pm$1.57 | 90.72$\pm$1.45
Graph Distillation | RDD | 84.68$\pm$0.40 | 73.63$\pm$0.50 | 89.38$\pm$0.44 | 92.74$\pm$0.78 | 81.84$\pm$1.48 | 89.70$\pm$0.93
TinyGNN | 82.79$\pm$0.57 | 72.67$\pm$0.72 | 88.72$\pm$0.42 | 92.20$\pm$0.67 | 79.22$\pm$1.69 | 89.24$\pm$1.24
CPR | 83.65$\pm$0.49 | 72.98$\pm$0.47 | 89.10$\pm$0.50 | 92.36$\pm$0.63 | 80.90$\pm$1.52 | 89.03$\pm$1.29
GLNN | 80.85$\pm$0.60 | 71.21$\pm$0.80 | 87.81$\pm$0.53 | 91.83$\pm$0.60 | 77.96$\pm$1.70 | 87.98$\pm$1.36
| GNN-SD | 81.85$\pm$0.55 | 71.69$\pm$0.61 | 87.80$\pm$0.50 | 92.07$\pm$0.48 | 77.66$\pm$1.85 | 87.80$\pm$1.52
MLP-based | MLP | 61.86$\pm$0.43 | 59.76$\pm$0.51 | 83.34$\pm$0.64 | 86.24$\pm$0.66 | 66.85$\pm$1.94 | 78.18$\pm$1.25
Graph-MLP | 81.45$\pm$0.52 | 72.87$\pm$0.70 | 88.16$\pm$0.70 | 91.85$\pm$0.49 | 77.23$\pm$1.76 | 87.64$\pm$1.37
LinkDist | 76.70$\pm$0.47 | 65.19$\pm$0.55 | 87.89$\pm$0.58 | 92.16$\pm$0.70 | 76.93$\pm$1.83 | 87.26$\pm$1.42
GSDN (ours) | 84.90$\pm$0.44 | 74.08$\pm$0.69 | 89.62$\pm$0.40 | 94.96$\pm$0.41 | 83.44$\pm$2.09 | 90.34$\pm$0.85

###  5.2 Evaluation on Robustness (Q2)

There has been some work pointing out that the performance of GNNs depends
heavily on the quality and quantity of the labels. To evaluate the robustness
of the proposed framework, we evaluate the model with extremely limited label
data and under label noise on the Cora and Citeseer datasets.

Table 2: Accuracy $\pm$ std (%) with extremely limited labels, with the best
results marked by bold.

Dataset | GCN | GAT | APPNP | DAGNN | RDD | CPR | GLNN | GNN-SD | Graph-MLP | GSDN (ours)
---|---|---|---|---|---|---|---|---|---|---
Cora | 5 labels | 73.10$\pm$0.87 | 74.85$\pm$0.74 | 76.39$\pm$0.95 | 79.02$\pm$0.94 | 76.11$\pm$0.81 | 75.92$\pm$0.90 | 73.85$\pm$0.92 | 74.32$\pm$0.85 | 78.43$\pm$0.78 | 80.22$\pm$0.87
10 labels | 77.52$\pm$0.63 | 80.10$\pm$0.57 | 79.99$\pm$0.72 | 81.99$\pm$0.62 | 79.68$\pm$0.54 | 79.20$\pm$0.64 | 77.94$\pm$0.52 | 78.55$\pm$0.64 | 79.60$\pm$0.49 | 82.80$\pm$0.45
15 labels | 79.47$\pm$0.56 | 80.55$\pm$0.68 | 80.70$\pm$0.44 | 82.72$\pm$0.50 | 80.45$\pm$0.46 | 80.12$\pm$0.51 | 79.16$\pm$0.48 | 79.89$\pm$0.55 | 80.32$\pm$0.57 | 83.40$\pm$0.53
Citeseer | 5 labels | 63.23$\pm$1.04 | 64.17$\pm$0.95 | 66.28$\pm$0.88 | 69.13$\pm$0.68 | 66.06$\pm$0.57 | 65.40$\pm$0.62 | 63.10$\pm$0.70 | 63.93$\pm$0.74 | 69.64$\pm$0.64 | 68.76$\pm$1.16
10 labels | 67.55$\pm$0.50 | 68.09$\pm$0.49 | 69.23$\pm$0.64 | 71.74$\pm$0.71 | 69.68$\pm$0.66 | 69.14$\pm$0.73 | 67.65$\pm$0.62 | 68.20$\pm$0.57 | 70.56$\pm$0.51 | 72.66$\pm$0.43
15 labels | 69.64$\pm$0.58 | 69.70$\pm$0.65 | 70.17$\pm$0.44 | 72.26$\pm$0.53 | 70.32$\pm$0.57 | 70.76$\pm$0.46 | 69.52$\pm$0.52 | 69.86$\pm$0.48 | 71.80$\pm$0.63 | 73.10$\pm$0.53

Performance with Severely Limited Labels. To evaluate the effectiveness of the
proposed GSDN framework when labeled data is severely limited, we randomly
select 5, 10, and 15 labeled samples per class for training, and the rest of
the training set is considered unlabeled. The classification performance on
the Cora and Citeseer datasets is reported in Table. 2, from which we can
observe that (1) The performance of all methods drops as the number of labeled
data decreases, but the accuracy of GSDN drops more slightly. When only a
limited number of labels are provided, GSDN outperforms all other baselines at
most label rates. For example, when trained with five labels per class, GSDN
outperforms GCN by 7.12% and 5.53% on the Cora and Citeseer datasets. (2)
While graph distillation methods perform well on clean data, as shown in
Table. 1, their performance gains are reduced when labeled data is extremely
limited. In contrast, MLP-based models, both Graph-MLP and GSDN, show great
advantages over GNN-based models under the label-limited setting.

Performance with Noisy Labels. We evaluate the robustness of GSDN to label
noise by injecting asymmetric noise into the labels, where the label $i$
$(0\leq i\leq C-1)$ of each training sample flips independently with
probability $r$ to another class $j=(i+1)\%C$, but with probability $1-r$
preserved as label $i$ (Tan et al., 2021). The performance with noisy labels
is reported in Fig. 4(a) and Fig. 4(b) at various noise ratios
$r\in\\{0\%,10\%,20\%,\cdots 60\%\\}$. It can be seen that as $r$ increases,
the accuracy of GSDN drops more slowly than other baselines, and GSDN is more
robust than other models under various noise ratios, especially under
extremely high noise ratios. For example, with $r=60\%$ label noise, GSDN
outperforms APPNP and DAGNN by 6.67% and 5.77% on the Citeseer dataset,
respectively. Due to space limitations, more evaluation results on noisy
labels, including comparisons with other graph distillation and MLP-based
models, are available in Appendix E.

(a) Performance on Cora

(b) Performance on Citeseer

(c) Inference time with different layers

Figure 4: (a)(b) Classification accuracy (%) under different label noise
ratios on the Cora and Citeseer datasets, respectively. (c) Inference time
with different layers on the Coauthor-CS dataset.

###  5.3 Evaluation on Inference Speed (Q3)

Commonly used inference acceleration techniques on GNNs include pruning,
quantizing, and neighbor sampling. With GCN as the backbone, we consider its
three variants: P-GCN, Q-GCN, and NS-GCN. A comparison with more approaches,
such as graph distillation and MLP-based models, in terms of inference speed,
can be seen in Fig. 1(c) and Appendix F. Moreover, since the focus of this
paper is on inference speed, we defer the results and analysis on training
time to Appendix G.

Comparison on Different Datasets. With the removal of neighborhood fetching,
the inference time of GSDN can be reduced from
$\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ of GCN to
$\mathcal{O}(|\mathcal{V}|dF)$. The inference time ($ms$) averaged over 30
sets of runs on four datasets is reported in Table. 3 with the acceleration
multiple w.r.t the vanilla GCN marked as ${\color[rgb]{0.4,0.71,0.376}green}$,
where all methods use $L=2$ layers and hidden dimension $F=16$. From Table. 3,
we have the following observations: (1) While APPNP and DAGNN improve a lot
over GCN in terms of classification accuracy, as shown in Table. 1, they
suffer from more severe inference latency. (2) Pruning and quantization are
not very effective on GNNs, given that data dependency has not been well
resolved. Besides, the neighbor sampling considers but does not completely
eliminate the neighborhood-fetching latency, so it infers faster than pruning
and quantization, yet still lags far behind our GSDN. (3) GSDN infers fastest
across four datasets.

Table 3: Inference time ($ms$), where three inference acceleration methods
speed up GCN, but still infer slower than GSDN. Note the acceleration multiple
w.r.t the vanilla GCN is marked as ${\color[rgb]{0.4,0.71,0.376}green}$.

Method | GCN | APPNP | DAGNN | P-GCN | Q-GCN | NS-GCN | GSDN (ours)
---|---|---|---|---|---|---|---
Cora | 402.7 | 731.8 | 647.6 | 372.9 (${\color[rgb]{0.4,0.71,0.376}1.08\times}$) | 383.5 (${\color[rgb]{0.4,0.71,0.376}1.05\times}$) | 97.7 (${\color[rgb]{0.4,0.71,0.376}4.12\times}$) | 4.6 (${\color[rgb]{0.4,0.71,0.376}87.54\times}$)
Citeseer | 383.4 | 700.5 | 727.5 | 316.9 (${\color[rgb]{0.4,0.71,0.376}1.21\times}$) | 361.4 (${\color[rgb]{0.4,0.71,0.376}1.06\times}$) | 105.6 (${\color[rgb]{0.4,0.71,0.376}3.63\times}$) | 4.3 (${\color[rgb]{0.4,0.71,0.376}89.16\times}$)
Coauthor-CS | 593.6 | 1099.1 | 672.4 | 498.8 (${\color[rgb]{0.4,0.71,0.376}1.19\times}$) | 566.3 (${\color[rgb]{0.4,0.71,0.376}1.05\times}$) | 129.2 (${\color[rgb]{0.4,0.71,0.376}4.59\times}$) | 7.9 (${\color[rgb]{0.4,0.71,0.376}75.14\times}$)
Coauthor-Phy | 1067.6 | 1467.4 | 708.6 | 922.3 (${\color[rgb]{0.4,0.71,0.376}1.16\times}$) | 997.8 (${\color[rgb]{0.4,0.71,0.376}1.07\times}$) | 269.3 (${\color[rgb]{0.4,0.71,0.376}3.96\times}$) | 12.8 (${\color[rgb]{0.4,0.71,0.376}83.41\times}$)

Comparison with Different Layers. As discussed earlier, to infer a single node
with a $L$-layer GNN on a graph with average node degree $R$, it requires
fetching and aggregating $\mathcal{O}(R^{L})$ nodes. To compare the
sensitivity of GSDN to layer depth with other inference acceleration methods,
we report the inference time ($ms$ in log-scale) of various methods at
different layer depths on the Coauthor-CS dataset in Fig. 4(c). It can be seen
that the inference time of GSDN only increases linearly with the layer depth,
but that of other baselines increases exponentially. Moreover, the speed gains
of pruning and quantization on GNNs are reduced as the layer depth increases,
and they approach the vanilla GCN when the layer depth is 4. In contrast, at
larger layer depths, the speed gain of the neighbor sampling gets enlarged
compared to the vanilla GCN. This demonstrates that neighborhood fetching is
one major source of inference latency as the layer depth increases, and the
linear complexity of GSDN has a significant advantage, especially when GNNs is
becoming deeper.

(a) Ablation Study

(b) Learning curves

(c) Mean cosine similarity curves

Figure 5: (a) Ablation study on four key model components. (b) Learning curves
of MLPs and GSDN on the Cora dataset, where the _logarithmized_ vertical
coordinate is the cross-entropy loss between the predicted and ground-truth
labels on the training or validation set. (c) Mean cosine similarity curves of
MLPs, GCNs, and GSDN with 1-hop and 2-hop neighbors on the Cora dataset.

###  5.4 Ablation Study (Q4)

Component Analysis. To evaluate the effectiveness of negative samples in Eq.
(4), mixup-like augmentation in Eq. (3), and label self-distillation in Eq.
(8), we conducted four sets of experiments: the model without (A) Negative
Samples (w/o NS); (B) mixup-like augmentation (w/o augment); (C) Label self-
Distillation (w/o LD); and (D) the full model. Besides, to evaluate the impact
of the negative distribution, we take the the nodal degree $d_{i}$ as a prior
and preset $P_{k}(v_{i})=\frac{d_{i}}{|\mathcal{E}|}$ in place of the default
uniform distribution in this paper, denoted as (E) w/o uniform. After
analyzing the results in Fig. 5(a), we can conclude: (1) Negative Samples and
mixup-like augmentation contribute to improving classification performance.
More importantly, applying them together can further improve performance on
top of each. (2) Label self-distillation helps to improve performance on top
of the stand-alone feature self-distillation. (3) Even without considering any
graph prior, presetting $P_{k}(\cdot)$ as uniform distribution is sufficient
to achieve comparable performance, so this paper defaults to the simplest
uniform distribution without considering other complex prior-based
distributions.

How GSDN Benefit from Neighborhood Self-Distillation. Next, we explore how
does GSDN benefit from neighborhood self-distillation? Existing MLP-based
models have shown that there do exist the optimal MLP parameters that enable
their performance to be competitive with GNNs on the attribute graph, but it
is hard to learn such parameters through a simple cross-entropy loss (Hu et
al., 2021). The proposed self-distillation helps to solve this problem with
two potential advantages: (1) alleviating overfitting and (2) introducing
inductive bias, i.e., graph topology (Zhang et al., 2021).

Firstly, we plot the training curves (with log-scale vertical coordinate) of
GSDN and MLPs on the Cora dataset in Fig. 5(b). We observe that the gap
between training and validation loss is smaller for GSDN than MLPs, which
indicates that GSDN helps to alleviate the overfitting trend of MLPs.
Secondly, we conjecture that the absence of inductive bias, e.g., graph
topology, is one of the major reasons why MLPs is inferior to GNNs in
inference accuracy. To illustrate it, we plot in Fig. 5(c) the average cosine
similarity of nodes with their 1-hop and 2-hop neighbors for MLPs, GCNs, and
GSDN on the Cora dataset. It can be seen that the average similarity with
1-hop neighbors is always higher than that with 2-hop neighbors throughout the
training process for MLPs, GCNs, and GSDN. More importantly, the average
similarity of GCNs and GSDN gradually increases with training, while that of
MLPs gradually decreases, which indicates that GSDN has introduced graph
topology as an inductive bias (as GCNs has done), while MLPs does not.
Finally, the GSDN enjoys the benefits of topology-awareness in training but
without neighborhood-fetching latency in inference.

##  6 Conclusion

Motivated by the complementary strengths and weaknesses of GNNs and MLPs, we
propose a novel MLP-based framework, namely Graph Self-Distillation on
Neighborhood (GSDN), where structural information is only implicitly used as
prior to guide knowledge self-distillation between the neighborhood and the
target. As a result, GSDN shifts a considerable amount of work from the
latency-sensitive inference stage to the latency-insensitive training state,
thus enjoying the benefits of graph topology-awareness in training but without
data dependency in inference. More importantly, we study GSDN properties
comprehensively by investigating how they benefit from neighborhood self-
distillation and how they are different from existing works. Extensive
experiments show the advantages of GSDN over existing methods in terms of
inference accuracy and inference efficiency.

## References

  * Chen et al. (2018) Jie Chen, Tengfei Ma, and Cao Xiao.  Fastgcn: fast learning with graph convolutional networks via importance sampling.  _arXiv preprint arXiv:1801.10247_ , 2018.
  * Chen et al. (2020) Yuzhao Chen, Yatao Bian, Xi Xiao, Yu Rong, Tingyang Xu, and Junzhou Huang.  On self-distilling graph neural network.  _arXiv preprint arXiv:2011.02255_ , 2020.
  * Defferrard et al. (2016) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  _arXiv preprint arXiv:1606.09375_ , 2016.
  * Giles et al. (1998) C Lee Giles, Kurt D Bollacker, and Steve Lawrence.  Citeseer: An automatic citation indexing system.  In _Proceedings of the third ACM conference on Digital libraries_ , pp. 89–98, 1998.
  * Gupta et al. (2015) Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan.  Deep learning with limited numerical precision.  In _International conference on machine learning_ , pp. 1737–1746. PMLR, 2015.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _Advances in neural information processing systems_ , pp. 1024–1034, 2017.
  * Han et al. (2015) Song Han, Jeff Pool, John Tran, and William J Dally.  Learning both weights and connections for efficient neural networks.  _arXiv preprint arXiv:1506.02626_ , 2015.
  * Hu et al. (2021) Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao.  Graph-mlp: Node classification without message passing in graph.  _arXiv preprint arXiv:2106.04051_ , 2021.
  * Iscen et al. (2019) Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum.  Label propagation for deep semi-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5070–5079, 2019.
  * Jia et al. (2020) Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken.  Redundancy-free computation for graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 997–1005, 2020.
  * Kipf & Welling (2016) Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Klicpera et al. (2018) Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann.  Predict then propagate: Graph neural networks meet personalized pagerank.  _arXiv preprint arXiv:1810.05997_ , 2018.
  * Li et al. (2021) Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun.  Training graph neural networks with 1000 layers.  _arXiv preprint arXiv:2106.07476_ , 2021.
  * Liu et al. (2020) Meng Liu, Hongyang Gao, and Shuiwang Ji.  Towards deeper graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 338–348, 2020.
  * Luo et al. (2021) Yi Luo, Aiguo Chen, Ke Yan, and Ling Tian.  Distilling self-knowledge from contrastive links to classify graph nodes without passing messages.  _arXiv preprint arXiv:2106.08541_ , 2021.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pp. 3111–3119, 2013.
  * Sen et al. (2008) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad.  Collective classification in network data.  _AI magazine_ , 29(3):93–93, 2008.
  * Shchur et al. (2018) Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann.  Pitfalls of graph neural network evaluation.  _arXiv preprint arXiv:1811.05868_ , 2018.
  * Tan et al. (2021) Cheng Tan, Jun Xia, Lirong Wu, and Stan Z Li.  Co-learning: Learning from noisy labels with self-supervision.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 1405–1413, 2021.
  * Tang et al. (2015) Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  Line: Large-scale information network embedding.  In _Proceedings of the 24th international conference on world wide web_ , pp. 1067–1077, 2015.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ , 2017.
  * Wang et al. (2019) Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tong He, George Karypis, Jinyang Li, and Zheng Zhang.  Deep graph library: A graph-centric, highly-performant package for graph neural networks.  _arXiv preprint arXiv:1909.01315_ , 2019.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_ , 2020\.
  * Yan et al. (2020) Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou.  Tinygnn: Learning efficient graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1848–1856, 2020.
  * Yang et al. (2021) Cheng Yang, Jiawei Liu, and Chuan Shi.  Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework.  In _Proceedings of the Web Conference 2021_ , pp. 1227–1237, 2021\.
  * Yang et al. (2020) Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang.  Distilling knowledge from graph convolutional networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7074–7083, 2020.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhang et al. (2019) Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3713–3722, 2019.
  * Zhang et al. (2021) Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah.  Graph-less neural networks: Teaching old mlps new tricks via distillation.  _arXiv preprint arXiv:2110.08727_ , 2021.
  * Zhang et al. (2020) Wentao Zhang, Xupeng Miao, Yingxia Shao, Jiawei Jiang, Lei Chen, Olivier Ruas, and Bin Cui.  Reliable data distillation on graph convolutional network.  In _Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data_ , pp. 1399–1414, 2020.
  * Zhou et al. (2020) Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  _AI Open_ , 1:57–81, 2020.

## Appendix

### A. Pseudo Code of GSDN

The pseudo-code of the proposed GSDN framework is summarized in Algorithm 1.

Algorithm 1 Algorithm for the proposed GSDN framework

1:Feature Matrix: $\mathbf{X}$; Edge Set: $\mathcal{E}$; # Batches: $B$; #
Epochs: $E$.

2:Predicted Labels $\mathcal{Y}_{U}$, Parameters
$\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$ and $f_{\theta}(\cdot)$.

3:Initialize parameters $\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$,
$f_{\theta}(\cdot)$, and $g_{\gamma}(\cdot)$.

4:for $epoch$ $\in$ {0,1,$\cdots$,$E-1$} do

5: for $b$ $\in$ {0,1,$\cdots$,$B-1$} do

6: Sample a mini-batch of edges $\mathcal{E}_{b}$ from $\mathcal{E}$;

7: Compute losses $\mathcal{L}_{feat}$ and $\mathcal{L}_{label}$ by Eq. (7)
and Eq. (8);

8: Sum up $\mathcal{L}_{feat}$ and $\mathcal{L}_{label}$ as total loss
$L_{total}$;

9: Update parameters by back-propagation of loss $L_{total}$.

10: end for

11:end for

12:Predict labels $\mathcal{Y}_{U}$ for those unlabeled nodes
$\mathcal{V}_{U}$.

13:return Predicted labels $\mathcal{Y}_{U}$, parameters
$\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$ and $f_{\theta}(\cdot)$.

### B. Rethink GNNs and GSDN from the perspective of information flow

Discussion on information flow. We rethink the information flow of GNNs and
GSDN from two perspectives, layer depth and training time. (1) Considering
message-passing-based GNNs, they rely on both training time and layer stacking
to aggregate information, with _the later playing a determining role_. Most
existing GNNs enlarge the receptive field to multi-hop neighbors by stacking
multiple GNN layers, such as graph convolution, to capture the long-distance
dependency between nodes. In other words, the information of nodes that are
multi-hop away flows to the target node along the unfolding path of the
stacked GNN layers, which can be considered as a kind of information flow
along the layer dimension. (2) In contrast, the layer depth of GSDN has no
effect on the receptive field. Instead, as the training proceeds, information
from the remote nodes gradually flows to the target node through the _cascaded
neighborhood feature self-distillation_ , which can be viewed as information
flow along the time dimension. (3) A comparison of these two types of
information flow is shown in Fig. A1, from which we find that at Layer 2, the
target node does not receive any information from 2-hop nodes. Instead, at
Time 2, not only does the target node receive information from 2-hop nodes,
but also more information can be gradually distilled into the target node as
training proceeds. In summary, while Eq. (4)(6) is defined on the 1-hop
neighborhood, it can still propagate messages to multi-hops away, along the
time dimension instead of stacking layers.

Figure A1: A comparison of two types of information flow, both of which work
to capture information from long-range nodes. However, compared to the layer
stacking of GNNs, GSDN works by cascaded neighborhood feature self-
distillation, along the time dimension instead of stacking layers.

### C. Dataset Statistics

_Six_ publicly available graph datasets are used to evaluate the proposed GSDN
framework. An overview summary of the statistical characteristics of datasets
is given in Tab. A1. For the three small-scale datasets, namely Cora and
Citeseer, we follow the data splitting strategy in (Kipf & Welling, 2016). For
the four large-scale datasets, namely Coauthor-CS, Coauthor-Physics, Amazon-
Photo, and Amazon-Computers, we follow (Zhang et al., 2021; Luo et al., 2021)
to randomly split the data into train/val/test sets, and each random seed
corresponds to a different splitting.

Table A1: Statistical information of the datasets.

Dataset | Cora | Citeseer | Amazon-Photo | Coauthor-CS | Coauthor-Phy | Amazon-Com |
---|---|---|---|---|---|---|---
$\\#$ Nodes | 2708 | 3327 | 7650 | 18333 | 34493 | 13752 |
$\\#$ Edges | 5278 | 4614 | 119081 | 81894 | 247962 | 245861 |
$\\#$ Features | 1433 | 3703 | 745 | 6805 | 8415 | 767 |
$\\#$ Classes | 7 | 6 | 8 | 15 | 5 | 10 |
Label Rate | 5.2% | 3.6% | 2.1% | 1.6% | 0.3% | 1.5% |

### D. Parameter Sensitivity Analysis (Q5)

To answer Q5, we evaluate the parameter sensitivity w.r.t two key
hyperparameters: trade-off weight $\lambda\in\\{0.0,0.1,0.3,0.5,0.8,1.0\\}$
and batch size $B\in\\{256,512,1024,2048,4096\\}$ in Fig. A2, from which we
can observe that (1) batch size $B$ is a dataset-specific hyperparameter. For
simple graphs with few nodes and edges, such as Cora, a small batch size,
$B=256$, can yield fairly good performance. However, for large-scale graphs
with more nodes and edges, such as Coauthor-Phy, the model performance usually
improves with the increase of batch size $B$. (2) When $\lambda$ is set to 0,
i.e., the neighborhood feature-level self-distillation is completely removed,
the performance of GSDN degrades to be close to that of MLPs. In contrast,
when $\lambda$ takes a non-zero value, the performance of GSDN improves as
$\lambda$ increases. However, when $\lambda$ becomes too large, it weakens the
benefit of label information, yielding lower performance gains. In practice,
we can usually determine $B$ and $\lambda$ by selecting the model with the
highest accuracy on the validation set through the grid search.

Figure A2: Parameter sensitivity analysis on the trade-off weight $\lambda$
and batch size $B$ on four datasets.

### E. More results on Performance with Noisy Labels.

The performance comparison with the graph distillation methods and MLP-based
models under noisy labels is reported in Fig. A3. As can be seen, the accuracy
of GSDN drops more slowly than other baselines as the noise ratio $r$
increases, and GSDN is more robust than other models under various label noise
ratios, especially under extremely high noise ratios. Besides, we find that
while graph distillation methods perform well on clean data, as shown in
Table. 1, their performance gains are reduced when the label noise ratio $r$
increases. In contrast, MLP-based models, both Graph-MLP and GSDN, show great
advantages over GNN-based models under extremely high noise ratios.

Figure A3: Accuracy (%) under different label noise ratios on the Cora and
Citeseer datasets.

### F. Model accuracy vs. Inference Time.

To provide a high-level picture of the trade-off between model accuracy and
inference time, we show the performance of various methods on the Cora dataset
in Fig. 3, where we can divide all methods into the following four categories
based on their inference accuracy and inference speed: (1) high-accuracy and
high-latency. Existing general GNN models and most GNN distillation models can
achieve high classification accuracy, but still suffer from inference latency
caused by data dependency. (2) low-accuracy and low-latency. The pure MLP-
based models and GLNN enjoy the benefit of low inference latency, but their
classification accuracy cannot match that of the current state-of-the-art
GNNs. (3) low-accuracy and high-latency. The inference acceleration methods,
such as pruning, quantization, and sampling, only have small gains in the
inference speed but greatly hurt the inference accuracy, as they have not
fully addressed the neighborhood fetching problem. (4) high-accuracy and low-
latency. The proposed GSDN framework matches the state-of-the-art GNNs in
terms of classification accuracy and also is comparable to MLPs in the
inference speed.

### I. Training Time

The training time complexity of the proposed GSDN framework is
$\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$, which is linear with respect to
the number of nodes $|\mathcal{V}|$ and edges $|\mathcal{E}|$, and is in the
same order of magnitude as GCN. The training time ($s$) averaged over 30 sets
of runs on four datasets is reported in Table. 3 with the time multiple w.r.t
the vanilla GCN marked as ${\color[rgb]{0.4,0.71,0.376}green}$, where all
methods use $L=2$ layers and hidden dimension $F=16$. From Table. A2, we have
the following observations: (1) Common inference acceleration methods,
including pruning, quantization and neighborhood sampling, not only help speed
up GNN inference, but also work for GNN training. (2) While GLNN, Graph-MLP
and GSDN have great speed advantages in the inference stage, they essentially
shift considerable work from the latency-sensitive inference stage, where time
reduction in milliseconds makes a huge difference, to the less latency-
insensitive training stage, where time cost in hours or days is often
tolerable. This explains why these methods require more training time compared
to the vanilla GCN. (3) Also as a MLP-based model, the training time of GSDN
is almost half of that of Graph-MLP, mainly because Graph-MLP performs
contrasting within higher-order neighborhoods, while GSDN only considers
first-order neighbors, which greatly reduces the computational burden in
training.

Table A2: Training time ($s$), where the time multiple w.r.t the vanilla GCN
is marked as ${\color[rgb]{0.4,0.71,0.376}green}$.

Method | GCN | APPNP | DAGNN | P-GCN | Q-GCN | NS-GCN | GLNN | Graph-MLP | GSDN (ours)
---|---|---|---|---|---|---|---|---|---
Cora | 12.8 | 25.2 | 33.3 | 8.1 | 7.5 | 6.3 | 20.4 (${\color[rgb]{0.4,0.71,0.376}1.59\times}$) | 56.3 (${\color[rgb]{0.4,0.71,0.376}4.40\times}$) | 30.1 (${\color[rgb]{0.4,0.71,0.376}2.35\times}$)
Citeseer | 23.5 | 34.7 | 42.4 | 15.2 | 13.7 | 12.6 | 31.8 (${\color[rgb]{0.4,0.71,0.376}1.35\times}$) | 71.8 (${\color[rgb]{0.4,0.71,0.376}3.06\times}$) | 43.6 (${\color[rgb]{0.4,0.71,0.376}1.86\times}$)
Coauthor-CS | 219.3 | 355.6 | 560.3 | 146.2 | 127.7 | 116.9 | 286.9 (${\color[rgb]{0.4,0.71,0.376}1.31\times}$) | 1029.6 (${\color[rgb]{0.4,0.71,0.376}4.69\times}$) | 648.64 (${\color[rgb]{0.4,0.71,0.376}2.95\times}$)
Coauthor-Phy | 507.2 | 799.7 | 682.9 | 334.9 | 322.8 | 284.4 | 656.2 (${\color[rgb]{0.4,0.71,0.376}1.29\times}$) | 1852.1 (${\color[rgb]{0.4,0.71,0.376}3.65\times}$) | 1076.8 (${\color[rgb]{0.4,0.71,0.376}2.12\times}$)


# AQUATK: An Audio Quality Assessment Toolkit

###### Abstract

Recent advancements in Neural Audio Synthesis (NAS) have outpaced the
development of standardized evaluation methodologies and tools. To bridge this
gap, we introduce AquaTk, an open-source Python library specifically designed
to simplify and standardize the evaluation of NAS systems. AquaTk offers a
range of audio quality metrics, including a unique Python implementation of
the basic PEAQ algorithm, and operates in multiple modes to accommodate
various user needs.

##  1 Introduction

Recent developments in Neural Audio Synthesis (NAS) are notable, but an
automated, standardized evaluation framework is lacking. To address this, we
introduce the Audio QUality Assessment Toolkit (AquaTk), an open-source Python
library designed to be framework-agnostic. AquaTk includes well-established
methods for evaluating the performance of audio synthesis models and codecs.

Traditional audio evaluation algorithms and metrics often require complex
setup procedures. For example, to compute the Fréchet Audio Distance (FAD),
one needs to clone the entire Google Research repository. While there is a
PyTorch version, there are variations in the results depending on the deep
learning framework.

AquaTk aims to resolve these challenges by offering:

  * •

An open-source toolkit for evaluating NAS algorithms,

  * •

A wide range of audio quality measures,

  * •

Testable implementations to ensure reliable, replicable outcomes,

  * •

Compatibility with popular deep learning frameworks such as TensorFlow,
Pytorch and JAX,

  * •

Command line and graphical interfaces and,

  * •

An extensible architecture for easy addition of new metrics.

Python’s widespread use in the Music Information Retrieval (MIR) and machine
learning communities makes a Python-based audio quality assessment library
both useful and necessary for evaluating NAS technologies. In order to ensure
replicability and reproducibility, we have written comprehensive tests where
possible to ensure that each user facing function behaves appropriately.

In the following demonstration, we will cover the library’s features, the
algorithms it includes, and its licensing.

##  2 Features of AQUATk

The library requires the following additional packages: numpy, openl3, tqdm,
panns-inference, jukemirlib, scikit-learn, streamlit. Installation also
includes PyTorch and TensorFlow as dependencies from openl3 and jukemirlib.

###  2.1 Distances

AquaTk includes the following metrics to compute difference between reference
and test signals:

  * •

Mean-squared error

  * •

Mean-absolute error

  * •

Cosine similarity

  * •

KL-divergence

  * •

SI-SDR

  * •

SNR

The library also supports embedding-based distances:

  * •

Fréchet Audio Distance [1] and,

  * •

Kernel Distances [2]

Both of which are usable with both default and user-defined embeddings.

###  2.2 Embedding extraction

AquaTk supports popular embeddings such as Openl3 [3], PANNSs[4], JukeMIR [5],
and VGGish[6].

###  2.3 Basic PEAQ

AquaTk offers a unique open-source implementation of the basic Perceptual
Evaluation of Audio Quality (PEAQ) algorithm [7], based on the PEAQb
package111https://github.com/akinori-ito/peaqb-fast. It enables the
computation of the Objective Difference Grade (ODG) using the FFT Ear Model
and the following Model Output Variables (MOVs):

AvgBwRef, AvgBwTst, NMRtotB, Average Distorted Block, Maximum probability of
detection, Harmonic error over time, RDF, WModDiff1b, Average modulation
difference, Average Modulation Difference with introduced modulations,
Loudness

PEAQ has regained attention in the realm of audio quality assessment,
particularly due to its perceptually relevant features. An example is the 2F
model, which uses PEAQ Model Output Variables (MOVs) to evaluate source
separation performance [8]. Early indications suggest that the performance of
our port is comparable to the existing PEAQb package.

##  3 Usage

AquaTk can be operated in one of three modes: as a library, a command-line
tool, or a browser-based app.

  1. 1.

Library Mode: This mode offers flexibility by granting users direct access to
the complete API and source code. Users can seamlessly integrate aquatk into
their existing machine learning pipelines.

  2. 2.

Command-Line Mode: Aimed at efficiency, this mode enables quick extraction of
embeddings and the computation of metrics on .wav file directories.

  3. 3.

Web Interface: Constructed with Streamlit, this user-friendly mode replicates
the functionalities of the command-line tool and enhances the experience with
in-app visualization features. An example can be seen in 1.

Figure 1: A graphical representation of AquaTK

In Library Mode, importing aquatk into an existing ML pipeline grants access
to the library’s suite of metrics. Additionally, using AquaTk in library mode
allows users to extend the package to better suit their needs.

Upon package installation, the aquatk command becomes available automatically.
The \--web flag launches the web interface, while various other flags can be
used for command-line operations.

##  4 Availability and license

We have opted to release AquaTk under the GPL license, in accordance with the
licensing terms of PeaqB. The source code will be publicly accessible via
GitHub222https://github.com/Ashvala/AQUA-Tk.

We actively encourage community involvement and contributions. One can
contribute to AquaTk either by forking the repository or by submitting pull
requests.

##  5 Conclusion

We present AquaTk, a Python-based, open-source library designed to address the
gap in standardized evaluation methods for Neural Audio Synthesis (NAS).
AquaTk supports a variety of quality metrics, including a python
implementation of the basic PEAQ algorithm. The library is versatile, with
three modes of operation—Library, Command-Line, and Web Interface—that cater
to different user needs. Released under the GPL license, AquaTk is openly
accessible and invites community collaboration.

Future work includes integrating additional metrics, improving scalability,
and extending the library for real-time evaluation tasks. By providing a
unified framework for evaluating NAS systems, AquaTk aims to become a valuable
resource for both researchers and practitioners in the field.

## References

  * [1] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms,” in _Interspeech 2019_ , pp. 2350–2354.
  * [2] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demystifying MMD GANs,” in _ICLR 2018_.
  * [3] J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings,” in _ICASSP 2019_. IEEE, pp. 3852–3856. [Online]. Available: https://ieeexplore.ieee.org/document/8682475/
  * [4] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,” vol. 28, pp. 2880–2894.
  * [5] R. Castellon, C. Donahue, and P. Liang, “CODIFIED AUDIO LANGUAGE MODELING LEARNS USEFUL REPRESENTATIONS FOR MUSIC INFORMATION RETRIEVAL,” in _ISMIR 2021_.
  * [6] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson. CNN Architectures for Large-Scale Audio Classification. [Online]. Available: http://arxiv.org/abs/1609.09430
  * [7] T. Thiede, “PEAQ–The ITU Standardfor ObjectiveMeasuremenot f Perceived Audio Quality,” vol. 48, no. 1, p. 27.
  * [8] M. Torcoli, T. Kastner, and J. Herre, “Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence,” vol. 29, pp. 1530–1541. [Online]. Available: https://ieeexplore.ieee.org/document/9388867/