# Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and
Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

Lirong Wu ${}^{1,2,^{*}}$, Haitao Lin${}^{1,2,^{*}}$, Yufei Huang1,2, Tianyu
Fan2, Stan.Z.Li${}^{1,^{\dagger}}$

###### Abstract

Recent years have witnessed the great success of Graph Neural Networks (GNNs)
in handling graph-related tasks. However, MLPs remain the primary workhorse
for practical industrial applications due to their desirable inference
efficiency and scalability. To reduce their gaps, one can directly distill
knowledge from a well-designed teacher GNN to a student MLP, which is termed
as GNN-to-MLP distillation. However, the process of distillation usually
entails a loss of information, and “which knowledge patterns of GNNs are more
likely to be left and distilled into MLPs?” becomes an important question. In
this paper, we first factorize the knowledge learned by GNNs into low- and
high-frequency components in the spectral domain and then derive their
correspondence in the spatial domain. Furthermore, we identified a potential
_information drowning_ problem for existing GNN-to-MLP distillation, i.e., the
high-frequency knowledge of the pre-trained GNNs may be overwhelmed by the
low-frequency knowledge during distillation; we have described in detail what
it represents, how it arises, what impact it has, and how to deal with it. In
this paper, we propose an efficient Full-Frequency GNN-to-MLP (FF-G2M)
distillation framework, which extracts both low-frequency and high-frequency
knowledge from GNNs and injects it into MLPs. Extensive experiments show that
FF-G2M improves over the vanilla MLPs by 12.6% and outperforms its
corresponding teacher GNNs by 2.6% averaged over six graph datasets and three
common GNN architectures. Codes are publicly available at:
https://github.com/LirongWu/FF-G2M.

## Introduction

In many real-world applications, including social networks, chemical
molecules, and citation networks, data can be naturally modeled as graphs.
Recently, the emerging Graph Neural Networks (GNNs) (Hamilton, Ying, and
Leskovec 2017; Kipf and Welling 2016; Veličković et al. 2017; Wu et al. 2021b,
2022b; Liu, Gao, and Ji 2020; Zhou et al. 2020; Liu et al. 2022; Xia et al.
2022) have demonstrated their powerful capability to handle various graph-
related tasks (Zhang and Chen 2018; Fan et al. 2019; Errica et al. 2019; Wu et
al. 2021a). However, practical deployments of GNNs in the industry are still
less popular due to inference efficiency and scalability challenges incurred
by data dependency (Jia et al. 2020; Zhang et al. 2021). In other words, GNNs
generally rely on message passing to aggregate features from the neighborhood,
but fetching and aggregating these nodes during inference can burden latency-
sensitive applications. In contrast, Multi-Layer Perceptrons (MLPs) involve no
data dependence between pairs of nodes and infer much faster than GNNs, but
often with less competitive performance. Motivated by these complementary
strengths and weaknesses, one solution to reduce their gaps is to perform GNN-
to-MLP knowledge distillation (Yang, Liu, and Shi 2021; Zhang et al. 2021;
Ghorbani et al. 2021; Gou et al. 2021), which extracts knowledge from a well-
trained teacher GNN and then distills it into a student MLP with the same
network architecture (e.g., layer number and layer size).

Most of the existing GNN-to-MLP distillation methods (Yang, Liu, and Shi 2021;
Zhang et al. 2021; Ghorbani et al. 2021) focus on special designs on either
student MLPs or teacher GNNs, but default to distill knowledge in a node-to-
node fashion. For example, CPF (Yang, Liu, and Shi 2021) combines Label
Propagation (LP) (Iscen et al. 2019) into the student MLPs to improve
classification performance and thus still suffers from the neighborhood-
fetching latency caused by label propagation, which defeats the original
intention of MLPs to be inference-efficient. In contrast, GLNN (Zhang et al.
2021) directly distills knowledge from arbitrary GNNs to vanilla MLPs with the
same network architecture. While the distilled MLPs of GLNN can be greatly
improved by employing more powerful teacher GNNs, the process of distillation
usually entails a loss of information (Kim et al. 2021), which may lead to
sub-optimal student MLPs. In this paper, we look away from specific
instantiations of teacher GNNs and student MLPs, but rather explore two
fundamental questions: (1) Can existing GNN-to-MLP distillation ensure that
sufficient knowledge is distilled from teacher GNNs to student MLPs? If not,
(2) Which knowledge patterns of GNNs are more likely to be distilled into
student MLPs?

Present Work. In this paper, we identify a potential _information drowning_
problem for existing GNN-to-MLP distillation, i.e., the high-frequency
knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency
knowledge during distillation. To illustrate this, we first factorize GNN
knowledge into low- and high-frequency components using graph signal
processing theory in the spectral domain and then derive their correspondence
in the spatial domain. Moreover, we conduct a comprehensive investigation of
the roles played by low- and high-frequency components in the distillation
process and describe in detail what _information drowning_ represents, how it
arises, what impact it has, and how to deal with it. Extensive experiments
have shown that high-frequency and low-frequency knowledge are complementary
to each other, and they can further improve performance on top of each other.
In this paper, we propose a novel Full-Frequency GNN-to-MLP (FF-G2M)
distillation framework, which extracts both low- and high-frequency knowledge
from teacher GNNs and injects it into student MLPs. As a simple and general
framework, FF-G2M outperforms other leading methods across various GNN
architectures and graph datasets in terms of classification performance.

## Related Work

Graph Neural Networks (GNNs). The early GNNs define graph convolution kernels
in the spectral domain (Bruna et al. 2013; Defferrard, Bresson, and
Vandergheynst 2016) based on the graph signal processing theory, known as
ChebyNet (Defferrard, Bresson, and Vandergheynst 2016) and Graph Convolutional
Networks (GCN) (Kipf and Welling 2016). The later GNNs directly define
updating rules in the spatial space and focus on the design of neighborhood
aggregation functions. For instance, GraphSAGE (Hamilton, Ying, and Leskovec
2017) employs a generalized induction framework to generate embeddings for
previously unseen nodes by aggregating known node features. Moreover, GAT
(Veličković et al. 2017) introduces the self-attention mechanism to assign
different importance scores to neighbors for better information aggregation.
We refer interested readers to the surveys (Liu, Gao, and Ji 2020; Wu et al.
2020; Zhou et al. 2020) for more GNN architectures.

Graph Knowledge Distillation. Despite the great progress, most existing GNNs
share the de facto design that relies on message passing to aggregate features
from neighborhoods, which may be one major source of latency in GNN inference.
To address this problem, there are previous works that attempt to distill
knowledge from large teacher GNNs to smaller student GNNs, termed as GNN-to-
GNN (Lassance et al. 2020; Wu et al. 2022a; Ren et al. 2021; Joshi et al.
2021). For example, the student model in RDD (Zhang et al. 2020) and TinyGNN
(Yan et al. 2020) is a GNN with fewer parameters but not necessarily fewer
layers than the teacher GNN, which makes both designs still suffer from the
neighborhood-fetching latency caused by data dependency.

To enjoy the low-latency of MLPs and high-accuracy of GNNs, the other branch
of graph knowledge distillation is to directly distill from large teacher GNNs
to student MLPs, termed as GNN-to-MLP. The existing work on GNN-to-MLP
distillation can be mainly divided into two branches: student MLPs-focused and
teacher GNNs-focused. The former branch, such as CPF (Yang, Liu, and Shi
2021), directly improves student MLPs by adopting deeper and wider network
architectures or incorporating label propagation, both of which burden the
inference latency. The other branch, such as GLNN (Zhang et al. 2021),
distills knowledge from teacher GNNs to vanilla MLPs with the same network
architectures but without other computing-consuming operations; while the
performance of their distilled MLPs can be indirectly improved by employing
more powerful GNNs, they still cannot match their corresponding teacher GNNs.
In this paper, we aim to develop a model-agnostic GNN-to-MLP distillation that
is applicable to various GNN architectures.

## Preliminaries

Notions. Let $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})$ be an
attributed graph, where $\mathcal{V}$ is the set of $N$ nodes with features
$\mathbf{X}=\left[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N}\right]\in\mathbb{R}^{N\times
d}$ and $\mathcal{E}$ denotes the edge set. Each node $v_{i}\in\mathcal{V}$ is
associated with a $d$-dimensional features vector $\mathbf{x}_{i}$, and each
edge $e_{i,j}\in\mathcal{E}$ denotes a connection between node $v_{i}$ and
$v_{j}$. The graph structure is denoted by an adjacency matrix
$\mathbf{A}\in[0,1]^{N\times N}$ with $\mathbf{A}_{i,j}=1$ if
$e_{i,j}\in\mathcal{E}$ and $\mathbf{A}_{i,j}=0$ if
$e_{i,j}\notin\mathcal{E}$. Consider a semi-supervised node classification
task where only a subset of node $\mathcal{V}_{L}$ with labels
$\mathcal{Y}_{L}$ are known, we denote the labeled set as
$\mathcal{D}_{L}=(\mathcal{V}_{L},\mathcal{Y}_{L})$ and unlabeled set as
$\mathcal{D}_{U}=(\mathcal{V}_{U},\mathcal{Y}_{U})$, where
$\mathcal{V}_{U}=\mathcal{V}\backslash\mathcal{V}_{L}$. The node
classification aims to learn a mapping
$\Phi:\mathcal{V}\rightarrow\mathcal{Y}$ with labeled data, so that it can be
used to infer the labels $\mathcal{Y}_{U}$.

Graph Neural Networks (GNNs). Most existing GNNs rely on message passing to
aggregate features from the neighborhood. A general GNN framework consists of
two key computations for each node $v_{i}$: (1) $\operatorname{AGGREGATE}$:
aggregating messages from neighborhood $\mathcal{N}_{i}$; (2)
$\operatorname{UPDATE}$: updating node representation from its representation
in the previous layer and aggregated messages. Considering a $L$-layer GNN,
the formulation of the $l$-th layer is as follows

$\displaystyle\mathbf{m}_{i}^{(l)}=$
$\displaystyle\operatorname{AGGREGATE}^{(l)}\left(\big{\\{}\mathbf{h}_{j}^{(l-1)}:v_{j}\in\mathcal{N}_{i}\big{\\}}\right)$
(1) $\displaystyle\mathbf{h}_{i}^{(l)}=$
$\displaystyle\operatorname{UPDATE}^{(l)}\left(\mathbf{h}_{i}^{(l-1)},\mathbf{m}_{i}^{(l)}\right)$

where $1\leq l\leq L$, $\mathbf{h}_{i}^{(0)}=\mathbf{x}_{i}$ is the input
feature, and $\mathbf{h}_{i}^{(l)}$ is the representation of node $v_{i}$ in
the $l$-th layer.

Multi-Layer Perceptrons (MLPs). To achieve efficient inference, the vanilla
MLPs (with the same network architecture as the teacher GNNs) are used as the
student model by default in this paper. For a $L$-layer MLP, the $l$-th layer
is composed of a linear transformation, an activation function
$\sigma=\mathrm{ReLu}(\cdot)$, and a dropout function
$\operatorname{Dropout}(\cdot)$, as

$\mathbf{z}^{(l)}_{i}=\operatorname{Dropout}\big{(}\sigma\big{(}\mathbf{z}^{(l-1)}_{i}\mathbf{W}^{(l-1)}\big{)}\big{)},\quad\mathbf{z}^{(0)}_{i}=\mathbf{x}_{i}$
(2)

where $\mathbf{W}^{(0)}\in\mathbb{R}^{d\times F}$ and
$\mathbf{W}^{(l)}\in\mathbb{R}^{F\times F}$ $(1\leq l<L)$ are weight matrices
with the hidden dimension $F$. In this paper, the network architecture of
MLPs, such as the layer number $L$ and layer size $F$, is set the same as that
of teacher GNNs.

GNN-to-MLP Knowledge Distillation. The knowledge distillation is first
introduced in (Hinton et al. 2015) to handle mainly image data, where
knowledge is transferred from a cumbersome teacher model to a simpler student
model. The later works on GNN-to-MLP distillation (Yang, Liu, and Shi 2021;
Zhang et al. 2021; Ghorbani et al. 2021) extend it to the graph domain by
imposing KL-divergence constraint $\mathcal{D}_{KL}(\cdot,\cdot)$ between the
softmax label distributions generated by teacher GNNs and student MLPs and
directly optimizing the objective function as follows

$\displaystyle\mathcal{L}_{\mathrm{KD}}=\frac{1}{|\mathcal{V}|}\sum_{i\in\mathcal{V}}\mathcal{D}_{KL}\left(\operatorname{softmax}\big{(}\mathbf{z}_{i}^{(L)}\big{)},\operatorname{softmax}\big{(}\mathbf{h}_{i}^{(L)}\big{)}\right)$
(3)

## Knowledge Factorization from the Perspective of Spectral and Spatial Domain

In this section, we first theoretically factorize the knowledge learned by
GNNs into low- and high-frequency components in the spectral domain based on
graph signal processing theory (Shuman et al. 2013). The normalized graph
Laplacian matrix of graph $\mathcal{G}$ is defined as
$\mathbf{L}\\!=\\!\mathbf{I}_{N}\\!-\\!\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}$,
where
$\widetilde{\mathbf{A}}\\!=\\!\mathbf{A}\\!+\\!\mathbf{I}_{N}\in\mathbb{R}^{N\times
N}$ is an adjacency matrix with self-loop,
$\widetilde{\mathbf{D}}\in\mathbb{R}^{N\times N}$ is a diagonal degree matrix
with $\widetilde{\mathbf{D}}_{i,i}=\sum_{j}\widetilde{\mathbf{A}}_{i,j}$, and
$\mathbf{I}_{N}$ denotes the identity matrix. Since $\mathbf{L}$ is a real
symmetric matrix, it can be eigendecomposed as
$\mathbf{L}=\mathbf{U}\Lambda\mathbf{U}^{\top}$, where
$\Lambda=\operatorname{diag}\left(\left[\lambda_{1},\lambda_{2},\cdots,\lambda_{N}\right]\right)$
with each eigenvalue $\lambda_{l}\in[0,2]$ corresponding to an eigenvectors
$\mathbf{u}_{l}$ in $\mathbf{U}$ (Chung and Graham 1997). According to graph
signal processing theory, we can directly take the eigenvector
$\left\\{\mathbf{u}_{l}\right\\}_{l=1}^{N}$ as bases. Given signal
$\mathbf{x}\in\mathbb{R}^{d}$, the graph Fourier transform and inverse Fourier
transform (Sandryhaila and Moura 2013; Ricaud et al. 2019) are defined as
$\widehat{\mathbf{x}}=\mathbf{U}^{\top}\mathbf{x}$ and
$\mathbf{x}=\mathbf{U}\widehat{\mathbf{x}}$. Thus, the convolutional $*_{G}$
between the signal $\mathbf{x}$ and convolution kernel $\mathcal{F}$ can be
defined as follows

$\mathcal{F}*_{G}\mathbf{x}=\mathbf{U}\left(\left(\mathbf{U}^{\top}\mathcal{F}\right)\odot\left(\mathbf{U}^{\top}\mathbf{x}\right)\right)=\mathbf{U}\mathbf{g}_{\theta}\mathbf{U}^{\top}\mathbf{x}$
(4)

where $\odot$ denotes the element-wise product and $\mathbf{g}_{\theta}$ is a
parameterized diagonal matrix. Most of the existing GNNs architectures can be
regarded as a special instantiation on the convolutional kernel $\mathcal{F}$
(i.e., the matrix $\mathbf{g}_{\theta}$). For example, GCN-Cheby parameterizes
$g_{\theta}$ with a polynomial expansion
$\mathbf{g}_{\theta}\\!=\\!\sum_{k=0}^{K-1}\alpha_{k}\Lambda^{k}$, and GCN
defines the convolutional kernel as
$\mathbf{g}_{\theta}\\!=\\!\mathbf{I}_{N}-\Lambda$. Considering a special
convolution kernel $\mathcal{F}_{A}\\!=\\!\mathbf{I}_{N}$, we have
$\mathcal{F}_{A}*_{G}\mathbf{x}\\!=\\!\mathbf{U}\mathbf{I}_{N}\mathbf{U}^{\top}\mathbf{x}\\!=\\!\mathbf{U}\widehat{\mathbf{x}}\\!=\\!\mathbf{x}$,
i.e., this is an identity mapping, where all information can be preserved.
Next, we decompose the graph knowledge into low-frequency and high-frequency
components (Bo et al. 2021a; Wu et al. 2019) by factorizing
$\mathcal{F}_{A}\\!=\\!\mathbf{I}_{N}$ as follows

$\displaystyle\mathcal{F}_{A}\\!=\\!\mathbf{I}_{N}\\!=\\!\frac{1}{2}\bigg{(}\big{(}\underbrace{\mathbf{I}_{N}+\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}}_{\text{Low-
Pass Filter}\
\mathcal{F}_{M}}\big{)}+\big{(}\underbrace{\mathbf{I}_{N}-\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}}_{\text{High-
pass Filter}\ \mathcal{F}_{H}}\big{)}\bigg{)}$

For a given signal $\mathbf{x}\in\mathbb{R}^{d}$, e.g., node feature, we have
$\mathcal{F}_{A}*_{G}\mathbf{x}=\frac{1}{2}\left(\mathcal{F}_{M}+\mathcal{F}_{H}\right)*_{G}\mathbf{x}=\frac{1}{2}(\mathcal{F}_{M}*_{G}\mathbf{x}+\mathcal{F}_{H}*_{G}\mathbf{x})=\mathbf{x}$,
which means that any signal $\mathbf{x}$ can be decomposed into the average of
two components $\mathcal{F}_{M}*_{G}\mathbf{x}$ and
$\mathcal{F}_{H}*_{G}\mathbf{x}$.

Analysis on the Spectral Domain. The Proposition 1 states what the two
components $\mathcal{F}_{M}*_{G}\mathbf{x}$ and
$\mathcal{F}_{H}*_{G}\mathbf{x}$ represent.

###### Proposition 1

The convolution kernel $\mathcal{F}_{M}$ works as a low-pass filter, which
filters out high-frequency information, and $\mathcal{F}_{M}*_{G}\mathbf{x}$
represents low-frequency knowledge; $\mathcal{F}_{H}$ works as a high-pass
filter, which filters out low-frequency information, and
$\mathcal{F}_{H}*_{G}\mathbf{x}$ represents high-frequency knowledge.

###### Proof 1

For a $L$-layer GNN, the signal $\mathbf{x}$ is filtered by the $L$-order
convolution kernel
$\mathcal{F}_{M}^{L}=(\mathbf{I}_{N}+\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}})^{L}=(2\mathbf{I}_{N}-\mathbf{L})^{L}$
to output
$\mathcal{F}_{M}^{L}*_{G}\mathbf{x}=\mathbf{U}(2\mathbf{I}_{N}-\Lambda)^{L}\mathbf{U}^{\top}\mathbf{x}$
with $g_{\theta}^{L}(\lambda_{i})=(2-\lambda_{i})^{L}$. As shown in Fig. 1,
$g_{\theta}^{L}(\lambda_{i})$ decreases monotonically in the range
$\lambda_{i}\in[0,2]$ and reaches $g_{\theta}^{L}(\lambda_{i}\\!=\\!2)=0$ at
$\lambda_{i}=2$, which mainly amplifies the low-frequency information and
filters out the high-frequency information. Similarly, the $L$-order
convolution kernel $\mathcal{F}_{H}^{L}$ has
$g_{\theta}^{L}(\lambda_{i})=\lambda_{i}^{L}$. As shown in Fig. 1,
$g_{\theta}^{L}(\lambda_{i})$ increases monotonically in the of range
$\lambda_{i}\in[0,2]$ and reaches $g_{\theta}^{L}(\lambda_{i}=0)=0$ at
$\lambda_{i}=0$, which mainly filters out the low-frequency information but in
turn amplifies the high-frequency information.

Figure 1: Eigenvalues vs. Amplitudes

Correspondence on the Spatial Domain. We have derived that
$\mathcal{F}_{M}*_{G}\mathbf{x}$ and $\mathcal{F}_{H}*_{G}\mathbf{x}$
represent mainly the low- and high-frequency components of signal
$\mathbf{x}$, and we next can derived their correspondences in the spatial
domain, as follows

$\displaystyle\mathcal{F}_{M}*_{G}\mathbf{x}_{i}\ \ \rightarrow\ \
\mathbf{x}_{i}^{(low)}=\mathbf{x}_{i}+\sum_{j\in\mathcal{N}_{i}}\frac{\mathbf{x}_{j}}{\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}}$
(5) $\displaystyle\mathcal{F}_{H}*_{G}\mathbf{x}_{i}\ \ \rightarrow\ \
\mathbf{x}_{i}^{(high)}=\mathbf{x}_{i}-\sum_{j\in\mathcal{N}_{i}}\frac{\mathbf{x}_{j}}{\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}}$

By the derivation in Eq. (5), the low-frequency knowledge
$\mathcal{F}_{M}*_{G}\mathbf{x}$ is the sum of node feature and its
neighborhood features in the spatial domain. On the other hand, the high-
frequency knowledge $\mathcal{F}_{H}*_{G}\mathbf{x}$ represents the
differences between the target node feature with its neighborhood features.
There have recently been some novel GNN models (Bo et al. 2021b; Pei et al.
2020; Zhu et al. 2020; Chien et al. 2021) that can capture both low- and high-
frequency information simultaneously or adaptively. However, in this paper, we
focus on the design of distillation objective functions and do not consider
indirect performance improvements by employing these more powerful but complex
GNNs. Instead, we consider the most commonly used GNNs, such as GCN (Kipf and
Welling 2016), GraphSAGE (Hamilton, Ying, and Leskovec 2017), and GAT
(Veličković et al. 2017), all of which rely on multi-layer message passing to
aggregate features of neighboring nodes that are multiple hops away, i.e.,
they essentially work as a low-pass filter $\mathcal{F}_{M}^{L}$ or its
variants.

## Roles Played by Low- and High-Frequency Knowledge during Distillation

### Rethinking the Core of Knowledge Distillation

We rethink the core of knowledge distillation from three shallow-to-deep
perspectives to highlight our motivations.

  * •

Firstly, knowledge distillation enables the representations of MLPs to “mimic”
those of GNNs as closely as possible by imposing KL-divergence constraints
between their softmax distribution probabilities. However, such a mimicking
(or fitting) process is inevitably accompanied by a loss of information,
especially high-frequency information, which explains why the performance of
student MLPs is always hard to match with that of teacher GNNs.

  * •

Secondly, for a neural network framework, any change in the final
representations is achieved indirectly by optimizing the mapping function,
i.e., the network parameters. In this sense, knowledge distillation
essentially optimizes the parameter matrices
$\\{\mathbf{W}^{(l)}\\}_{l=0}^{L-1}$ of the student MLPs to make it
functionally approximates the convolution kernel of the teacher GNNs, which
makes the student MLPs also serve as a low-pass filter
$\widetilde{\mathcal{F}}_{M}^{L}$ for graph data.

  * •

Finally, the low-pass filter in the spectral domain is equivalent to
neighborhood aggregation in the spatial domain as derived in Eq. (5), which in
essence can be considered as a special use of the graph topology.

(a) Mean Cosine Similarity

(b) Pairwise Distance Differences

(c) Spectral Analysis

(d) Spatial Analysis

Figure 2: (a) Mean cosine similarity (the higher, the better) between nodes
with their first-order neighbors on Cora. (b) Pairwise distance differences
(the lower, the better) between teacher GCNs and student MLPs on Cora. (c)(d)
Illustrations of how the high-frequency information drowning arises and what
potential impact it has in the spectral and spatial domains, respectively.

To explore the roles played by graph topology during GNN-to-MLP distillation,
we plot the mean cosine similarity of nodes with their first-order neighbors
for vanilla GCNs, vanilla MLPs, and Distilled MLPs (GLNN) on the Cora dataset
in Fig. 2(a), from which we observe that the mean similarity of GCNs and GLNN
gradually increases with training, while that of vanilla MLPs gradually
decreases, which indicates that knowledge distillation has introduced graph
topology as an inductive bias (as GCNs has done), while vanilla MLPs do not.
As a result, the distilled MLPs can enjoy the benefits of topology-awareness
in training but without neighborhood-fetching latency in inference.

### High-Frequency Information Drowning

Next, we discuss a potential high-frequency information drowning problem from
both spectral and spatial domains, i.e., the high-frequency information of the
pre-trained GNNs may be overwhelmed by the low-frequency knowledge during the
process of GNN-to-MLP knowledge distillation.

How information drowning arises? From the perspective of spectral domain, the
knowledge distillation optimizes the network parameters of the student MLPs to
make it functionally approximate the convolution kernel of the teacher GNNs,
i.e., $\widetilde{\mathcal{F}}_{M}^{L}\\!\approx\\!\mathcal{F}_{M}^{L}$. The
information loss induced by such approximation may be inconsequential for
high-amplitude low-frequency information but can be catastrophic for those
high-frequency information with very low amplitude, as shown in Fig. 2(c). As
a result, compared to low-frequency information, high-frequency information is
more likely to be drowned by these optimization errors.

What impact does information drowning have? From the perspective of spatial
domain, the information drowning may lead to distilled MLPs that, despite
preserving neighborhood smoothing well, can easily neglect differences between
nodes, such as pairwisde distances. To illustrate this, we consider a target
node $v_{i}$ and its two neighboring nodes $v_{j}$ and $v_{k}$ in Fig. 2(d),
where they are mapped closely by GNNs. In the process of knowledge
distillation, the representations of these three nodes may be mapped around
the representations of teacher GNNs, i.e., they are still mapped closely with
most of their low-frequency information preserved; however, the relative
distances between nodes, i.e., their high-frequency information, may be
drowned dramatically. For example, node $v_{i}$ is adjacent to node $v_{j}$
but far from node $v_{k}$ in the representation space of teacher GNNs.
However, in the representation space of student MLPs, node $v_{i}$ becomes
closer to node $v_{k}$ and farther from node $v_{j}$.

The curves of the pairwise distance differences between the teacher GCNs and
the student MLP in Fig. 2(b) show that common knowledge distillation (e.g.,
GLNN) is not good at capturing high frequency information, compared to our
proposed FF-G2M. Moreover, extensive qualitative and quantitative experiments
have been provided to demonstrate the harmfulness of the identified high-
frequency information drowning problem in the experimental section. The
detailed experimental settings, including hyperparameters and evaluation
metric definitions, are available in Appendix B&E.

Figure 3: Illustration of the Full-Frequency GNN-to-MLP (FF-G2M) distillation
framework, where the dotted red lines denote the predicted class-boundary, the
solid black lines denote feature aggregation from the neighborhood, and the
dashed black lines denote the distillation of knowledge (neighborhood features
and pairwise distances) from teacher GNNs to student MLPs.

## Full-Frequency GNN-to-MLP (FF-G2M) Knowledge Distillation

The above discussions reached two important insights: (1) the inductive bias
of graph topology plays an important role, and (2) it is mainly the low-
frequency knowledge of graph data that has been distilled from the teacher
GNNs to the student MLPs. Inspired by these two insights, we propose Low-
Frequency Distillation (LFD) and Hign-Frequency Distillation (HFD) to fully
capture the low-frequency and high-frequency knowledge learned by GNNs,
respectively. An high-level overview of the proposed Full-Frequency GNN-to-MLP
(FF-G2M) framework is shown in Fig. 3.

### Low-Frequency Distillation (LFD)

The node representations of teacher GNNs are generated by explicit message
passing, so it mainly captures the low-frequency information of the graph data
as analyzed earlier. Unlike aggregating features from neighborhoods as in
GNNs, we directly distill (diffuse) knowledge from teacher GNNs into the
neighborhoods of student MLPs in order to better utilize topological
information and low-frequency knowledge captured by GNNs, formulated as
follows

$\mathcal{L}_{\mathrm{LFD}}\\!=\\!\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{V}}\sum_{j\in\mathcal{N}_{i}\cup
i}\\!\mathcal{D}_{KL}\Big{(}\sigma(\mathbf{z}_{j}^{(L)}/\tau_{1}),\sigma(\mathbf{h}_{i}^{(L)}/\tau_{1})\Big{)}$
(6)

where $\tau_{1}$ is the low-frequency distillation temperature, and
$\sigma=\operatorname{softmax}(\cdot)$ denotes an activation function.

### High-Frequency Distillation (HFD)

As derived in Eq. (5), the high-frequency components in the spectral domain
represent the differences between node feature and its neighborhood features
in the spatial domain. Inspired by this, we propose High-Frequency
Distillation (HFD), a GNN knowledge objective that trains student MLPs to
preserve the neighborhood pairwise differences from the representation space
of teacher GNNs. The neighborhood pairwise differences around node $v_{i}$ are
defined as the differences between the target node feature $\mathbf{s}_{i}$
and its neighborhood features $\\{\mathbf{s}_{j}\ |\ j\in\mathcal{N}_{i}\\}$,
which can be computed by the kernel
$\mathcal{K}\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)=\left|\mathbf{s}_{i}-\mathbf{s}_{j}\right|$,
where $|\cdot|$ denotes the element-wise absolute values. The high-frequency
distillation trains the student model to mimic the neighborhood pairwise
differences from the teacher GNNs via KL-divergence constraints, which can be
defined as follows

$\displaystyle\mathcal{L}_{\mathrm{HFD}}\\!=\\!\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{V}}\sum_{j\in\mathcal{N}_{i}}\mathcal{D}_{\mathrm{KL}}\Big{(}\sigma\big{(}\mathcal{K}\big{(}\mathbf{z}_{i}^{(L)},\mathbf{z}_{j}^{(L)}\big{)}/\tau_{2}\big{)},$
(7)
$\displaystyle\sigma\big{(}\mathcal{K}\big{(}\mathbf{h}_{i}^{(L)},\mathbf{h}_{j}^{(L)}\big{)}/\tau_{2}\big{)}\Big{)}$

where $\mathcal{K}\left(\cdot,\cdot\right)$ denotes the element-wise absolute
values, and $\tau_{2}$ is the high-frequency distillation temperature.

### Training Strategy

The pseudo-code of the FF-G2M framework is summarized in Appendix C. To
achieve GNN-to-MLP knowledge distillation, we first pre-train the teacher GNNs
with the classification loss
$\mathcal{L}_{\mathrm{label}}=\frac{1}{|\mathcal{V}_{L}|}\sum_{i\in\mathcal{V}_{L}}\mathcal{H}\big{(}y_{i},\sigma(\mathbf{h}_{i}^{(L)})\big{)}$,
where $\mathcal{H}(\cdot)$ denotes the cross-entropy loss and $y_{i}$ is the
ground-truth label of node $v_{i}$. Finally, the total objective function to
distill the low- and high-frequency knowledge from the teacher GNNs into the
student MLPs is defined as follows

$\mathcal{L}_{\mathrm{total}}=\frac{\lambda}{|\mathcal{V}_{L}|}\sum_{i\in\mathcal{V}_{L}}\mathcal{H}\big{(}y_{i},\sigma(\mathbf{z}_{i}^{(L)})\big{)}+\big{(}1-\lambda\big{)}\big{(}\mathcal{L}_{\mathrm{LFD}}+\mathcal{L}_{\mathrm{HFD}}\big{)}$

where $\lambda$ is the weights to balance the influence of the classification
loss and two knowledge distillation losses. The time complexity analysis of
FF-G2M is available in Appendix D.

### Discussion and Comparision

In this subsection, we compare the proposed FF-G2M framework with the commonly
used node-to-node distillation (e.g., GLNN) in Fig. 3. While the node-to-node
distillation can map neighboring nodes closely in the representation space of
MLPs, i.e., preserving low-frequency knowledge, it completely confounds the
relative distance between node pairs, i.e., high-frequency knowledge is
drowned, leading to a different (incorrect) class-boundary with the teacher
GNNs. In terms of the proposed FF-G2M framework, the Low-Frequency
Distillation distills (diffuses) the features aggregated from the neighborhood
in the teacher GNNs back into their neighborhood of the student MLPs to better
utilize the extracted low-frequency knowledge. Besides, High-Frequency
Distillation directly distills the neighborhood pairwise differences from
teacher GNNs into the student MLPs to better capture the high-frequency
knowledge patterns, i.e., the relative positions between pairs of nodes.

Table 1: Classificatiom accuracy $\pm$ std (%) on six real-world datasets,
where we consider three different GNN architectures (GCN, GraphSAGE, and GAT)
as the teacher and pure MLPs as the student model. The best metrics are marked
by bold.

Teacher GNNs | Method | Cora | Citeseer | Pubmed | Amazon-Photo | Coauthor-CS | Coauthor-Phy | Average$\uparrow$
---|---|---|---|---|---|---|---|---
GCN | Vanilla GCN | 82.2 $\pm$ 0.5 | 71.6 $\pm$ 0.4 | 79.3 $\pm$ 0.3 | 91.8 $\pm$ 0.6 | 89.9 $\pm$ 0.7 | 91.9 $\pm$ 1.2 | -
Vanilla MLP | 59.7 $\pm$ 1.0 | 60.7 $\pm$ 0.5 | 71.5 $\pm$ 0.5 | 77.4 $\pm$ 1.2 | 87.5 $\pm$ 1.4 | 89.2 $\pm$ 0.9 | -
GLNN | 82.8 $\pm$ 0.5 | 72.7 $\pm$ 0.4 | 80.2 $\pm$ 0.6 | 91.4 $\pm$ 1.0 | 92.7 $\pm$ 1.0 | 93.2 $\pm$ 0.5 | -
FF-G2M | 84.3 $\pm$ 0.4 | 74.0 $\pm$ 0.5 | 81.8 $\pm$ 0.4 | 94.2 $\pm$ 0.4 | 93.8 $\pm$ 0.5 | 94.4 $\pm$ 0.9 | -
$\Delta_{GCN}$ | 2.1 | 2.4 | 2.5 | 2.4 | 3.9 | 2.5 | 2.63
$\Delta_{MLP}$ | 24.6 | 13.3 | 10.3 | 16.8 | 6.3 | 5.2 | 12.75
$\Delta_{GLNN}$ | 1.5 | 1.3 | 1.6 | 2.8 | 1.1 | 1.2 | 1.58
GraphSAGE | Vanilla SAGE | 82.5 $\pm$ 0.6 | 70.9 $\pm$ 0.6 | 77.9 $\pm$ 0.4 | 92.0. $\pm$ 0.6 | 89.7 $\pm$ 1.0 | 92.2 $\pm$ 0.9 | -
Vanilla MLP | 59.7 $\pm$ 1.0 | 60.7 $\pm$ 0.5 | 71.5 $\pm$ 0.5 | 77.4 $\pm$ 1.2 | 87.5 $\pm$ 1.4 | 89.2 $\pm$ 0.9 | -
GLNN | 82.7 $\pm$ 0.8 | 70.5 $\pm$ 0.5 | 79.7 $\pm$ 0.6 | 91.0 $\pm$ 0.9 | 92.5 $\pm$ 1.0 | 93.0 $\pm$ 0.7 | -
FF-G2M | 84.5 $\pm$ 0.8 | 73.1 $\pm$ 1.2 | 80.9 $\pm$ 0.5 | 94.1 $\pm$ 0.5 | 93.6 $\pm$ 0.5 | 94.5 $\pm$ 1.1 | -
$\Delta_{SAGE}$ | 2.0 | 2.2 | 3.0 | 2.1 | 3.9 | 2.3 | 2.58
$\Delta_{MLP}$ | 24.8 | 12.4 | 9.4 | 16.7 | 6.1 | 5.3 | 12.45
$\Delta_{GLNN}$ | 1.8 | 2.6 | 1.2 | 3.1 | 1.1 | 1.5 | 1.88
GAT | Vanilla GAT | 81.9 $\pm$ 1.0 | 71.2 $\pm$ 0.5 | 78.6 $\pm$ 0.4 | 91.4 $\pm$ 0.8 | 90.5 $\pm$ 0.8 | 92.3 $\pm$ 1.5 | -
Vanilla MLP | 59.7 $\pm$ 1.0 | 60.7 $\pm$ 0.5 | 71.5 $\pm$ 0.5 | 77.4 $\pm$ 1.2 | 87.5 $\pm$ 1.4 | 89.2 $\pm$ 0.9 | -
GLNN | 82.1 $\pm$ 0.7 | 70.6 $\pm$ 0.8 | 80.4 $\pm$ 1.0 | 91.7 $\pm$ 0.7 | 92.7 $\pm$ 0.9 | 92.7 $\pm$ 0.6 | -
FF-G2M | 84.0 $\pm$ 0.7 | 73.5 $\pm$ 0.8 | 81.1 $\pm$ 0.6 | 93.9 $\pm$ 0.7 | 94.0 $\pm$ 0.7 | 94.7 $\pm$ 1.2 | -
$\Delta_{GAT}$ | 2.1 | 2.3 | 2.5 | 2.5 | 3.5 | 2.4 | 2.55
$\Delta_{MLP}$ | 24.3 | 12.8 | 9.6 | 16.5 | 6.5 | 5.5 | 12.53
$\Delta_{GLNN}$ | 1.9 | 2.9 | 0.7 | 2.2 | 1.3 | 2.0 | 1.83

## Experiments

Datasets. The effectiveness of the FF-G2M framework is evaluated on six public
real-world datasets, including Cora (Sen et al. 2008), Citeseer (Giles,
Bollacker, and Lawrence 1998), Pubmed (McCallum et al. 2000), Coauthor-CS,
Coauthor-Physics, and Amazon-Photo (Shchur et al. 2018). For each dataset,
following the data splitting settings of (Kipf and Welling 2016; Liu, Gao, and
Ji 2020), we select 20 nodes per class to construct a training set, 500 nodes
for validation, and 1000 nodes for testing. A statistical overview of these
datasets is placed in Appendix A. Besides, we defer the implementation details
and hyperparameter settings for each dataset to Appendix B and supplementary
materials.

Baselines. Three basic components in knowledge distillation are (1) teacher
model, (2) student model, and (3) distillation loss. As a model-agnostic
general framework, FF-G2M can be combined with any teacher GNN architecture.
In this paper, we consider three types of teacher GNNs, including GCN (Kipf
and Welling 2016), GraphSAGE (Hamilton, Ying, and Leskovec 2017), and GAT
(Veličković et al. 2017). As for the student model, we default to using pure
MLPs (with the same network architecture as the teacher GNNs) as the student
model for a fair comparison. Finally, the focus of this paper is on designing
distillation objectives rather than powerful teacher and student models.
Therefore, we only take GLNN (Zhang et al. 2021) as an important baseline to
compare FF-G2M with the conventional node-to-node distillation approach. The
experiments of all baselines and FF-G2M are implemented based on the standard
implementation in the DGL library (Wang et al. 2019) using PyTorch 1.6.0
library on NVIDIA V100 GPU. Each set of experiments is run five times with
different random seeds, and the average are reported as metrics.

### Classification Performance Comparison

This paper aims to explore which knowledge patterns should and how to be
distilled into student MLPs, rather than designing more powerful teacher GNNs.
Therefore, we consider three classical GNNs, including GCN, GraphSAGE, and
GAT, as the teacher models and distill their knowledge into MLPs with the same
network architecture. The experimental results on six datasets are reported in
Table. 1, from which we can make the following observations: (1) In general,
more powerful teacher GNNs can lead to student MLPs with better classification
performance. However, such improvements are usually very limited and do not
work for all datasets and GNN architectures. For example, on the Citeseer
dataset, the performance of GLNN drops over the vanilla implementation of
teacher GNNs by 0.4% (GraphSAGE) and 0.6% (GAT), respectively. (2) The
proposed FF-G2M framework can consistently improve the performance of student
MLPs across three GNN architectures on all six datasets. For example, FF-G2M
can outperform vanilla teacher GNNs by 2.63% (GCN), 2.58% (GraphSAGE), and
2.55% (GAT) averaged over six datasets, respectively.

(a) Pre-trained GCNs

(b) Vanilla MLPs

(c) Distilled MLPs (GLNN)

(d) Distilled MLPs (FF-G2M)

Figure 4: Representation 2D-Visualizations (by UMAP (McInnes, Healy, and
Melville 2018)) of the teacher model and three student models on Cora. Each
node is colored by its ground-truth label, and the numbers around the nodes
denote the node ids. Table 2: Classificatiom accuracy $\pm$ std (%) on six
real-world datasets. The best metrics are marked by bold.

Method | Cora | Citeseer | Pubmed | Amazon-Photo | Coauthor-CS | Coauthor-Phy
---|---|---|---|---|---|---
Vanilla GCN | 82.2 $\pm$ 0.5 | 71.6 $\pm$ 0.4 | 79.3 $\pm$ 0.3 | 91.8 $\pm$ 0.6 | 89.9 $\pm$ 0.7 | 91.9 $\pm$ 1.2
Vanilla MLP | 59.7 $\pm$ 1.0 | 60.7 $\pm$ 0.5 | 71.5 $\pm$ 0.5 | 77.4 $\pm$ 1.2 | 87.5 $\pm$ 1.4 | 89.2 $\pm$ 0.9
GLNN (Zhang et al. 2021) | 82.8 $\pm$ 0.5 | 72.7 $\pm$ 0.4 | 80.2 $\pm$ 0.6 | 91.4 $\pm$ 1.0 | 92.7 $\pm$ 1.0 | 93.2 $\pm$ 0.5
Low-Frequency KD w/ $\mathcal{L}_{\mathrm{LFD}}$ | 83.4 $\pm$ 0.9 | 73.7 $\pm$ 0.6 | 81.0 $\pm$ 0.5 | 92.1 $\pm$ 0.8 | 93.2 $\pm$ 0.8 | 93.7 $\pm$ 0.8
High-Frequency KD w/ $\mathcal{L}_{\mathrm{HFD}}$ | 68.5 $\pm$ 0.8 | 63.2 $\pm$ 0.7 | 74.4 $\pm$ 0.4 | 82.5 $\pm$ 1.3 | 89.3 $\pm$ 1.7 | 91.0 $\pm$ 1.6
FF-G2M (full model) | 84.3 $\pm$ 0.4 | 74.0 $\pm$ 0.5 | 81.8 $\pm$ 0.4 | 94.2 $\pm$ 0.4 | 93.8 $\pm$ 0.5 | 94.4 $\pm$ 0.9

### Qualitative and Quantitative Analysis

Extensive qualitative and quantitative experiments are conducted to explore
the existence and harmfulness of the information drowning problem and how to
solve it by FF-G2M.

#### Qualitative Analysis on Visualizations.

We consider GCNs as the teacher model and compare its visualization with that
of vanilla MLPs, GLNN, and FF-G2M on the Cora dataset (due to space
limitations, more results can be found in Appendix F). We select a target node
(id 27 for Cora) and analyze its relative position relationship with its
neighbors in Fig.4, from which we observe that: (1) The vanilla MLPs map
neighboring nodes apart, which indicates that it is not even good at capturing
low-frequency information. (2) GLNN fails to capture the relative positions
between the target node and its neighbors, i.e., high-frequency information.
(3) FF-G2M well preserves the relative positions between nodes while mapping
neighboring nodes closely, which suggests that it is good at capturing both
low- and high-frequency information. For example, on the Cora dataset, the
target node (id 27) is the closest to node (id 1810) and the farthest from
node (id 2678) in the visualizations of both teacher GCNs and FF-G2M’s student
MLPs.

#### Quantitative Analysis on Evaluation Metrics.

To study what knowledge patterns of GNNs are actually distilled into MLPs
during GNN-to-MLP distillation, we consider both (1) low-frequency knowledge,
measured by the mean cosine similarity of nodes with their first-order
neighbors, and (2) high-frequency knowledge, measured by KL-divergence between
the pairwise distances of teacher GNNs and student MLPs, respectively. The
detailed mathematical definitions of these two evaluation metrics are
available in Appendix E. From the experimental results on the Cora dataset
reported in Fig. 5, we make four observations: (1) The vanilla MLP does not
consider the inductive bias of the graph topology at all and thus fails to
capture the low-and high-frequency knowledge in the graph data. (2) GLNN is
capable of successfully capturing low-frequency information, i.e.,
neighborhood smoothing, but is not good at capturing high-frequency knowledge,
i.e., difference information between pairs of nodes. (3) The proposed low- and
high-frequency distillation has an advantage over GLNN in capturing one type
of individual frequency but lags behind in another frequency. (4) The proposed
FF-G2M combines both the two distillation and is better at capturing both low-
and high-frequency knowledge than GLNN, especially the latter.

(a) Low-frequency Knowledge

(b) High-frequency Knowledge

Figure 5: (a) Curves of mean cosine similarity (the higher, the better)
between nodes with their first-order neighbors. (b) Curves of pairwise
distance differences (the lower, the better) between the teacher GNNs and
student MLPs.

### Roles of Low- and High-frequency Knowledge

To evaluate the roles played by low- and high-frequency knowledge in GNN-to-
MLP distillation, we consider distillation with only
$\mathcal{L}_{\mathrm{LFD}}$ and $\mathcal{L}_{\mathrm{HFD}}$, in addition to
the full FF-G2M model. The experiments (with GCNs as the teacher model) on six
datasets are reported in Table. 2, from which we observe that: (1) The
proposed low-frequency distillation $\mathcal{L}_{\mathrm{LFD}}$ makes fuller
use of the graph topology and the low-frequency information from GNNs, and in
turn outperforms GLNN that adopts node-to-node distillation on all six
datasets. (2) While both low- and high-frequency distillation can work alone
to improve the performance of vanilla MLPs, the former plays a _primary_ role
and the latter a _secondary_ (auxiliary) role. More importantly, these two
distillations are complementary to each other and can further improve
performance on top of each other. (3) The FF-G2M (full model) considers both
low- and high-frequency distillation and is capable of capturing full-
frequency knowledge, and thus can far outperform GLNN on all six datasets.

## Conclusion

In this paper, we factorize the knowledge learned by GNNs into low- and high-
frequency components in the spectral and spatial domains and then conduct a
comprehensive investigation on their roles played in GNN-to-MLP distillation.
Our key finding is existing GNN-to-MLP distillation may suffer from a
potential _information drowning_ problem, i.e., the high-frequency knowledge
of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge
during distillation. Therefore, we propose a novel Full-Frequency GNN-to-MLP
(FF-G2M) knowledge distillation framework, which extracts both low- and high-
frequency knowledge from GNNs and injects it into MLPs. As a simple but
general framework, FF-G2M outperforms other leading methods across various GNN
architectures and graph datasets. Limitations still exist; for example, this
paper pays little attention to the special designs on teacher GNNs, and
designing more expressive teachers to directly capture full-frequency
knowledge may be another promising direction.

## Acknowledgement

This work is supported in part by Ministry of Science and Technology of the
People’s Republic of China (No. 2021YFA1301603) and National Natural Science
Foundation of China (No. U21A20427).

## References

  * Bo et al. (2021a) Bo, D.; Wang, X.; Shi, C.; and Shen, H. 2021a.  Beyond Low-frequency Information in Graph Convolutional Networks.  _arXiv preprint arXiv:2101.00797_.
  * Bo et al. (2021b) Bo, D.; Wang, X.; Shi, C.; and Shen, H. 2021b.  Beyond Low-frequency Information in Graph Convolutional Networks.  _Association for the Advance of Artificial Intelligence (AAAI)_.
  * Bruna et al. (2013) Bruna, J.; Zaremba, W.; Szlam, A.; and LeCun, Y. 2013.  Spectral networks and locally connected networks on graphs.  _arXiv preprint arXiv:1312.6203_.
  * Chien et al. (2021) Chien, E.; Peng, J.; Li, P.; and Milenkovic, O. 2021.  Adaptive Universal Generalized PageRank Graph Neural Network.  _International Conference on Learning Representations (ICLR)_.
  * Chung and Graham (1997) Chung, F. R.; and Graham, F. C. 1997.  _Spectral graph theory_.  American Mathematical Soc.
  * Defferrard, Bresson, and Vandergheynst (2016) Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016.  Convolutional neural networks on graphs with fast localized spectral filtering.  _arXiv preprint arXiv:1606.09375_.
  * Errica et al. (2019) Errica, F.; Podda, M.; Bacciu, D.; and Micheli, A. 2019.  A fair comparison of graph neural networks for graph classification.  _arXiv preprint arXiv:1912.09893_.
  * Fan et al. (2019) Fan, W.; Ma, Y.; Li, Q.; He, Y.; Zhao, E.; Tang, J.; and Yin, D. 2019.  Graph neural networks for social recommendation.  In _The world wide web conference_ , 417–426.
  * Ghorbani et al. (2021) Ghorbani, M.; Bahrami, M.; Kazi, A.; Soleymani Baghshah, M.; Rabiee, H. R.; and Navab, N. 2021.  GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 709–718. Springer.
  * Giles, Bollacker, and Lawrence (1998) Giles, C. L.; Bollacker, K. D.; and Lawrence, S. 1998.  CiteSeer: An automatic citation indexing system.  In _Proceedings of the third ACM conference on Digital libraries_ , 89–98.
  * Gou et al. (2021) Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021.  Knowledge distillation: A survey.  _International Journal of Computer Vision_ , 129(6): 1789–1819.
  * Hamilton, Ying, and Leskovec (2017) Hamilton, W.; Ying, Z.; and Leskovec, J. 2017.  Inductive representation learning on large graphs.  In _Advances in neural information processing systems_ , 1024–1034.
  * Hinton et al. (2015) Hinton, G.; Vinyals, O.; Dean, J.; et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Iscen et al. (2019) Iscen, A.; Tolias, G.; Avrithis, Y.; and Chum, O. 2019.  Label propagation for deep semi-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5070–5079.
  * Jia et al. (2020) Jia, Z.; Lin, S.; Ying, R.; You, J.; Leskovec, J.; and Aiken, A. 2020.  Redundancy-Free Computation for Graph Neural Networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 997–1005.
  * Joshi et al. (2021) Joshi, C. K.; Liu, F.; Xun, X.; Lin, J.; and Foo, C.-S. 2021.  On Representation Knowledge Distillation for Graph Neural Networks.  _arXiv preprint arXiv:2111.04964_.
  * Kim et al. (2021) Kim, T.; Oh, J.; Kim, N.; Cho, S.; and Yun, S.-Y. 2021.  Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation.  _arXiv preprint arXiv:2105.08919_.
  * Kipf and Welling (2016) Kipf, T. N.; and Welling, M. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Lassance et al. (2020) Lassance, C.; Bontonou, M.; Hacene, G. B.; Gripon, V.; Tang, J.; and Ortega, A. 2020\.  Deep geometric knowledge distillation with graphs.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 8484–8488. IEEE.
  * Liu, Gao, and Ji (2020) Liu, M.; Gao, H.; and Ji, S. 2020.  Towards deeper graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 338–348.
  * Liu et al. (2022) Liu, Z.; Luo, Y.; Wu, L.; Li, S.; Liu, Z.; and Li, S. Z. 2022.  Are Gradients on Graph Structure Reliable in Gray-box Attacks?  _arXiv preprint arXiv:2208.05514_.
  * Luo et al. (2021) Luo, Y.; Chen, A.; Yan, K.; and Tian, L. 2021.  Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages.  _arXiv preprint arXiv:2106.08541_.
  * McCallum et al. (2000) McCallum, A. K.; Nigam, K.; Rennie, J.; and Seymore, K. 2000.  Automating the construction of internet portals with machine learning.  _Information Retrieval_ , 3(2): 127–163.
  * McInnes, Healy, and Melville (2018) McInnes, L.; Healy, J.; and Melville, J. 2018.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_.
  * Pei et al. (2020) Pei, H.; Wei, B.; Chang, K. C.-C.; Lei, Y.; and Yang, B. 2020.  Geom-gcn: Geometric graph convolutional networks.  _arXiv preprint arXiv:2002.05287_.
  * Ren et al. (2021) Ren, Y.; Ji, J.; Niu, L.; and Lei, M. 2021.  Multi-task Self-distillation for Graph-based Semi-Supervised Learning.  _arXiv preprint arXiv:2112.01174_.
  * Ricaud et al. (2019) Ricaud, B.; Borgnat, P.; Tremblay, N.; Gonçalves, P.; and Vandergheynst, P. 2019.  Fourier could be a data scientist: From graph Fourier transform to signal processing on graphs.  _Comptes Rendus Physique_ , 20(5): 474–488.
  * Sandryhaila and Moura (2013) Sandryhaila, A.; and Moura, J. M. 2013.  Discrete signal processing on graphs: Graph fourier transform.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , 6167–6170. IEEE.
  * Sen et al. (2008) Sen, P.; Namata, G.; Bilgic, M.; Getoor, L.; Galligher, B.; and Eliassi-Rad, T. 2008\.  Collective classification in network data.  _AI magazine_ , 29(3): 93–93.
  * Shchur et al. (2018) Shchur, O.; Mumme, M.; Bojchevski, A.; and Günnemann, S. 2018.  Pitfalls of graph neural network evaluation.  _arXiv preprint arXiv:1811.05868_.
  * Shuman et al. (2013) Shuman, D. I.; Narang, S. K.; Frossard, P.; Ortega, A.; and Vandergheynst, P. 2013\.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains.  _IEEE signal processing magazine_ , 30(3): 83–98.
  * Veličković et al. (2017) Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_.
  * Wang et al. (2019) Wang, M.; Zheng, D.; Ye, Z.; Gan, Q.; Li, M.; Song, X.; Zhou, J.; Ma, C.; Yu, L.; Gai, Y.; He, T.; Karypis, G.; Li, J.; and Zhang, Z. 2019.  Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.  _arXiv preprint arXiv:1909.01315_.
  * Wu et al. (2019) Wu, F.; Souza, A.; Zhang, T.; Fifty, C.; Yu, T.; and Weinberger, K. 2019.  Simplifying graph convolutional networks.  In _International conference on machine learning_ , 6861–6871. PMLR.
  * Wu et al. (2021a) Wu, L.; Lin, H.; Gao, Z.; Tan, C.; Li, S.; et al. 2021a.  GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction.  _arXiv preprint arXiv:2106.11133_.
  * Wu et al. (2021b) Wu, L.; Lin, H.; Gao, Z.; Tan, C.; Li, S.; et al. 2021b.  Self-supervised on Graphs: Contrastive, Generative, or Predictive.  _arXiv preprint arXiv:2105.07342_.
  * Wu et al. (2022a) Wu, L.; Lin, H.; Huang, Y.; and Li, S. Z. 2022a.  Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks.  In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., _Advances in Neural Information Processing Systems_.
  * Wu et al. (2022b) Wu, L.; Lin, H.; Xia, J.; Tan, C.; and Li, S. Z. 2022b.  Multi-level disentanglement graph neural network.  _Neural Computing and Applications_ , 1–15.
  * Wu et al. (2020) Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; and Philip, S. Y. 2020.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_.
  * Xia et al. (2022) Xia, J.; Wu, L.; Chen, J.; Hu, B.; and Li, S. Z. 2022.  SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation.  In _Proceedings of the ACM Web Conference 2022_ , 1070–1079.
  * Yan et al. (2020) Yan, B.; Wang, C.; Guo, G.; and Lou, Y. 2020.  TinyGNN: Learning Efficient Graph Neural Networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 1848–1856.
  * Yang, Liu, and Shi (2021) Yang, C.; Liu, J.; and Shi, C. 2021.  Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework.  In _Proceedings of the Web Conference 2021_ , 1227–1237.
  * Zhang and Chen (2018) Zhang, M.; and Chen, Y. 2018.  Link prediction based on graph neural networks.  _arXiv preprint arXiv:1802.09691_.
  * Zhang et al. (2021) Zhang, S.; Liu, Y.; Sun, Y.; and Shah, N. 2021.  Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation.  _arXiv preprint arXiv:2110.08727_.
  * Zhang et al. (2020) Zhang, W.; Miao, X.; Shao, Y.; Jiang, J.; Chen, L.; Ruas, O.; and Cui, B. 2020.  Reliable data distillation on graph convolutional network.  In _Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data_ , 1399–1414.
  * Zhou et al. (2020) Zhou, J.; Cui, G.; Hu, S.; Zhang, Z.; Yang, C.; Liu, Z.; Wang, L.; Li, C.; and Sun, M. 2020.  Graph neural networks: A review of methods and applications.  _AI Open_ , 1: 57–81.
  * Zhu et al. (2020) Zhu, J.; Yan, Y.; Zhao, L.; Heimann, M.; Akoglu, L.; and Koutra, D. 2020.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs.  _Advances in Neural Information Processing Systems_ , 33.

## Appendix

### A. Dataset Statistics

_Six_ publicly available graph datasets are used to evaluate the FF-G2M
framework. An overview summary of the statistical characteristics of datasets
is given in Tab. A1. For the three small-scale datasets, namely Cora,
Citeseer, and Pubmed, we follow the data splitting strategy in (Kipf and
Welling 2016). For the three large-scale datasets, namely Coauthor-CS,
Coauthor-Phy, and Amazon-Photo, we follow (Zhang et al. 2021; Luo et al. 2021)
to randomly split the data, and each random seed corresponds to a different
splitting.

Table A1: Statistical information of the datasets.

Dataset | Cora | Citeseer | Pubmed | Amazon-Photo | Coauthor-CS | Coauthor-Phy
---|---|---|---|---|---|---
$\\#$ Nodes | 2708 | 3327 | 19717 | 7650 | 18333 | 34493
$\\#$ Edges | 5278 | 4614 | 44324 | 119081 | 81894 | 247962
$\\#$ Features | 1433 | 3703 | 500 | 745 | 6805 | 8415
$\\#$ Classes | 7 | 6 | 3 | 8 | 15 | 5
Label Rate | 5.2% | 3.6% | 0.3% | 2.1% | 1.6% | 0.3%

### B. Hyperparameters and Search Space

All baselines and our approach are implemented based on the standard
implementation in the DGL library (Wang et al. 2019) using the PyTorch 1.6.0
library with Intel(R) Xeon(R) Gold 6240R @ 2.40GHz CPU and NVIDIA V100 GPU.
The following hyperparameters are set for all datasets: learning rate $lr$ =
0.01 ($lr$ = 0.001 for Amazon-Photo); weight decay $decay$ = 5e-4; Maximum
Epoch $E$ = 500; Layer number $L$ = 2 ($L$ = 3 for Cora and Amazon-Photo);
distillation temperature $\tau_{1}$ = 1.0. The other dataset-specific
hyperparameters are determined by a hyperparameter search tool - NNI for each
dataset, including hidden dimension $F$, loss weight $\lambda$, dropout ratio
$R$, and distillation temperature $\tau_{2}$. The hyperparameter search space
is shown in Tab. A2, and the model with the highest validation accuracy is
selected for testing. The best hyperparameter choices for each dataset and GNN
architecture are available in the supplementary materials.

Table A2: Hyperparameter search space. Hyperparameters | Search Space
---|---
Hidden Dimension $F$ | [128, 256, 512, 1024, 2048]
Loss Weight $\lambda$ | [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]
Dropout Ratio $R$ | [0.3, 0.4, 0.5, 0.6]
Temperature $\tau_{2}$ | [1.0, 1.1, 1.2, 1.3, 1.4, 1.5]

### C. Pseudo-code

The pseudo-code of the proposed FF-G2M framework is summarized in Algorithm.
1.

### D. Complexity Analysis

The training time complexity of FF-G2M mainly comes from two parts: (1) GNN
training $\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ and (2) knowledge
distillation $\mathcal{O}(|\mathcal{E}|F)$, where $d$ and $F$ are the
dimensions of input and hidden spaces. The total time complexity
$\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ is linear w.r.t the number of
nodes $|\mathcal{V}|$ and edges $|\mathcal{E}|$, which is in the same order as
GCN and GLNN. Besides, the inference time of FF-G2M can be reduced from
$\mathcal{O}(|\mathcal{V}|dF+|\mathcal{E}|F)$ to
$\mathcal{O}(|\mathcal{V}|dF)$, which is as fast as MLP, due to the removal of
neighborhood aggregation.

Algorithm 1 Algorithm for the Full-Frequency GNN-to-MLP knowledge distillation
framework

1:Feature Matrix: $\mathbf{X}$; Edge Set: $\mathcal{E}$; # Epochs: $E$.

2:Predicted Labels $\mathcal{Y}_{U}$ and network parameters of student MLPs
$\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$.

3:Randomly initialize the parameters of GNNs and MLPs.

4:Compute GNN representations and pre-train the GNNs until convergence by
$\mathcal{L}_{\mathrm{label}}$.

5:for $epoch$ $\in$ {0, 1, $\cdots$, $E-1$} do

6: Compute GNN representations $\\{\mathbf{h}_{i}^{(L)}\\}_{i=1}^{N}$ from the

7: pre-trained GNNs and freeze it;

8: Compute MLP representations $\\{\mathbf{z}_{i}^{(L)}\\}_{i=1}^{N}$ and
calcu-

9: late the total loss $\mathcal{L}_{\mathrm{total}}$;

10: Update model parameters $\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$ by back propa-

11: gation of loss $\mathcal{L}_{\mathrm{total}}$.

12:end for

13:Predicted labels $\mathcal{Y}_{U}$ for those unlabeled nodes
$\mathcal{V}_{U}$.

14:return Predicted labels $\mathcal{Y}_{U}$ and the network parameters of
student MLPs $\\{\mathbf{W}^{l}\\}_{l=0}^{L-1}$.

### E. Definitions on Evaluation Metrics

As derived in Eq. (5), the low-frequency knowledge can be represented by the
sum of the target node feature and its neighborhood features in the spatial
domain, which essentially encourages neighborhood smoothing. In this paper,
the low-frequency knowledge is measured by the mean cosine similarity of nodes
with their 1-order neighbors, as follows

$\textit{mean cosine
similarit}=\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{V}}\sum_{j\in\mathcal{N}_{i}}\frac{\mathbf{s}_{i}\cdot\mathbf{s}_{j}}{\left|\mathbf{s}_{i}\right|\left|\mathbf{s}_{j}\right|}$

where $\mathbf{s}_{i}$ and $\mathbf{s}_{j}$ are the representations of node
$v_{i}$ and $v_{j}$, and we set $\mathbf{s}_{i}=\mathbf{h}_{i}^{(L)}$ for
teacher GNNs and $\mathbf{s}_{i}=\mathbf{z}_{i}^{(L)}$ for student MLPs. On
the other hand, the high-frequency knowledge can be represented as the
differences between the target node feature with its neighborhood features in
the spatial domain. In this paper, the low-frequency knowledge is measured by
the KL-divergence between the pairwise distances of teacher GNNs and student
MLPs, which is defined as

$\displaystyle\textit{KL-
divergence}=\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{V}}\sum_{j\in\mathcal{N}_{i}}\operatorname{softmax}\Big{(}\big{|}\mathbf{z}_{j}^{(L)}-$
$\displaystyle\mathbf{z}_{i}^{(L)}\big{|}\Big{)}\log\frac{\operatorname{softmax}\Big{(}\big{|}\mathbf{z}_{j}^{(L)}-\mathbf{z}_{i}^{(L)}\big{|}\Big{)}}{\operatorname{softmax}\Big{(}\big{|}\mathbf{h}_{j}^{(L)}-\mathbf{h}_{i}^{(L)}\big{|}\Big{)}}$

### F. More Qualitative Visualizations.

We consider GCNs as the teacher model and compare its visualization with that
of vanilla MLPs, GLNN, and FF-G2M on the Citeseer dataset. We select a target
node (id 557 for Citeseer) and analyze its relative position relationship with
its neighbors in Fig.A1. _The analyses presented in the paper for the Cora
dataset still hold true for the Citeseer dataset._

(a) Pre-trained GCNs (b) Vanilla MLPs

(c) Distilled MLPs (GLNN) (d) Distilled MLPs (FF-G2M)

Figure A1: Representation 2D-Visualizations (by UMAP (McInnes, Healy, and
Melville 2018)) of the teacher model and three student models on Citeseer.
Each node is colored by its ground-truth label, and the numbers around nodes
denote the node ids.


* Oosterhuis and de Rijke (2021b) Harrie Oosterhuis and Maarten de de Rijke. 2021b.  Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank. In _Proceedings of the Web Conference 2021_. 158–170.
  * Ovaisi et al. (2020) Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva. 2020.  Correcting for Selection Bias in Learning-to-rank Systems. In _Proceedings of The Web Conference 2020_. 1863–1873.
  * Qin and Liu (2013) Tao Qin and Tie-Yan Liu. 2013.  Introducing LETOR 4.0 datasets.  _arXiv preprint arXiv:1306.2597_ (2013).
  * Radlinski et al. (2008) Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008\.  How Does Clickthrough Data Reflect Retrieval Quality?. In _Proceedings of the 17th ACM Conference on Information and Knowledge Management_. ACM, 43–52.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting Clicks: Estimating the Click-Through Rate for New Ads. In _Proceedings of the 16th International Conference on World Wide Web_. 521–530.
  * Robins et al. (1994) James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994\.  Estimation of Regression Coefficients when Some Regressors are not Always Observed.  _Journal of the American statistical Association_ 89, 427 (1994), 846–866.
  * Saito (2020) Yuta Saito. 2020\.  Doubly Robust Estimator for Ranking Metrics with Post-Click Conversions. In _Fourteenth ACM Conference on Recommender Systems_. 92–100.
  * Saito and Joachims (2021) Yuta Saito and Thorsten Joachims. 2021.  Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent Advances. In _Fifteenth ACM Conference on Recommender Systems_. 828–830.
  * Saito et al. (2020) Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. 2020.  Unbiased Recommender Learning from Missing-not-at-Random Implicit Feedback. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 501–509.
  * Schuth et al. (2016) Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. 2016.  Multileave Gradient Descent for Fast Online Learning to Rank. In _Proceedings of the Ninth ACM International Conference on Web Search and Data Mining_. 457–466.
  * Singh and Joachims (2018) Ashudeep Singh and Thorsten Joachims. 2018.  Fairness of Exposure in Rankings. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2219–2228.
  * Singh and Joachims (2019) Ashudeep Singh and Thorsten Joachims. 2019.  Policy Learning for Fairness in Ranking. In _Advances in Neural Information Processing Systems_. 5426–5436.
  * Strehl et al. (2010) Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. 2010.  Learning from Logged Implicit Exploration Data. In _Advances in Neural Information Processing Systems_ , J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (Eds.), Vol. 23. Curran Associates, Inc.
  * Student (1908) Student. 1908.  The Probable Error of a Mean.  _Biometrika_ (1908), 1–25.
  * Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al. 1998\.  _Introduction to Reinforcement Learning_. Vol. 135.  MIT press Cambridge.
  * Ustimenko and Prokhorenkova (2020) Aleksei Ustimenko and Liudmila Prokhorenkova. 2020.  StochasticRank: Global Optimization of Scale-Free Discrete Functions. In _International Conference on Machine Learning_. PMLR, 9669–9679.
  * Vardasbi et al. (2020a) Ali Vardasbi, Maarten de Rijke, and Ilya Markov. 2020a.  Cascade Model-Based Propensity Estimation for Counterfactual Learning to Rank. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2089–2092.
  * Vardasbi et al. (2020b) Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020b.  When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_.
  * Wang et al. (2021) Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. 2021\.  Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 481–489.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to Rank with Selection Bias in Personal Search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. ACM, 115–124.
  * Wang et al. (2018a) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018a.  Position Bias Estimation for Unbiased Learning to Rank in Personal Search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. ACM, 610–618.
  * Wang et al. (2018b) Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018b.  The LambdaLoss Framework for Ranking Metric Optimization. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. ACM, 1313–1322.
  * Williams (1992) Ronald J Williams. 1992\.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.  _Machine Learning_ 8, 3-4 (1992), 229–256.
  * Yan et al. (2022) Le Yan, Zhen Qin, Honglei Zhuang, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2022\.  Revisiting Two Tower Models for Unbiased Learning to Rank. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Madrid, Spain) _(SIGIR ’22)_. ACM.
  * Yuan et al. (2020) Bowen Yuan, Yaxu Liu, Jui-Yang Hsia, Zhenhua Dong, and Chih-Jen Lin. 2020.  Unbiased Ad Click Prediction for Position-Aware Advertising Systems. In _Fourteenth ACM Conference on Recommender Systems_. 368–377.
  * Yue and Joachims (2009) Yisong Yue and Thorsten Joachims. 2009.  Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem. In _Proceedings of the 26th Annual International Conference on Machine Learning_. ACM, 1201–1208.
  * Zhuang et al. (2021) Honglei Zhuang, Zhen Qin, Xuanhui Wang, Michael Bendersky, Xinyu Qian, Po Hu, and Dan Chary Chen. 2021.  Cross-Positional Attention for Debiasing Clicks. In _Proceedings of the Web Conference 2021_. 788–797.
  * Zhuang and Zuccon (2020) Shengyao Zhuang and Guido Zuccon. 2020.  Counterfactual Online Learning to Rank. In _European Conference on Information Retrieval_. Springer, 415–430.

  *[LTR]: learning-to-rank
  *[IPS]: inverse-propensity-scoring
  *[DR]: doubly-robust
  *[RPS]: ratio-propensity-scoring
  *[CTR]: click-through-rate
  *[DCG]: discounted cumulative gain
  *[DM]: direct-method
  *[CV]: covariate
  *[EM]: expectation-maximization