$\mathbb{P}_{j}$. The intersection of $F_{j}$ with $H_{j}$ generates the class
group of $F_{j}$, so it is irreducible. General members of the pencil
$\mathcal{L}_{j}$ on $\mathbb{P}_{j}$ generated by $F_{j}$ and $dH_{j}$ are
then quasi-smooth terminal hypersurfaces of $\mathbb{P}_{j}$ of degree
$d_{j}$, too. By [CP17, Theorem 1.1.10] they are all birationally rigid.

By Theorem 1.4 general members of the pencil $\mathcal{L}_{j}$ on
$\mathbb{P}_{j}$ are realized as general fibers of a Mori fiber space
$p_{j}\colon V_{j}\rightarrow\mathbb{P}^{1}$, which contains
$\mathbb{A}^{4}\cong\mathbb{P}_{j}\setminus H_{j}$. Assume that two Mori fiber
spaces for $j=1,2$ are weakly square birational equivalent. Then there exists
a birational map $\chi\colon V_{1}\dashrightarrow V_{2}$ and an isomorphism
$\varphi\colon\mathbb{P}^{1}\rightarrow\mathbb{P}^{1}$ of the base curve such
that $p_{2}\circ\chi=\varphi\circ p_{1}$. Then for a general point
$t\in\mathbb{P}^{1}$, $\chi$ induces a birational map between
$(V_{1})_{t}=p_{1}^{-1}(t)$ and $(V_{2})_{\varphi(t)}=p_{2}^{-1}(\varphi(t))$.
Since for a general $t$ these threefolds are isomorphic to general members of
$\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ respectively, which are birationally
rigid, it follows that general members of $\mathcal{L}_{1}$ are isomorphic to
general members of $\mathcal{L}_{2}$. In particular, the self-intersections of
their respective anti-canonical divisors are equal and their singularities are
the same. By [IF00, §16.6] this implies that $\bar{a}_{1}=\bar{a}_{2}$. ∎

Finally, the following example gives a proof of Corollary 1.5.

######  Example 6.12.

Let $n=4$ and let the quadruple $\bar{a}=(a_{1},a_{2},a_{3},a_{4})$ be one of
those with numbers:

${\rm No.}\ 97-102,107-110,116,117$

in [Oka19, Table 1]. Then a very general quasi-smooth hypersurface
$F\subset\mathbb{P}(1,\bar{a})$ of degree $d=\sum_{j=1}^{4}a_{j}-\alpha$ is a
$\mathbb{Q}$-factorial terminal Fano threefold that is not stably rational if
$\alpha=3$ for No. $107-110$ or if $\alpha=5$ for No. 116, 117 or if
$\alpha=2$ otherwise. Fixing any such quadruple and choosing a very general
hypersurface $F$ of indicated degree $d$ we obtain by Theorem 1.4 a Mori fiber
completion of $\mathbb{A}^{4}$, $\pi\colon V\rightarrow\mathbb{P}^{1}$, whose
fibers are isomorphic to the members of the pencil $\langle F,dH\rangle$ other
than $dH$, hence whose very general fibers are not stably rational.

## References

  * [BCHM10] Caucher Birkar, Paolo Cascini, Christopher D. Hacon, and James McKernan, _Existence of minimal models for varieties of log general type_ , J. Amer. Math. Soc. 23 (2010), no. 2, 405–468.
  * [Bea77] Arnaud Beauville, _Variétés de Prym et jacobiennes intermédiaires_ , Ann. Sci. École Norm. Sup. (4) 10 (1977), no. 3, 309–391.
  * [BS13] Manuel Blickle and Karl Schwede, _$p^{-1}$ -linear maps in algebra and geometry_, Commutative algebra, Springer, New York, 2013, pp. 123–205.
  * [CDP18] Ivan Cheltsov, Adrien Dubouloz, and Jihun Park, _Super-rigid affine Fano varieties_ , Compos. Math. 154 (2018), no. 11, 2462–2484.
  * [CFST16] Giulio Codogni, Andrea Fanelli, Roberto Svaldi, and Luca Tasin, _Fano varieties in Mori fibre spaces_ , Int. Math. Res. Not. IMRN (2016), no. 7, 2026–2067.
  * [CFST18] by same author, _A note on the fibres of Mori fibre spaces_ , Eur. J. Math. 4 (2018), no. 3, 859–878.
  * [CG72] C. Herbert Clemens and Phillip A. Griffiths, _The intermediate Jacobian of the cubic threefold_ , Ann. of Math. (2) 95 (1972), 281–356.
  * [Che05] Ivan Cheltsov, _Birationally rigid Fano varieties_ , Uspekhi Mat. Nauk 60 (2005), no. 5(365), 71–160.
  * [CLS11] David A. Cox, John B. Little, and Henry K. Schenck, _Toric varieties_ , Graduate Studies in Mathematics, vol. 124, American Mathematical Society, Providence, RI, 2011.
  * [Cor00] Alessio Corti, _Singularities of linear systems and $3$-fold birational geometry_, Explicit birational geometry of 3-folds, London Math. Soc. Lecture Note Ser., vol. 281, Cambridge Univ. Press, Cambridge, 2000, pp. 259–312.
  * [CP17] Ivan Cheltsov and Jihun Park, _Birationally rigid Fano threefold hypersurfaces_ , Mem. Amer. Math. Soc. 246 (2017), no. 1167, v+117.
  * [CPR00] Alessio Corti, Aleksandr Pukhlikov, and Miles Reid, _Fano $3$-fold hypersurfaces_, Explicit birational geometry of 3-folds, London Math. Soc. Lecture Note Ser., vol. 281, Cambridge Univ. Press, Cambridge, 2000, pp. 175–258.
  * [dF13] Tommaso de Fernex, _Birationally rigid hypersurfaces_ , Invent. Math. 192 (2013), no. 3, 533–566.
  * [Dim92] Alexandru Dimca, _Singularities and topology of hypersurfaces_ , Universitext, Springer-Verlag, New York, 1992.
  * [DK17] Adrien Dubouloz and Takashi Kishimoto, _Explicit biregular/birational geometry of affine threefolds: completions of $\mathbb{A}^{3}$ into del Pezzo fibrations and Mori conic bundles_, Algebraic varieties and automorphism groups, Adv. Stud. Pure Math., vol. 75, Math. Soc. Japan, Tokyo, 2017, pp. 49–71.
  * [DK18] by same author, _Cylinders in del Pezzo fibrations_ , Israel J. Math. 225 (2018), no. 2, 797–815.
  * [Dol82] Igor Dolgachev, _Weighted projective varieties_ , Group actions and vector fields (Vancouver, B.C., 1981), Lecture Notes in Math., vol. 956, Springer, Berlin, 1982, pp. 34–71.
  * [Dol12] Igor V. Dolgachev, _Classical algebraic geometry_ , Cambridge University Press, Cambridge, 2012, A modern view.
  * [FA92] _Flips and abundance for algebraic threefolds_ , Société Mathématique de France, Paris, 1992, Papers from the Second Summer Seminar on Algebraic Geometry held at the University of Utah, Salt Lake City, Utah, August 1991, Astérisque No. 211 (1992).
  * [Fuj81] Takao Fujita, _On the structure of polarized manifolds with total deficiency one. II_ , J. Math. Soc. Japan 33 (1981), no. 3, 415–434.
  * [Fuk19] Takeru Fukuoka, _Refinement of the classification of weak Fano threefolds with sextic del Pezzo fibrations_ , arXiv:1903.06872, 2019.
  * [Ful98] William Fulton, _Intersection theory_ , second ed., Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics], vol. 2, Springer-Verlag, Berlin, 1998.
  * [Fur90] Mikio Furushima, _Complex analytic compactifications of ${\bf C}^{3}$_, vol. 76, 1990, Algebraic geometry (Berlin, 1988), pp. 163–196.
  * [Fur93] by same author, _The complete classification of compactifications of ${\bf C}^{3}$ which are projective manifolds with the second Betti number one_, Math. Ann. 297 (1993), no. 4, 627–662.
  * [Gro61] A. Grothendieck, _Éléments de géométrie algébrique. III. Étude cohomologique des faisceaux cohérents. I_ , Inst. Hautes Études Sci. Publ. Math. (1961), no. 11, 167.
  * [Gro65] by same author, _Éléments de géométrie algébrique. IV. Étude locale des schémas et des morphismes de schémas. II_ , Inst. Hautes Études Sci. Publ. Math. (1965), no. 24, 231.
  * [Gro67] by same author, _Éléments de géométrie algébrique. IV. Étude locale des schémas et des morphismes de schémas IV_ , Inst. Hautes Études Sci. Publ. Math. (1967), no. 32, 361.
  * [Gro68] Alexander Grothendieck, _Cohomologie locale des faisceaux cohérents et théorèmes de Lefschetz locaux et globaux $(SGA$ $2)$_, North-Holland Publishing Co., Amsterdam; Masson & Cie, Éditeur, Paris, 1968, Augmenté d’un exposé par Michèle Raynaud, Séminaire de Géométrie Algébrique du Bois-Marie, 1962, Advanced Studies in Pure Mathematics, Vol. 2.
  * [Har77] Robin Hartshorne, _Algebraic geometry_ , Springer-Verlag, New York-Heidelberg, 1977, Graduate Texts in Mathematics, No. 52.
  * [Hir54] Friedrich Hirzebruch, _Some problems on differentiable and complex manifolds_ , Ann. of Math. (2) 60 (1954), 213–236.
  * [IF00] A. R. Iano-Fletcher, _Working with weighted complete intersections_ , Explicit birational geometry of 3-folds, London Math. Soc. Lecture Note Ser., vol. 281, Cambridge Univ. Press, Cambridge, 2000, pp. 101–173.
  * [IM71] V. A. Iskovskih and Ju. I. Manin, _Three-dimensional quartics and counterexamples to the Lüroth problem_ , Mat. Sb. (N.S.) 86(128) (1971), 140–166.
  * [IP99] V. A. Iskovskikh and Yu. G. Prokhorov, _Fano varieties_ , Algebraic geometry, V, Encyclopaedia Math. Sci., vol. 47, Springer, Berlin, 1999, pp. 1–247.
  * [Ish18] Shihoko Ishii, _Introduction to singularities_ , Springer, Tokyo, 2018, Second edition.
  * [Isk80] V. A. Iskovskikh, _Anticanonical models of three-dimensional algebraic varieties_ , J. Sov. Math. 13 (1980), 745–814.
  * [Kas13] Alexander Kasprzyk, _Classifying terminal weighted projective space_ , arXiv:1304.3029, 2013.
  * [Kaw88] Yujiro Kawamata, _Crepant blowing-up of $3$-dimensional canonical singularities and its application to degenerations of surfaces_, Ann. of Math. (2) 127 (1988), no. 1, 93–163.
  * [KM92] János Kollár and Shigefumi Mori, _Classification of three-dimensional flips_ , J. Amer. Math. Soc. 5 (1992), no. 3, 533–703.
  * [KM98] János Kollár and Shigefumi Mori, _Birational geometry of algebraic varieties_ , Cambridge Tracts in Mathematics, vol. 134, Cambridge University Press, Cambridge, 1998, With the collaboration of C. H. Clemens and A. Corti, Translated from the 1998 Japanese original.
  * [Kol95] János Kollár, _Shafarevich maps and automorphic forms_ , M. B. Porter Lectures, Princeton University Press, Princeton, NJ, 1995.
  * [Kol96] by same author, _Rational curves on algebraic varieties_ , Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics], vol. 32, Springer-Verlag, Berlin, 1996.
  * [Kol97] by same author, _Singularities of pairs_ , Algebraic geometry—Santa Cruz 1995, Proc. Sympos. Pure Math., vol. 62, Amer. Math. Soc., Providence, RI, 1997, pp. 221–287.
  * [Kol13] by same author, _Singularities of the minimal model program_ , Cambridge Tracts in Mathematics, vol. 200, Cambridge University Press, Cambridge, 2013, With a collaboration of Sándor Kovács.
  * [KSC04] János Kollár, Karen E. Smith, and Alessio Corti, _Rational and nearly rational varieties_ , Cambridge Studies in Advanced Mathematics, vol. 92, Cambridge University Press, Cambridge, 2004.
  * [Man86] Yu. I. Manin, _Cubic forms_ , second ed., North-Holland Mathematical Library, vol. 4, North-Holland Publishing Co., Amsterdam, 1986, Algebra, geometry, arithmetic, Translated from the Russian by M. Hazewinkel.
  * [Mat89] Hideyuki Matsumura, _Commutative ring theory_ , second ed., Cambridge Studies in Advanced Mathematics, vol. 8, Cambridge University Press, Cambridge, 1989, Translated from the Japanese by M. Reid.
  * [Mat02] Kenji Matsuki, _Introduction to the Mori program_ , Universitext, Springer-Verlag, New York, 2002.
  * [Mor75] Shigefumi Mori, _On a generalization of complete intersections_ , J. Math. Kyoto Univ. 15 (1975), no. 3, 619–646.
  * [Oka19] Takuzo Okada, _Stable rationality of orbifold Fano 3-fold hypersurfaces_ , J. Algebraic Geom. 28 (2019), no. 1, 99–138.
  * [Pro91] Yu. G. Prokhorov, _Fano threefolds of genus $12$ and compactifications of ${\bf C}^{3}$_, Algebra i Analiz 3 (1991), no. 4, 162–170.
  * [Pro94] Yuri G. Prokhorov, _Compactifications of ${\bf C}^{4}$ of index $3$_, Algebraic geometry and its applications (Yaroslavl’, 1992), Aspects Math., E25, Friedr. Vieweg, Braunschweig, 1994, pp. 159–169.
  * [Pro16] Yu. G. Prokhorov, _Singular Fano manifolds of genus 12_ , Mat. Sb. 207 (2016), no. 7, 101–130.
  * [PS88] Thomas Peternell and Michael Schneider, _Compactifications of ${\bf C}^{3}$. I_, Math. Ann. 280 (1988), no. 1, 129–146.
  * [Puk98] Aleksandr V. Pukhlikov, _Birational automorphisms of Fano hypersurfaces_ , Invent. Math. 134 (1998), no. 2, 401–426.
  * [Puk13] Aleksandr Pukhlikov, _Birationally rigid varieties_ , Mathematical Surveys and Monographs, vol. 190, American Mathematical Society, Providence, RI, 2013\.
  * [PZ18] Yuri Prokhorov and Mikhail Zaidenberg, _Fano-Mukai fourfolds of genus 10 as compactifications of $\mathbb{C}^{4}$_, Eur. J. Math. 4 (2018), no. 3, 1197–1263.
  * [Rei80] Miles Reid, _Canonical $3$-folds_, Journées de Géometrie Algébrique d’Angers, Juillet 1979/Algebraic Geometry, Angers, 1979, Sijthoff & Noordhoff, Alphen aan den Rijn—Germantown, Md., 1980, pp. 273–310.
  * [SS85] Bernard Shiffman and Andrew John Sommese, _Vanishing theorems on complex manifolds_ , Progress in Mathematics, vol. 56, Birkhäuser Boston, Inc., Boston, MA, 1985.
  * [Tak00] Shigeharu Takayama, _Simple connectedness of weak Fano varieties_ , J. Algebraic Geom. 9 (2000), no. 2, 403–407.
  * [Zha06] Qi Zhang, _Rational connectedness of log ${\bf Q}$-Fano varieties_, J. Reine Angew. Math. 590 (2006), 131–142.


# Bag of Tricks for Node Classification with
Graph Neural Networks

Yangkun Wang1†, Jiarui Jin1†, Weinan Zhang1‡, Yong Yu1, Zheng Zhang2, David
Wipf2‡ 1Shanghai Jiao Tong University, 2Amazon<EMAIL_ADDRESS>jinjiarui97, wnzhang<EMAIL_ADDRESS>zhaz<EMAIL_ADDRESS>
(2021)

###### Abstract.

Over the past few years, graph neural networks (GNN) and label propagation-
based methods have made significant progress in addressing node classification
tasks on graphs. However, in addition to their reliance on elaborate
architectures and algorithms, there are several key technical details that are
frequently overlooked, and yet nonetheless can play a vital role in achieving
satisfactory performance. In this paper, we first summarize a series of
existing tricks-of-the-trade, and then propose several new ones related to
label usage,111For the label trick in particular, please see our more
detailed, follow-up analysis (Wang et al., 2020). loss function formulation,
and model design that can significantly improve various GNN architectures. We
empirically evaluate their impact on final node classification accuracy by
conducting ablation studies and demonstrate consistently-improved performance,
often to an extent that outweighs the gains from more dramatic changes in the
underlying GNN architecture. Notably, many of the top-ranked models on the
Open Graph Benchmark (OGB) leaderboard and KDDCUP 2021 Large-Scale Challenge
MAG240M-LSC benefit from these techniques we initiated.

Graph Neural Networks, Node Classification, OGB Leaderboard

††copyright: rightsretained††conference: The 6th International Workshop on
Deep Learning on Graphs: Methods and Applications; August 14 - 18, 2021;
Virtual Conference††journalyear: 2021††price: ††isbn: ††doi: 22footnotetext:
Work done during internship at AWS Shanghai AI Lab.33footnotetext:
Corresponding authors.

##  1\. Introduction

Recently, machine learning tasks involving graphs have received increasing
attention, among which node classification is one of the most prominent
examples. Since the remarkable success of graph convolution networks (GCN)
(Kipf and Welling, 2016), many high-performance GNN designs have been proposed
to address the node classification problem, such as graph attention networks
(GAT) (Veličković et al., 2017) and GraphSAGE (Hamilton et al., 2017). At the
same time, we have witnessed a steady improvement in model accuracy as
demonstrated on the Open Graph Benchmark (OGB) leaderboard (Hu et al., 2020).
For example, the top-$1$ test accuracy for node classification on the ogbn-
arxiv dataset has improved from $70.1\%$ (based on node2vec (Grover and
Leskovec, 2016)) to $74.1\%$ (based on GAT).

However, these advances are not derived exclusively from the development of
model architectures. Refinements including data processing, loss function
design and negative sampling also play a major role. Specifically, for semi-
supervised learning, it is worthwhile to explore the effective use of the
information contained in node features and/or labels. In this context, a
common approach is to train GNN models that make predictions based on node
features and model parameters; however, this strategy cannot directly utilize
existing label information (beyond their influence on model parameters through
training). In contrast, label propagation algorithms (LPA) (Zhu, 2005) spread
label information to make predictions, but cannot exploit node features.
Although many recent attempts (Klicpera et al., 2018; Huang et al., 2020)
propose to integrate node features and label information by combining GNN and
LPA, these approaches suffer from the inherent limitation that LPA requires
neighboring nodes to share similar labels and cannot be applied to graphs with
edge features.

In this paper, we propose a series of novel techniques covering both label
usage and architecture design. Specifically, we first develop a sampling
technique that enables GNNs to leverage random subsets of _original labels_ as
a model input. Based on this, we also design an iterative enhancement which
utilizes the _predicted labels_ from the previous iteration as input for
further training. Additionally, we propose a robust loss function and describe
different variants of GAT designs. We evaluate these modifications and tricks
on multiple GNN architectures and datasets, demonstrating that they often lead
to significant improvement in node classification accuracy.

Notably, as of Jul. 2, 2021, all of the top 10 models on the ogbn-arxiv
leaderboard, including AGDN (Sun and Wu, 2020), C&S (Huang et al., 2020), FLAG
(Kong et al., 2020) and UniMP (Shi et al., 2020), have applied these methods
or minor variations thereof. Moreover, on the more challenging ogbn-proteins
dataset, we can obtain an ROC-AUC of 0.8765, which at the time of our post to
the OGB leaderboard, outperformed all prior methods. And our label usage ideas
in particular, which we were the first to propose for improving node
classification,222Please see our original submission to the OGB leaderboard
for ogbn-arxiv on Sept. 5, 2020 at
https://ogb.stanford.edu/docs/leader_nodeprop/ and the corresponding original
code at this github repo. have been followed by UniMP (Shi et al., 2020) among
others, and have now been adopted in many submissions to the OGB leaderboard.
Overall, these techniques continue to be widely adopted, as an evidenced by
the KDDCUP 2021 Large-Scale Challenge MAG240M-LSC (Hu et al., 2021), e.g., the
released results333https://ogb.stanford.edu/kddcup2021/results/. indicate that
all of the top 3 approaches benefit from techniques we initiated.

##  2\. Background

Given a graph $G=(V,E)$, where $V=\\{v_{1},v_{2},\ldots,v_{N}\\}$ is the set
of nodes and $E$ is the set of edges, we denote $\bm{A}$ as the adjacency
matrix and $\bm{D}$ as the diagonal degree matrix. We assume that we have node
features $\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N})^{T}$ and one-hot encoded label
matrix $\bm{Y}=(\bm{y}_{1},\ldots,\bm{y}_{N})^{T}\in\mathbb{R}^{N\times C}$,
with $C$ being the number of classes. Each node is associated with a feature
vector $\bm{x}_{i}$ and label $\bm{y}_{i}$, assuming that only the first $M$
nodes $\bm{y}_{1},\bm{y}_{2},\ldots,\bm{y}_{M}$ can be observed during
training. For each dataset
$\mathcal{D}=\\{v_{i},\bm{x}_{i},\bm{y}_{i}\\}_{i=1}^{N}$ associated with a
graph $G$, we have the training set $\mathcal{D}_{train}$
($|\mathcal{D}_{train}|=M$) and the test set $\mathcal{D}_{test}$. The goal of
the node classification task is to predict the labels of unlabeled nodes.
Given the loss function $\ell(\hat{\bm{y}}_{i},\bm{y}_{i})$, the optimization
objective is to minimize the aggregated cost
$\mathcal{L}(\bm{\theta})=\sum_{i=1}^{M}\ell(\hat{\bm{y}}_{i},\bm{y}_{i})$,
where $\hat{\bm{y}}$ indicates the predicted label and $\bm{\theta}$ indicates
model parameters.

Label Propagation Algorithm. LPA is a semi-supervised algorithm that predicts
unlabeled nodes by propagating the observed labels across the edges of the
graph, with the underlying assumption that two nodes connected by an edge in
the graph are likely to have the same label. Letting
$\bm{S}=\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}$ be the symmetric
normalized adjacency, LPA solves a linear system
$\bm{Y}^{*}=(1-\lambda)(\bm{I}-\lambda\bm{S})^{-1}\bm{Y}$ by iteratively
computing $\bm{Y}^{(k+1)}=\lambda\bm{S}\bm{Y}^{(k)}+(1-\lambda)\bm{Y}^{(0)}$,
where $\bm{Y}^{(0)}$ is the label matrix of training nodes, padded with zeros
for test nodes. While effective in many circumstances, LPA does not make use
of node features as do the GNN models described next.

Graph Neural Networks. GNNs are a family of multi-layer feed-forward neural
networks that transform and propagate layer-wise features across graph edges.
Among these models, a GCN architecture is widely adopted, relying on the
layer-wise propagation rule

(1)
$\bm{X}^{(l+1)}=\sigma(\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}\bm{X}^{(l)}\bm{W}^{(l)}),$

where $\bm{W}^{(l)}$ denotes a trainable weight matrix of the $l$-th layer,
$\sigma(\cdot)$ is an activation function, and $\bm{X}^{(l)}$ represents the
$l$-th layer node representations. GAT models further leverage masked self-
attention layers to implicitly assign different weights to different
neighboring nodes. Assuming $(v_{i},v_{j})\in E$ is an edge, then the layer-
wise propagation rule of GAT is as follows:

(2)
$\begin{gathered}\alpha_{ij}^{(l)}=\frac{\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{W}^{(l)}\bm{x}_{i}^{(l)}\parallel\bm{W}^{(l)}\bm{x}_{j}^{(l)}]\right)\right)}{\sum_{r\in\mathcal{N}(v_{i})}\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{W}^{(l)}\bm{x}_{i}^{(l)}\parallel\bm{W}^{(l)}\bm{x}_{r}^{(l)}]\right)\right)},\\\
\bm{x}^{(l+1)}_{i}=\sigma\left(\sum_{v_{j}\in\mathcal{N}(v_{i})}\alpha^{(l)}_{ij}\bm{W}^{(k)}\bm{x}^{(l)}_{j}\right),\end{gathered}$

where $\bm{a}$ is a trainable weight vector, $\mathcal{N}(v_{i})$ denotes the
neighbors of node $v_{i}$, and $\parallel$ represents the concatenation
operation. Note that unlike LPA, when inferring the labels of test nodes, GNN
models do not make explicit use of the ground-truth labels of training nodes.

Combinations of Label and Feature Propagation. Since both LPA based on
spreading observed labels and GNN architectures that propagate node features
often achieve promising performance, it is worth exploring combinations
thereof to potentially overcome their respective limitations. However, one of
the major challenges is that simple combinations can lead to trivial
degenerate solutions when these labels are provided as input to trainable
models. Although many recent attempts have been made to circumvent this
problem, they still have various limitations. For example, APPNP (Klicpera et
al., 2018) does not actually propagate ground-truth training labels (only
predicted labels), while C&S (Huang et al., 2020) propagates ground-truth
labels but only during inference; it is not trained end-to-end. Instead, we
propose an approach in Section 4.1 that allows parallel propagation of node
features and labels during both training and inference stages. Based on this,
we further design a novel label reuse strategy on graphs, which propagates not
only the true labels of training nodes but also the predicted labels of test
nodes.

##  3\. Existing Tricks

Among many useful strategies, here we briefly discuss sampling, data
augmentation, renormalization, and residual connections, which can all be
applied in various settings to improve performance.

Sampling. Sampling techniques (Chen et al., 2018; Zou et al., 2019; Hamilton
et al., 2017) are often essential for the efficient training of GNNs. For
example, recent methods such as FastGCN (Chen et al., 2018) and LADIES (Zou et
al., 2019) investigate layer-wise and layer-dependent importance sampling.
Additionally, negative sampling methods (Mikolov et al., 2013), as first
proposed to serve as a simplified version of noise contrastive estimation, can
also play an important role, and are now widely adopted in web-scale graph
mining approaches such as PinSAGE (Ying et al., 2018).

Data Augmentation. For semi-supervised node classification tasks on graphs,
over-fitting and over-smoothing (Li et al., 2018) are two main obstacles in
training GNNs. In order to surmount these obstacles, the DropEdge method (Rong
et al., 2019) randomly removes a certain number of edges from the input graph,
acting like a data augmenter and a message-passing reducer. In addition to
modifying graph structures, another direction is inspired by the recent
success of adopting adversarial training in computer vision (Xie et al., 2020)
by adding gradient-based adversarial perturbations to the input features,
while keeping graph structures unchanged (Kong et al., 2020).

Renormalization. The renormalization trick was introduced in GCN models (Kipf
and Welling, 2016) to alleviate the numerical instabilities and gradient
explosion brought about by repeated application of Eq. (1) during training
with many layers. Specifically, we replace
$\bm{I}+\bm{D}^{-\frac{1}{2}}\bm{A}\bm{D}^{-\frac{1}{2}}$ with
$\tilde{\bm{D}}^{-\frac{1}{2}}\tilde{\bm{A}}\tilde{\bm{D}}^{-\frac{1}{2}}$,
where $\tilde{\bm{A}}=\bm{A}+\bm{I}$ and $\tilde{\bm{D}}=\bm{D}+\bm{I}$.

GCN with Residual Connections. A primitive form of GCN whose linear connection
with different parameters added to the message passing formulation was
introduced (Kipf and Welling, 2016). Subsequently, there has also been a body
of work using broader forms of residual connections (Rossi et al., 2020; Li et
al., 2020). One variant we find to be stable and robust adds a linear
connection with free parameters to GCN with the renormalization trick:

(3)
$\bm{X}^{(l+1)}=\sigma\left(\tilde{\bm{D}}^{-\frac{1}{2}}\tilde{\bm{A}}\tilde{\bm{D}}^{-\frac{1}{2}}\bm{X}^{(l)}\bm{W}_{0}^{(l)}+\bm{X}^{(l)}\bm{W}_{1}^{(l)}\right).$

This form can avoid the gradient instabilities with proper initialization of
$\bm{W}$, and moreover makes the GCN more expressive and overcomes the over-
smoothing issue, since the linear component in Eq. (3) retains the node
representations distinguishable even with infinitely many propagation layers.

##  4\. A New Bag of Tricks

###  4.1. Label Usage

Label as Input. For semi-supervised classification tasks, apart from the graph
$G$ and the feature matrix $\bm{X}$, we also have access to the label matrix
$\bm{Y}$, in which some nodes have missing labels and need to be predicted.
However, outside of LPA, prior work seldom considers the explicit use of
ground-truth label information during the inference of test node labels.
Instead, the label information is usually regarded only as the target for the
supervised training of GNN models. However, when the training accuracy is
below $100\%$, the label information of the misclassified samples is not
contained in the model, despite the fact that they can provide additional
information during inference. Additionally, samples misclassified by the model
have the potential to mislead their neighbors.

While some approaches are proposed to address this problem by combining GNN
with LPA, they have their own shortcomings as mentioned in Section 2.
Additionally, LPA relies heavily on the smoothness assumption that adjacent
nodes tend to share similar labels. In contrast, we propose a novel sampling
technique that allows parametric GNN models to learn interrelationships
between labels by taking label information as input. The advantages of our
method are as follows:

  * •

Capable of propagating features and labels during _both training and inference
stages_.

  * •

Does not explicitly rely on the smoothness assumption of LPA, and can be
conveniently adapted to various GNN architectures capable of handling
heterogeneous and heterophily graphs where this assumption may break down (Zhu
et al., 2020; Busbridge et al., 2019; Pei et al., 2019; Schlichtkrull et al.,
2018; Yang et al., 2021).

  * •

Can be trained end-to-end, while avoiding the model learning trivial
degenerate solutions, i.e., an identity mapping whereby the ground-truth
labels merely pass directly from input to output training nodes.

Our method starts with a random split of $\mathcal{D}_{train}$ into several
sub-datasets. For simplicity, we consider the case of two sub-datasets here,
denoted as $\mathcal{D}^{L}_{train}$ and $\mathcal{D}^{U}_{train}$,
respectively. Next, we set to zero the labels of $\mathcal{D}^{U}_{train}$,
and learn to predict their original values. Specifically, the input for
$\mathcal{D}^{L}_{train}$ contains both features and labels, while the input
for $\mathcal{D}^{U}_{train}$ contains only features, where the labels used as
inputs are set to zero-valued null vectors. During the final inference
procedure, all labels in the training set are used as inputs to the model. We
summarize this training procedure in Algorithm 1, where $f_{\bm{\theta}}$
denotes an arbitrary GNN model with parameters $\bm{\theta}$; for further
analysis of this label trick, please see (Wang et al., 2020).

0: $G$, $\bm{X}$, $\bm{Y}$, the recycling times $R$

1: for each epoch do

2: Obtain $\mathcal{D}^{L}_{train},\mathcal{D}^{U}_{train}$ by randomly
splitting $\mathcal{D}_{train}$

3:
$\bm{y}^{L}_{i}\leftarrow\begin{cases}\bm{y}_{i},&(v_{i},\bm{x}_{i},\bm{y}_{i})\in\mathcal{D}^{L}_{train}\\\
\mathbf{0},&\text{otherwise}\\\ \end{cases}$

4: $\hat{\bm{Y}}^{(0)}\leftarrow
f_{\bm{\theta}}(\bm{X}\parallel\bm{Y}^{L},\bm{A})$

5: for $k\leftarrow 1$ to $R$ do

6:
$\bm{y}_{i}^{(k-1)}\leftarrow\begin{cases}\bm{y}_{i},&(v_{i},\bm{x}_{i},\bm{y}_{i})\in\mathcal{D}^{L}_{train}\\\
\hat{\bm{y}}_{i}^{(k-1)},&\text{otherwise}\\\ \end{cases}$

7: $\hat{\bm{Y}}^{(k)}\leftarrow
f_{\bm{\theta}}(\bm{X}\parallel\bm{Y}^{(k-1)},\bm{A})$

8: end for

9: Compute $\mathcal{L}(\bm{\theta})$ and update $\bm{\theta}$ via back
propagation.

10: end for

Algorithm 1 Label Usage for Graph Neural Networks

Augmentation with Label Reuse. We further propose label reuse, which recycles
the predicted soft labels of the previous iteration and uses them labels as
input. In this case, the labels of $\mathcal{D}_{train}^{U}$ and all test
nodes are not assigned with zero-valued null vectors but the predicted results
of the previous iteration. In Algorithm 1, line 5 to line 8 presents the label
reuse procedure.

Loss | $\rho(z)$ |  $\rho(\phi_{logit}(v))$
---|---|---
Logistic | $z$ | $\log(1+\exp(-v))$
Exponential | $\exp(z)-1$ | $\exp(-v)$
Sigmoid | $1-\exp(-z)$ | $\dfrac{1}{1+\exp(v)}$
Savage | $(1-\exp(-z))^{2}$ | $\dfrac{1}{(1+\exp(v))^{2}}$
$\mathcal{L}_{q}$ | $\frac{1}{q}(1-\exp(-qz))$ | $\frac{1}{q}\left(1-\dfrac{1}{(1+\exp(-v))^{q}}\right)$
Loge | $\log(\epsilon+z)-\log\epsilon$ | $\log(\epsilon+\log(1+\exp(-v)))-\log\epsilon$

Table 1. Loss functions with different $\rho(\cdot)$.

---
Figure 1. Visualization of various margin-based losses $\phi$ (left) and their
corresponding derivatives $\phi^{\prime}$ (right).

###  4.2. Robust Loss Function for Classification

In binary classification scenarios, given feature space $\mathcal{X}$ and
label space $\mathcal{\bm{Y}}=\\{-1,+1\\}$, we aim to learn a classifier $g$
that maps $\bm{x}\in\mathcal{X}$ to $\mathcal{\bm{Y}}$. The classifier follows
the decision rule $g(\bm{x})=\text{sign}(f(\bm{x}))$ for some mapping $f$ from
$\mathcal{X}$ to $\mathbb{R}$. The optimization objective is to minimize the
risk, defined as

(4)
$R_{\phi}(f):=\mathbb{E}_{\mathcal{D}}[\ell(f(\bm{x}),y)]=\mathbb{E}_{\mathcal{D}}[\phi(yf(\bm{x}))],$

where $\ell(f(\bm{x}),y)$ is the loss function and
$\phi:\mathbb{R}\rightarrow\mathbb{R}^{+}$ is known as the margin-based loss
function. In this setting, choosing the loss function corresponds to choosing
$\phi(\cdot)$.

A straightforward choice for $\phi(\cdot)$ is the 0-1 loss

(5) $\ell_{0/1}(f(\bm{x}),y)=\phi_{0/1}(yf(\bm{x}))\coloneqq H(-yf(\bm{x})),$

where $H(\cdot)$ denotes the Heaviside step function. However,
$\phi_{0/1}(\cdot)$ is a discontinuous function and is therefore
computationally challenging to optimize. As a result, instead of directly
optimizing the 0-1 loss, we turn to using $\phi(\cdot)$ as an upper bound of
$\phi_{0/1}(\cdot)$, often referred to as the calibrated surrogate loss, for
the optimization objective.

Being the most commonly used loss function for classification, the logistic
loss, denoted as $\phi_{logit}(\cdot)$, provides a convex upper bound for
$\phi_{0/1}(\cdot)$, which takes the form

(6) $\phi_{logit}(v)=\log(1+\exp(-v)).$

While the logistic loss performs satisfactorily in most cases, it suffers from
sensitivity to outliers, whereas non-convex loss functions could be more
robust (Masnadi-shirazi and Vasconcelos, 2009). Motivated by this, we consider
weakening the convexity condition, and thereby designing a quasi-convex loss
to contribute robustness:

(7) $\phi_{\rho-logit}(v):=\rho(\phi_{logit}(v)),$

where $\rho:\mathbb{R}^{+}\rightarrow\mathbb{R}^{+}$ is a non-decreasing
function.

Some loss functions have been proposed to achieve outlier robustness, e.g.,
the Savage loss (Masnadi-shirazi and Vasconcelos, 2009) and $\mathcal{L}_{q}$
loss (Zhang and Sabuncu, 2018), which can be interpreted as choosing a
suitable $\rho(\cdot)$. Table 1 summarizes different $\rho(\cdot)$ of these
and other loss functions discussed.

Here, we propose the Loge loss as a more preferable possibility for
$\rho(\cdot)$:

(8) $\rho_{{loge}}(z):=\log(\epsilon+z)-\log\epsilon,$

where $\epsilon$ is a tunable parameter and is fixed to $1-\log 2$ throughout
this paper so that $\left.\frac{d^{2}}{dv^{2}}\phi_{{loge}}(v)\right|_{v=0}=0$
with $\phi_{{loge}}(v):=\rho_{{loge}}(\phi_{logit}(v))$. This implies that the
derivative of the Loge loss reaches its maximum magnitude at $z=0$, which is
exactly the decision boundary for classification tasks. Since the derivative
of $\phi(\cdot)$ is the weight of the data sample (Leistner et al., 2009),
this may facilitate the optimization of classification accuracy.

Figure 1 illustrates the visualization of Loge loss and other losses along
with their derivatives. As can be seen, our Loge loss meets the following
criteria:

  * •

While it helps to prevent outliers from dominating the training loss, it is
nonetheless unbounded in such a way that the derivative converges slowly to
$0$ as $v$ decreases, meaning that it still provides non-negligible gradient
signals for the misclassified samples as desired.

  * •

The maximum gradient magnitude occurs at $v=0$, which enhances the gradient
signal near the decision boundary. The only other loss function with an
analogously maximal gradient at $0$ is the Sigmoid loss; however, its tail
converges to $0$ very fast, which can lead to a vanishing gradient problem.

  * •

For correctly classified samples (i.e., $v>0$), the derivative converges to
$0$ relatively quickly as with other loss functions; however, in this regime
the gradient signal is less critical.

The Loge loss can also be extended for multi-class classification tasks. For
this purpose, we formulate the labels in a one-hot fashion, where both
$\bm{y}$ and $\hat{\bm{y}}$ are one-hot vectors, $\hat{y}_{i}$ denotes the
value of the $i$-th element in $\hat{\bm{y}}$, and the predicted value of the
target class is denoted by $\hat{y}_{class}$, meaning that the subscript class
refers to the index of the nonzero element of $\bm{y}$. The Loge loss can then
be formulated as

(9)
$\ell_{{loge}}(\hat{\bm{y}},\bm{y})=\log\left(\epsilon-\log\frac{\exp({\hat{y}}_{class})}{\sum_{i=1}^{C}\exp({\hat{y}_{i}})}\right)-\log\epsilon.$

###  4.3. Tweaking the GAT Architecture

GAT with Symmetric Normalized Adjacency Matrix. We find the symmetric
normalized adjacency matrix in GCN improves the performance at times, and yet
GAT is not a natural extension of GCN. In order to better connect GAT with
GCN, we first define the unnormalized attention matrix
$\bm{A}_{att}=\bm{D}\bm{\alpha}$, with $\bm{\alpha}$ described in Eq. (2).
Then the message passing rule with self-loops becomes

(10)
$\bm{X}^{(l+1)}=\sigma\left(\tilde{\bm{D}}^{-\frac{1}{2}}\tilde{\bm{A}}_{att}\tilde{\bm{D}}^{-\frac{1}{2}}\bm{X}^{(l)}\bm{W}_{0}^{(l)}+\bm{X}^{(l)}\bm{W}_{1}^{(l)}\right),$

where $\tilde{\bm{A}}_{att}=\bm{I}+\bm{A}_{att}$. Note that when
$\bm{A}_{att}=\bm{A}$, this variant is equivalent to Eq. (3) of GCN.

Other GAT Variants. The attention mechanism of the original GAT is described
in Eq. (2). By replacing $\bm{a}^{T}\bm{W}$ with $\bm{a}^{T}$, the computation
of attention value in Eq. (2) can be simplified to

(11)
$\alpha_{ij}=\frac{\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{x}_{i}\parallel\bm{x}_{j}]\right)\right)}{\sum_{r\in\mathcal{N}(v_{i})}\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{x}_{i}\parallel\bm{x}_{r}]\right)\right)},$

where for simplicity we henceforth omit the layer-wise superscripts. Another
variant is the non-interactive GAT, which performs similarly to and at times
better than the original form, and can be expressed as

(12)
$\alpha_{ij}=\frac{\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}\bm{x}_{j}\right)\right)}{\sum_{r\in\mathcal{N}(v_{i})}\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}\bm{x}_{r}\right)\right)}.$

We also propose a GAT variant that exploits the edge features in the graph:

(13)
$\alpha_{ij}=\frac{\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{x}_{i}^{V}\parallel\bm{x}_{j}^{V}\parallel\bm{x}_{ij}^{E}]\right)\right)}{\sum_{r\in\mathcal{N}(v_{i})}\exp\left(\mathrm{LeakyReLU}\left(\bm{a}^{T}[\bm{x}_{i}^{V}\parallel\bm{x}_{r}^{V}\parallel\bm{x}_{ij}^{E}]\right)\right)},$

where $\bm{x}^{V}$ and $\bm{x}^{E}$ denote node and edge features
respectively. The time complexity of computing one layer of a single-headed
GAT with $C^{V}$ node features, $C^{E}$ edge features and $F$ filters is
$O(|V|C^{V}F+|E|C^{E}+|E|F)$.

##  5\. Experiments

In this section, we examine the performance of each method through ablation
experiments, reporting the mean classification accuracy for multi-class
classification tasks and Area Under the ROC Curve (ROC-AUC) for binary
classification tasks. We choose three commonly used citation network datasets,
Cora, Citeseer and Pubmed (Sen et al., 2008), and three relatively large
datasets from OGB (Hu et al., 2020), ogbn-arxiv, ogbn-proteins and ogbn-
products, as well as Reddit, a dataset of posts from the Reddit
website.444https://snap.stanford.edu/graphsage/. Some statistics of these
datasets are presented in Table 2. Since ogbn-proteins is a dataset with edge
features and ogbn-products is a huge dataset, we adopt neighbor sampling for
them due to memory constraints. For Cora, Pubmed and Citeseer, we report the
average scores and standard deviations after 100 runs, and for the relatively
larger datasets ogbn-arxiv, ogbn-proteins, ogbn-products and Reddit, we report
mean scores and standard deviations after 10 runs. All experiments were
implemented using the Deep Graph Library (DGL) (Wang et al.,
2019).555Reproducible code based on DGL with instructions is available at
https://github.com/espylapiza/Bag-of-Tricks-for-Node-Classification-with-
Graph-Neural-Networks.

Table 2. Datasets statistics, where label rate denotes the proportion of
labeled nodes used for training to the total nodes.

Dataset | #Nodes | #Edges | Metric | Label rate
---|---|---|---|---
Cora | 2,708 | 5,429 | Accuracy | 5.2%
Citeseer | 3,327 | 4,732 | Accuracy | 3.6%
Pubmed | 1,9717 | 44,338 | Accuracy | 0.03%
Reddit | 232,965 | 114,615,892 | Accuracy | 65.9%
Arxiv | 169,343 | 1,166,243 | Accuracy | 53.7%
Proteins | 132,534 | 39,561,252 | ROC-AUC | 65.4%
Products | 2,449,029 | 61,859,140 | Accuracy | 8.0%

Table 3. Accuracy results (as measured by classification accuracy and ROC-AUC
for ogbn-arxiv and ogbn-proteins, respectively) of different datasets and
models in terms of label usage and GAT variant. GCN+linear indicates the GCN
variant with a residual connection of Eq. (3). GAT* indicates the GAT variant
that incorporates the edge features.

Dataset | Model | Label Usage |  Accuracy(%)
---|---|---|---
Arxiv | GCN | – | 72.48 ± 0.11
Arxiv | GCN | label as input | 72.64 ± 0.10
Arxiv | GCN | label reuse | 72.78 ± 0.17
Arxiv | GCN+linear | – | 72.74 ± 0.13
Arxiv | GCN+linear | label as input | 73.13 ± 0.14
Arxiv | GCN+linear | label reuse | 73.22 ± 0.13
Arxiv | GAT | – | 73.20 ± 0.16
Arxiv | GAT | label as input | 73.24 ± 0.10
Arxiv | GAT | label reuse | 73.43 ± 0.13
Arxiv | GAT(norm.adj.) | – | 73.59 ± 0.14
Arxiv | GAT(norm.adj.) | label as input | 73.66 ± 0.11
Arxiv | GAT(norm.adj.) | label reuse | 73.91 ± 0.12
Arxiv | GAT(norm.adj.) | label reuse+C&S | 73.95 ± 0.12
Arxiv | AGDN | – | 73.75 ± 0.21
Arxiv | AGDN | label as input | 73.98 ± 0.09
Proteins | GCN | – | 80.07 ± 0.95
Proteins | GCN | label as input | 80.80 ± 0.56
Proteins | GAT* | – | 87.47 ± 0.16
Proteins | GAT* | label as input | 87.65 ± 0.08

Label Usage. There are two principal factors that determine the benefits of
using labels as inputs during training. One is the proportion of graph nodes
with labels available for training, and the other is the training accuracy. We
investigate the performance of label as input and label reuse on datasets with
a relatively large proportion of training set and low training accuracy. The
results are reported in Table 3. Here our approach improves the performance
consistently with only a small increase in parameters. Furthermore, we can
further improve the performance by combining our method with C&S (Huang et
al., 2020).

Loss Functions. We evaluate the performance of our loss function on datasets
with classification accuracy as the metric. Results are reported in Table 4,
where each model is trained with the same hyperparameters, varying only the
loss functions. As shown, while the robust Savage loss performs well on some
small datasets, it performs considerably worse on larger datasets. Meanwhile,
the Loge loss outperforms other losses on most datasets.

Table 4. Comparative results of loss functions on different datasets and
models, where $\epsilon$ of the Loge loss is $1-\log 2$.

Dataset | Model |  Accuracy(%)
---|---|---
|  | Logistic | Savage | Loge
Cora | MLP | 59.72 ± 1.01 | 61.10 ± 0.91 | 60.39 ± 0.74
Cora | GCN | 82.26 ± 0.84 | 81.65 ± 0.74 | 82.60 ± 0.83
Citeseer | MLP | 57.75 ± 1.05 | 59.60 ± 0.92 | 59.07 ± 0.98
Citeseer | GCN | 71.13 ± 1.12 | 71.10 ± 1.22 | 72.49 ± 1.12
Pubmed | MLP | 73.15 ± 0.68 | 73.39 ± 0.62 | 72.93 ± 0.65
Pubmed | GCN | 78.89 ± 0.71 | 78.91 ± 0.63 | 78.93 ± 0.69
Reddit | MLP | 72.98 ± 0.09 | 68.64 ± 0.29 | 73.12 ± 0.09
Reddit | GCN | 95.22 ± 0.04 | 92.29 ± 0.48 | 95.18 ± 0.03
Arxiv | MLP | 56.18 ± 0.14 | 51.97 ± 0.20 | 56.72 ± 0.15
Arxiv | GCN | 71.77 ± 0.34 | 68.47 ± 0.32 | 72.43 ± 0.16
Arxiv | GAT | 73.08 ± 0.26 | 69.58 ± 1.00 | 73.20 ± 0.16
Arxiv | GAT(norm.adj.) | 73.29 ± 0.17 | 69.22 ± 1.48 | 73.59 ± 0.14
Products | MLP | 62.90 ± 0.16 | 58.13 ± 1.03 | 63.20 ± 0.13
Products | GAT | 80.99 ± 0.16 | 77.48 ± 0.14 | 81.39 ± 0.14

Table 5. Results of GAT variant. GAT+norm.adj. corresponds to GAT with
symmetric normalized adjacency.

Dataset |  Accuracy(%)
---|---
| vanilla GAT | GAT+norm.adj.
Cora | 83.41 ± 0.74 | 83.72 ± 0.74
Citeseer | 71.92 ± 0.92 | 72.25 ± 1.04
Pubmed | 78.43 ± 0.64 | 78.77 ± 0.54
Reddit | 96.97 ± 0.04 | 97.06 ± 0.05
Arxiv | 73.20 ± 0.16 | 73.59 ± 0.14

GAT Variants. To explore the effect of the symmetric normalized adjacency
matrix on GAT, we compare its performance with the original GAT on 5 datasets.
The results are reported in Table 5. We see that GAT with a normalized
adjacency matrix achieves higher performance on all datasets. Nevertheless, we
recommend choosing the appropriate adjacency matrix for different datasets. In
Table 3, our GAT variant that incorporates the edge features outperforms all
prior methods applied to the ogbn-proteins dataset by a significant margin at
the time of our post to the OGB leaderboard.

##  6\. Conclusion

In this paper, we present a new framework for combining feature and label
propagation, propose a robust loss function, and investigate several tricks
for training deep GNNs with promising performance. These techniques can be
applied to various GNN models, which generally only require minor
modifications to the data processing, loss function, or architecture.

##  7\. Acknowledgements

We thank the support of National Natural Science Foundation of China (Grant
No. 61702327, 61772333, 61632017) and Wu Wen Jun Honorary Doctoral
Scholarship, AI Institute, Shanghai Jiao Tong University.

## References

  * (1)
  * Busbridge et al. (2019) Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. 2019.  Relational graph attention networks. In _arXiv:1904.05811_.
  * Chen et al. (2018) Jie Chen, Tengfei Ma, and Cao Xiao. 2018.  Fastgcn: fast learning with graph convolutional networks via importance sampling. In _arXiv:1801.10247_.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks. In _KDD_.
  * Hamilton et al. (2017) William L Hamilton, Rex Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs. In _arXiv:1706.02216_.
  * Hu et al. (2021) Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. 2021\.  Ogb-lsc: A large-scale challenge for machine learning on graphs. In _arXiv:2103.09430_.
  * Hu et al. (2020) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020\.  Open graph benchmark: Datasets for machine learning on graphs. In _arXiv:2005.00687_.
  * Huang et al. (2020) Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. 2020.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks. In _arXiv:2010.13993_.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks. In _arXiv:1609.02907_.
  * Klicpera et al. (2018) Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. 2018.  Predict then propagate: Graph neural networks meet personalized pagerank. In _arXiv:1810.05997_.
  * Kong et al. (2020) Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. 2020\.  Flag: Adversarial data augmentation for graph neural networks. In _arXiv:2010.09891_.
  * Leistner et al. (2009) Christian Leistner, Amir Saffari, Peter M Roth, and Horst Bischof. 2009.  On robustness of on-line boosting-a competitive study. In _ICCV Workshops_.
  * Li et al. (2020) Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. 2020\.  Deepergcn: All you need to train deeper gcns. In _arXiv:2006.07739_.
  * Li et al. (2018) Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018.  Deeper insights into graph convolutional networks for semi-supervised learning. In _AAAI_.
  * Masnadi-shirazi and Vasconcelos (2009) Hamed Masnadi-shirazi and Nuno Vasconcelos. 2009.  On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost. In _NeurIPS_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality. In _NIPS_.
  * Pei et al. (2019) Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2019.  Geom-GCN: Geometric Graph Convolutional Networks. In _ICML_.
  * Rong et al. (2019) Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2019\.  Dropedge: Towards deep graph convolutional networks on node classification. In _arXiv:1907.10903_.
  * Rossi et al. (2020) Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. 2020\.  Sign: Scalable inception graph neural networks. In _arXiv:2004.11198_.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _ESWC_.
  * Sen et al. (2008) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008\.  Collective classification in network data. In _AI magazine_.
  * Shi et al. (2020) Yunsheng Shi, Zhengjie Huang, Wenjin Wang, Hui Zhong, Shikun Feng, and Yu Sun. 2020\.  Masked label prediction: Unified message passing model for semi-supervised classification. In _arXiv:2009.03509_.
  * Sun and Wu (2020) Chuxiong Sun and Guoshi Wu. 2020.  Adaptive Graph Diffusion Networks with Hop-wise Attention. In _arXiv:2012.15024_.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks. In _arXiv:1710.10903_.
  * Wang et al. (2019) Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. 2019\.  Deep graph library: A graph-centric, highly-performant package for graph neural networks. In _arXiv:1909.01315_.
  * Wang et al. (2020) Yangkun Wang, Jiarui Jin, Weinan Zhang, Yongyi Yang, Jiuhai Chen, Quan Gan, Yong Yu, Zheng Zhang, Zengfeng Huang, and David Wipf. 2020\.  Why Propagate Alone? Parallel Use of Labels and Features on Graphs. In _arXiv:2006.11468_.
  * Xie et al. (2020) Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. 2020\.  Adversarial examples improve image recognition. In _CVPR_.
  * Yang et al. (2021) Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. 2021.  Graph Neural Networks Inspired by Classical Iterative Algorithms. In _arXiv:2103.06064_.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018\.  Graph convolutional neural networks for web-scale recommender systems. In _KDD_.
  * Zhang and Sabuncu (2018) Zhilu Zhang and Mert R Sabuncu. 2018.  Generalized cross entropy loss for training deep neural networks with noisy labels. In _arXiv:1805.07836_.
  * Zhu et al. (2020) Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020\.  Beyond homophily in graph neural networks: Current limitations and effective designs. In _arXiv:2006.11468_.
  * Zhu (2005) Xiaojin Jerry Zhu. 2005\.  Semi-supervised learning literature survey.
  * Zou et al. (2019) Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. 2019\.  Layer-dependent importance sampling for training deep and large graph convolutional networks. In _arXiv:1911.07323_.