# Learn Layer-wise Connections in Graph Neural Networks

Lanning Wei1,3, Huan Zhao1, Zhiqiang He2,3 14Paradigm Inc., China, 2Lenovo,
China3Institute of Computing Technology, Chinese Academy of Sciences
weilanning18z@ict.ac.cn;zhaohuan<EMAIL_ADDRESS>
###### Abstract.

In recent years, Graph Neural Networks (GNNs) have shown superior performance
on diverse applications on real-world datasets. To improve the model capacity
and alleviate the over-smoothing problem, several methods proposed to
incorporate the intermediate layers by layer-wise connections. However, due to
the highly diverse graph types, the performance of existing methods vary on
diverse graphs, leading to a need for data-specific layer-wise connection
methods. To address this problem, we propose a novel framework LLC (Learn
Layer-wise Connections) based on neural architecture search (NAS) to learn
adaptive connections among intermediate layers in GNNs. LLC contains one novel
search space which consists of 3 types of blocks and learnable connections,
and one differentiable search algorithm to enable the efficient search
process. Extensive experiments on five real-world datasets are conducted, and
the results show that the searched layer-wise connections can not only improve
the performance but also alleviate the over-smoothing problem.111Lanning is a
research intern in 4Paradigm. This paper has been accepted by
DLG-KDD’21.

Graph Neural Networks, Neural Architecture Search, Over-smoothing

##  1\. Introduction

In recent years, Graph Neural Networks(GNNs) have been widly used due to its
promising performance on various graph-based datasets, e.g., chemistry (Gilmer
et al., 2017), bioinformatics (Ying et al., 2018), text categorization
(Rousseau et al., 2015), and recommendation (Xiao et al., 2019). Most GNNs
follow a neighborhood aggregation schema, also called message passing (Gilmer
et al., 2017), which learns the embeddings of a node by aggregating the
embeddings of its neighborhoods. Representative GNN models are GCN (Kipf and
Welling, 2016), GraphSAGE (Hamilton et al., 2017), GAT (Veličković et al.,
2018) and GIN (Xu et al., 2019). In practice, multiple GNN layers tend to be
stacked to improve the model capacity and thus the final performance. However,
it leads to a chain-like architecture by stacking GNN layers, which only uses
the node representations of the previous layer and is proved to provide
limited improvements (Li et al., 2019). Besides, the node representations
become indistinguishable along with the network goes deeper, i.e., increasing
the stacked layers, which is called the over-smoothing problem (Li et al.,
2018). The problem is that it fails to make full use of the information
contained in the intermediate layers, which are important for improving the
model capacity and alleviating the over-smoothing problem (Xu et al., 2018;
Fey, 2019; Li et al., 2019).

Recently, to address this problem, several methods are proposed to use the
intermediate layers by constructing the layer-wise connections. For example,
JKNet (Xu et al., 2018) and SIGN (Rossi et al., 2020) connect all the
intermediate layers at the end of GNNs, ResGCN (Li et al., 2019) summarizes
the previous two layers as ResNet (He et al., 2016), and DenseGCN (Li et al.,
2019) concatenates the features of all the previous layers as DenseNet (Huang
et al., 2017). Despite the success of these pre-defined layer-wise
connections, in reality, graphs are from highly diverse domains, e.g.,
chemistry (Gilmer et al., 2017), bioinformatics (Ying et al., 2018), text
categorization (Rousseau et al., 2015), and social networks (Xu et al., 2019),
none of these methods can adapt to various tasks. To further quantify this
problem, we design an experiment based on GraphGym (You et al., 2020), a GNN
benchmark to evaluate the different design dimensions of GNNs, to evaluate
existing layer-wise connection methods. As shown in Fig. 1, different
connections are selected and fused in each layer. 222The experiment details
can be found in Section 4.1. We sample 180 settings on the node classification
task, and we can see that no single architecture can win in all cases on 14
diverse datasets (The rank 1 distribution is nearly uniform). Therefore, it
leads to a straightforward need to obtain data-specific layer-wise connections
in GNN architecture design.

Figure 1. Top 3 rank distributions. Left: The rank distributions of different
connection selection methods in GNNs. Right: the rank distributions of
different connect functions used in merging these layers.

For this problem, two questions need to be considered: how to select the
connections and how to fuse these selected previous layers in each
intermediate layer. To address these problems, we turn to neural architecture
search (NAS) in learning data-specific and SOTA architectures in CNNs(Zoph and
Le, 2017; Liu et al., 2019; Xie et al., 2018) and GNNs (Gao et al., 2020; Zhou
et al., 2019; Zhao et al., 2020; Zhao et al., 2021; Li and King, 2020; Li et
al., 2021; Cai et al., 2021). For example, AutoGraph (Li and King, 2020)
learns to select the connections in each intermediate layer, SNAG (Zhao et
al., 2020) and SANE (Zhao et al., 2021) learn to select and fuse connections
at the end of GNNs. However, most of these methods focus on searching for
different aggregation functions, while ignoring the layer-wise connections due
to the incomplete search space of GNNs.

Thus, in this work we propose one framework LLC (Learn Layer-wise Connections)
to learn the connections among layers adaptively. Firstly, we design one
search space to represent this problem with one Directed Acyclic Graph (DAG),
it contains 1 pre-process block, 1 post-process block and several GNN blocks.
Apart from the input block, each block can select connections from all the
previous blocks and choose one fusion function to merge these inputs. Then, on
top of the designed search space, we develop a differentiable search algorithm
to make the learning efficient and effective. The goal of LLC is to improve
the model capacity and alleviate the over-smoothing problem by learning layer-
wise connections. In this way, our method can be integrated with any existing
GNN models. And experiments on various datasets demonstrate the effectiveness
of the searched layer-wise connection methods by LLC.

To summarize, the contributions of this work are as follows:

  * •

In this paper, we proposed one framework LLC in learning layer-wise
connections to address the model capacity and the over-smoothing problems.

  * •

By modeling this problem as a graph neural architecture search problem, we
design a novel search space that can cover existing human-designed layer-wise
connection methods. On top of the search space, we develop an efficient search
algorithm to obtain data-specific layer-wise connection methods.

  * •

Extensive experiments show that the searched layer-wise connections can not
only improve the performance but also alleviate the over-smoothing problem.

Notations We represent a graph as $G=(\textbf{A},\textbf{H})$ ,where
$\textbf{A}\in\mathbb{R}^{N\times N}$ is the adjacency matrix of this graph
and $\textbf{H}\in\mathbb{R}^{N\times d}$ is the node features. $N$ is the
node number. $\widetilde{\mathcal{N}}(v)=\\{v\\}\cup\\{u|\textbf{A}_{uv}\neq
0\\}$ represents set of the self-contained first-order neigbors of node $v$,
and $\textbf{h}_{v}^{l}$ denotes the features of node $v$ in $l$-th layer.

##  2\. Related Work

###  2.1. Layer-wise Connections in GNNs

General GNNs are constructed by stacking aggregators and each layer is
connected with the previous layer merely. It leads to the over-smoothing and
performance drop problem on deep networks. To alleviate the over-smoothing
problem and improve the model capacity, layer-wise connections are used in
GNNs. DenseGCN (Li et al., 2019) concatenates all the previous layers, and
ResGCN (Li et al., 2019) summarizes the previous two layers. These 2 methods
utilize connections in each layer, and more methods focus on fuse all the
intermediate layers at the end of GNNs directly. JKNet (Xu et al., 2018)
integrates all the intermediate layers with concatenation, LSTM and maximum;
SIGN (Rossi et al., 2020) uses different adjacency matrix $\textbf{A}^{n}$ in
each layer and fuses these layers with concatenation.These pre-defined
connections cannot handle the diverse graph data as shown in Figure 1.
However, our method LLC can learn layer-wise connections adaptively and
improve the model capacity as well as alleviate the over-smoothing problem.

###  2.2. Graph Neural Architecture Search

NAS methods were proposed to automatically find SOTA CNN architectures in a
pre-defined search space and representative methods are (Zoph and Le, 2017;
Real et al., 2017; Pham et al., 2018; Liu et al., 2019; Xie et al., 2018; Real
et al., 2019). Very recently, researchers tried to automatically design GNN
architectures by NAS, and there are several pioneering works. Based on
Reinforcement Learning (RL), which sample architectures with RNN controller
and updated with policy gradient (Zoph and Le, 2017; Pham et al., 2018),
GraphNAS (Gao et al., 2020) and Auto-GNN (Zhou et al., 2019) learn to design
aggregation operations with attention function, attention head number,
embedding size, etc.; SNAG (Zhao et al., 2020) provides extra connections
selection and layer aggregations learning in the output node. Based on
Evolutionary Algorithm (EA), which select parent architecture from the search
space and generate new architectures with mutation (Real et al., 2017) and
crossover (Real et al., 2019), AutoGraph (Li and King, 2020) learns to select
connections in each intermediate layers. These RL and EA based methods need
thousands of architecture evaluations which are time-consuming and
computationally expensive. Differentiable methods (Liu et al., 2019; Xie et
al., 2018) construct an over-parameterized network (supernet) and optimize
this supernet with gradient descent due to the continuous relaxation of the
search space. DSS (Li et al., 2021) and GNAS (Cai et al., 2021) learn to
design aggregators in each layer. SANE (Zhao et al., 2021), the first method
to apply differentiable NAS in design GNNs, provide the aggregator learning
and extra skip connections and layer aggregations learning for the output
node.

More graph neural architecture search methods can be found in (Zhang et al.,
2021). Compared with these methods focusing on searching for different
aggregation functions, our method LLC try to search for data-specific layer-
wise connections.

##  3\. Methods

###  3.1. Overview

In this section, we elaborate on the proposed framework LLC, which can solve
the layer-wise connection learning problem in GNNs. It contains the designed
search space and the differentiable search algorithm.

We design one novel search space with the DAG in Fig. 2(a) to learn
connections, and it contains one input block, one output block and several GNN
blocks. Based on the learnable layer-wise connections which denoted as dashed
edges, the blocks select and fuse connections with as shown in Figure 2(b).
After learning finished, we can construct the GNN architectures by connecting
blocks with the learned connections. To learn more flexible GNNs rather than
chain-structure GNNs stacking message passing layers, the connection from the
previous block is learnable either. With this search space, more flexible
architectures can be obtained with LLC.

Figure 2. The designed search space. (a) We use a 4-layer architecture
backbone as an example, one input block 0, one output block 5 and 4 GNN
blocks. The connections among these blocks are learnable. (b) In $i$-th GNN
block, the previous layers are selected and then merged with a fusion
operation $f$, then the results are used by the following GNN layer.

###  3.2. Search Space

We use a DAG to represent the search space as shown in Fig. 2(a), the dashed
edges represent the learnable connections, there exist 2 candidate operations
IDENTITY and ZERO in each dashed edge which corresponding to the selection and
not selection stage. With the input graph $G=(\textbf{A}^{I},\textbf{H}^{I})$,
block 0 processes the input features $\textbf{H}^{I}$ with a 2-layer MLPs
(Multilayer Perceptrons), and the output of block 0 can be represented as
$\textbf{H}^{0}=\sigma(\textbf{W}_{1}\sigma(\textbf{W}_{0}\textbf{H}^{I}))$.
GNN blocks (block $1,2,3,4$ in Figure 2(a)) select and fuse these connections
firstly, then update node embeddings with GNN layers as shown in Figure2 (b).
Block 5 is the post-process block, it selects and fuses connections as the GNN
block, and then generates the output results with a 2-layer MLPs.

In this paper, based on the aggregators which have been explored extensively,
we learn to select and fuse connections with the selection and fusion
operations as shown in Table 1. These fusion operations produce one new
feature based on these connected layers with the summation, average, maximum,
concatenation, LSTM cell and attention mechanism, which demonstrated as SUM,
MEAN, MAX, CONCAT, LSTM and ATT, respectively.

Discussions. With the design of this search space, it is capable to obtain the
multi-branches architectures as (Rossi et al., 2020; Abu-El-Haija et al.,
2019), which can expand the chain-structure in general GNNs. Therefore, LLC is
more flexible than (Zhao et al., 2020; Zhao et al., 2021; Li and King, 2020)
which search connections based on the chain-structure GNNs.

Table 1. The selection set $\mathcal{O}_{e}$ and the fusion set $\mathcal{O}_{f}$ we used in LLC. | Operations
---|---
Selection set $\mathcal{O}_{e}$ |  ZERO, IDENTITY
Fusion set $\mathcal{O}_{f}$ |  SUM, MEAN, MAX, CONCAT, LSTM, ATT

###  3.3. Differentiable Search Algorithm

Continuous relaxation. The search algorithm is used to search architectures
from the search space shown in Figure 2. To enable the usage of gradient
descent to accelerate the search process, continuous relaxation is used to
make the search space continuous, thus the discrete selection of operations is
relaxed by a weighted summation of all possible operations as

(1)
$\displaystyle\bar{o}(x)=\sum\nolimits_{k=1}^{\left|\mathcal{O}\right|}c_{k}o_{k}(x),$

$c_{k}\in(0,1)$ is the weight of $k$-th operation $o_{k}(\cdot)$, it is
generated by one relaxation function $c_{k}=g(\bm{\alpha})$ and $\alpha_{k}$
is the corresponding learnable supernet parameter for $c_{k}$.

For block $j$, the connection selection result from block $i$ can be
represented as the weighted summation of 2 selection operations as shown in

(2)
$\displaystyle\bar{o}^{ij}(\textbf{x}_{i})=\sum\nolimits_{k=1}^{\left|\mathcal{O}_{e}\right|}c_{k}^{ij}o_{k}^{ij}(\textbf{x}_{i})=c_{1}^{ij}\textbf{0}+c_{2}^{ij}\textbf{x}_{i}=c_{2}^{ij}\textbf{x}_{i}.$

These selection results are integrated with fusion operations as shown in

(3)
$\displaystyle\bar{o}^{j}(\textbf{x})=\sum\nolimits_{k=1}^{\left|\mathcal{O}_{f}\right|}c_{k}^{j}o_{k}^{j}(\textbf{x})=\sum\nolimits_{k=1}^{\left|\mathcal{O}_{f}\right|}c_{k}^{j}f_{k}^{j}(\\{\bar{o}^{ij}(\textbf{x}_{i})|i=0,\cdots,j-1\\}),$

where $f_{k}^{j}$ is the $k$-th candicate fusion operation in block $j$. After
learning the layer-wise connections in block $j$, the results
$\bar{o}^{j}(\textbf{x})$ will be utilized by GNN layers or MLPs.

If the ZERO operation is selected, block $j$ can still obtain the features
from block $i$. Due to we only have ZERO and IDENTITY in the selection
operations, if block $i$ does not select any connections, the input and output
for this block should be a zero tensor. It has a large influence on layer-wise
connection learning since the successors can also obtain the information from
block $i$ no matter which operations are selected.

Thus, we use the Gumbel-Softmax (Jang et al., 2017) as our relaxation method
to alleviate this problem, as shown in

(4) $\displaystyle c_{k}$
$\displaystyle=\frac{\exp((\log\bm{\alpha}_{k}+\textbf{G}_{k})/\lambda)}{\sum_{j=1}^{\left|\mathcal{O}\right|}\exp((\log\bm{\alpha}_{j}+\textbf{G}_{j})/\lambda)},$

$\textbf{G}_{k}=-\log(-\log(\textbf{U}_{k}))$ is the Gumble random variable,
and $\textbf{U}_{k}$ is a uniform random variable, $\lambda$ is the
temperature of softmax. It is designed to approximate discrete distribution in
a differentiable manner and shown useful for supernet training in NAS (Xie et
al., 2018; Dong and Yang, 2019). By choosing a small $\lambda$, the weight
$c_{i}$ becomes one-hot, and the results will become a zero tensor if we do
not select any connections. That is, the IDENTITY operation will have a
smaller influence on the connection learning results when the ZERO operation
is chosen.

In this paper, we directly use the Gumble-Softmax as our relaxation function,
an alternative way is to add a temperature in Softmax function as
$c_{i}=\frac{\exp(\alpha_{i}/\lambda)}{\sum\nolimits_{j=1}^{\left|\mathcal{O}\right|}\exp(\alpha_{j}/\lambda)}$,
we will leave the comparisons into the future work.

Optimization with gradient descent. Based on the continuous relaxation in Eq.
(2) and Eq. (3), we can calculate the mixed results step by step as shown in
Figure 2(a). After we obtain the final representations from the output block
5, we calculate the cross-entropy loss on the input data and optimize the
supernet. Thus, LLC is to solve a bi-level optimization problem as:

(5) $\displaystyle\min\nolimits_{\bm{\alpha}\in\mathcal{A}}$
$\displaystyle\;\mathcal{L}_{\text{val}}(\textbf{w}^{*}(\bm{\alpha}),\bm{\alpha}),$
(6) $\displaystyle\ \text{\;s.t.\;}\textbf{w}^{*}(\bm{\alpha})$
$\displaystyle=\operatorname*{arg\,min}\nolimits_{\textbf{w}}\mathcal{L}_{\text{tra}}(\textbf{w},\bm{\alpha}),$

where $\mathcal{A}$ represents the search space, $\mathcal{L}_{\text{tra}}$
and $\mathcal{L}_{\text{val}}$ are the training and validation loss,
respectively. $\bm{\alpha}$ is the learnable supernet parameter, w is the
operation parameter, and $\textbf{w}^{*}(\bm{\alpha})$ represents the
corresponding operation parameter after training. In search process, we update
$\bm{\alpha}$ and w alternately based on the above continuous relaxation. It
lead to an efficient search process with gradient descent. Deriving Process.
After the search process is finished, we preserve the operations with the
largest weights in each module, from which we obtain the searched
architecture.

##  4\. Experiments

Table 2. Performance comparisons of our method and all baselines. We report the mean test accuracy and the standard deviation with 10 repeat runs. The best results of each base GNN model are in boldface. “L2” and “L4” means the number of layers of the base GNN architecture, respectively. GNN Layer | Connection Method | Cora | PubMed | DBLP | Computer | Physics
---|---|---|---|---|---|---
SAGE | Stacking (L2) | 0.8609(0.0050) | 0.8896(0.0029) | 0.8358(0.0033) | 0.9114(0.0030) | 0.9642(0.0011)
Stacking (L4) | 0.8568(0.0061) | 0.8823(0.0028) | 0.8383(0.0032) | 0.9052(0.0042) | 0.9597(0.0014)
ResGCN (L4) | 0.8566(0.0052) | 0.8899(0.0025) | 0.8339(0.0030) | 0.9151(0.0018) | 0.9631(0.0017)
DenseGCN (L4) | 0.8668(0.0059) | 0.8942(0.0027) | 0.8330(0.0073) | 0.9074(0.0051) | 0.9648(0.0014)
JKNet (L4) | 0.8647(0.0060) | 0.8921(0.0029) | 0.8394(0.0062) | 0.9121(0.0030) | 0.9656(0.0005)
LLC (L4) | 0.8772(0.0050) | 0.8948(0.0023) | 0.8440(0.0015) | 0.9173(0.0020) | 0.9662(0.0010)
GAT | Stacking (L2) | 0.8592(0.0072) | 0.8756(0.0022) | 0.8434(0.0026) | 0.9149(0.0021) | 0.9576(0.0016)
Stacking (L4) | 0.8616(0.0055) | 0.8573(0.0034) | 0.8429(0.0041) | 0.8673(0.0874) | 0.9347(0.0393)
ResGCN (L4) | 0.8466(0.0092) | 0.8756(0.0044) | 0.8411(0.0034) | 0.8569(0.1618) | 0.9567(0.0028)
DenseGCN (L4) | 0.8531(0.0086) | 0.8867(0.0019) | 0.8343(0.0037) | 0.9130(0.0037) | 0.9616(0.0006)
JKNet (L4) | 0.8655(0.0046) | 0.8971(0.0016) | 0.8373(0.0035) | 0.9180(0.0023) | 0.9620(0.0009)
LLC (L4) | 0.8777(0.0006) | 0.8978(0.0023) | 0.8490(0.0015) | 0.9192(0.0008) | 0.9678(0.0002)

###  4.1. Experimental Settings

Datasets. In this part, we evaluate our method on 5 real-world datasets with
diverse types and graph size as shown in Table 3. We split the dataset with
60% for training, 20% for validation and 20% for test with the considering of
supernet training and evaluation.

Table 3. Statistics of the 5 real-world datasets in our experiments. | #Nodes | #Edges | #Features | #Classes | Type
---|---|---|---|---|---
Cora | 2,708 | 5,278 | 1,433 | 7 | Citation
PubMed | 19,717 | 44,324 | 500 | 3 | Citation
DBLP | 17,716 | 105,734 | 1,639 | 4 | Citation
Computer | 13,381 | 245,778 | 767 | 10 | Co-purchase
Physics | 34,493 | 495,924 | 8,415 | 5 | Co-authorship

Baselines. In our experiment, we propose one framework LLC to learn layer-wise
connections in GNNs. We evaluate our method based on GraphSAGE and GAT, 2
widely used methods in learning node representations. We further provide 5
layer-wise connection variants to make comparisons: 2 and 4 layers GNNs
without layer-wise connections, 4-layer baselines ResGCN, DenseGCN and JKNet.

Implementation details. We tune these baselines and the searched architectures
with the same hyperparameters space and same epoch on HyperOpt
333https://github.com/hyperopt/hyperopt. After finetune, we report the test
accuracy and the standard deviation with 10 repeat runs.

Setups for Figure 1. In the GraphGym experiment, to enable the layer-wise
connection learning, we set the message passing layer numbers as [3,4,5,6]. We
provide 7 baselines: Ori: which has no layer-wise connections; JK, Res and
Dense: which use the same layer-wise connections as JKNet, ResGCN and
DenseGCN; 3 extra baselines Dense3, Dense5 and Dense8: which randomly select
the layer-wise connections based on DenseGCN with probability 0.3, 0.5 and
0.8, these 3 baselines are generated before evaluation.

###  4.2. Performance Comparisons

As shown in Table 2, there is no absolute winner from 5 baselines on these
diverse datasets. 3 variants using layer-wise connections generally have a
better performance than 2 stacking baselines, which indicates the
effectiveness of connections among intermediate layers. LLC consistently
outperforms all baselines on all datasets, which demonstrates that our method
can improve the model performance based on the adaptive layer-wise
connections.

Searched architectures. We further visualize the searched layer-wise
connections in Figure 3. For simplicity, we only show the results on Cora and
Computer. Based on the searched connections, the corresponding searched
architectures can be represented as Figure 4. We emphasize several interesting
observations in the following:

  * •

The results are different for different GNN models on different datasets,
demonstrating the necessity of data-specific connection methods.

  * •

On Cora, the searched connections actually lead to multi-branch GNN
architectures as shown in Figure 4(a), which is quite different from most
existing human-designed GNN architectures, i.e., chain-structure by stacking
GNN layers. It provides more flexible and expressive GNN architectures than
existing human-designed and searched ones.

  * •

More interestingly, the input block $0$ is selected by most blocks, which
demonstrates the importance of the input features for the final performance.

  * •

On Computer with GraphSAGE, the searched connection method leads to a densely
connected GNN architecture similar to the DenseGCN as shown in Figure 4(b).
While the performance of LLC on Computer is better than DenseGCN, we attribute
the performance gain to the usage of the fusing functions.

###  4.3. Alleviating the Over-smoothing Problem

Along with the increasing layer numbers, connected node pairs have gradually
similar receptive fields and lead to similar node representations. That is,
node representations become indistinguishable as the network goes deeper. MAD
(Metric for Smoothness) (Chen et al., 2020) is used to measure the smoothness
of the features. We show the comparisons of test accuracy and test MAD in
Figure 5 on different layers. Along with the layer increase, LLC has less
performance drop and the higher MAD values based on baselines, which
demonstrates the effectiveness of LLC in alleviating the over-smoothing
problem.

Figure 3. The searched layer-connections on Cora and Computer datasets with 2
base GNN architectures: GraphSAGE and GAT. Figure 4. Based on the searched
connections in Figure 3, we construct the corresponding architectures with
GraphSAGE. On Cora dataset, $4$-th GNN block is removed since it is not used
in the output block. Figure 5. The comparisons of test accuracy and test MAD
on different layers, and darker colors mean larger and better values. “L4”
represents a 4 layer GNN, and so on.

##  5\. Conclusion and Future Work

In this paper, we propose one framework LLC to address the layer-wise
connections learning problem in GNNs. This framework contains one novel search
space which consists of 3 types of blocks and learnable connections, and one
differentiable search algorithm to enable an efficient search process. The
performance has shown that LLC can achieve the SOTA performance with the
adaptive layer-wise connections and can alleviate the over-smoothing problem.
In future work, we will investigate the influence of different aggregation
functions and the continuous relaxation methods, and learn in-depth the
connections between the searched architectures and the graph properties.

## References

  * (1)
  * Abu-El-Haija et al. (2019) Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019.  Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In _ICML_. PMLR, 21–29.
  * Cai et al. (2021) Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. 2021.  Rethinking Graph Neural Network Search from Message-passing.  _CVPR_ (2021).
  * Chen et al. (2020) Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020.  Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In _AAAI_ , Vol. 34. 3438–3445.
  * Dong and Yang (2019) Xuanyi Dong and Yi Yang. 2019.  Searching for a robust neural architecture in four gpu hours. In _CVPR_. 1761–1770.
  * Fey (2019) Matthias Fey. 2019\.  Just jump: Dynamic neighborhood aggregation in graph neural networks.  _arXiv preprint arXiv:1904.04849_ (2019).
  * Gao et al. (2020) Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. 2020.  Graphnas: Graph neural architecture search with reinforcement learning. In _IJCAI_.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017\.  Neural Message Passing for Quantum Chemistry. In _ICML_. 1263–1272.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs. In _NeurIPS_. 1024–1034.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017.  Densely connected convolutional networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4700–4708.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.  _ICLR_ (2017).
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _ICLR_ (2016).
  * Li et al. (2019) Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019.  Deepgcns: Can gcns go as deep as cnns?. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9267–9276.
  * Li et al. (2018) Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018.  Deeper insights into graph convolutional networks for semi-supervised learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Li and King (2020) Yaoman Li and Irwin King. 2020.  AutoGraph: Automated Graph Neural Network. In _ICONIP_. 189–201.
  * Li et al. (2021) Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. 2021\.  One-shot Graph Ne‘ural Architecture Search with Dynamic Search Space.  (2021).
  * Liu et al. (2019) Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019\.  DARTS: Differentiable architecture search.  _ICLR_ (2019).
  * Pham et al. (2018) Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018.  Efficient Neural Architecture Search via Parameter Sharing. In _ICML_. 4092–4101.
  * Real et al. (2019) Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019.  Regularized evolution for image classifier architecture search. In _AAAI_ , Vol. 33. 4780–4789.
  * Real et al. (2017) Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. 2017\.  Large-scale evolution of image classifiers. In _ICML_. PMLR, 2902–2911.
  * Rossi et al. (2020) Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. 2020\.  Sign: Scalable inception graph neural networks.  _arXiv preprint arXiv:2004.11198_ (2020).
  * Rousseau et al. (2015) François Rousseau, Emmanouil Kiagias, and Michalis Vazirgiannis. 2015.  Text categorization as a graph classification problem. In _ACL_. 1702–1712.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks.  _ICLR_ (2018).
  * Xiao et al. (2019) Wenyi Xiao, Huan Zhao, Haojie Pan, Yangqiu Song, Vincent W Zheng, and Qiang Yang. 2019\.  Beyond personalization: Social content recommendation for creator equality and consumer satisfaction. In _KDD_. 235–245.
  * Xie et al. (2018) Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2018\.  SNAS: stochastic neural architecture search.  _ICLR_ (2018).
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019\.  How powerful are graph neural networks?  _ICLR_ (2019).
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018.  Representation Learning on Graphs with Jumping Knowledge Networks. In _ICML_. 5453–5462.
  * Ying et al. (2018) Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018\.  Hierarchical graph representation learning with differentiable pooling. In _NeurIPS_. 4800–4810.
  * You et al. (2020) Jiaxuan You, Zhitao Ying, and Jure Leskovec. 2020.  Design space for graph neural networks.  _NeurIPS_ 33 (2020).
  * Zhang et al. (2021) Ziwei Zhang, Xin Wang, and Wenwu Zhu. 2021.  Automated Machine Learning on Graphs: A Survey.  _arXiv preprint arXiv:2103.00742_ (2021).
  * Zhao et al. (2020) Huan Zhao, Lanning Wei, and Quanming Yao. 2020.  Simplifying Architecture Search for Graph Neural Network.
  * Zhao et al. (2021) Huan Zhao, Quanming Yao, and Weiwei Tu. 2021.  Search to aggregate neighborhood for graph neural network. In _ICDE_.
  * Zhou et al. (2019) Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. 2019\.  Auto-gnn: Neural architecture search of graph neural networks.
  * Zoph and Le (2017) Barret Zoph and Quoc V Le. 2017.  Neural architecture search with reinforcement learning.  _ICLR_ (2017).