capability to support multiple modes of computation necessitates generality,
which may prevent us from getting the optimal performance for each specific
case.

Computation/communication pattern and sparsity rate disparity. The computation
and communication requirements of different steps in a GNN may vary
significantly due to the distinct sparsity rates of the matrices involved and
their irregular access pattern. For instance, the adjacency matrix is an
ultra-sparse matrix, whereas the weight matrix is dense. The node embeddings
in general are dense but if they are generated by activation functions such as
ReLU, they may create a sparse matrix, but the sparsity rate is much lower
than that of the adjacency matrix. Additionally, an end-to-end GNN application
may utilize other computation patterns such as a Multi-Layer Perceptron (MLP),
which creates a dense multiplication unit to process the node/graph
embeddings. These differences in computation and communication patterns pose a
significant challenge to GNN hardware designers, as they must address the
following sources of disparity and irregular access patterns:

  * •

Graph (adjacency matrix) sparsity: The graph is a large and ultra-sparse
matrix and poses challenges to the memory design. For instance, a graph such
as Orkut (Mislove et al., 2007) comprises over 3 million nodes and more than
223 million edges, yet the density is only $2.5\times 10^{-5}$. The large
graph size requires an on-chip and off-chip memory hierarchy for storage,
while the sparsity translates to irregular accesses of graph elements that
span multiple memory hierarchies.

  * •

Sparsity introduced by activation operators: The introduction of sparsity by
activation operators is a relevant aspect to consider for GNN hardware design.
Specifically, activation operators like ReLU can produce sparsity in the node
embeddings or features. To enable efficient computation on the node
embeddings, it is desirable to address both graph and feature sparsity. This
can be achieved by implementing zero elimination or bypassing techniques in
the computation units. By doing so, we can accelerate the computation and
avoid wasting computation cycles on zero-valued operands.

  * •

Coordination between modules: The differences in data layout caused by
sparsity necessitate the coordination of computation modules. Specifically,
the Aggregation step in GNNs often requires sparse matrix-vector
multiplication (SpMV) or SPMM modules, which typically shuffle the data layout
to enable efficient processing. On the other hand, the Update (also called
transformation) step usually works on continuous matrix/vector layouts.
Consequently, an accelerator must coordinate the data layout between modules.
As mentioned before, in some GNNs, the weights or embeddings in the Update
stage may also be sparse. Therefore, to further reduce processing latency, the
accelerator must further coordinate the computation pattern and sparsity of
the Update module with the SpMV and SpMM modules.

Dynamic input and network structure. Given the dynamic nature of graphs, the
architectural decisions for GNNs are highly dependent on the specific
characteristics of the input graph and the model’s hyperparameters. The memory
and computation requirements of a GNN accelerator are significantly impacted
by factors such as the size of the graph, its level of sparsity, and the
dimensions of the vectors involved. Therefore, the features and properties of
each input graph can be highly distinct from one another, including the
following:

  * •

Different scales and structure of graphs: Real-world graphs exhibit a wide
range of node counts, ranging from a few nodes (e.g., 10) (Riesen and Bunke,
2008) to millions (Mislove et al., 2007). This variation in scale results in
differences in the computation resources needed to perform GNN computations.
Moreover, recent studies have demonstrated that graph size can significantly
impact the selection of scheduling and optimization techniques (Sohrabizadeh
et al., 2022b; Geng et al., 2020). Additionally, the varying number of
neighbors of nodes in the graph introduces a workload imbalance in processing
them.

  * •

Dealing with vectors with various sizes per node: Traditional graph algorithms
like BFS, SSSP, and PageRank typically assign a single scalar value to each
node. In contrast, GNNs utilize long feature vectors for nodes. This
difference in representation affects the memory access pattern and
parallelism. The use of long vectors allows for intra-node parallelism and
data reuse opportunities when applying the same weight matrix to all node
embeddings. A GNN is composed of multiple layers that can operate on vectors
of different sizes. This leads to changes in the access pattern and the
required degree of parallelism across different layers. These differences
create additional opportunities for customization in the hardware design of
GNN accelerators.

Given the special features of graphs and GNNs, designers must carefully
consider how to address the aforementioned critical factors when designing
customized hardware accelerators. These include selecting appropriate data
structures and hardware architectures that can handle the large and irregular
data structures involved in graph processing, optimizing parallelism and
exploiting the available sparsity to achieve high-performance computing,
balancing the workload across the processing units, minimizing communication
overheads between processing units, and adjusting the flow of data to avoid
potential bottlenecks. Additionally, designers must also consider how to
optimize the accelerator for the specific GNN model and its corresponding
computation and communication patterns. Furthermore, given the wide range of
GNN models, each with its unique characteristics and performance requirements,
the designer must consider the trade-off between generality (flexibility),
scalability, and specialization to achieve optimal performance for a given
application.

###  6.2. Summary of The Existing Customized Accelerators

The accelerators presented in this section employ distinct approaches to
address the aforementioned challenges, influenced in part by the neural
networks and datasets they target. Such a trade-off can affect the attainable
peak performance and the generalizability of the proposed accelerators.
Naturally, as the application scope of the work narrows, more opportunities
arise for customization of the accelerator, thereby improving its performance,
albeit at the expense of flexibility to adapt to new experimental settings.
These accelerators are primarily intended for deployment, focusing on
inference rather than training. The constantly changing network architecture
during training presents challenges for adapting the hardware design or
requiring regeneration and/or reconfiguration. Consequently, customized
accelerators designed for the training stage are rare (Zeng and Prasanna,
2020; Chen et al., 2021c).

The summarized works are presented in Table 6, which compares them based on
five main features: graph size, target GNNs, support for layer customization
based on the dynamic structure of the input, the approach to handle the
inherent sparsity of the target application, and levels of parallelization
employed. The prevalent parallelization schemes used in GNN accelerators
include parallelizing the feature dimension for all stages, parallelizing
nodes in the Update stage, and pipelining the main stages of each layer for
intra-layer parallelization. Some approaches also involve exploiting parallel
processing of multiple edges in the Aggregation stage, pipelining the
execution of multiple layers (inter-layer), or processing multiple graphs in
parallel (batch). In the remainder of this section, the related works are
categorized based on the flexibility of their workload. Section 6.3 provides a
summary of the notable works that propose accelerators to target multiple GNN
algorithms. Conversely, another category of works concentrates mainly on the
operations of GCNs, which are one of the most widely used GNN algorithms.
These works further customize the microarchitecture for GCNs, and their
details are presented in Section 6.4.

Table 6. Customized Accelerators Properties Work |  Graph Size |  Target GNNs |  Layer Customization |  Sparsity Support | Parallelization
---|---|---|---|---|---
EnGN (Liang et al., 2020b) | Large, | GCN, GRN |  | Reorganize edges | Intra-layer, Node-level
Ultra | Gated GCN | ✗ | for aggregation | Feature-level, Edge-level
Large | R-GCN |  |  |
| GraphSage |  |  |
Rubik (Chen et al., 2021c) | Small, | GIN | ✗ | Pre-process to | Intra-layer, Node-level
Large | GraphSage |  | reorder the graph | Feature-level, Edge-level
HyGCN (Yan et al., 2020a) |  | GCN |  | Window sliding and | Intra-layer, Node-level
Large | GraphSage | ✗ | shrinkage + sampler as | Feature-level, Edge-level
| GIN |  |  a sparsity eliminator | (if feature dimension is
| DiffPool |  | tool for aggregation | small)
FlowGNN (Sarkar et al., 2022) | Small | GCN, GIN |  | On-the-fly multicasting | Intra-layer, Node-level
Large | GAT, PNA | ✗ | for data distribution | Feature-level, Edge-level
| DGN, VN |  |  |
BlockGNN (Zhou et al., 2021a) | Large | GS-Pool |  | Normal SIMD process | Intra-layer, Node-level
| GCN, GAT | ✗ |  | Feature-level, Edge-level
| GGCN |  |  |
DeepBurning -GL (Liang et al., 2020a) | Large | GCN |  | Degree-aware caching | Intra-layer, Node-level
| GS-Pool | ✓ |  | Feature-level, Edge-level
| EdgeConv |  |  |
AWB-GCN (Geng et al., 2020) |  |  |  | Load-balancing |
|  |  | for both stages: | Inter-layer, Intra-layer
Large | GCN | ✓ |  distribution smoothing, | Node-level, Edge-level
|  |  |  remote switching, |
|  |  | evil row remapping |
StreamGCN (Sohrabizadeh et al., 2022b) |  |  |  | Pre-process for | Inter-layer, Intra-layer
Small | GCN | ✓ | aggregation - In-situ | Feature-level, Node-level
|  |  | support for update to | Edge-level, Batch
|  |  | prune zeros on-the-fly |
GraphACT (Zeng and Prasanna, 2020) |  |  |  | Redundancy reduction | Intra-layer, Node-level
Small | GCN | ✗ | to reduce the number of | Feature-level
|  |  | edges for aggregation |
Zhang et al. (Zhang et al., 2020b) |  |  |  | Pre-process to partition | Intra-layer, Node-level
Large | GCN | ✗ | and reorder the graph, | Feature-level, Edge-level
|  |  | and redundancy reduction |
BoostGCN (Zhang et al., 2021a) |  |  |  | 3-D tiling of edges, | Intra-layer, Node-level
Large | GCN | ✗ | nodes, and features; | Feature-level, Edge-level
|  |  |  sort-and-combine unit |
G-CoS (Zhang et al., 2021b) | Small | GCN, GraphSage |  | On-the-fly multicasting | Intra-layer, Node-level
Large | GAT, GIN | ✓ | for data distribution | Feature-level
GCoD (You et al., 2022) |  | GCN, GIN |  | Workload balanced denser, | Intra-layer, Node-level
Small | GAT | ✗ | light workload sparser | Feature-level, Edge-level
I-GCN (Geng et al., 2021) | Small | GCN |  | On the fly reorder | Intra-layer, Node-level
Large | GraphSage | ✓ | and islandize the graph | Feature-level

###  6.3. Accelerators for General Workloads

As previously stated, a fundamental decision when designing a customized
accelerator is to determine the range of applications it aims to support. In
this section, we provide an overview of the works that propose a flexible
accelerator capable of targeting multiple GNN algorithms. These works either
create a unified architecture that handles all computation stages, as
discussed in Section 6.3.1, or develop a specialized engine for the primary
computation stages due to the distinct computation and communication patterns,
as summarized in Section 6.3.2.

####  6.3.1. All-Stage Unified Architecture.

The works in this category propose a unified architecture to process multiple
stages of computation, despite their division into distinct phases, such as
the Aggregation and Update steps. The proposed accelerators in this category
develop a singular engine capable of handling these stages, even though they
have different requirements in terms of computation and communication. We
elaborate on their approaches below:

EnGN (Liang et al., 2020b) targets graph convolution network (GCN) (Kipf and
Welling, 2016), relational GCN (R-GCN) (Schlichtkrull et al., 2018), gated GCN
(Dauphin et al., 2017), GraphSage (Hamilton et al., 2017a), and graph
recurrent network (Salehinejad et al., 2017). To process them, the operations
are divided into three stages: feature extraction, aggregation, and update.
The authors develop a Neural Graph Processing Unit (NGPU) as a unified
architecture for executing them. These stages are processed as pipelined
matrix multiplications of the feature matrix, neural weights, and adjacency
matrix. The NGPU uses a 32-bit fixed-point $128\times 16$ systolic array as
the main computation unit, which maps nodes to different rows and their
features to different columns. To perform the aggregation, the authors propose
a ring-edge-reduce dataflow that connects processing elements (PEs) in the
same column with a ring so that the results can be passed through and added
together based on the adjacency matrix. As the adjacency matrix is sparse, the
authors reorganize the edges to reduce the number of idle PEs during the
execution of aggregation. Additionally, high-degree nodes are cached to
decrease the number of off-chip transactions since they are reused across
multiple operations. EnGN analyzes the effect of different schedules on the
number of operations and develops a dimension-aware stage reordering strategy,
which changes the order of execution between the aggregation and feature
extraction stage based on the input and output dimensions, to reduce the
number of operations. Furthermore, it splits the graphs into 2D tiles to fit
them into on-chip resources and develops an analytical formula to decide
whether for a given layer, a column-order traversal of the tiles results in
fewer I/O operations or their row-order traversal.

Rubik (Chen et al., 2021c) develops an accelerator tailored to train GIN and
GraphSage. To enhance graph locality and data reuse, the proposed design
leverages a pre-processor that reorders the graph to group nodes with common
neighbors. The graph subsequently passes through a hierarchical accelerator
architecture, consisting of a PE array interconnected via a 2D-mesh network
NoC. Each PE embedded in this architecture features a multiply-and-accumulate
(MAC) array. To map the graph and features to the accelerator, the authors
presented graph-level and node-level mappings. The former entails mapping a
window of nodes to one PE, allowing for task-level parallelism, with PEs
working on distinct node windows. The latter involves tiling the dense vector-
matrix multiplication for the feature update (aggregation and update) using
the MAC array. Rubik incorporates a global buffer for the PE array, two
private caches (for data and instruction) in each PE, and register files (RFs)
in every MAC to facilitate data reuse. G-CoS (Zhang et al., 2021b) is an
automated framework that aims to optimize the performance and efficiency of
GNNs by co-searching for the best GNN structures and accelerators. This
framework consists of two components: (1) a one-shot co-search algorithm for
GNN structures and their matched accelerators, and (2) a generic accelerator
search space that can be applied to various GNN structures. The accelerator
search space in G-CoS is based on a template comprising multiple sub-
accelerators that can handle both sparse and dense matrix multiplications. By
configuring the settings of each sub-accelerator, such as tiling sizes, tiling
order, and interconnection style, they can be specialized to process different
clusters of GNN operators with distinct data sizes and sparsity, which
ultimately enhances the hardware efficiency. In addition, G-CoS employs local
buffers assigned to the intermediate features, index for sparse features (in
COO format), and weights and intermediate outputs for each sub-accelerator.
These buffers can be interconnected to allow for data sharing and reuse across
different sub-accelerators, reducing the need for costly off-chip memory
access. G-CoS is a pioneering effort towards automating the search for both
GNN model structure and accelerator design knobs.

GCoD (You et al., 2022) is a co-design framework that addresses the challenge
of extreme sparsity in GNN inference by optimizing both the algorithm and
hardware accelerator. At the algorithm level, GCoD polarizes the graphs into
either denser or sparser local neighborhoods without compromising model
accuracy. This polarization results in adjacency matrices with two levels of
workload, which greatly improves regularity and ease of acceleration. On the
hardware level, GCoD integrates a dedicated two-pronged accelerator comprising
a dense branch and a sparse branch processor. The dense branch processor
employs a chunk-based micro-architecture with several heterogeneous computing
modules to balance the workload across different subgraphs from the polarized
adjacency matrix. This balance is achieved by allocating computing resources
and bandwidth proportionally based on the data volume and operation size
involved in each subgraph’s processing. The sparse branch processor
accelerates the remaining sparse workloads, which constitute a small portion
of nonzero data, mostly on-chip. To store data in a sparse format, the sparse
branch processor uses the CSC format, which significantly reduces the storage
overhead of adjacency matrices. GCoD’s sparse branch processor also employs a
query-based weight forwarding mechanism that flexibly shares on-chip input
data from the dense branch processor to enhance data reuse. The weight
forwarding is performed on-demand of the sparse branch to achieve more
efficient control, and it accesses around 63% of the data, which drastically
reduces the off-chip access cost. Overall, GCoD’s algorithm-hardware co-design
framework offers better accuracy-efficiency trade-offs than optimizing each
aspect separately. By polarizing the graphs into denser and sparser local
neighborhoods and integrating a two-pronged accelerator with a dense and
sparse branch processor, GCoD provides a comprehensive solution to tackle the
challenge of extreme sparsity in GNN inference.

I-GCN (Geng et al., 2021) proposes a new approach called "islandization" to
enhance data locality in GNNs. By identifying clusters of nodes with strong
internal connections, called "islands," I-GCN aims to minimize off-chip memory
accesses and improve on-chip data reuse. To support islandization, I-GCN’s
hardware architecture includes an Island Locator and an Island Consumer. The
Island Locator searches the graph from multiple nodes in parallel to locate
highly-connected nodes termed as hubs. From these hubs, multiple tailored
breadth-first-search engines are used to identify islands. The Island Consumer
then combines and aggregates the islands and hubs, with a focus on reusing
aggregation results among nodes with shared neighbors. To optimize the
aggregation process, a ring-based on-chip network is used to distribute
partial results and detect reuse opportunities among processing elements
(PEs), and an in-network reduction scheme is employed to minimize network
communication latency. Overall, I-GCN’s islandization approach and hardware
design achieve better data locality and efficiency in GNNs.

####  6.3.2. Dedicated Module per Stage.

In contrast to the previous category, the works in this category propose a
specialized engine for the primary computation stages of the GNN algorithms.
By adopting this approach, each engine can be customized according to the
requirements of each stage. It can also facilitate the concurrent execution of
different stages.

HyGCN (Yan et al., 2020a) targets GCN, GraphSage, GINConv (Xu et al., 2018a),
and DiffPool (Ying et al., 2018b) and treats their operations as two main
stages, aggregation, and combination (Update in Eq. 1). Due to the distinctive
execution patterns of the aggregation and combination stages, the proposed
accelerator employs a separate processing engine for each stage and processes
them in a dataflow manner. Specifically, the Aggregation engine utilizes
parallel single-instruction multiple-data (SIMD) cores to exploit the intra-
vertex parallelism resulting from the long feature vectors assigned to each
vertex by the GNN algorithms. If the feature vector length is smaller than the
total number of cores, the unassigned cores are allocated to other nodes,
thereby enabling edge-level parallelism for this stage. As the graph
connections are sparsely distributed, a sparsity eliminator is employed to
identify a region in the adjacency matrix of the corresponding graph partition
that stores the effective edges, using a window sliding and shrinkage
technique. To further reduce the computation complexity, the aggregation
operation is performed on a sampled set of neighbors. The sparsity eliminator
and sampler also help to avoid fetching feature vectors of the nodes that are
not connected to any edge in the current graph partition. Subsequently, the
Combination engine treats the computation as a dense matrix-vector
multiplication (MVM) and is realized as a group of systolic arrays that can
work independently or cooperatively depending on the configuration. Each
systolic array processes a small group of nodes, with the feature vectors
(weights) of the nodes flowing through the row (column) dimension.

FlowGNN (Sarkar et al., 2022) proposes a generic dataflow architecture for
accelerating GNNs that can flexibly support the majority of message-passing
GNN algorithms, including GCN, GINConv, PNA (Corso et al., 2020), GAT
(Veličković et al., 2017), DGN (Beaini et al., 2021), and GNN with virtual
node (VN) (Gilmer et al., 2017). The authors argued that common pre-processing
techniques used to exploit data locality are not feasible for real-time
applications with millions of input graphs with varied structures. Thus, they
directly take the graph in COO format without any pre-processing to reorganize
the data. FlowGNN’s architecture is based on the idea that any graph-related
functions can be expressed with pair-wise message passing. It divides the
operations into three steps: 1) the gather phase, which aggregates messages
from neighbors; 2) the node transformation (NT) phase, which applies the
update function (or feature transformation function) to messages; and 3) the
scatter phase, which constructs messages for the next layer by applying the
respective message transformation function (e.g., applying the edge weights)
on the node embeddings produced in step two. The computations are divided into
these steps to integrate edge embeddings and different aggregation functions.
The gather and scatter phases handle per-edge computations and are implemented
as a single unit (MP unit) that follows the NT unit in a dataflow fashion. The
NT unit handles per-node computations. As such, the MP (NT) unit consists of
parallel PEs that parallelize the edges (nodes). Each of the PEs further
parallelizes the features. To avoid bank conflicts, each of the PEs in the MP
unit processes one particular bank of edges. FlowGNN uses an NT-to-MP adapter
to perform on-the-fly multicasting of the node embeddings generated by the NT
unit to their right PE in the MP unit.

BlockGNN (Zhou et al., 2021a) aims to compress the update stage (feature
transformation) of GNN models for GraphSage-Pool (GS-Pool), GCN, GGCN
(Marcheggiani and Titov, 2017), and GAT that require a large weight memory,
high computing resources, and high computing latency when the feature size is
large. To achieve this, they leverage block-circulant weight matrices, which
are configured in the form of block-circulant matrices that can be accelerated
by Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT).
This transformation reduces the computation intensity and required storage for
the update stage. The computation engine of BlockGNN is composed of two main
units, the CirCore unit for the update stage and the VPU (vector processing
unit) for the aggregation stage and non-linear operations. The CirCore unit is
a three-stage pipeline, consisting of an FFT unit that converts data from the
spatial domain to the spectral domain, a systolic array that performs element-
wise product and accumulation, and an IFFT unit that transforms the results
back to the spatial domain. The VPU is organized as a SIMD unit with m lanes,
each with a parallel factor of 16.

DeepBurning-GL (Liang et al., 2020a) is an automated framework designed to
facilitate the development of hardware accelerators for GNNs. The framework
takes a GNN model as input, which is designed using software frameworks such
as DGL and PyG, and generates a hardware accelerator for the target FPGA
platform. For the hardware design, the authors provide three templates. The
first template is the GNN computation template that handles dense and sparse
multiplications. Dense multiplications are implemented via a systolic array or
a dot production array, while sparse multiplications are handled by an array
of SIMD units or a ring-reduce topology (similar to EnGN). The second template
is the memory template, which consists of a distributed on-chip buffer, cache,
or degree-aware hierarchy on-chip buffer to support both regular and irregular
memory accesses. The third template is the graph manipulation template, which
supports graph sampling and reconstruction in various GNN models. To adapt to
the specific characteristics of the target model and FPGA platform, the
authors develop analytical models that analyze the processing requirements and
adopt a design space exploration to tune hardware parameters for the required
templates based on the graph properties, network architecture, and resource
constraints. The authors validate their framework on various GNN algorithms,
including GCN, GS-Pool, R-GCN, and EdgeConv (Wang et al., 2019a).

###  6.4. Accelerators for Special Workload

The literature in this field endeavors to direct their efforts toward a more
specialized algorithm. More specifically, it concentrates on a singular GNN
algorithm (here, mostly GCN), and subsequently tailors the proposed solutions
accordingly. Given that the target model may contain multiple layers, the
first subgroup develops a deep pipeline with custom layer modifications, which
are elaborated upon in Section 6.4.1. Conversely, the second subgroup adopts a
fixed hardware approach for all layers, which is described in detail in
Section 6.4.2.

####  6.4.1. Layer-Customizable Deep Pipeline.

The studies presented in this section argue that a fundamental optimization in
a GCN accelerator should involve reducing the expenses associated with memory
transactions. To this end, they construct a deep pipeline that circumvents the
need for off-chip (global) memory transactions concerning intermediate results
among distinct GCN layers. Through the allocation of specific engines for each
layer, they can additionally tailor hardware parameters based on the
individual workload of each layer.

AWB-GCN (Geng et al., 2020) argues that many real-world graphs follow the
power-law distribution, which implies that the number of nodes with a given
degree x follows a proportional relationship with $x^{-\beta}$ for a constant
$\beta$. This leads to a situation where a small number of rows/columns in the
adjacency matrix contain the majority of non-zero elements, while the rest
contain only a few non-zero elements. The authors also note that ReLU
activation leads to an abundant number of zero elements, making it more
efficient to treat the feature transformation step (Update step) as a sparse
computation. In response to these observations, the authors propose an
accelerator architecture called Autotuning-Workload-Balancing (AWB) GCN, which
includes distribution smoothing, evil row remapping, and remote switching.
Since GCNs have linear aggregation and update functions, the authors propose
executing the update stage before the aggregation stage since this approach
helps reduce the number of non-zero operations. This is because by this
change, in both steps, we are dealing with two SpMM as opposed to an SpMM and
a dense matrix multiplication. AWB-GCN’s architecture supports inter-layer
parallelism in addition to the commonly used intra-layer parallelism and is
based on the inner-product matrix multiplication. The authors partition the
graph and implement a task distributor to navigate the data to idle PEs to
balance the workload. The architectural optimizations are designed to overcome
the load imbalance problem. Distribution smoothing is local and tries to
balance the load among neighboring PEs within up to three hops. The remote
switching addresses regional clustering by exchanging the workloads of under-
and overloaded PEs. Evil row remapping distributes rows with the most non-
zeros to the most under-loaded PEs.

StreamGCN (Sohrabizadeh et al., 2022b) presents an efficient and flexible GCN
accelerator for streaming small graphs - from the DRAM, the host CPU, and
through the network - and exploiting all the available sparsity. The authors
argue that in memory-bound applications like this, it is essential to minimize
memory transactions, and they propose a scheduling mechanism to achieve this.
In this scheduling, input data is fetched from global memory (DRAM) only once
and is reused for all corresponding computations before being evicted. The
proposed GCN accelerator includes dedicated modules for each layer, which not
only allows customization of parallelization based on the workload of each
layer but also enables inter-layer pipelining. As a result, the intermediate
results are directly passed to the modules for the next layer through on-chip
FIFOs which further helps with decreasing the number of DRAM transactions. To
avoid read-after-write (RAW) dependencies in the aggregation unit, the edges
are pre-processed and reordered. Both the aggregation unit and the feature
transformation unit contain multiple SIMD PE with the SIMD dimension
parallelizing the node features. The PEs in the aggregation unit process
different pairs of neighbors in parallel with their respective edge weight
(edge parallelism), while in the feature transformation unit, they parallelize
the nodes. These units are connected in a dataflow fashion. Similar to AWB-
GCN, StreamGCN treats the feature transformation step as a sparse computation.
However, due to the smaller size of the target graphs, the method is based on
outer-product-based multiplication. This scheduling is shown to reduce the
number of pipeline stalls due to RAW dependencies. Since the intermediate
results are to directly be processed as they are being produced, StreamGCN has
in-situ sparsity support by developing a mechanism to prune the zero elements
of the node embeddings on-the-fly, while they are being generated. This is
realized by a pruner and an arbiter unit. The pruner pre-fetches the results
at a higher rate than the next layer’s processing rate to pass non-zero
elements through FIFOs. The arbiter then checks them for RAW dependencies and
dispatches enough elements to fill the PEs. Overall, the proposed StreamGCN
architecture is ideal for real-time or near-real-time graph search and
similarity computation for many biological, chemical, or pharmaceutical
applications.

####  6.4.2. All-Layer Unified Architecture.

As seen in the previous section, a GCN algorithm consists of multiple layers
that have distinct features (e.g., node embedding dimension). In contrast to
the works detailed in Section 6.4.1, the studies presented in this category
put forth the idea of constructing a more adaptable architecture across
distinct layers and utilizing the same engines for all layers. Since the works
in this category focus on a specific workload, namely GCN, they offer greater
possibilities for customization than those presented in Section 6.3, as
indicated below:

GraphACT (Zeng and Prasanna, 2020) develops an accelerator for training GCNs
targeting small graphs on CPU-FPGA heterogeneous systems. The small graphs are
created by sampling subgraphs of the input graph to be able to fit them into
the on-chip resources. GraphACT does not work directly with the normalized
adjacency matrix and rather defines three types of multiplications for GCNs:
1) dense (weights) - dense (embeddings), 2) binary sparse (adjacency matrix) -
dense, 3) diagonal (degree matrix) - dense. The first multiplication is
performed using a 2D systolic array called the Weight Transformation module,
where nodes are mapped to the row dimension and output features to the column
dimension. The Feature Aggregation module implements steps two and three. The
third multiplication (Step three) is treated as scaling the rows of the second
matrix in the multiplication with minimal hardware overhead. GraphACT
implements a redundancy reduction technique to reduce the computation cost of
the second step, which seeks to pre-compute the repeated aggregations on CPU.
This is possible since GraphACT directly works with the adjacency matrix in
its binary (not normalized) format. The redundancy reduction algorithm
identifies the common pairs of neighbors and merges each of the most repeated
ones in a node. This is done as a pre-processing step on CPU since it is
communication-intensive. The Feature Aggregation module has a 1D accumulator
array that parallelizes the feature dimension. It adds the features of the
identified node pairs and performs the aggregation based on the adjacency
matrix of the merged graph (generated after merging the node pairs). Finally,
it scales the node features based on the node degrees (Step 3). Both the
Feature Aggregation and Weight Transformation modules are reused for all the
$L$ layers of the GCN.

Zhang et al. (Zhang et al., 2020b) focus on accelerating GCN inference for
large graphs. The approach begins by partitioning the input data into smaller
sizes that can fit into the on-chip resources for processing. Following this,
a redundancy reduction scheme, such as GraphACT, and a node-reordering phase
are applied to decrease global memory access. Upon completion of the pre-
processing stage, the graph is passed to a hardware architecture that
incorporates two primary modules: Aggregation and Weight Transformation. The
Aggregation module employs parallel vector accumulators (VAs) for sparse
matrix multiplication in the GCN aggregation process. Here, each VA
parallelizes the feature dimension and different VAs process distinct edges.
The Weight Transformation module leverages a systolic array to implement the
update stage. These modules are bi-directionally connected to enable different
modes of scheduling, where the order of executing the aggregation and update
stage can vary based on the GCN and dataset properties.

BoostGCN (Zhang et al., 2021a) presents a framework aimed at optimizing GCN
inference on FPGAs. The hardware architecture of BoostGCN includes two primary
modules, namely the feature aggregation module (FAM) and the feature update
module (FUM). These modules are interconnected through internal buffers that
cache intermediate results, and the connection between them is bi-directional,
as in Zhang et al.’s work. To reduce memory traffic and address workload
imbalance, the authors propose Partition-Centric Feature Aggregation (PCFA),
which leverages tiling in three dimensions: edges, nodes, and features. FAM
operates on multiple edges in parallel and further parallelizes the feature
dimension for them. Additionally, FAM uses a sort-and-combine (SaC) unit to
sort the inputs and combine them if two of them have the same destination,
which reduces congestion. BoostGCN employs two types of FUM based on the
sparsity of the feature matrix (node embeddings) – Sparse-FUM and Dense-FUM.
Sparse-FUM uses the same structure as FAM, with a format transformation module
in the beginning that transforms the input to COO format. Dense-FUM, on the
other hand, implements a 2D systolic array.

Table 7. Performance and Validation Methodology of the Customized Accelerators Work | Validation Methodology |  End-to-end Measurement | Reported Speedup
---|---|---|---
EnGN (Liang et al., 2020b) | ASIC (TSMC 14nm with HBM 2.0 at 1GHz) | ✗ |  $1802.9\times$, $19.75\times$, and $2.97\times$ over CPU
cycle-accurate simulator |  |  (PyG $\&$ DGL), GPU (PyG $\&$ DGL), and HyGCN
Rubik (Chen et al., 2021c) | ASIC (45nm at 500MHz) | ✗ |  $3.42-46.7\times$, and $1.3-14.16\times$ over
cycle-accurate simulator |  |  PyG-GPU, and Eyeriss (Chen et al., 2016)-like accelerator
HyGCN (Yan et al., 2020a) | ASIC (TSMC 12nm with HBM at 1GHz) | ✗ |  $1509\times$, and $6.5\times$ over
cycle-accurate simulator |  | PyG-CPU, and PyG-GPU
FlowGNN (Sarkar et al., 2022) | FPGA (Xilinx Alveo U50 at 300MHz) | ✓ |  $24-254\times$, $1.3-477\times$, and $1.26\times$ over
on-board evaluation |  | PyG-CPU, PyG-GPU, and I-GCN
BlockGNN (Zhou et al., 2021a) | FPGA (Xilinx ZC706 at 100MHz) | ✗ |  $2.3\times$, and $4.2-8.3\times$ over
on-board evaluation and analytical modeling |  | CPU (TensorFlow-based), and HyGCN
DeepBurning -GL (Liang et al., 2020a) | FPGA (Xilinx ZC706, KCU1500, and Alveo U50) | ✗ |  $7.43-346.98\times$, $3.7-16.5\times$, and $6.28\times$ over
on-board evaluation at 100, 200, and 200MHz |  | DGL-CPU, DGL-GPU, and HyGCN on FPGA
AWB-GCN (Geng et al., 2020) | FPGA (Intel Stratix 10 SX) | ✗ |  $3255\times$, $80.3\times$, $5.1\times$ over
on-board evaluation |  | PyG-CPU, PyG-GPU, and HyGCN
StreamGCN (Sohrabizadeh et al., 2022b) | FPGA (Xilinx KU15P, Alveo U50 and U280) | ✓ |  $8.2-18.2\times$, and $12.1-26.9\times$ over
on-board evaluation at 201, 279, 290MHz |  | PyG-CPU, and PyG-GPU
GraphACT (Zeng and Prasanna, 2020) | FPGA (Xilinx Alveo U200 at 200MHz) |  ✓ (training speed) |  $12-15\times$, and $1.1-1.5\times$ over
on-board evaluation |  | CPU, and GPU
Zhang et al. (Zhang et al., 2020b) | FPGA (Xilinx Alveo U200 at 250MHz) | ✗ |  $30\times$, and $2\times$ over
on-board evaluation |  | CPU, and GPU - TensorFlow and C++ (Cuda)
BoostGCN (Zhang et al., 2021a) | FPGA (Intel Stratix 10 GX 10M at 250MHz) | ✗ |  $100\times$, $30\times$, $3$-$45\times$ over CPU (PyG $\&$ DGL),
on-board evaluation |  |  GPU (PyG $\&$ DGL), and FPGA ((Zhang et al., 2020b) $\&$ HyGCN)
G-CoS (Zhang et al., 2021b) |  FPGA (Xilinx VCU128 at 330 MHz) on-board evaluation | ✗ |  $5.52\times$, and $1.92\times$ speedups over
| HyGCN, AWB-GCN
GCoD (You et al., 2022) |  FPGA (Xilinx VCU128 at 330 MHz) on-board evaluation | ✗ |  $7.8\times$, and $2.5\times$ over HyGCN, AWB-GCN
I-GCN (Geng et al., 2021) |  FPGA (Intel Stratix 10 SX at 330 MHz) on-board evaluation | ✗ |  $368\times$, $453\times$, $16\times$, and $5.7\times$ over PyG-GPU, DGL-GPU, SIGMA, and AWB-GCN

Table 7 shows a review of accelerators, focusing on their validation
methodology and reported speedup. These accelerators have been implemented on
FPGAs or simulated as an ASIC design. A prevalent baseline for reporting the
accelerator’s performance is CPU and GPU, with most works utilizing the state-
of-the-art implementation of PyG for this purpose. The reported results may
vary regarding whether they exclusively measure the runtime of the execution
of the main kernel for their accelerator or include other associated overheads
and present an end-to-end runtime.

##  7\. GNN Acceleration: Special Graphs and Special GNNs

In the sections above we talked about acceleration techniques for the regular
message-passing GNNs on regular graphs. However, many practical problems are
better modeled with special graph types, e.g., heterogeneous graphs and
dynamic graphs (both defined below), and solved with more sophisticated GNN
architectures, e.g., heterogeneous GNNs and GNNs with attention. Many methods
we discussed above may be applied to these special cases if their specialties
are ignored, e.g, treating a heterogeneous graph as if it is homogeneous.
However, the result would be suboptimal such as having high accuracy loss. In
this section, we discuss techniques for handling these special cases
especially.

###  7.1. GNN Acceleration on Heterogeneous Graphs

The graphs we talked about so far have nodes/edges of the same type, and they
are called homogeneous graphs. The complex graphs in the real world are often
associated with various types of nodes and edges (Dong et al., 2020; Javari et
al., 2020; Huang et al., 2022), establishing a heterogeneous structure. For
example in a citation graph, nodes can be authors, venues, and papers, and
edges can denote an author published a paper, a paper published in a venue,
etc. Different types of nodes and edges have substantially different meanings
from each other and it is better to distinguish them explicitly. Heterogeneous
graphs as a new data structure are defined below.

###### Definition 7.1.

A heterogeneous graph is defined as a directed graph
${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}})$ associated with a node type
mapping function $\phi:{\mathcal{V}}\rightarrow{\mathcal{A}}$ and an edge type
mapping function $\tau:{\mathcal{E}}\rightarrow{\mathcal{R}}$. Each node
$v\in{\mathcal{V}}$ belongs to one node type $\phi(v)\in{\mathcal{A}}$ and
each edge $e\in{\mathcal{E}}$ belongs to one edge type
$\tau(e)\in{\mathcal{R}}$.

Many GNNs are specially designed for heterogeneous graphs (Hu et al., 2020a;
Zhang et al., 2019b; Long et al., 2021). They face similar scalability
challenges as regular GNNs on homogeneous graphs and require acceleration. In
the previous sections, we discussed acceleration methods that could be used
for GNNs on heterogeneous graphs by disregarding their heterogeneity (i.e.,
different node and edge types). However, applying acceleration methods in a
homogeneous manner by ignoring heterogeneity often leads to a significant
decrease in performance. For example, for sampling algorithms, if a message
passing step sample node neighbors uniformly without considering the node
types, some infrequent types are likely missed out. Completely missing one
type of node can downgrade the GNN performance a lot (Zhang et al., 2019b). A
better sampling algorithm on heterogeneous graphs should perform a weighted
sampling to include neighbor nodes of all types. Similarly for model pruning,
heterogeneous GNNs often have separate model weights in the update function
for each edge types (Schlichtkrull et al., 2018), and ignoring heterogeneity
may prune the weights completely for some edge types, which makes the GNN
unable to pass messages that type of edges and downgrade the performance.
Therefore, accelerating GNNs while taking into account heterogeneity presents
additional challenges and requires special treatment to capture heterogeneity.
In the following, we will provide a summary of recent research on modeling and
accelerating GNNs on heterogeneous graphs.

HetGNN (Zhang et al., 2019b) is a GNN on heterogeneous graphs that introduces
a novel heterogeneous neighbor sampler. The sampler belongs to the node-wise
sampling category discussed in Section 3.2.1 but overcomes a special challenge
on heterogeneous graphs. On heterogeneous graphs, node types in a target
node’s receptive field can be very imbalanced, e.g., some types are very
infrequent compared to other types. HetGNN argues that a regular sampler may
miss some less common node types completely and result in insufficient node
representations. Therefore, instead of randomly sampling, the HetGNN sampler
first runs random walks with restart (RWR) in the neighborhood and then
chooses nodes to form the computation graph according to their hit frequency
by RWR. The sampler will also balance node types, such that the top frequent
nodes of all types are chosen. With this RWR-based and type-balanced sampler,
HetGNN can achieve fast training with good accuracy performance.

HGT (Hu et al., 2020a) is a transformer-based GNN on heterogeneous graphs,
which also introduces an efficient sampler to make the model handle Web-scale
graph data with billions of edges. The HGT samples subgraphs in mini-batches.
The sampler is able to 1) maintain a similar number of nodes and edges for
each type in each subgraph and 2) keep the sampled subgraph dense.
Specifically, for each node type it maintains a node budget and iteratively
samples the same number of nodes with an importance sampling strategy to
reduce variance. The sampling probability for each node type is calculated
based on the square of the cumulative normalized degree of other nodes that
have already been sampled in its budget. The model is shown to achieve state-
of-the-art results on the Open Academic Graph (OAG) which contains millions of
nodes and billions of edges.

Pigeon (Wu et al., 2023) is an intermediate representation (IR) and code
generator for end-to-end training and inference of relational graph neural
networks (RGNNs) (Schlichtkrull et al., 2018) on heterogeneous graphs. Pigeon
addresses the performance challenges posed by RGNNs by decoupling the model
semantics, data layout, and operator-specific schedule, and expressing these
opportunities to allow them to be integrated into the design space as integral
elements. The proposed solution includes a compact tensor materialization
scheme and a linear operator fusion pass to achieve both training and
inference. The compact tensor materialization scheme reduces the resources
spent on computing and storing common subexpressions in RGNNs. Specifically,
certain edge data are determined by source node features and edge types, and
instead of computing and storing such data for each edge, the system computes
and stores the data once for each (edge type, unique node index) pair. This
reduces the amount of computation and storage required for common
subexpressions. The linear operator fusion pass switches the order of linear
operators, such as linear layers and inner products. When a linear operator is
followed by another linear operator, their order may be switched to reduce the
strength.

###  7.2. GNN Acceleration on Dynamic Graphs

Spatio-temporal data appears frequently in scientific research and real-world
applications (Huang et al., 2020; Sanchez-Gonzalez et al., 2020). Dynamic
graphs have shown great performance for modeling such spatio-temporal data to
solve problems like molecular dynamics in biology (Yang et al., 2021a),
information diffusion over social networks (Wang et al., 2021c), and epidemic
spread in public health (Huang et al., 2021a). Dynamic graphs are graphs that
can change over time. Changes include the addition and removal of nodes and
edges to capture the evolving nature of the data. The evolving nature has
imposed additional challenges for corresponding learning and reasoning on
dynamic graphs as opposed to static graphs. Therefore, researchers have
developed special spatio-temporal GNNs to jointly model the graph structural
information on the spatial aspect and the evolving dynamics on the temporal
aspect. The regular message passing in Equation (1) has been modified as is
shown in Equation (18) for spatio-temporal GNNs, where the node
representations are updated by both their spatial neighbors and their
historical states.

(18)
$\displaystyle{\bm{h}}_{v}^{t}=\textsc{UPDATE}({\bm{h}}_{v}^{t-k:t-1},\textsc{AGGR}(\\{\phi({\bm{h}}_{v}^{t-1},{\bm{h}}_{u}^{t-1})|(u^{t-1},v^{t-1}\in{\mathcal{E}}^{t-1}\\}))$

Here ${\mathcal{E}}^{t-1}$ denotes the edges for graph at timestamp $t-1$, and
sometimes the edges can be latent and need to be inferred first before the
message passing.

Spatio-temporal GNNs are often applied to computation-heavy problems like the
simulation of large-scale dynamical systems (Sanchez-Gonzalez et al., 2020),
which have similar scalability concerns as regular GNNs on static graphs and
require acceleration. However, spatio-temporal GNN acceleration incurs special
challenges because model training needs to be done chronologically and graph
operations (modification and sampling) need to consider both spatial and
temporal neighbors. For example, the sampling methods introduced in Section
3.2 may be individually applied to dynamic graphs at each timestamp $t$, but
they can fall short in performance as they only consider spatial properties of
nodes but omit the temporal influence. In this section, we summarize recent
works on spatio-temporal GNN acceleration, which are often applied for
learning and reasoning over large-scale dynamic graphs.

TGL (Zhou et al., 2022) proposes a unified framework for accelerating
continuous-time temporal GNN training. It designs a temporal-CSR data
structure for rapid access of temporal neighborhood candidates, and a parallel
sampler is further equipped to accelerate temporal neighborhood sampling.
Also, it proposes a novel random chunk scheduling technique for training
acceleration where simply increasing the batch size would result in the
obsolete node memory issue. TGL (Zhou et al., 2022) shows its great capability
in handling large-scale dynamic graphs where it achieves an average of
173$\times$ speedup on a multi-core CPU compared with the baselines.

MTGNN (Wu et al., 2020a) applies GNNs to model the spatio-temporal graph for
solving the multivariate time series prediction problem, where variables are
dependent on each other but their dependence (graph structure) is latent.
Naively assuming a fully-connected graph structure will result in $N^{2}$
edges and impose huge GNN computation time. MTGNN proposes a graph learning
module to automatically extract the uni-directed relations among nodes, which
greatly simplifies the structure and accelerates GNN computation. MTGNN also
applies clustering to reduce memory usage by the intermediate states of nodes
and save memory access time. Specifically, the edge search space is reduced by
first randomly separating nodes into clusters and then only forming edges
within each cluster. The clustering process for nodes is repeated in each
iteration, which allows every pair of nodes in the same cluster to be possibly
connected.

GraphODEs (Huang et al., 2020, 2021a) combine the expressive power of GNNs
with the principled ordinary differential equation (ODE) modeling for learning
dynamical systems. LG-ODE (Huang et al., 2020) is a latent ODE generative
model, which models multi-agent system dynamics with fixed graph structure in
a continuous manner. The ODE function in LG-ODE is a GNN to capture the
continuous interaction among agents, and the latent initial states for ODEs
are inferred through another GNN-based encoder. Towards acceleration, existing
discrete neural models learn a fixed-step transition function that takes the
system state at time $t$ as input to predict the state at time $t+1$. An
adaptive ODE solver will automatically adjust the times, where the graph
interaction module is called to provide the balance between model performance
and time consumption. Additionally, LG-ODE is especially superior at making
long-range predictions. It is able to adjust its prediction length without re-
feeding its previous predictions at the same fixed step during training, which
greatly accelerates model inference. Moreover, since LG-ODE is only capable of
handling dynamic graphs with evolving nodes but fixed edges, the follow-up
work CG-ODE (Huang et al., 2021a) is proposed to jointly model the evolution
of nodes and edges through two coupled ODE functions.

GN-ETA (Derrow-Pinion et al., 2021) proposes a large-scale GNN model for
estimating the time of arrival (ETA) in real time. Supersegment-based
acceleration is applied so that the complex interaction of traffic roads and
the traffic conditions evolution can be joint modeled over time. In
particular, GN-ETA segments the traffic network into several supersegments and
is trained to predict both the across-supersegments time and within-
supersegments time. For model inference, the sequence of supersegments will be
put in a route sequentially where the later supersegements would utilize the
prediction time of an earlier one to determine its relevant prediction
horizons for scaling up. Additionally, GN-ETA adopts the MetaGradients
methodology (Xu et al., 2018b) to stabilize the training process raised by
uneven query batches that are commonly seen in real-life applications. GN-ETA
is able to handle web-scale applications like Google Maps. Recently, the
follow-up work CompactETA (Fu et al., 2020) proposes to encode temporal
information using positional encoding instead of recurrent networks to further
accelerate the model.

Overall, spatio-temporal GNNs for dynamic graphs are extensions of static GNNs
with special treatment for the temporal aspect. Traditional methods usually
employ recurrent neural networks to capture the evolving nature of networks
over time, but are usually time-consuming and fail to capture the long-term
dependency. Recent advances to use positional encoding and neural ODEs, etc
have shown promising results in speeding up dynamic GNNs. The works discussed
above accelerate spatio-temporal GNNs in various ways to scale up them to
computation-heavy dynamic system modeling and web-scale applications.

The proposed hardware accelerators in Section 6.2 have demonstrated the
capability to support various graph topologies, provided that the graph is
within the scale specified by the accelerator. This is attributed to the
implementation of preprocessing techniques, such as graph re-ordering and
tiling, within these works. However, challenges arise in supporting dynamic
and heterogeneous graphs when the algorithm utilized to process them is
altered. To address this issue without necessitating a complete redesign of
the accelerator for each algorithmic change, software-hardware co-design can
be extended. On the software front, an analyzer can be developed to evaluate
the requirements of the graph and target algorithm and determine the optimal
partitioning, mapping, and scheduling of operations to enable efficient
execution on the hardware. On the hardware side, options include moving
towards more general design architectures or supporting multiple specialized
architectures that the software component can select based on the application
requirements.

##  8\. Future Directions

While significant progress has been achieved, the investigation into GNN
acceleration remains in its early stages and has not kept pace with the rapid
expansion of graph data. Numerous promising research avenues exist within this
field, particularly in terms of acceleration from a COTS system and customized
hardware perspective, which have only recently begun to garner attention.
Additionally, there is a need to explore acceleration techniques tailored for
heterogeneous and dynamic graphs. This section highlights several captivating
research directions that warrant further exploration.

Better Graph Modification Algorithms Improving the efficiency of graph
modification methods, either coarsening, sparsification, or condensation,
represents a promising avenue for future research. In many graph datasets,
nodes often have an excessive number of neighbors, which may provide redundant
information or even noise. To accelerate GNN training, it may be possible to
remove these neighbors more aggressively at the pre-processing stage. For
example, through sparsification without preserving the global graph properties
as practical GNNs are trained for preserving locality. Graph condensation is
another promising technique that can enhance training efficiency, but it
currently requires running the training process at least once to identify the
best condensed graph. If the optimal graph can be determined earlier, e.g.,
identify a good condensed graph by training only a few epochs, this technique
would become much more useful. Good graph modification methods can also
benefit the automating designing and tuning GNN architectures. In practice,
preprocessed simpler graphs will significantly accelerate the whole pipeline
of GNN development.

Better Sampling Algorithms There are still several potential improvements for
future research in sampling methods for GNNs. One potential direction is to
investigate hierarchical sampling methods that can exploit the multi-scale
structure of graphs to improve sampling efficiency and capture multi-scale
features of graphs. Hierarchical sampling can be particularly useful for
large-scale graphs with complex structures, such as social networks, where
nodes can be organized into clusters or communities at different levels.
Another possibility is to further investigate the sampling method for dynamic
graphs, where nodes and edges change over time. Individually applying sampling
to dynamic graphs at each timestamp is still not very efficient and fails to
capture the temporal dynamics of nodes and edges on dynamic graphs.

Combined Inference Acceleration Algorithms Inference acceleration methods,
such as pruning, quantization, and distillation, are often orthogonal to each
other and can be combined to further improve the speed. Some initial steps
have already been taken, such as leveraging distillation when ambitiously
quantizing weights to binary. As the inference stage is more flexible
training, e.g., no concerns like training stability and model convergence,
these methods can be more easily combined in various ways. For instance,
pruning and quantization can be combined to achieve a sparse model with low
precision. Additionally, pruning can be used in conjunction with distillation
to further refine the distilled student model.

COTS Systems Existing work mainly focuses on improving the run-time efficiency
of GNNs. In contrast, the memory requirement can often be the bottleneck of
accelerating GNNs on large-scale data, which has not been well studied yet.
Various techniques including rematerialization, quantization, and data pruning
can be considered for reducing the memory overhead. In addition, because most
GNN models have irregular computation and data access patterns, the advanced
training/inference algorithms can be system unfriendly. This makes system-
algorithm co-design for system-friendly GNN algorithms a promising research
direction. Finally, full-stack development for GNN systems is still missing in
the existing literature. Combining existing techniques at different levels
(e.g., algorithm, system, hardware) towards an ultra-efficient GNN system is
another interesting research topic.

Customized Hardware As explained in Section 6, memory access presents a
significant bottleneck in GNN acceleration. Thus, a key aspect of accelerating
the processing of GNNs is to minimize the amount of data that must be fetched
from off-chip memory. In addition to the techniques employed by the works
reviewed, such as data reuse, on-chip memory caching, and dataflow
optimization, exploring approaches such as pruning, quantization, and
compression may reduce the volume of data fetched from off-chip memory.
Furthermore, a critical requirement is to develop a more advanced automated
design generator that facilitates GNN accelerator design. Techniques such as
Neural Architecture Search (NAS)(Lin et al., 2021), learning-based models for
accelerated hardware design evaluation(Sohrabizadeh et al., 2022a), and
intelligent design space exploration to navigate through design
candidates(Sohrabizadeh et al., 2022c; Mirhoseini et al., 2021) can be
suitably adapted for GNN accelerator design.

##  9\. Conclusion

In conclusion, we have presented a systematic view of acceleration methods for
GNNs on large-scale datasets. Our survey covers GNN training and inference
algorithms, efficient GNN systems, and customized hardware for GNNs.
Additionally, we have discussed the special cases when GNNs are applied to
heterogeneous graphs and dynamic graphs. Acceleration methods pave the way for
more efficient GNN pipelines, enabling the use of larger GNN models for
solving more complex tasks on graphs.

###### Acknowledgements.

We thank Linghao Song for the helpful discussions and comments throughout this
work. We thank Xiang Song and Kezhao Huang for providing helpful feedback on
an early draft of the paper. This work is partially supported by NSF (2211557,
1937599, 2119643), NASA, SRC, Okawa Foundation Grant, Amazon Research Awards,
Cisco Research Grant, Picsart Gifts, and Snapchat Gifts.

## References

  * (1)
  * Abadal et al. (2021) Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, and Eduard Alarcón. 2021\.  Computing graph neural networks: A survey from algorithms to accelerators.  _ACM Computing Surveys (CSUR)_ 54, 9 (2021), 1–38.
  * Alon and Yahav (2021) Uri Alon and Eran Yahav. 2021.  On the Bottleneck of Graph Neural Networks and its Practical Implications. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=i80OPhOCVH2
  * Bahri et al. (2021) Mehdi Bahri, Gaétan Bahl, and Stefanos Zafeiriou. 2021\.  Binary graph neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9492–9501.
  * Bai et al. (2021) Jiyang Bai, Yuxiang Ren, and Jiawei Zhang. 2021.  Ripple walk training: A subgraph-based training framework for large and deep graph neural network. In _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Beaini et al. (2021) Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. 2021.  Directional graph networks. In _International Conference on Machine Learning_. PMLR, 748–758.
  * Benczúr and Karger (1996) András A Benczúr and David R Karger. 1996.  Approximating st minimum cuts in Õ (n 2) time. In _Proceedings of the twenty-eighth annual ACM symposium on Theory of computing_. 47–55.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013\.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ (2013).
  * Bravo Hermsdorff and Gunderson (2019) Gecia Bravo Hermsdorff and Lee Gunderson. 2019.  A unifying framework for spectrum-preserving graph sparsification and coarsening.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006\.  Model compression. In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_. 535–541.
  * Cai et al. (2021) Chen Cai, Dingkang Wang, and Yusu Wang. 2021.  Graph Coarsening with Neural Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=uxpzitPEooJ
  * Cai et al. (2023) Zhenkun Cai, Qihui Zhou, Xiao Yan, Da Zheng, Xiang Song, Chenguang Zheng, James Cheng, and George Karypis. 2023\.  DSP: Efficient GNN Training with Multiple GPUs. In _Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming_. 392–404.
  * Chen et al. (2018a) Jie Chen, Tengfei Ma, and Cao Xiao. 2018a.  Fastgcn: fast learning with graph convolutional networks via importance sampling.  _arXiv preprint arXiv:1801.10247_ (2018).
  * Chen et al. (2017) Jianfei Chen, Jun Zhu, and Le Song. 2017.  Stochastic training of graph convolutional networks with variance reduction.  _arXiv preprint arXiv:1710.10568_ (2017).
  * Chen et al. (2020a) Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020a.  Simple and Deep Graph Convolutional Networks. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, 1725–1735.  https://proceedings.mlr.press/v119/chen20v.html
  * Chen et al. (2018b) Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018b.  $\\{$TVM$\\}$: An automated $\\{$End-to-End$\\}$ optimizing compiler for deep learning. In _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_. 578–594.
  * Chen et al. (2021b) Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. 2021b.  A unified lottery ticket hypothesis for graph neural networks. In _International Conference on Machine Learning_. PMLR, 1695–1706.
  * Chen et al. (2021c) Xiaobing Chen, Yuke Wang, Xinfeng Xie, Xing Hu, Abanti Basak, Ling Liang, Mingyu Yan, Lei Deng, Yufei Ding, Zidong Du, et al. 2021c.  Rubik: A hierarchical architecture for efficient graph neural network training.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ (2021).
  * Chen et al. (2021a) Yuzhao Chen, Yatao Bian, Xi Xiao, Yu Rong, Tingyang Xu, and Junzhou Huang. 2021a.  On Self-Distilling Graph Neural Network. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 2278–2284.  https://doi.org/10.24963/ijcai.2021/314 Main Track.
  * Chen et al. (2016) Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016.  Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks.  _ACM SIGARCH Computer Architecture News_ 44, 3 (2016), 367–379.
  * Chen et al. (2020b) Zhaodong Chen, Mingyu Yan, Maohua Zhu, Lei Deng, Guoqi Li, Shuangchen Li, and Yuan Xie. 2020b.  fuseGNN: accelerating graph convolutional neural network training on GPGPU. In _2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 1–9.
  * Chiang et al. (2019) Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. 2019\.  Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 257–266.
  * Ching et al. (2015) Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, and Sambavi Muthukrishnan. 2015\.  One trillion edges: Graph processing at facebook-scale.  _Proceedings of the VLDB Endowment_ 8, 12 (2015), 1804–1815.
  * Cong et al. (2020) Weilin Cong, Rana Forsati, Mahmut Kandemir, and Mehrdad Mahdavi. 2020.  Minimal variance sampling with provable guarantees for fast training of graph neural networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1393–1403.
  * Corso et al. (2020) Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. 2020\.  Principal neighbourhood aggregation for graph nets.  _Advances in Neural Information Processing Systems_ 33 (2020), 13260–13271.
  * Dauphin et al. (2017) Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017.  Language modeling with gated convolutional networks. In _International conference on machine learning_. PMLR, 933–941.
  * Deng et al. (2020) Chenhui Deng, Zhiqiang Zhao, Yongyu Wang, Zhiru Zhang, and Zhuo Feng. 2020.  GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=r1lGO0EKDH
  * Deng and Zhang (2021) Xiang Deng and Zhongfei Zhang. 2021.  Graph-Free Knowledge Distillation for Graph Neural Networks.  arXiv:2105.07519 [cs.LG]
  * Derrow-Pinion et al. (2021) Austin Derrow-Pinion, Jennifer She, David Wong, Oliver Lange, Todd Hester, Luis Perez, Marc Nunkesser, Seongjae Lee, Xueying Guo, Brett Wiltshire, et al. 2021\.  Eta prediction with graph neural networks in google maps. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3767–3776.
  * Dong et al. (2021) Jialin Dong, Da Zheng, Lin F Yang, and Geroge Karypis. 2021\.  Global neighbor sampling for mixed CPU-GPU training on giant graphs.  _arXiv preprint arXiv:2106.06150_ (2021).
  * Dong et al. (2020) Yuxiao Dong, Ziniu Hu, Kuansan Wang, Yizhou Sun, and Jie Tang. 2020.  Heterogeneous Network Representation Learning.. In _IJCAI_ , Vol. 20. 4861–4867.
  * Dou et al. (2020) Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S. Yu. 2020\.  Enhancing Graph Neural Network-Based Fraud Detectors against Camouflaged Fraudsters. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ (Virtual Event, Ireland) _(CIKM ’20)_. Association for Computing Machinery, New York, NY, USA, 315–324.  https://doi.org/10.1145/3340531.3411903
  * Esser et al. (2019) Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2019\.  Learned step size quantization.  _arXiv preprint arXiv:1902.08153_ (2019).
  * Feng et al. (2020) Boyuan Feng, Yuke Wang, Xu Li, Shu Yang, Xueqiao Peng, and Yufei Ding. 2020.  Sgquant: Squeezing the last bit on graph neural networks with specialized quantization. In _2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)_. IEEE, 1044–1052.
  * Fey and Lenssen (2019) Matthias Fey and Jan E. Lenssen. 2019.  Fast Graph Representation Learning with PyTorch Geometric. In _ICLR Workshop on Representation Learning on Graphs and Manifolds_.
  * Fey et al. (2021) Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. 2021.  Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In _International Conference on Machine Learning_. PMLR, 3294–3304.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ (2018).
  * Fu et al. (2020) Kun Fu, Fanlin Meng, Jieping Ye, and Zheng Wang. 2020\.  CompactETA: A Fast Inference System for Travel Time Prediction. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3337–3345.
  * Fu et al. (2022) Qiang Fu, Yuede Ji, and H Howie Huang. 2022.  TLPGNN: A Lightweight Two-Level Parallelism Paradigm for Graph Neural Network Computation on GPU. In _Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing_. 122–134.
  * Gandhi and Iyer (2021) Swapnil Gandhi and Anand Padmanabha Iyer. 2021.  P3: Distributed deep graph learning at scale. In _15th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 21)_. 551–568.
  * Geng et al. (2020) Tong Geng, Ang Li, Runbin Shi, Chunshu Wu, Tianqi Wang, Yanfei Li, Pouya Haghi, Antonino Tumeo, Shuai Che, Steve Reinhardt, et al. 2020\.  AWB-GCN: A graph convolutional network accelerator with runtime workload rebalancing. In _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 922–936.
  * Geng et al. (2021) Tong Geng, Chunshu Wu, Yongan Zhang, Cheng Tan, Chenhao Xie, Haoran You, Martin Herbordt, Yingyan Lin, and Ang Li. 2021.  I-GCN: A graph convolutional network accelerator with runtime locality enhancement through islandization. In _MICRO-54: 54th annual IEEE/ACM international symposium on microarchitecture_. 1051–1063.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017\.  Neural message passing for quantum chemistry. In _International conference on machine learning_. PMLR, 1263–1272.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks. In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_. 855–864.
  * Gupta et al. (2015) Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015.  Deep learning with limited numerical precision. In _International conference on machine learning_. PMLR, 1737–1746.
  * Hamilton et al. (2017a) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a.  Inductive representation learning on large graphs. In _Advances in neural information processing systems_. 1024–1034.
  * Hamilton et al. (2017b) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017b.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ 30 (2017).
  * Han et al. (2015a) Song Han, Huizi Mao, and William J Dally. 2015a.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _arXiv preprint arXiv:1510.00149_ (2015).
  * Han et al. (2015b) Song Han, Jeff Pool, John Tran, and William J. Dally. 2015b.  Learning Both Weights and Connections for Efficient Neural Networks. In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ (Montreal, Canada) _(NIPS’15)_. MIT Press, Cambridge, MA, USA, 1135–1143.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the Knowledge in a Neural Network.  arXiv:1503.02531 [stat.ML]
  * Hu et al. (2021) Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. 2021\.  OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs.  _arXiv preprint arXiv:2103.09430_ (2021).
  * Hu et al. (2020b) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020b.  Open Graph Benchmark: Datasets for Machine Learning on Graphs.  _CoRR_ abs/2005.00687 (2020).  arXiv:2005.00687 https://arxiv.org/abs/2005.00687
  * Hu et al. (2020c) Yuwei Hu, Zihao Ye, Minjie Wang, Jiali Yu, Da Zheng, Mu Li, Zheng Zhang, Zhiru Zhang, and Yida Wang. 2020c.  Featgraph: A flexible and efficient backend for graph neural network systems. In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_. IEEE, 1–13.
  * Hu et al. (2020a) Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020a.  Heterogeneous graph transformer. In _Proceedings of The Web Conference 2020_. 2704–2710.
  * Huang et al. (2018) Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. 2018\.  Adaptive sampling towards fast graph representation learning.  _Advances in neural information processing systems_ 31 (2018).
  * Huang et al. (2022) Zijie Huang, Zheng Li, Haoming Jiang, Tianyu Cao, Hanqing Lu, Bing Yin, Karthik Subbian, Yizhou Sun, and Wei Wang. 2022.  Multilingual Knowledge Graph Completion with Self-Supervised Adaptive Graph Alignment. In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Huang et al. (2020) Zijie Huang, Yizhou Sun, and Wei Wang. 2020.  Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations. In _Advances in Neural Information Processing Systems_.
  * Huang et al. (2021a) Zijie Huang, Yizhou Sun, and Wei Wang. 2021a.  Coupled Graph ODE for Learning Interacting System Dynamics. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_.
  * Huang et al. (2021b) Zengfeng Huang, Shengzhong Zhang, Chong Xi, Tang Liu, and Min Zhou. 2021b.  Scaling Up Graph Neural Networks Via Graph Coarsening. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery amp; Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 675–684.  https://doi.org/10.1145/3447548.3467256
  * Huawei (2020) Huawei. 2020.  _MindSpore_.  https://e.huawei.com/en/products/cloud-computing-dc/atlas/mindspore
  * Hubara et al. (2018) Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2018\.  Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations.  _Journal of Machine Learning Research_ 18, 187 (2018), 1–30.  http://jmlr.org/papers/v18/16-456.html
  * Javari et al. (2020) Amin Javari, Zhankui He, Zijie Huang, J. Jacob Durai Raj, and Kevin Chen-Chuan Chang. 2020.  Weakly Supervised Attention for Hashtag Recommendation using Graph Data.  _Proceedings of The Web Conference 2020_ (2020).
  * Jia et al. (2020a) Zhihao Jia, Sina Lin, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2020a.  Improving the accuracy, scalability, and performance of graph neural networks with roc.  _Proceedings of Machine Learning and Systems_ 2 (2020), 187–198.
  * Jia et al. (2020b) Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. 2020b.  Redundancy-free computation for graph neural networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 997–1005.
  * Jiang et al. (2021) Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. 2021.  Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models.  _Journal of cheminformatics_ 13, 1 (2021), 1–23.
  * Jiang and Luo (2022) Weiwei Jiang and Jiayun Luo. 2022.  Graph neural network for traffic forecasting: A survey.  _Expert Systems with Applications_ (2022), 117921.
  * Jin et al. (2022) Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, and Bing Yin. 2022.  Condensing Graphs via One-Step Gradient Matching. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 720–730.
  * Jin et al. (2021) Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. 2021\.  Graph Condensation for Graph Neural Networks.  _arXiv preprint arXiv:2110.07580_ (2021).
  * Kaler et al. (2022) Tim Kaler, Nickolas Stathas, Anne Ouyang, Alexandros-Stavros Iliopoulos, Tao Schardl, Charles E Leiserson, and Jie Chen. 2022\.  Accelerating training and inference of graph neural networks with fast sampling and pipelining.  _Proceedings of Machine Learning and Systems_ 4 (2022), 172–189.
  * Karypis and Kumar (1998) George Karypis and Vipin Kumar. 1998.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs.  _SIAM J. Sci. Comput._ 20, 1 (1998), 359–392.  https://doi.org/10.1137/S1064827595287997
  * Kernighan and Lin (1970) Brian W Kernighan and Shen Lin. 1970.  An efficient heuristic procedure for partitioning graphs.  _The Bell system technical journal_ 49, 2 (1970), 291–307.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Krishnamoorthi (2018) Raghuraman Krishnamoorthi. 2018\.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  _arXiv preprint arXiv:1806.08342_ (2018).
  * Li et al. (2021) Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 2021.  Training graph neural networks with 1000 layers. In _International conference on machine learning_. PMLR, 6437–6449.
  * Li et al. (2020) Jiayu Li, Tianyun Zhang, Hao Tian, Shengmin Jin, Makan Fardad, and Reza Zafarani. 2020\.  Sgcn: A graph sparsifier based on graph convolutional networks. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 275–287.
  * Liang et al. (2020a) Shengwen Liang, Cheng Liu, Ying Wang, Huawei Li, and Xiaowei Li. 2020a.  Deepburning-gl: an automated framework for generating graph neural network accelerators. In _2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 1–9.
  * Liang et al. (2020b) Shengwen Liang, Ying Wang, Cheng Liu, Lei He, LI Huawei, Dawen Xu, and Xiaowei Li. 2020b.  Engn: A high-throughput and energy-efficient accelerator for large graph neural networks.  _IEEE Trans. Comput._ 70, 9 (2020), 1511–1525.
  * Lin et al. (2022) Haiyang Lin, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Shirui Pan, Wenguang Chen, and Yuan Xie. 2022.  A Comprehensive Survey on Distributed Training of Graph Neural Networks.  _arXiv preprint arXiv:2211.05368_ (2022).
  * Lin et al. (2021) Yujun Lin, Mengtian Yang, and Song Han. 2021.  Naas: Neural accelerator architecture search. In _2021 58th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 1051–1056.
  * Lin et al. (2020) Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu. 2020.  Pagraph: Scaling gnn training on large graphs via computation-aware caching. In _Proceedings of the 11th ACM Symposium on Cloud Computing_. 401–415.
  * Liu et al. (2021a) Tianfeng Liu, Yangrui Chen, Dan Li, Chuan Wu, Yibo Zhu, Jun He, Yanghua Peng, Hongzheng Chen, Hongzhi Chen, and Chuanxiong Guo. 2021a.  Bgl: Gpu-efficient gnn training by optimizing graph data i/o and preprocessing.  _arXiv preprint arXiv:2112.08541_ (2021).
  * Liu et al. (2021b) Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, and Dongrui Fan. 2021b.  Sampling methods for efficient training of graph convolutional networks: A survey.  _IEEE/CAA Journal of Automatica Sinica_ 9, 2 (2021), 205–234.
  * Liu et al. (2020) Ziqi Liu, Zhengwei Wu, Zhiqiang Zhang, Jun Zhou, Shuang Yang, Le Song, and Yuan Qi. 2020.  Bandit samplers for training graph neural networks.  _Advances in Neural Information Processing Systems_ 33 (2020), 6878–6888.
  * Long et al. (2021) Qingqing Long, Lingjun Xu, Zheng Fang, and Guojie Song. 2021\.  HGK-GNN: Heterogeneous Graph Kernel Based Graph Neural Networks. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 1129–1138.  https://doi.org/10.1145/3447548.3467429
  * Loukas (2019) Andreas Loukas. 2019\.  Graph Reduction with Spectral and Cut Guarantees.  _J. Mach. Learn. Res._ 20, 116 (2019), 1–42.
  * Loukas and Vandergheynst (2018) Andreas Loukas and Pierre Vandergheynst. 2018.  Spectrally Approximating Large Graphs with Smaller Graphs. In _Proceedings of the 35th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer Dy and Andreas Krause (Eds.). PMLR, 3237–3246.  https://proceedings.mlr.press/v80/loukas18a.html
  * Ma et al. (2019a) Lingxiao Ma, Zhi Yang, Youshan Miao, Jilong Xue, Ming Wu, Lidong Zhou, and Yafei Dai. 2019a.  $\\{$NeuGraph$\\}$: Parallel Deep Neural Network Computation on Large Graphs. In _2019 USENIX Annual Technical Conference (USENIX ATC 19)_. 443–458.
  * Ma et al. (2019b) Lingxiao Ma, Zhi Yang, Youshan Miao, Jilong Xue, Ming Wu, Lidong Zhou, and Yafei Dai. 2019b.  $\\{$NeuGraph$\\}$: Parallel Deep Neural Network Computation on Large Graphs. In _2019 USENIX Annual Technical Conference (USENIX ATC 19)_. 443–458.
  * Manning (2009) Christopher D Manning. 2009\.  _An introduction to information retrieval_.  Cambridge university press.
  * Mao et al. (2021) Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. 2021\.  UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 1253–1262.
  * Marcheggiani and Titov (2017) Diego Marcheggiani and Ivan Titov. 2017.  Encoding sentences with graph convolutional networks for semantic role labeling.  _arXiv preprint arXiv:1703.04826_ (2017).
  * Mirhoseini et al. (2021) Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. 2021\.  A graph placement methodology for fast chip design.  _Nature_ 594, 7862 (2021), 207–212.
  * Mislove et al. (2007) Alan Mislove, Massimiliano Marcon, Krishna P Gummadi, Peter Druschel, and Bobby Bhattacharjee. 2007\.  Measurement and analysis of online social networks. In _Proceedings of the 7th ACM SIGCOMM conference on Internet measurement_. 29–42.
  * Mohoney et al. (2021) Jason Mohoney, Roger Waleffe, Henry Xu, Theodoros Rekatsinas, and Shivaram Venkataraman. 2021\.  Marius: Learning Massive Graph Embeddings on a Single Machine. In _15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)_. USENIX Association, 533–549.  https://www.usenix.org/conference/osdi21/presentation/mohoney
  * Mostafa (2022) Hesham Mostafa. 2022\.  Sequential aggregation and rematerialization: Distributed full-batch training of graph neural networks on large graphs.  _Proceedings of Machine Learning and Systems_ 4 (2022), 265–275.
  * Nagel et al. (2021) Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021.  A white paper on neural network quantization.  _arXiv preprint arXiv:2106.08295_ (2021).
  * Peng et al. (2022) Jingshu Peng, Zhao Chen, Yingxia Shao, Yanyan Shen, Lei Chen, and Jiannong Cao. 2022\.  Sancus: staleness-aware communication-avoiding full-graph decentralized training in large-scale graph neural networks.  _Proceedings of the VLDB Endowment_ 15, 9 (2022), 1937–1950.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. 701–710.
  * Rahman et al. (2021) Md Khaledur Rahman, Majedul Haque Sujon, and Ariful Azad. 2021\.  Fusedmm: A unified sddmm-spmm kernel for graph embedding and graph neural networks. In _2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. IEEE, 256–266.
  * Ramezani et al. (2022) Morteza Ramezani, Weilin Cong, Mehrdad Mahdavi, Mahmut Kandemir, and Anand Sivasubramaniam. 2022\.  Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=FndDxSz3LxQ
  * Reed (1993) Russell Reed. 1993\.  Pruning algorithms-a survey.  _IEEE transactions on Neural Networks_ 4, 5 (1993), 740–747.
  * Riesen and Bunke (2008) Kaspar Riesen and Horst Bunke. 2008.  IAM graph database repository for graph based pattern recognition and machine learning. In _Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)_. Springer, 287–297.
  * Saad (2003) Yousef Saad. 2003\.  _Iterative methods for sparse linear systems_.  SIAM.
  * Salehinejad et al. (2017) Hojjat Salehinejad, Sharan Sankar, Joseph Barfett, Errol Colak, and Shahrokh Valaee. 2017.  Recent advances in recurrent neural networks.  _arXiv preprint arXiv:1801.01078_ (2017).
  * Sanchez-Gonzalez et al. (2020) Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. 2020.  Learning to simulate complex physics with graph networks. In _International Conference on Machine Learning_. PMLR, 8459–8468.
  * Sarkar et al. (2022) Rishov Sarkar, Stefan Abi-Karam, Yuqi He, Lakshmi Sathidevi, and Cong Hao. 2022.  FlowGNN: A Dataflow Architecture for Universal Graph Neural Network Inference via Multi-Queue Streaming.  _arXiv preprint arXiv:2204.13103_ (2022).
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _European semantic web conference_. Springer, 593–607.
  * Sen et al. (2008a) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008a.  Collective classification in network data.  _AI magazine_ 29, 3 (2008), 93–93.
  * Sen et al. (2008b) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008b.  Collective classification in network data.  _AI magazine_ 29, 3 (2008), 93–93.
  * Serafini and Guan (2021) Marco Serafini and Hui Guan. 2021.  Scalable Graph Neural Network Training: The Case for Sampling.  _ACM SIGOPS Operating Systems Review_ 55, 1 (2021), 68–76.
  * Serrano et al. (2009) M Ángeles Serrano, Marián Boguná, and Alessandro Vespignani. 2009.  Extracting the multiscale backbone of complex weighted networks.  _Proceedings of the national academy of sciences_ 106, 16 (2009), 6483–6488.
  * Shao et al. (2022) Yingxia Shao, Hongzheng Li, Xizhi Gu, Hongbo Yin, Yawen Li, Xupeng Miao, Wentao Zhang, Bin Cui, and Lei Chen. 2022.  Distributed Graph Neural Network Training: A Survey.  _arXiv preprint arXiv:2211.00216_ (2022).
  * Sohrabizadeh et al. (2022a) Atefeh Sohrabizadeh, Yunsheng Bai, Yizhou Sun, and Jason Cong. 2022a.  Automated Accelerator Optimization Aided by Graph Neural Networks. In _2022 59th ACM/IEEE Design Automation Conference (DAC)_.
  * Sohrabizadeh et al. (2022b) Atefeh Sohrabizadeh, Yuze Chi, and Jason Cong. 2022b.  StreamGCN: Accelerating Graph Convolutional Networks with Streaming Processing. In _2022 IEEE Custom Integrated Circuits Conference (CICC)_. IEEE, 1–8.
  * Sohrabizadeh et al. (2020) Atefeh Sohrabizadeh, Jie Wang, and Jason Cong. 2020.  End-to-End Optimization of Deep Learning Applications. In _FPGA_. 133–139.
  * Sohrabizadeh et al. (2022c) Atefeh Sohrabizadeh, Cody Hao Yu, Min Gao, and Jason Cong. 2022c.  AutoDSE: Enabling Software Programmers to Design Efficient FPGA Accelerators.  _ACM Transactions on Design Automation of Electronic Systems (TODAES)_ 27, 4 (2022), 1–27.
  * Song et al. (2022) Linghao Song, Yuze Chi, Atefeh Sohrabizadeh, Young-kyu Choi, Jason Lau, and Jason Cong. 2022\.  Sextans: A streaming accelerator for general-purpose sparse-matrix dense-matrix multiplication. In _Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_. 65–77.
  * Spielman and Srivastava (2011) Daniel A Spielman and Nikhil Srivastava. 2011.  Graph sparsification by effective resistances.  _SIAM J. Comput._ 40, 6 (2011), 1913–1926.
  * Spielman and Teng (2014) Daniel A Spielman and Shang-Hua Teng. 2014.  Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems.  _SIAM J. Matrix Anal. Appl._ 35, 3 (2014), 835–885.
  * Srivastava et al. (2020) Nitish Srivastava, Hanchen Jin, Jie Liu, David Albonesi, and Zhiru Zhang. 2020.  Matraptor: A sparse-sparse matrix multiplication accelerator based on row-wise product. In _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 766–780.
  * Tailor et al. (2020) Shyam A Tailor, Javier Fernandez-Marques, and Nicholas D Lane. 2020\.  Degree-quant: Quantization-aware training for graph neural networks.  _arXiv preprint arXiv:2008.05000_ (2020).
  * Thorpe et al. (2021) John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, et al. 2021\.  Dorylus: Affordable, Scalable, and Accurate $\\{$GNN$\\}$ Training with Distributed $\\{$CPU$\\}$ Servers and Serverless Threads. In _15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)_. 495–514.
  * Tian et al. (2020) Chao Tian, Lingxiao Ma, Zhi Yang, and Yafei Dai. 2020\.  Pcgcn: Partition-centric processing for accelerating graph convolutional network. In _2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. IEEE, 936–945.
  * Tripathy et al. (2020) Alok Tripathy, Katherine Yelick, and Aydın Buluç. 2020\.  Reducing communication in graph neural network training. In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_. IEEE, 1–14.
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Von Luxburg (2007) Ulrike Von Luxburg. 2007\.  A tutorial on spectral clustering.  _Statistics and computing_ 17, 4 (2007), 395–416.
  * Waleffe et al. (2022) Roger Waleffe, Jason Mohoney, Theodoros Rekatsinas, and Shivaram Venkataraman. 2022.  MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks.  _arXiv preprint arXiv:2202.02365_ (2022).
  * Wan et al. (2022a) Cheng Wan, Youjie Li, Ang Li, Nam Sung Kim, and Yingyan Lin. 2022a.  BNS-GCN: Efficient full-graph training of graph convolutional networks with partition-parallelism and random boundary node sampling.  _Proceedings of Machine Learning and Systems_ 4 (2022).
  * Wan et al. (2022b) Cheng Wan, Youjie Li, Cameron R Wolfe, Anastasios Kyrillidis, Nam Sung Kim, and Yingyan Lin. 2022b.  PipeGCN: Efficient full-graph training of graph convolutional networks with pipelined feature communication.  _arXiv preprint arXiv:2203.10428_ (2022).
  * Wang et al. (2021b) Jie Wang, Licheng Guo, and Jason Cong. 2021b.  Autosa: A polyhedral compiler for high-performance systolic arrays on fpga. In _The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_. 93–104.
  * Wang et al. (2019b) Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2019b.  Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.  _arXiv preprint arXiv:1909.01315_ (2019).
  * Wang et al. (2021c) Ruijie Wang, Zijie Huang, Shengzhong Liu, Huajie Shao, Dongxin Liu, Jinyang Li, Tianshi Wang, Dachun Sun, Shuochao Yao, and Tarek Abdelzaher. 2021c.  Dydiff-vae: A dynamic variational framework for information diffusion prediction. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 163–172.
  * Wang et al. (2021a) Yuke Wang, Boyuan Feng, Gushu Li, Shuangchen Li, Lei Deng, Yuan Xie, and Yufei Ding. 2021a.  $\\{$GNNAdvisor$\\}$: An Adaptive and Efficient Runtime System for $\\{$GNN$\\}$ Acceleration on $\\{$GPUs$\\}$. In _15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)_. 515–531.
  * Wang et al. (2019a) Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2019a.  Dynamic graph cnn for learning on point clouds.  _Acm Transactions On Graphics (tog)_ 38, 5 (2019), 1–12.
  * Wu et al. (2023) Kun Wu, Mert Hidayetoğlu, Xiang Song, Sitao Huang, Da Zheng, Israt Nisa, and Wen-mei Hwu. 2023.  PIGEON: Optimizing CUDA Code Generator for End-to-End Training and Inference of Relational Graph Neural Networks.  _arXiv preprint arXiv:2301.06284_ (2023).
  * Wu et al. (2020b) Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2020b.  Graph neural networks in recommender systems: a survey.  _ACM Computing Surveys (CSUR)_ (2020).
  * Wu et al. (2021) Yidi Wu, Kaihao Ma, Zhenkun Cai, Tatiana Jin, Boyang Li, Chenguang Zheng, James Cheng, and Fan Yu. 2021\.  Seastar: vertex-centric programming for graph neural networks. In _Proceedings of the Sixteenth European Conference on Computer Systems_. 359–375.
  * Wu et al. (2020a) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020a.  Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020\.  Self-training with noisy student improves imagenet classification. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10687–10698.
  * Xie et al. (2022) Zhiqiang Xie, Minjie Wang, Zihao Ye, Zheng Zhang, and Rui Fan. 2022.  Graphiler: Optimizing Graph Neural Networks with Message Passing Data Flow Graph.  _Proceedings of Machine Learning and Systems_ 4 (2022), 515–528.
  * Xu et al. (2018a) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018a.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_ (2018).
  * Xu et al. (2018b) Zhongwen Xu, Hado P van Hasselt, and David Silver. 2018b.  Meta-gradient reinforcement learning.  _Advances in neural information processing systems_ 31 (2018).
  * Yan et al. (2020b) Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. 2020b.  TinyGNN: Learning Efficient Graph Neural Networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_. 1848–1856.
  * Yan et al. (2020a) Mingyu Yan, Lei Deng, Xing Hu, Ling Liang, Yujing Feng, Xiaochun Ye, Zhimin Zhang, Dongrui Fan, and Yuan Xie. 2020a.  Hygcn: A gcn accelerator with hybrid architecture. In _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 15–29.
  * Yang et al. (2021b) Cheng Yang, Jiawei Liu, and Chuan Shi. 2021b.  Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In _Proceedings of the Web Conference 2021_. 1227–1237.
  * Yang et al. (2022) Jianbang Yang, Dahai Tang, Xiaoniu Song, Lei Wang, Qiang Yin, Rong Chen, Wenyuan Yu, and Jingren Zhou. 2022\.  GNNLab: a factored system for sample-based GNN training over GPUs. In _Proceedings of the Seventeenth European Conference on Computer Systems_. 417–434.
  * Yang et al. (2021a) Shuwen Yang, Ziyao Li, Guojie Song, and Lingsheng Cai. 2021a.  Deep Molecular Representation Learning via Fusing Physical and Chemical Information.  _Advances in Neural Information Processing Systems_ 34 (2021), 16346–16357.
  * Yang et al. (2020) Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. 2020.  Distilling knowledge from graph convolutional networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7074–7083.
  * Yao and Li (2021) Kai-Lang Yao and Wu-Jun Li. 2021.  Blocking-based Neighbor Sampling for Large-scale Graph Neural Networks. In _International Joint Conference on Artificial Intelligence_.
  * Ye et al. (2022) Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. 2022.  SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning.  _arXiv preprint arXiv:2207.04606_ (2022).
  * Ying et al. (2018a) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018a.  Graph convolutional neural networks for web-scale recommender systems. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 974–983.
  * Ying et al. (2018b) Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018b.  Hierarchical graph representation learning with differentiable pooling.  _Advances in neural information processing systems_ 31 (2018).
  * You et al. (2022) Haoran You, Tong Geng, Yongan Zhang, Ang Li, and Yingyan Lin. 2022.  Gcod: Graph convolutional network acceleration via dedicated algorithm and accelerator co-design. In _2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 460–474.
  * You et al. (2021) Haoran You, Zhihan Lu, Zijian Zhou, Yonggan Fu, and Yingyan Lin. 2021.  Early-Bird GCNs: Graph-Network Co-Optimization Towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets.  _arXiv preprint arXiv:2103.00794_ (2021).
  * Yu et al. (2022) Haiyang Yu, Limei Wang, Bokun Wang, Meng Liu, Tianbao Yang, and Shuiwang Ji. 2022\.  Graphfm: Improving large-scale gnn training via feature momentum. In _International Conference on Machine Learning_. PMLR, 25684–25701.
  * Zeng and Prasanna (2020) Hanqing Zeng and Viktor Prasanna. 2020.  GraphACT: Accelerating GCN training on CPU-FPGA heterogeneous platforms. In _Proceedings of the 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_. 255–265.
  * Zeng et al. (2021a) Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, and Ren Chen. 2021a.  Decoupling the Depth and Scope of Graph Neural Networks.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Zeng et al. (2019) Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. 2019\.  Graphsaint: Graph sampling based inductive learning method.  _arXiv preprint arXiv:1907.04931_ (2019).
  * Zeng et al. (2021b) Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. 2021b.  Accurate, efficient and scalable training of Graph Neural Networks.  _J. Parallel and Distrib. Comput._ 147 (2021), 166–183.
  * Zhang et al. (2021a) Bingyi Zhang, Rajgopal Kannan, and Viktor Prasanna. 2021a.  BoostGCN: A framework for optimizing GCN inference on FPGA. In _2021 IEEE 29th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_. IEEE, 29–39.
  * Zhang et al. (2020b) Bingyi Zhang, Hanqing Zeng, and Viktor Prasanna. 2020b.  Hardware acceleration of large scale gcn inference. In _2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP)_. IEEE, 61–68.
  * Zhang et al. (2019b) Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. 2019b.  Heterogeneous Graph Neural Network. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ _(KDD ’19)_. 793–803.
  * Zhang et al. (2020a) Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xianzheng Song, Zhibang Ge, Zhiqiang Zhang, Lin Wang, Jun Zhou, Yang Shuang, et al. 2020a.  Agl: a scalable system for industrial-purpose graph machine learning.  _arXiv preprint arXiv:2003.02454_ (2020).
  * Zhang et al. (2022b) Hengrui Zhang, Zhongming Yu, Guohao Dai, Guyue Huang, Yufei Ding, Yuan Xie, and Yu Wang. 2022b.  Understanding gnn computational graph: A coordinated computation, io, and memory perspective.  _Proceedings of Machine Learning and Systems_ 4 (2022), 467–484.
  * Zhang et al. (2019a) Jiani Zhang, Xingjian Shi, Shenglin Zhao, and Irwin King. 2019a.  Star-gcn: Stacked and reconstructed graph convolutional networks for recommender systems.  _arXiv preprint arXiv:1905.13129_ (2019).
  * Zhang et al. (2022a) Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. 2022a.  Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=4p6_5HBWPCw
  * Zhang et al. (2018) Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin, Jinjun Xiong, Wen-mei Hwu, and Deming Chen. 2018.  DNNBuilder: an automated tool for building high-performance DNN hardware accelerators for FPGAs. In _ICCAD_. IEEE, 1–8.
  * Zhang et al. (2021b) Yongan Zhang, Haoran You, Yonggan Fu, Tong Geng, Ang Li, and Yingyan Lin. 2021b.  G-CoS: Gnn-accelerator co-search towards both better accuracy and efficiency. In _2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 1–9.
  * Zhao et al. (2020) Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik, and Pietro Lio. 2020\.  Learned low precision graph neural networks.  _arXiv preprint arXiv:2009.09232_ (2020).
  * Zheng et al. (2020b) Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. 2020b.  Robust graph representation learning via neural sparsification. In _International Conference on Machine Learning_. PMLR, 11458–11468.
  * Zheng et al. (2020a) Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, and George Karypis. 2020a.  Distdgl: distributed graph neural network training for billion-scale graphs. In _2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3)_. IEEE, 36–44.
  * Zheng et al. (2021b) Da Zheng, Xiang Song, Chengru Yang, Dominique LaSalle, Qidong Su, Minjie Wang, Chao Ma, and George Karypis. 2021b.  Distributed hybrid CPU and GPU training for graph neural networks on billion-scale graphs.  _arXiv preprint arXiv:2112.15345_ (2021).
  * Zheng et al. (2021a) Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian. 2021a.  Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods.  _arXiv preprint arXiv:2111.04840_ (2021).
  * Zhou et al. (2021b) Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng, Rajgopal Kannan, and Viktor Prasanna. 2021b.  Accelerating large scale real-time GNN inference using channel pruning.  _arXiv preprint arXiv:2105.04528_ (2021).
  * Zhou et al. (2022) Hongkuan Zhou, Da Zheng, Israt Nisa, Vasileios Ioannidis, Xiang Song, and George Karypis. 2022\.  TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs.  _arXiv preprint arXiv:2203.14883_ (2022).
  * Zhou et al. (2021a) Zhe Zhou, Bizhao Shi, Zhe Zhang, Yijin Guan, Guangyu Sun, and Guojie Luo. 2021a.  BlockGNN: Towards Efficient GNN Acceleration Using Block-Circulant Weight Matrices. In _2021 58th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 1009–1014.
  * Zhu et al. (2019) Rong Zhu, Kun Zhao, Hongxia Yang, Wei Lin, Chang Zhou, Baole Ai, Yong Li, and Jingren Zhou. 2019\.  Aligraph: a comprehensive graph neural network platform.  _arXiv preprint arXiv:1902.08730_ (2019).
  * Zitnik and Leskovec (2017) Marinka Zitnik and Jure Leskovec. 2017.  Predicting multicellular function through multi-layer tissue networks.  _Bioinformatics_ 33, 14 (2017), i190–i198.
  * Zou et al. (2019) Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. 2019\.  Layer-dependent importance sampling for training deep and large graph convolutional networks.  _Advances in neural information processing systems_ 32 (2019).