#  LSM-GNN: Large-scale Storage-based Multi-GPU GNN Training by Optimizing
Data Transfer Scheme

Jeongmin Brian Park  UIUCUSA<EMAIL_ADDRESS>,  Kun Wu  UIUCUSA
<EMAIL_ADDRESS>,  Vikram Sharma Mailthody  NVIDIAUSA
<EMAIL_ADDRESS>,  Zaid Qureshi  NVIDIAUSA<EMAIL_ADDRESS>,  Scott
Mahlke  NVIDIAUSA<EMAIL_ADDRESS>and  Wen-mei Hwu  NVIDIA/UIUCUSA
<EMAIL_ADDRESS>
###### Abstract.

Graph Neural Networks (GNNs) are widely used today in recommendation systems,
fraud detection, and node/link classification tasks. Real world GNNs continue
to scale in size and require a large memory footprint for storing graphs and
embeddings that often exceed the memory capacities of the target GPUs used for
training. To address limited memory capacities, traditional GNN training
approaches use graph partitioning and sharding techniques to scale up across
multiple GPUs within a node and/or scale out across multiple nodes. However,
this approach suffers from the high computational costs of graph partitioning
algorithms and inefficient communication across GPUs.

To address these overheads, we propose Large-scale Storage-based Multi-GPU GNN
framework (LSM-GNN), a storage-based approach to train GNN models that
utilizes a novel communication layer enabling GPU software caches to function
as a system-wide shared cache with low overheads. LSM-GNN incorporates a
hybrid eviction policy that intelligently manages cache space by using both
static and dynamic node information to significantly enhance cache
performance. Furthermore, we introduce the Preemptive Victim-buffer Prefetcher
(PVP), a mechanism for prefetching node feature data from a Victim Buffer
located in CPU pinned-memory to further reduce the pressure on the storage
devices. Experimental results show that despite the lower compute capabilities
and memory capacities, LSM-GNN in a single node with two GPUs offer superior
performance over two-node-four-GPU Dist-DGL baseline and provides up to
3.75$\times$ speed up on end-to-end epoch time while running large-scale GNN
training.

##  1\. INTRODUCTION

Graph neural networks (GNNs) have emerged as an effective paradigm for
learning rich relation and interaction information among input nodes and
edges, leading to improved generalization performance over traditional machine
learning techniques. Consequently, GNNs have gained significant traction in
recent years and demonstrated their efficacy in graph-based machine learning
applications, such as recommendation (gnn-recommendation, ; pinner_sage, ),
fraud detection (gnn_fraud, ; fdgar, ; gnn_fraud3, ; gnn_fraud4, ), node
classification (GCN, ; Graphsage, ; GAT, ; LazyGCN, ), and link prediction
(gnn_linkpredict, ; fewshot, ; LP_system, ).

To cater to this growing interest, new open-source frameworks such as PyTorch
Geometric (PyG) (pyg, ), Spektral (Spektral, ), and Deep Graph Library (DGL)
(dgl, ) have been developed to provide performant GNN execution. To increase
performance, they incorporate optimizations, such as message-passing for
aggregating feature information across related graph nodes, and graph-specific
neural network computation layers. These optimizations improve the speed of
all stages of GNN training (sampling, feature aggregation, and model training)
when the entire graph and embedding data can fit within a single GPU’s memory.

The challenge escalates with real-world graphs that can scale to massive
proportions, far exceeding the memory capacities of traditional systems. For
example, the Pinterest user-to-item graph contains over 2 billion nodes and 17
billion edges, totaling a data size of 18 TB (pinner_sage, ). Such sizes
render it impractical to load the entire graph and embeddings into the memory
of one GPU or even the host CPU. Sharding the graph and embeddings across many
CPU and/or GPU memories limits scalability. In a multi-GPU environment,
scalability is limited because each GPU needs to access most of its required
features from other GPU’s or CPU’s memories, resulting in high network
communication overhead.

To reduce communication overhead, state-of-the-art distributed GNN training
often partitions the graph with well-known graph partition algorithms like
METIS (Metis, ). However, graph partitioning algorithms are notoriously time-
consuming and their execution time increases exponentially with graph size
resulting in significant preprocessing overhead that must be amortized over
the training time. Furthermore, the graph partitioning algorithm consumes a
significant amount of memory for temporary data and can run out of memory
while processing large graphs even on high-memory capacity CPUs. For example,
when partitioning a medium-sized heterogeneous IGBH-medium (IGB, ) graph in
DistDGL (distdgl, ), it takes more than 220 GB of memory, surpassing 5 times
the size of the dataset. On a node with 4$\times$ AMD “Interlagos” CPUs,
partitioning IGBH-medium into four partitions via METIS takes more than 5
times the time of random partitioning, as detailed in Section 2.2.

An alternative approach is to use storage-based GNN training, where prior
works (GIDS, ; Ginex, ; MariusGNN, ) maintain graph data in storage and feed
sampled data to the GPU on demand. This method is cost-effective and obviates
the need for extensive additional resources dedicated to GNN training.
However, the feature aggregation stage is predominantly constrained by the
bandwidth of storage. To effectively scale storage-based GNN training in
multi-GPU environments, a proportional increase in the number of SSDs is
necessary to ensure adequate storage bandwidth for each GPU. However, it may
not be practical to increase the number of SSDs in many systems. Moreover,
altering the SSD configurations to accommodate system changes necessitates
either data replication or reformatting. Such operations introduce substantial
overhead, especially for large-scale graphs. Thus, a naive adaption of the
storage-based GNN training to multi-GPU systems is not practical.

To effectively support storage-based multi-GPU training, we introduce the
Large-scale Storage-based Multi-GPU GNN framework (LSM-GNN). LSM-GNN
accelerates the feature aggregation with limited storage bandwidth by
significantly improving hardware resource utilization, including GPU memory,
CPU memory, and interconnect bandwidth, in multi-GPU systems. LSM-GNN
leverages a novel communication layer that allows GPU software caches to
operate as a system-wide shared cache. This approach avoids the use of high-
overhead, system-scope operations, aiming to maximize collective cache
capacity and minimize redundant storage accesses while maintaining peak
software cache bandwidth.

Furthermore, LSM-GNN enhances GPU software cache utilization through a hybrid
eviction policy. This policy smartly evicts cold cache-lines by exploiting
both the normalized reverse page-rank value of each node (static information)
and the next reuse iteration (dynamic information). To gather dynamic
information, LSM-GNN pre-executes a configurable number of graph sampling
stages, thereby tracking the nodes sampled in subsequent iterations.

Finally, we introduce the novel Preemptive Victim-buffer Prefetcher (PVP) to
efficiently prefetch previously evicted node feature data. Within LSM-GNN,
when node feature data is evicted from the software cache, the cache line,
along with its dynamic information, is moved to a victim buffer in CPU memory.
It is then prefetched back to GPU memory as needed. Given that PVP operates
without requiring GPU resources and can be executed asynchronously, it
facilitates prefetching during the model training stage, a period when GPU
PCIe ingress bandwidth is significantly underutilized.

When compared to the previous distributed training approaches, the optimized
caching scheme in LSM-GNN enables each GPU to find more of its required graph
data in its GPU memory and/or local CPU memory (victim cache), which helps
improve the speed, efficiency, and scalability of multi-GPU GNN training
pipelines. Overall, we make the following key contributions:

  * •

We introduce an efficient large-scale storage-based multi-GPU GNN training
framework without the requirement of higher storage bandwidth.

  * •

We design a novel communication layer that orchestrates the GPU’s independent
software caches into a shared cache that further lowers the bandwidth pressure
on storage, without using the expensive (low-throughput) system-scope
operations.

  * •

We present a hybrid cache eviction policy that leverages both static graph
information and dynamic node access patterns to significantly improve cache
hit ratios, thus reducing the pressure on the storage.

  * •

We propose the PVP, which efficiently moves the evicted node feature data
likely to be reused into CPU memory, enabling effective prefetching that
further lowers storage contention.

##  2\. BACKGROUND

###  2.1. GNNs and GNN Training Pipeline

Inspired by the success of convolutional neural networks (CNNs) (CNN0, ),
people started to apply similar filters to graphs (lecunGCN, ;
hamilton2017inductive, ; ying2019pinsage, ; GCNPierre, ; kipf2017semi, ;
kipf2016variational, ; pmlr-v48-niepert16, ) and refer to such approaches as
Graph Neural Networks (GNNs). In GNNs, the graph adjacency matrix and node
features are used as input during forward propagation. For example, the
forward propagation formula of a GCN layer is defined as
$h^{out}=\sigma\left(A^{*}h^{in}W^{(l)}\right)$, where $h^{in}$ is the input
node features, $h^{out}$ is the output node features, $W$ is the weights in
this layer to be learned during the training process, and $A^{*}$ is the
adjacency matrix normalized with the in degree and out degree of nodes.

GraphSAGE (hamilton2017inductive, ) proposes neighbor sampling along with
minibatch, greatly reducing the memory footprint. A GraphSAGE model includes
two to three layers, which can be mean, pooling, LSTM, etc. Figure 1 shows an
example of neighbor sampling on node 9. Node indices are represented in
hexadecimal. Neighbors and 2-hop neighbors of node 9 are sampled, constituting
the input of the second layer and input of the first layer. Consequently, the
sampled node features are scattered in the node feature tensor, as the
illustration on the left shows. Graph Attention Network (GAT) (GAT, )
introduces learned self-attention masks into GraphSage layers to further
enhance inference accuracy and will be the model used for evaluating the
proposed optimizations in LSM-GNN.

Figure 1. An example demonstrating the GraphSAGE neighbor sampling approach
for output node 9. There are two layers in the model. On the left, the data
layout of node attributes tensor, feats, of this graph is shown.

Due to the large scale of graphs in many enterprise use cases, GNN model
training usually adopts mini-batching. Each iteration in mini-batch training
consists of three major stages: sampling, feature aggregation, and model
training. In the sampling stage, the system samples the nodes and edges to
create the subgraph for a mini-batch. Then, the feature data for the sampled
nodes and edges are aggregated into a dense feature tensor in the feature
aggregation stage. The sampled subgraphs with the feature data tensor are
transferred to the GPU for the model training stage to produce gradients and
update the model parameters.

###  2.2. Distributed/Multi-GPU GNN Training

For large-scale graphs, GNN training is often executed in multi-GPU or
distributed systems to address the memory capacity limitation and decrease the
E2E training time by parallelizing the training process with multiple GPUs. In
this case, the graph is partitioned across all nodes or GPUs, and the graph
data is transferred over the network when other nodes or GPUs request the data
during sampling or feature aggregation stages. To parallelize the GNN
training, mini-batches are often distributed to compute nodes where each node
has a full replicate of all the model weights and executes the forward
propagation and backward propagation individually (distdgl, ) and then
aggregates the model weights. Another approach is leveraging model parallelism
where the model is split into several parts and different computing nodes
compute each part (imagenet, ; model_parall1, ).

The state-of-the-art distributed/multi-GPU GNN training encounters two major
challenges. First, it incurs an extremely high cost for large-scale GNN
training, as the cumulative memory capacity of GPUs or CPUs must surpass the
total size of the graph dataset, which is often orders of magnitude higher
than GPU memory capacity. Second, the partitioning of graphs across multiple
nodes and GPUs introduces significant network communication overhead during
the feature aggregation phase. To mitigate the communication costs, previous
studies have employed the METIS-based graph partitioning algorithm (distdgl, ;
wangFlexGraphFlexibleEfficient2021, ; dorylus, ).

However, graph partitioning algorithms, e.g., METIS, incur substantial
preprocessing overhead. For example, it takes 457.55 seconds for DistDGL to
partition the heterogeneous IGBH-medium (IGB, ) graph, which only involves
26.0M nodes and 249M edges, into four partitions on a node with 4$\times$ AMD
“Interlagos” CPU. By contrast, under the same configurations, the random
partition only costs 83.43 seconds. Additionally, the runtime of METIS-based
algorithms scales exponentially with the size of the graph. Previous studies
(linComprehensiveSurveyDistributed2022, ; caiDGCLEfficientCommunication2021, )
have proposed optimizations to reduce network communication overheads by
overlapping communication with computation or smartly planning communication
before actual transfer. However, the network communication overhead remains
the main bottleneck, leading to grossly low GPU utilization.

Another approach to mitigate the communication overhead in the distributed GNN
training is to leverage static GPU cache.

###  2.3. Storage-based GNN Training

In contrast to partitioning graphs across multiple nodes and GPUs, several
studies (GIDS, ; Ginex, ; MariusGNN, ) have explored enabling GNN frameworks
to fetch data directly from storage systems. Ginex (Ginex, ) and Marius GNN
(MariusGNN, ) aim to hide the storage latency through in-memory caching and
pipelining of storage accesses using CPU orchestrated approaches while GIDS
(GIDS, ) enables GPU threads to access feature data directly, leveraging the
massive parallelism of GPUs to mitigate storage latency. Unlike conventional
distributed GNN training frameworks, storage-based GNN training obviates the
need for additional resources or graph partitioning. However, none of these
previous studies support GNN training in multi-GPU systems as the feature
aggregation performance is dependent on the storage bandwidth, which may not
be scalable with the number of GPUs or nodes in the system in practice.

###  2.4. Scoped Memory Consistency Model

Since Volta architecture (PTXFormal, ), NVIDIA GPUs use the Scoped Memory
Consistency Model. NVIDIA’s Scoped Memory Consistency Model is weakly ordered
and utilizes scoped synchronization primitives like threading block (.cta), at
the device or current GPU (.gpu), and across the whole system (.sys) levels to
facilitate interthread communication through memory, allowing threads within
the same thread block to synchronize efficiently while not mandating data race
freedom, resulting in a more intricate set of rules. Concurrent algorithms can
exploit memory consistency operations using the GPU memory model to select if
they want to quality with the .acquire, .release semantics or .relaxed or
.weak in addition to the memory scope to improve their performance. This work
exploits these memory consistency capabilities to achieve higher performance
when accessing cache metadata.

##  3\. MOTIVATION

The state-of-the-art storage-based GPU GNN training systems face significant
scalability challenges in multi-GPU settings due to limited storage bandwidth,
particularly during the feature aggregation stage.

Figure 2 illustrates the disparity in feature aggregation times when employing
1, 2, and 4 SSDs for storage-based GNN training. Notably, the time required
for feature aggregation with a single SSD is nearly four times longer than
when four SSDs are used. As more GPUs are used, the pressure on the SSD will
further increase and the aggregation stage for each GPU will take longer,
negating the benefit of using more GPUs. This storage bandwidth bottleneck
significantly hinders scalability as the system transitions from 1 to 4 GPUs
with a fixed number of SSDs, emphasizing the need for a more efficient
approach to scale the number of GPUs that can perform the feature aggregation
stage simultaneously with a fixed number of GPUs.

Previous studies (GIDS, ; bam, ) address this problem by connecting multiple
SSDs to a single GPU, thereby increasing the collective storage bandwidth
available. However, this approach is not a practical approach in multi-GPU
systems due to two main challenges. Firstly, achieving the necessary ingress
bandwidth for all GPUs requires a linear increase in the number of SSDs as a
function of the number of GPUs. For instance, fully matching the intake
bandwidth of one A100 GPU with the high-performance Intel Optane SSDs
necessitates five SSDs per GPU, scaling to 40 SSDs for an eight-GPU system.
This requirement becomes even more daunting with less performant SSDs like
those from Samsung 980pro, potentially exceeding 60 SSDs. Secondly,
accommodating changes in SSD configurations necessitates either duplicating or
reformatting data across all SSDs, a process that introduces considerable
overhead when a large number of SSDs are involved, especially given the large
scale of graph data involved in GNN training.

The challenge of limited storage bandwidth is anticipated to intensify based
on the growth trend of interconnect and hardware resources over recent
decades. Figure 3 compares the growth rates of SSD read bandwidth, PCIe
bandwidth, GPU memory bandwidth, and GPU compute throughput
(userbenchmarkSSDUserBenchmarks1072, ; epochParameterComputeData, ;
wikipediaTensorProcessingUnit2024, ;
jouppiInDatacenterPerformanceAnalysis2017, ; techpowerupGPUSpecsDatabase2024,
). The slowest growth rate of storage bandwidth starkly indicates that the
challenge of limited storage bandwidth in multi-GPU settings will become
increasingly critical for storage-based multi-GPU GNN training systems.

To this end, we develop the LSM-GNN that efficiently utilizes available
resources to scale the feature aggregation stage with limited storage
bandwidth.

Figure 2. Breakdown of GNN training time for storage-based GNN training,
illustrating the effect of varying the number of SSDs. The feature aggregation
time with a single SSD connected is 3.8$\times$ higher than with four SSDs
connected. One GPU is used in this measurement. Figure 3. Trend of recent GPUs
for deep learning and consumer-grade SSDs. Compared with other select metrics,
the bandwidth of SSDs has the lowest growth rate. We collect the inter-device
(P2P) bandwidth, PCIe bandwidth, memory bandwidth, and floating-point
throughput of Nvidia 100-level GPUs since Kepler (K100) and Google TPUs
(epochParameterComputeData, ; wikipediaTensorProcessingUnit2024, ;
jouppiInDatacenterPerformanceAnalysis2017, ; techpowerupGPUSpecsDatabase2024,
). We sampled the 4kB read bandwidth of recent consumer-grade SSDs with 1TB
capacity from UserBenchmark (userbenchmarkSSDUserBenchmarks1072, ).

##  4\. LSM-GNN SYSTEM DESIGN

The overall system design of LSM-GNN is illustrated in Figure 4. As LSM-GNN is
a storage-based GNN training framework, feature data is stored in SSDs and
directly fetched by GPU threads. To reduce the pressure on the storage, LSM-
GNN employs a 32-way set associative GPU software cache to temporarily store
the feature data of recently accessed nodes. LSM-GNN leverages the window
buffering technique from GIDS (GIDS, ) to store the list of sampled nodes to
be used in the next iterations in the window buffer, which provides node
access pattern information to the cache and allows the node features to be
used in these future iterations to be preserved if they are evicted from the
cache. To orchestrate the individual GPU software caches into a shared cache,
LSM-GNN has a communication layer for GPUs to receive their needs from other
GPUs’ software caches. Finally, the graph structure data is pinned in the CPU
memory (system memory) to enable GPU threads to directly fetch graph data with
Unified Virtual Addressing (UVA) during graph sampling while Preemptive Victim
Buffer Prefetcher (PVP) stores reusable evicted cache-lines from GPU and
transfers them back to GPU when they are needed.

Figure 4. Illustration of the System Design of LSM-GNN.

###  4.1. Communication Layer

Efficiently managing GPU memory across multiple GPUs is critical for
increasing the effective bandwidth of feature aggregation beyond limited
storage bandwidth in a storage-based GNN training framework. Similar to
previous studies (GIDS, ; bam, ), LSM-GNN utilizes GPU memory as a software
cache to capture the spatial and temporal locality of node access patterns to
minimize storage accesses.

However, due to the randomness of the sampling process, cache capacity becomes
a critical factor in achieving a high cache hit ratio. In a multi-GPU
environment, achieving this involves orchestrating the GPUs’ software caches
into a shared cache. The node features are shared and assigned to the GPUs so
that the node features can reside in more than one GPU’s software cache. The
benefits of the shared cache system are particularly substantial in systems
equipped with NVLink, as NVLink offers significantly higher bandwidth compared
to traditional storage and PCIe bandwidth.

A naive implementation of the shared cache would enable each GPU to directly
access and manage each other’s software cache. However, this approach faces
two major challenges. First, any changes to the data in the shared cache must
be visible to all GPUs, requiring system-scope cache management operations to
ensure memory consistency. These cache management operations require system-
scope memory operations, whose latency is significantly higher and throughput
is significantly lower than their device-scope counterparts, limiting the
effective bandwidth of the cache.

Figure 5 illustrates the impact of cache management operation scope on the
software cache’s effective bandwidth. As shown, the effective bandwidth of the
cache for cache hit (Hot) is around 750 GBps with the device-scope memory
operations and drops to 110 GBps with the system-scope memory operations.
Similarly, for cache misses (Cold), the cache achieves an effective bandwidth
of 124 GBps for device-scope memory operations and only 42 GBps for system-
scope memory operations. These results demonstrate how the implementation of
shared cache design with system-scope operations substantially limits the
cache access bandwidth.

The second issue with the naive approach to implementing a shared cache arises
from the overhead associated with the Python Global Interpreter Lock (GIL). In
order to facilitate a basic shared cache, GPU threads must be able to directly
access caches on other GPUs to retrieve cache lines. This necessitates that
GPU threads share the same address space, implying that multi-GPU training
should utilize multi-threading, with each thread dedicated to managing a
single GPU. However, both DGL and PyG advocate for the use of Distributed Data
Parallel (DDP) over Data Parallel. DDP leverages multi-processing parallelism,
which is preferred due to the substantial overhead incurred by multi-threading
parallelism as a result of the GIL (DDP, ).

Figure 5. Comparison of GPU software cache effective bandwidth using two
scopes: device and system for cache management operations.

To address these problems, we propose a novel communication layer that
orchestrates GPUs’ software caches into a system-wide shared software cache.
Figure 6 illustrates the stages within this layer. Initially, each GPU
generates a list of sampled nodes for its mini-batch through the sampling
process and then splits the list into sub-lists using a hash function, which
determines the cache-lines assigned to each GPU software cache. Although our
implementation utilizes a straightforward striding function, more complex hash
functions, such as those derived from graph partitioning algorithms, could
further reduce communication overhead.

Each GPU then communicates its sub-lists to its assigned GPUs. Since each sub-
list consists of only node IDs the overhead of this communication is
negligible. Each GPU then independently performs the feature aggregation
process for the sampled nodes requested by all GPUs. This pre-distribution of
accesses to the GPU software caches ensures only the local GPU threads are
accessing the software cache, eliminating the need for system-scope memory
operations. After feature data is fetched, they are transferred to the
requesting GPU which assembles the original mini-batch corresponding to the
sampled nodes, ensuring the model training stage remains unaffected by this
optimized communication layer.

Figure 6. The Communication Layer process with 2 GPUs.

To facilitate this system, a global barrier synchronization is introduced
after the node ID lists are divided, ensuring all GPUs progress in lockstep.
While such synchronization might typically introduce significant overhead in
multi-GPU setups, in our context, it proves minimally disruptive. This is
because GPUs are already synchronized post-training during gradient
accumulation, which minimizes the latency of the additional barrier
synchronization. Furthermore, the communication layer abstracts the storage
layout, allowing for flexible storage configuration tailored to specific
hardware setups, thereby enhancing the system’s adaptability and efficiency.

###  4.2. Hybrid Eviction Policy

The cache hit ratio during the feature aggregation process for large-scale GNN
training often remains low, due to the vast size of graph datasets and the
inherent randomness of the sampling process. Thus, it is crucial to
incorporate application-specific information to guide the software cache
toward more efficient exploitation of data locality.

The cache can leverage two types of information to efficiently exploit
locality. The first type is static information about the graph structure.
Previous studies (DataTiering, ) demonstrated that the static information of
the graph, such as out-degree or weighted reverse page rank score, can
effectively distinguish frequently accessed nodes from less frequently
accessed nodes. Thus, the software cache can leverage this information to
minimize evicting frequently used cache lines.

The second type is dynamic information. Previous work, GIDS (GIDS, ),
leverages the timing flexibility of the graph sampling process to gather the
list of nodes sampled in upcoming iterations and store them in the window
buffer. To avoid adding latency to the cache eviction operations, LSM-GNN
scans the sampled nodes in the window buffer to determine the next reuse
iteration for the cache-lines that currently reside in the cache before the
feature aggregation stage.

Although dynamic information provides accurate critical information for the
node access pattern, the cost of scanning the next reuse time for all cache-
lines in the cache can introduce significant overhead, especially for large
window buffer sizes (number of iterations to foresee). To mitigate the impact
of this overhead, LSM-GNN’s software cache updates the dynamic information for
a configurable number of iterations, rather than for every iteration. The
frequency of such updates is determined based on the cache-size, batch size,
and window buffer size.

Both static and dynamic information offer insights into the unique
characteristics of node access patterns. Thus, LSM-GNN improves the cache
efficiency of GPU software by implementing a hybrid eviction policy that
leverages both types of information. The hybrid eviction policy classifies
each eviction candidate into four priority levels based on their dynamic
information. If there is more than one eviction candidate at the same priority
level, the eviction policy selects the one with the lowest static priority
value for eviction. The eviction policy first searches for candidates that are
in the lowest priority level and goes to the next level if there are no
candidates in the current level.

The lowest priority is assigned to nodes with no anticipated reuse within the
window buffer, suggesting that their feature data will not be required in the
foreseeable future. Thus, the probability of reusing these cache lines is low.

The next priority level is the nodes whose dynamic information value falls
below a specific threshold. This threshold is configurable, but by default, it
is defined as 1/8 of the window buffer size. This priority level indicates
that the candidates are expected to be reused within the window buffer but not
in the immediate future, as suggested by the threshold value.

Candidates at the third priority level consist of recently inserted nodes.
Since these nodes have just been added to the cache, they lack dynamic
information. Given their recent insertion, there’s a potential for these nodes
to be reused in subsequent iterations, granting them a higher priority over
the first two groups of candidates.

The highest priority is assigned to nodes whose reuse value exceeds the
specified threshold. These candidates are expected to be reused in the near
future, making it crucial to retain them in the cache.

This hierarchical approach ensures efficient cache management, prioritizing
the retention of nodes likely needed in the imminent iterations while
facilitating the eviction of those less likely to be reused. Moreover, static
information helps the eviction policy to minimize and avoid evicting hot nodes
even if they fall into lower priority levels. The flexibility of the hybrid
policy allows adjustment of thresholds and priority levels, tailoring cache
behavior to the specific demands of the GNN training process.

###  4.3. Preemptive Victim-Buffer Prefetcher

Traditional GNN feature data prefetchers, which are primarily based on the
CPU, attempt to preload feature data for upcoming iterations while the current
iteration is being processed by the GPU. However, as evidenced by previous
studies (pytorch-direct, ; GIDS, ), these prefetching methods fall short of
matching the GPU’s model training throughput, leading to little performance
improvement. Moreover, attempts at GPU-based prefetching can lead to resource
contention due to the significant GPU resources, e.g., GPU memory, required to
exploit the GPU’s massive parallelism to hide the storage latency as GPU
resource is already utilized for model training. This challenge has rendered
previous prefetching approaches ineffective within storage-based GNN training
frameworks.

Thus, LSM-GNN enhances the efficiency of the feature aggregation phase with
the introduction of the PVP, aimed at optimizing interconnect bandwidth
utilization across the GNN training pipeline. The PVP is specifically designed
to improve bandwidth usage during periods when PCIe bandwidth is
underutilized, such as during the model training stage without increasing GPU
resource contention. It achieves this by preemptively managing the eviction
and prefetching of cache lines, guided by dynamic information.

Figure 7. Preemptive Victim-Buffer Prefetcher (PVP).

Figure 7 illustrates the architecture of the PVP. On the CPU side, the
architecture includes data and metadata victim buffers tasked with managing
evicted cache-line data and their associated metadata, respectively. This
setup ensures efficient handling of data that is no longer retained within the
GPU’s cache. On the GPU side, the PVP is equipped with counters to track the
number of victim cache-lines present in the buffer. GPU threads use atomic
operations to increment these counters each time an evicted cache-line is
enqueued into a victim buffer. Additionally, a prefetching buffer is
designated for the temporary storage of prefetched data. This setup and the
PVP Metadata on the CPU side enable CPU threads to directly transfer data to
the appropriate entries of the prefetching buffer on the GPU side through a
CUDA API call (cudaMemcpyAsync), streamlining the prefetching process without
access to the GPU’s software cache. Moreover, the CPU to GPU transfer is done
at the training stage of the pipeline, when the PCIe ingress bandwidth is not
utilized and thus does not cause extra resource contention.

####  4.3.1. Eviction Process with PVP

The LSM-GNN PVP introduces an additional eviction stage during the feature
aggregation stage, leveraging the next reuse timestamp for each node’s feature
data in the LSM-GNN GPU software cache. Figure 8 illustrates this eviction
stage with PVP.

Upon evicting a node’s feature data from the GPU software cache, the leader
thread first checks the next reuse iteration. If the cache-line has the
information for the next reuse iteration, a GPU thread calls an atomic
operation to increment the corresponding queue counter (1), which tracks the
number of evicted node feature data currently stored in the victim buffer.
This incremented counter value identifies the appropriate index for storing
the data within the corresponding victim buffer. In scenarios where the
counter value surpasses the buffer’s capacity or if no reuse iteration is
marked, the cache-line data is simply discarded. Otherwise, the obtained
counter value is shared among the threads in the same warp, guiding the
collective transfer of the cache line into the prefetcher buffer in the CPU
memory (2). For instance, with a next reuse timestamp of 4 and a queue counter
at 5, the evicted cache line is placed into the 5th position of the fourth
victim buffer.

Importantly, the additional eviction stage does not increase resource
contention. GPU PCIe egress bandwidth is not utilized during the feature
aggregation stage for the GPU-oriented feature aggregation process. Thus,
there is no contention on the PCIe egress bandwidth. Moreover, since the
number of concurrently evicting cache lines is proportional to the number of
accesses to the storage, the latency of evicting cache-line to the CPU victim
buffer is hidden when there are a sufficient number of storage accesses.

Figure 8. Example of the eviction process with PVP.

####  4.3.2. Eviction Policy with PVP

In Section 4.2, we introduce a hybrid eviction policy designed to improve the
GPU software cache hit ratio by utilizing both static and dynamic information.
However, when PVP is activated, a slight modification to this policy is
necessary to fully harness its performance benefits.

Under the original hybrid eviction policy, cache-lines not expected to be
reused within the window buffer’s scope are prioritized for eviction. These
cache-lines, lacking tags indicating their next reuse iteration, are directly
discarded. This approach can lead to missing potential reuse opportunities
outside the window buffer scope. To address this, LSM-GNN swaps the two lowest
priority levels in the hybrid eviction policy when PVP is enabled. This
modification prioritizes the cache lines tagged with the next reuse iteration
but below the threshold for the eviction so that they can be evicted into the
victim buffers. This modified policy allows more of the cache-lines, which are
not reused within the window buffer’s scope, to remain in the software cache.
This strategy enhances both PVP efficiency and overall GPU cache utilization
by ensuring that cache-lines with foreseeable reuse but below the threshold
are recycled through the PVP path.

####  4.3.3. PVP’s Metadata

Once victim cache-lines are prefetched and temporarily stored in the
prefetching buffer, they must be copied back to the mini-batch. However, since
the mini-batch is not sorted by node ID, locating the appropriate position for
the prefetched data within the mini-batch would necessitate a search
operation, potentially introducing significant overhead. To mitigate this, the
system manages metadata including the batch ID, GPU ID, and next reuse
iteration within the software cache. This metadata is evicted to the victim
buffer alongside the cache-line when it is moved to the victim buffer.
Consequently, when GPU threads retrieve data from the prefetching buffer, they
efficiently identify the corresponding batch index for placing the feature
data into the appropriate entry of the mini-batch data buffer.

To minimize memory usage and reduce the number of operations required during
the dynamic information update stage, PVP adopts a specific metadata
representation. The first 2 bytes indicate the next reuse iteration, setting a
limit on the window buffer size to no more than 65,536. The next byte
identifies the GPU ID that requested the sampled node. The final 5 bytes
denote the index within the mini-batch. Since the initial 2 bytes represent
the reuse time, atomicMin operations are utilized during the dynamic
information update stage to record the next reuse iteration efficiently.

##  5\. EVALUATION

###  5.1. Experimental Setup

Environment. We compare LSM-GNN and the state-of-the-art baseline storage-
based GNN framework using the system described in Table 1. This system has
2$\times$ NVIDIA A100-40GB GPUs and the GPUs are connected with 3 NVLink
bridges. For the baseline distributed GNN framework, we evaluated on a two
nodes system where each node is equipped with 2$\times$ AMD “Milan” EPYC 7763
CPU, and 2TB memory. Each node has 2$\times$ A100 PCIe 40GB, with PCIe Gen 4
and NVLink.

Table 1. Configuration used to evaluate LSM-GNN. Configuration |  Specification
---|---
CPU |  AMD EPYC 7702 64-Core Processor
Memory |  1TB DDR4
GPU |  NVIDIA A100 HBM2 40GB
NVLink |  3 bridges, 270 GBps peak Bandwidth
S/W |  Ubuntu 20.04 LTS, NVIDIA Driver 470.103
CUDA 12.1, DGL 2.0.0, Pytorch 2.0.1
SSDs |  Intel Optane SSDs, PCIe Gen 4 Interconnect

Datasets. To assess the performance of LSM-GNN on large-scale graph datasets,
we conducted experiments using five real-world datasets: IGB-Full (IGB, ),
IGBH-Full (IGB, ), IGB-Medium (IGB, ), ogbn-papers100M (ogbn_paper, ), and
MAG240M (MAG, ) as shown in Table 2. As ogbn-papers100M and MAG240M datasets
are relatively small, we only tested with a 4 GB GPU software cache size. We
used the IGB-Full dataset by default in the evaluations if not explicitly
mentioned.

Table 2. Real-world dataset used for evaluating LSM-GNN.  Dataset |  Graph Type |  # Nodes |  # Edges |  Feature Size
---|---|---|---|---
papers100M |  Homogeneous |  111M |  1.62B |  128
IGB-Full |  Homogeneous |  269M |  4.00B |  1024
IGB-Medium |  Homogeneous |  10.4M |  120B |  1024
MAG240M |  Heterogeneous |  244M |  1.73B |  768
IGBH-Full |  Heterogeneous |  547M |  5.81B |  1024

Model: We assessed LSM-GNN’s performance with neighborhood sampling
(Graphsage, ) with fanout values (5,2,2,2) and 4-layer GAT (GAT, ) model whose
hidden channel dimension is 512. Finally, we set the batch size as 2048 for
all evaluations.

LSM-GNN Configuration: In the default configuration, we allocated 16 GB of GPU
device memory for each GPU software cache. For the hybrid eviction and dynamic
eviction policies, the default configuration is 256 iterations for the window
buffer size with 4 iterations for the window buffer update period. For PVP,
each Victim buffer in the CPU memory can capture up to 16K cache-lines.

Measuring Execution Time: When working with large graph datasets, the training
process can be excessively long. Therefore, we conducted the evaluations by
measuring the execution time for a sum of 100 iterations after a warm-up stage
of 1,000 iterations.

Baseline:  To compare LSM-GNN with storage-based GNN frameworks, we extended
GIDS to work in multi-GPU systems (M-GIDS), which does not incorporate any of
the LSM-GNN techniques. To reduce cache metadata for the GPU software cache,
we integrated a 32-way set associative cache with a Round-Robin eviction
policy into M-GIDS. For distributed GNN training, we compared against Dist-DGL
(distdgl, ).

###  5.2. Impact of the Communication Layer

In this section, we investigate the effects of the Communication Layer on the
GPU software cache hit ratios and feature aggregation times. Our evaluation
compares LSM-GNN with the GIDS based multi-GPU GNN training framework
(M-GIDS), and both systems are configured with two SSDs (one SSD per GPU).
Performance metrics were gathered using two distinct eviction policies: the
Round-Robin (RR) eviction policy and the hybrid (H) eviction policy.

For both LSM-GNN and M-GIDS, the GPU software caches are set at sizes of 4 GB,
8 GB, and 16 GB. The hybrid eviction policy was evaluated with a window buffer
with a size of 256, and the Preemptive Victim-buffer Prefetcher was disabled
for this assessment. Additionally, for LSM-GNN, the feature aggregation time
includes communication time created by the communication layer.

Figure 9 illustrates the cache hit ratios and feature aggregation times for
M-GIDS and LSM-GNN with the Communication Layer. As shown, the Communication
Layer allows LSM-GNN to achieve cache hit ratios, and thus time, comparable to
those of the baseline configuration with a cache size that is twice as large.
This is achieved by enabling the GPU software caches to function collectively
as a system cache.

The feature aggregation time significantly decreases with larger cache sizes
and with the hybrid eviction policy. When the cache size is set to 4 GB using
the Round-Robin eviction policy, the feature aggregation times are similar.
However, with a 16 GB cache under the hybrid eviction policy, the time is
reduced by 1.18$\times$. This is because the feature aggregation time is
proportional to the cache miss rate, and a larger cache with a more efficient
eviction policy enables the cache to exploit spatial and temporal locality,
resulting a higher differences in cache miss rate.

Furthermore, the communication time remains constant, irrespective of the
cache capacity and hit ratio, accounting for only 0.21 seconds, which is less
than 5% of the feature aggregation time in the worst-case scenario.
Nevertheless, for the communication layer to contribute to performance
enhancement, the feature aggregation time must decrease by more than 5% due to
increased cache capacity.

###  5.3. Impact of the Cache Eviction Policy

This section explores the impact of different eviction strategies on cache
performance. We evaluated the cache hit ratio and feature aggregation time
using four distinct eviction policies: 1) Round-Robin, 2) Static Information-
Based, 3) Dynamic Information-Based, and 4) Hybrid Information-Based eviction
policies. In this evaluation, the reverse PageRank value represents static
information, while the next reuse iteration serves as dynamic information. The
static information-based eviction policy prioritizes evicting the cache line
with the lowest static value, whereas the dynamic information-based eviction
policy targets the cache line that has the longest time until the next reuse
(a cache line with no expected reuse has the highest eviction priority). We
conducted experiments with three different GPU sizes: 4 GB, 8 GB, and 16 GB,
and two SSD configurations: 2 SSDs and 4 SSDs in the system. For all tests, we
standardized the GPU software cache configuration to a 32-way set associative
cache.

Figure 9. The impact of higher cache capacity achieved by the communication
layer. LSM-GNN achieves up to 1.18$\times$ speed up for the feature
aggregation compared to the baseline framework with independent GPU software
caches mainly because of better cache hit ratios.

Figure 10 illustrates the hit ratios and feature aggregation times across
these eviction policies. In all configurations, the hybrid eviction policy
yields the highest cache hit ratio, while the Round-Robin policy results in
the lowest cache hit ratio. The hybrid eviction policy significantly improves
the cache hit ratio compared to the Round-Robin eviction policy, increasing it
from 0.23 to 0.48 when the cache size is 16 GB and achieves a 1.41$\times$
speedup with 2 SSDs and a 1.38$\times$ speedup with 4 SSDs.

Figure 10. Impact of the cache eviction policies on LSM-GNN software cache.
RR, S, D, and H stand for Round-Robind, Static information based, Dynamic
information based, and Hybrid eviction policies, respectively. D1 and H1 have
a smaller window buffer size (64) while D2 and H2 are evaluated with a window
buffer size of 256. The hybrid eviction policy can increase the cache hit
ratio from 0.23 to 0.48 when the cache size is 16 GB, resulting in
1.41$\times$ speed up for the feature aggregation process.

The performance gain from the hybrid cache eviction policy is greater in the
case with 2 SSDs than with 4 SSDs. This is because the feature aggregation
process is bounded by storage bandwidth. Thus, contention for storage accesses
is more detrimental to the aggregation process when the storage bandwidth is
lower. Thus, the performance gain will be higher in a system where a single
SSD is connected to two GPUs or SSDs with lower read bandwidth. Furthermore,
the performance gain of the hybrid eviction policy compared to the Round-Robin
eviction policy is greater when the cache capacity is larger. For example,
with 2 SSDs in the system, the feature aggregation process speed ups are
1.22$\times$, 1.29$\times$, and 1.41$\times$ for GPU software caches of 4 GB,
8 GB, and 16 GB, respectively. This is because a higher cache capacity enables
the cache to capture more node access locality, increasing the advantage over
the Round-Robin eviction policy.

The hybrid eviction policy outperforms both the static and dynamic
information-based eviction policies in any setting. It specifically surpasses
the static information-based policy as cache size increases. The static
eviction policy achieves cache hit ratios of 0.19, 0.27, and 0.37, while the
hybrid eviction policy attains 0.25, 0.33, and 0.48 for cache sizes of 4 GB, 8
GB, and 16 GB, respectively. Compared to the dynamic eviction policy, the
performance gain from the hybrid policy is more notable when the window buffer
size is smaller. These characteristics are observed when compared against both
static and dynamic eviction policies because dynamic information provides more
accurate insights into node access patterns when there is adequate cache size
to capture temporal locality. Thus, the performance of the dynamic eviction
policy heavily depends on both the window buffer size and cache capacity. The
hybrid eviction policy leverages both static and dynamic information,
resulting in a minimized performance decline with smaller window buffer sizes.

Figure 11.  Cache hit ratio and feature aggregation performance comparison
between PVP and the baseline. LSM-GNN achieves a higher hit ratio by capturing
the reusable evicted cache-lines in the Vicitm-buffer, especially with the
hybrid eviction policy.

These results demonstrate that the hybrid eviction policy effectively balances
static and dynamic information according to the system configuration.
Consequently, users can leverage this policy across various resource
availability and GNN training pipelines, ensuring optimal performance
regardless of the constraints.

###  5.4. Preemptive Victim-Buffer Prefetcher’s Impact

In this evaluation, we assess the impact of the Preemptive Victim-buffer
Prefetcher (PVP) on GPU software cache performance and feature aggregation. We
measured cache hit ratios and feature aggregation times across all
configurations previously examined in the cache eviction policy assessment.

Figure 11 presents a comparison of hit ratios and feature aggregation
throughputs for LSM-GNN both with and without PVP enabled. For the dynamic and
hybrid eviction policy, we compared when the window buffer size is 256. Across
all configurations, enabling PVP consistently enhances performance.
Specifically, with the hybrid eviction policy, PVP elevates the hit ratio from
0.25, 0.33, and 0.48 to 0.52, 0.60, and 0.69 for cache sizes of 4 GB, 8 GB,
and 16 GB, respectively. Additionally, feature aggregation throughput
increases by 1.54$\times$, 1.53$\times$, and 1.68$\times$ for cache sizes of 4
GB, 8 GB, and 16 GB, respectively, when 2 SSDs are utilized. With 4 SSDs, the
throughput improvements are 1.38$\times$, 1.46$\times$, and 1.51$\times$,
correspondingly. These results demonstrate PVP’s capability to efficiently
prefetch evicted cache lines likely to be reused in subsequent iterations,
storing them in the Victim-buffer for efficient CPU-to-GPU data transfer by a
CPU thread.

The performance gains from PVP are less pronounced for the static and dynamic
information-based eviction policies than for the Round-Robin and hybrid
eviction policies. This discrepancy arises because these policies are more
inclined to evict cache lines lacking dynamic information. Specifically, the
dynamic information-based eviction policy prioritizes the eviction of cache
lines anticipated to have no subsequent reuse, making these lines ineligible
for transfer to the Victim-buffer. Conversely, while the static information-
based eviction policy does exhibit a slightly higher performance gain from PVP
compared to the dynamic eviction policy, it too prioritizes the eviction of
cache lines with lower static values, which are less likely to be present in
the window buffer. Therefore, the likelihood of evicting cache lines without
dynamic information is lower with the Round-Robin and hybrid eviction
policies, resulting in reduced performance benefits from PVP.

Figure 12. Epoch time comparison of LSM-GNN, M-GIDS, and Dist-DGL on
homogeneous and heterogeneous graphs.

###  5.5. Overall Performance Comparison

Figure 12 presents one epoch time for Dist-DGL, M-GIDS, and LSM-GNN across
both homogeneous and heterogeneous graph datasets. Dist-DGL is executed with
two DGX-A100 nodes each with two GPUs while M-GIDS and LSM-GNN are evaluated
with two A100 GPUs and each GPU is connected to an SSD. For smaller datasets,
such as ogbn-papers100M and IGB-Medium, 4 GB GPU memory is allocated for each
GPU software cache in the case of M-GIDS and LSM-GNN. Due to substantial
memory requirements, Dist-DGL encounters an out-of-memory error when
processing the IGB-Full, IGBH-Full, and MAG240M datasets. Some cases trigger
OOM errors due to insufficient collective GPU memory of the 4 GPUs used. All
systems reach the same convergence properties as LSM-GNN or M-GIDS does not
make any changes to the algorithms involved.

Despite the lower compute capabilities and memory capacities, M-GIDS and LSM-
GNN in a single node with two GPUs offer superior performance over Dist-DGL
baseline as shown in Figure 12. LSM-GNN achieves up to a 1.54$\times$ speedup
over M-GIDS and a 3.75$\times$ speedup over the widely used Dist-DGL. This
superior performance of LSM-GNN can be attributed to novel data management
strategies, including the communication layer, a hybrid eviction policy, and a
Preemptive Victim-buffer Prefetcher (PVP). These strategies effectively
alleviate storage pressure and optimize the use of GPU resources, thereby
minimizing the impact of the limited storage bandwidth. The substantial
speedup observed in comparison to Dist-DGL primarily results from the
elimination of high network communication overheads for the distributed GNN
training frameworks. Moreover, using fewer nodes equipped with expensive GPUs
not only lowers training expenses but also significantly cuts down on power
costs over time providing order of magnitude in total cost of ownership
savings at scale.

##  6\. RELATED WORK

GIDS (GIDS, ), Ginex (Ginex, ), and MariousGNN (MariusGNN, ) propose storage-
based GNN training that can hide storage latency. However, due to the limited
storage bandwidth and hardware resource requirement, they are not scalable in
multi-GPU cases.

Previous studies (DataTiering, ; R1_pagerank, ) show PageRank can be used to
estimate the frequency of accesses during node sampling, improving GPU memory
utilization. ROC (ROC, ), AliGraph (AliGraph, ), PaGraph (PaGraph, ), NextDoor
(jangdaAcceleratingGraphSampling2021, ), and Ginex (Ginex, ) leverage static
in-memory cache to reduce communication overhead. However, unlike LSM-GNN’s
hybrid eviction policy, these caches are solely dependent on static
information. GIDS (GIDS, ) leverages software cache to reduce redundant
accesses. However, the cache eviction policy is dependent on the number of
accesses in the next iterations, increasing the contention on available cache-
lines due to cache-line pinning.

Previous works (G3, ; dorylus, ; R5_sampling, ; BGL, ; Neugraph, ; P3, ; DSP,
; R10, ; GNN:I/O, ) propose a multi-GPU training system for large-scale GNN
training. However, they require significant additional hardware resources to
partition the graph across nodes or GPUs. Moreover, they are not scalable
solutions due to high network communication overhead during the feature
aggregation stage. Flexgraph (wangFlexGraphFlexibleEfficient2021, ), Dorylus
(dorylus, ), ByteGNN (ByteGNN, ), Pytorch-BigGraph (Pytorch-biggraph, ), and
Cluster-GCN (Cluster-GCN, ) partition graph with METIS based graph
partitioning algorithm, causing a significant preprocessing overhead.

There are also works optimizing GPU performance for the sampling stage and/or
the training stage for sampling-based GNN training, including GNNadvisor
(wangGNNAdvisorAdaptiveEfficient2021, ), BNS-GCN (bns-gcn, ), and GNNlab
(GNNLab, ). Our work is orthogonal to the optimizations they have proposed.

##  7\. CONCLUSION

Scaling large-scale storage-based GNNs Training in multi-GPU systems is a
challenging task due to the limited storage bandwidth during the feature
aggregation stage. In this paper, we propose LSM-GNN, a large-scale storage-
based multi-GPU GNN training framework, that efficiently utilizes GPU and CPU
available hardware resources to scale the feature aggregation process in the
multi-GPU system without changing storage configuration. LSM-GNN efficiently
leverage GPU software caches by orchestrating each GPU cache as a shared
system cache without slow system-scope operations by distributing node
accesses with the communication layer. LSM-GNN then substantially increases
the cache hit ratio by the hybrid eviction policy which exploits both static
and dynamic information about the node access pattern. Finally, LSM-GNN
leverages PVP, which temporally stores the evicted cache-line data that will
be reused in the future iteration and enables high throughput feature data
prefetching without increasing contention on GPU resources. All in all, LSM-
GNN achieves up to 3.75$\times$ end-to-end training time speed up compared to
the state-of-the-art frameworks.

## References

  * [1] Aleksandar Bojchevski, Johannes Gasteiger, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann.  Scaling graph neural networks with approximate pagerank.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 2464–2473, New York, NY, USA, 2020. Association for Computing Machinery.
  * [2] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun.  Spectral networks and locally connected networks on graphs.  In International Conference on Learning Representations (ICLR2014), CBLS, April 2014, 2014.
  * [3] Zhenkun Cai, Xiao Yan, Yidi Wu, Kaihao Ma, James Cheng, and Fan Yu.  DGCL: An efficient communication library for distributed GNN training.  In Proceedings of the Sixteenth European Conference on Computer Systems, EuroSys ’21, pages 130–144, New York, NY, USA, April 2021. Association for Computing Machinery.
  * [4] Zhenkun Cai, Qihui Zhou, Xiao Yan, Da Zheng, Xiang Song, Chenguang Zheng, James Cheng, and George Karypis.  Dsp: Efficient gnn training with multiple gpus.  In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP ’23, page 392–404, 2023.
  * [5] Karanbir Singh Chahal, Manraj Singh Grover, Kuntal Dey, and Rajiv Ratn Shah.  A hitchhiker’s guide on distributed training of deep neural networks.  Journal of Parallel and Distributed Computing, 137:65–76, 2020.
  * [6] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh.  Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 257–266, New York, NY, USA, 2019. Association for Computing Machinery.
  * [7] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 3844–3852, Red Hook, NY, USA, 2016. Curran Associates Inc.
  * [8] Epoch.  Parameter, Compute and Data Trends in Machine Learning.  https://epochai.org/mlinputs/visualization.
  * [9] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin.  Graph neural networks for social recommendation.  In The World Wide Web Conference, WWW ’19, page 417–426, New York, NY, USA, 2019. Association for Computing Machinery.
  * [10] Matthias Fey and Jan Eric Lenssen.  Fast graph representation learning with pytorch geometric, 2019.
  * [11] Swapnil Gandhi and Anand Padmanabha Iyer.  P3: Distributed deep graph learning at scale.  In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 551–568. USENIX Association, July 2021.
  * [12] Victor Garcia and Joan Bruna.  Few-shot learning with graph neural networks, 2018.
  * [13] Daniele Grattarola and Cesare Alippi.  Graph neural networks in tensorflow and keras with spektral [application notes].  Comp. Intell. Mag., 16(1):99–106, feb 2021.
  * [14] William L. Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [15] William L. Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [16] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open Graph Benchmark: Datasets for Machine Learning on Graphs.  http://arxiv.org/abs/2005.00687, February 2021.
  * [17] Abhinav Jangda, Sandeep Polisetty, Arjun Guha, and Marco Serafini.  Accelerating Graph Sampling for Graph Machine Learning using GPUs, May 2021.
  * [18] Zhihao Jia, Sina Lin, Mingyu Gao, Matei Zaharia, and Alex Aiken.  Improving the accuracy, scalability, and performance of graph neural networks with roc.  Proceedings of Machine Learning and Systems, 2:187–198, 2020.
  * [19] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon.  In-Datacenter Performance Analysis of a Tensor Processing Unit.  In Proceedings of the 44th Annual International Symposium on Computer Architecture, pages 1–12, Toronto ON Canada, June 2017. ACM.
  * [20] Tim Kaler, Nickolas Stathas, Anne Ouyang, Alexandros-Stavros Iliopoulos, Tao Schardl, Charles E. Leiserson, and Jie Chen.  Accelerating training and inference of graph neural networks with fast sampling and pipelining.  In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 172–189, 2022.
  * [21] George Karypis and Vipin Kumar.  Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices, 1997.
  * [22] Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, and Wen mei Hwu.  Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research, 2023.
  * [23] Thomas N Kipf and Max Welling.  Variational graph auto-encoders.  NIPS Workshop on Bayesian Deep Learning, 2016.
  * [24] Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  In Proceedings of the 5th International Conference on Learning Representations, ICLR ’17, 2017.
  * [25] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations (ICLR), 2017.
  * [26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
  * [27] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551, 1989.
  * [28] Claire Songhyun Lee, V Hewes, Giuseppe Cerati, Jim Kowalkowski, Adam Aurisano, Ankit Agrawal, Alok Choudhary, and Wei-Keng Liao.  A case study of data management challenges presented in large-scale machine learning workflows.  In 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 71–81, 2023.
  * [29] Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich.  Pytorch-biggraph: A large scale graph embedding system.  In A. Talwalkar, V. Smith, and M. Zaharia, editors, Proceedings of Machine Learning and Systems, volume 1, pages 120–131, 2019.
  * [30] Haiyang Lin, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Shirui Pan, Wenguang Chen, and Yuan Xie.  A Comprehensive Survey on Distributed Training of Graph Neural Networks, November 2022.
  * [31] Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu.  Pagraph: Scaling gnn training on large graphs via computation-aware caching.  In Proceedings of the 11th ACM Symposium on Cloud Computing, SoCC ’20, page 401–415, New York, NY, USA, 2020. Association for Computing Machinery.
  * [32] Tianfeng Liu, Yangrui Chen, Dan Li, Chuan Wu, Yibo Zhu, Jun He, Yanghua Peng, Hongzheng Chen, Hongzhi Chen, and Chuanxiong Guo.  BGL: GPU-Efficient GNN training by optimizing graph data I/O and preprocessing.  In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 103–118, Boston, MA, April 2023. USENIX Association.
  * [33] Zhiwei Liu, Yingtong Dou, Philip S. Yu, Yutong Deng, and Hao Peng.  Alleviating the inconsistency problem of applying graph neural network to fraud detection.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1569–1572, New York, NY, USA, 2020. Association for Computing Machinery.
  * [34] Daniel Lustig, Sameer Sahasrabuddhe, and Olivier Giroux.  A formal analysis of the nvidia ptx memory consistency model.  Proc. ASPLOS, 2019.
  * [35] Lingxiao Ma, Zhi Yang, Youshan Miao, Jilong Xue, Ming Wu, Lidong Zhou, and Yafei Dai.  Neugraph: Parallel deep neural network computation on large graphs.  In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’19, page 443–457, USA, 2019. USENIX Association.
  * [36] Microsoft.  Microsoft Academic Graph.  https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/.
  * [37] Seung Won Min, Kun Wu, Mert Hidayetoglu, Jinjun Xiong, Xiang Song, and Wen-mei Hwu.  Graph neural network training and data tiering.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 3555–3565, New York, NY, USA, 2022. Association for Computing Machinery.
  * [38] Seung Won Min, Kun Wu, Sitao Huang, Mert Hidayetoğlu, Jinjun Xiong, Eiman Ebrahimi, Deming Chen, and Wen mei Hwu.  Pytorch-direct: Enabling gpu centric data access for very large graph neural network training with irregular accesses, 2021.
  * [39] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov.  Learning convolutional neural networks for graphs.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2014–2023, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * [40] Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovec.  Pinnersage: Multi-modal user embedding framework for recommendations at pinterest.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’20, page 2311–2320, New York, NY, USA, 2020. Association for Computing Machinery.
  * [41] Jeongmin Brian Park, Vikram Sharma Mailthody, Zaid Qureshi, and Wen mei Hwu.  Accelerating sampling and aggregation operations in gnn frameworks with gpu initiated direct storage accesses, 2023.
  * [42] Yeonhong Park, Sunhong Min, and Jae W. Lee.  Ginex: Ssd-enabled billion-scale graph neural network training on a single machine via provably optimal in-memory caching.  Proc. VLDB Endow., 15(11):2626–2639, jul 2022.
  * [43] Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seungwon Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, and Wen-mei Hwu.  Gpu-initiated on-demand high-throughput storage access in the bam system architecture.  In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, page 325–339, New York, NY, USA, 2023. Association for Computing Machinery.
  * [44] Morteza Ramezani, Weilin Cong, Mehrdad Mahdavi, Anand Sivasubramaniam, and Mahmut T. Kandemir.  Gcn meets gpu: Decoupling ”when to sample” from ”how to sample”.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [45] Andrea Rossi, Donatella Firmani, Paolo Merialdo, and Tommaso Teofili.  Explaining link prediction systems based on knowledge graph embeddings.  In Proceedings of the 2022 International Conference on Management of Data, SIGMOD ’22, page 2062–2075, New York, NY, USA, 2022. Association for Computing Machinery.
  * [46] Shen Li.  Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.2.1+cu121 documentation.  https://pytorch.org/tutorials/intermediate/ddp_tutorial.html, February 2024.
  * [47] Shihui Song and Peng Jiang.  Rethinking graph data placement for graph neural network training on multiple gpus.  In Proceedings of the 36th ACM International Conference on Supercomputing, ICS ’22, New York, NY, USA, 2022. Association for Computing Machinery.
  * [48] TechPowerUp.  GPU Specs Database.  https://www.techpowerup.com/gpu-specs/, February 2024.
  * [49] John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu.  Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads.  In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 495–514. USENIX Association, July 2021.
  * [50] UserBenchmark.  SSD UserBenchmarks - 1072 Solid State Drives Compared.  https://ssd.userbenchmark.com.
  * [51] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks, 2018.
  * [52] Roger Waleffe, Jason Mohoney, Theodoros Rekatsinas, and Shivaram Venkataraman.  Mariusgnn: Resource-efficient out-of-core training of graph neural networks.  In Proceedings of the Eighteenth European Conference on Computer Systems, page 144–161. Association for Computing Machinery, 2023.
  * [53] Cheng Wan, Youjie Li, Ang Li, Nam Sung Kim, and Yingyan Lin.  Bns-gcn: Efficient full-graph training of graph convolutional networks with partition-parallelism and random boundary node sampling.  In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 673–693, 2022.
  * [54] Xinchen Wan, Kaiqiang Xu, Xudong Liao, Yilun Jin, Kai Chen, and Xin Jin.  Scalable and efficient full-graph gnn training for large graphs.  Proc. ACM Manag. Data, 1(2), jun 2023.
  * [55] Jianyu Wang, Rui Wen, Chunming Wu, Yu Huang, and Jian Xiong.  Fdgars: Fraudster detection via graph convolutional networks in online app review system.  In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 310–316, New York, NY, USA, 2019. Association for Computing Machinery.
  * [56] Lei Wang, Qiang Yin, Chao Tian, Jianbang Yang, Rong Chen, Wenyuan Yu, Zihang Yao, and Jingren Zhou.  FlexGraph: A flexible and efficient distributed framework for GNN training.  In Proceedings of the Sixteenth European Conference on Computer Systems, EuroSys ’21, pages 67–82, New York, NY, USA, April 2021. Association for Computing Machinery.
  * [57] Yuke Wang, Boyuan Feng, Gushu Li, Shuangchen Li, Lei Deng, Yuan Xie, and Yufei Ding.  GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUs.  In The 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’21), July 2021.
  * [58] Wikipedia.  Tensor Processing Unit, February 2024.
  * [59] Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan.  Energy-based out-of-distribution detection for graph neural networks, 2023.
  * [60] Jianbang Yang, Dahai Tang, Xiaoniu Song, Lei Wang, Qiang Yin, Rong Chen, Wenyuan Yu, and Jingren Zhou.  Gnnlab: A factored system for sample-based gnn training over gpus.  In Proceedings of the Seventeenth European Conference on Computer Systems, EuroSys ’22, page 417–434, New York, NY, USA, 2022. Association for Computing Machinery.
  * [61] Chang Ye, Yuchen Li, Bingsheng He, Zhao Li, and Jianling Sun.  Gpu-accelerated graph label propagation for real-time fraud detection.  In Proceedings of the 2021 International Conference on Management of Data, SIGMOD ’21, page 2348–2356, New York, NY, USA, 2021. Association for Computing Machinery.
  * [62] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec.  Graph convolutional neural networks for web-scale recommender systems.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 974–983, New York, NY, USA, 2018. Association for Computing Machinery.
  * [63] Muhan Zhang and Yixin Chen.  Link prediction based on graph neural networks.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 5171–5181, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * [64] Chenguang Zheng, Hongzhi Chen, Yuxuan Cheng, Zhezheng Song, Yifan Wu, Changji Li, James Cheng, Hao Yang, and Shuai Zhang.  Bytegnn: Efficient graph neural network training at large scale.  Proc. VLDB Endow., 15(6):1228–1242, feb 2022.
  * [65] Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, and George Karypis.  Distdgl: Distributed graph neural network training for billion-scale graphs.  In 2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3), pages 36–44, 2020.
  * [66] Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, and George Karypis.  Distdgl: Distributed graph neural network training for billion-scale graphs, 2021.
  * [67] Rong Zhu, Kun Zhao, Hongxia Yang, Wei Lin, Chang Zhou, Baole Ai, Yong Li, and Jingren Zhou.  Aligraph: A comprehensive graph neural network platform.  Proc. VLDB Endow., 12(12):2094–2105, aug 2019.