memory that arise between adjacent tensors. This problem is particularly
pronounced during the training of LLMs due to the varying lifetimes of
different tensors and the inefficient memory allocation and deallocation
schemes of general deep learning frameworks, such as PyTorch [240] and
TensorFlow [326]. Furthermore, memory optimization techniques like
recomputation and offloading exacerbate the issue by introducing more frequent
and irregular memory allocation and deallocation requests [308, 306, 299]. The
fragmentation problem could cause high peak memory and out-of-memory (OOM)
errors limiting the batch size and overall training efficiency.
Defragmentation efforts are proposed to mitigate these issues through memory
management techniques.

####  6.3.1 Tensor-based Defragmentation

Deep learning frameworks typically use a caching allocator with a memory pool
to enable fast memory allocation and deallocation without requiring device
synchronization. Several approaches have been proposed to reduce memory
fragmentation based on the tensor allocation and deallocation scheme in the
caching allocator. ROAM [306] co-optimizes the execution order of operators
and tensor allocation by considering tensors’ lifetimes and sizes. It
introduces an efficient tree-based algorithm to search for an execution plan
that maximizes tensor reuse and reduces data fragmentation. ROAM has been
evaluated in single-GPU scenarios, specifically with the largest model being
the 1.5B GPT-2 XL [5], but it has not yet been tested in distributed training
scenarios with larger models, where computation graphs can become
significantly larger. Imanishi et al. [307] present an offline optimization
approach by modeling tensor allocation as a 2D bin-packing problem. In this
model, each tensor allocation is represented as a vertically movable
rectangle, reflecting periodic allocation patterns during model training. They
propose a heuristic algorithm using simulated annealing to optimize the
topological ordering of allocations, aiming to minimize fragmentation. While
effective, this method may struggle with scalability issues when applied to
LLMs due to the high number of allocations and complex patterns involved.
MegTaiChi [298] and Coop [299] consider memory fragmentation issues when
evicting activation tensors for reducing memory consumption.

####  6.3.2 VMM-based Defragmentation

GMLake [308] and PyTorch expandable segments [309] propose to mitigate
fragmentation by utilizing the virtual memory management (VMM) functions of
the low-level CUDA driver application programming interface. This low-level
API provides developers with direct control over the GPU’s virtual memory
operations, such as reserving, mapping, and managing virtual memory addresses.
Building on this, GMLake [308] introduces a virtual memory stitching mechanism
that consolidates non-contiguous memory blocks into larger ones through
virtual memory address mapping, minimizing data movement and copying.
Similarly, PyTorch’s expandable segments [309] enable allocated memory
segments to be expanded to larger sizes for reuse. Both approaches are
transparent to different models and memory-efficient training techniques and
can be seamlessly integrated into existing deep learning frameworks.
Furthermore, GMLake demonstrates excellent scalability on multi-GPUs with
minimal overhead and does not require modification to user code. PyTorch-v2.1
has also integrated expandable segments.

###  6.4 Offloading

To enable efficient training of LLMs on fewer GPUs, various works leveraging
swap and offload methods have been proposed. These techniques transfer parts
of the computation and data from GPU memory to external resources, which are
inexpensive and slower but enjoy vast capacity.

####  6.4.1 CPU Offloading

Numerous studies have proposed methods to efficiently utilize CPU memory to
enhance distributed LLM training. These techniques can be broadly categorized
into two main approaches: Static Offloading and Dynamic Offloading.

Static Offloading. Static offloading methods involve a predetermined
allocation of model components between GPU and CPU memory. L2L [310] manages
and moves tensors layer by layer. L2L synchronously fetches tensors required
for the upcoming computational layers into GPU memory while keeping the
tensors for the remaining layers stored in CPU memory. L2L allows scaling the
models to arbitrary depth but fails to scale across multi-GPUs. In contrast,
ZeRO-Offload [311] concentrates on multi-GPU training. It holds model
parameters on GPU, and stores optimizer states and gradients on CPU memory. In
addition, it offloads optimizer update computation to the CPU. This method
enables the training of up to 70B models with 16 V100s. However, ZeRO-Offload
can leave some GPU memory unused and suffers from slow CPU optimizer updates
[312]. To address this issue, Elixir [312] employs a search engine to find the
optimal combination of memory partitioning and offloading by leveraging pre-
runtime model profiling. Unlike ZeRO-Offload, Elixir effectively utilizes all
available GPU memory by partitioning both the model states and optimizer
chunks between GPU and CPU. Mobius [315] tackles multi-GPU training on
commodity servers with limited inter-GPU bandwidth and high communication
contention by introducing a pipeline parallelism scheme. This scheme assigns
each GPU multiple stages and dynamically swaps them between GPU and CPU
memory. Additionally, Mobius optimizes communication through prefetching and
cross-mapping to reduce overhead and contention. Yuan et al. [301] propose to
mitigate the activation bottleneck by offloading and reloading activations at
the granularity of pipeline stages while maximizing the overlap between
activation transmission with computation, thereby avoiding slowing the
training process. Compared to other offloading efforts, this work focuses more
on improving the balance between computation and memory utilization rather
than training with extremely tight memory budgets.

####  6.4.2 Dynamic Offloading

Dynamic offloading methods adaptively allocate partitions of model or tensors
between GPU and CPU memory based on real-time optimization of memory
utilization and data transmission. STRONGHOLD [318] proposes to dynamically
offload model states between GPU and CPU memory and maintain a suitable
working window size to minimize GPU stalls during offloading. Harmony [316]
employs a heuristic-based scheduler to map computation and model states to
physical devices. Harmony reduces the overhead for offloading with reduced
swaps and fast peer-to-peer swaps. TMOF [317] introduces disjoint swapping and
bidirectional overlapping coordination mechanisms to prevent PCIe channel
contention in swapping and offloading. For MoE models, MPipeMoE[327] designs
an adaptive and memory-efficient pipeline parallelism algorithm. Specifically,
MPipeMoE employs efficient memory reusing strategies by eliminating memory
redundancies and an adaptive selection component to decide whether to offload
or recompute the required tensors to reduce memory requirements.

To facilitate better memory management, some studies have proposed systems
that break tensors into finer-grained units. TSPLIT [313] and PatrickStar
[314] are two dynamic memory management systems that optimize peak GPU memory
usage. TSPLIT splits tensors into micro-tensors and performs operations at the
micro-tensor level, enabling precise and dynamic memory operations.
PatrickStar organizes model data into memory chunks that are dynamically
distributed between CPU and GPU memory and optimizes CPU-GPU data transmission
as well as bandwidth utilization. Additionally, TSPLIT uses a model-guided
planning algorithm to find optimal memory configurations for each tensor,
while PatrickStar employs runtime memory tracing, chunk eviction strategies,
and device-aware operator placement to further minimize data movement between
CPU and GPU.

####  6.4.3 SSD Offloading

To facilitate the training of trillion-scale LLMs, where methods solely
relying on CPU offloading are insufficient, several works have been proposed
for offloading data to both CPU memory and NVMe SSDs during training. ZeRO-
Infinity [319] offloads all the partitioned model states to CPU or NVMe memory
and offloads activation only to CPU memory. This method supports training
models with up to 32T parameters on 32 nodes (a total of 512 V100s). However,
the CPU offloading for activations still requires extensive CPU memory. For
instance, approximately 0.76 TB of CPU memory is needed to store activation
checkpoints for training a 10T model, and around 4 TB for 100T models. Fuyou
[322] focuses on training LLMs on commodity servers with limited CPU memory
capacity and a single GPU. Compared to ZeRO-Infinity, Fuyou further offloads
the activations to SSDs and incorporates SSD-CPU communication as an
additional optimization dimension. It also proposes a synchronous out-of-core
CPU optimizer that overlaps with the backward propagation stage and introduces
an automatic activation swapping mechanism, thereby maximizing GPU
utilization. Smart-Infinity [321] proposes to reduce the secondary storage
bandwidth requirements by using near-storage processing devices for parameter
update. MoESys[323, 324] combines various storage devices (GPU, CPU memory,
and SSDs) to save the sparse parameter states and dense parameter states and
propose a 2D prefetch scheduling strategy to MoE training so that the
computation of parameters can be overlapped with the scheduling.

##  7 Communication Optimizations

Figure 13: Communication traffic heatmap for InternLM-2 102B pre-training
using 128 GPUs during a single iteration, with tensor parallelism (TP) size 8,
pipeline parallelism (PP) size 4, data parallelism (DP) size 4 and ZeRO stage
1 (ZeRO-1) size 4. The prioritization of topology arrangement is TP >DP/ZeRO-1
>PP. There are four different data traffic loads: 1the AllReduce of TP; 2
missingmissing3 ReduceScatter/AllGather of DP/ZeRO-1; 4Send/Recv of PP. The
communication for TP utilizes the fully-connected topology of NVSwitch,
resulting in sixteen dense square traffic patterns along the diagonals in the
diagram, with each pattern representing a node. The cross-node communication
traffic for DP and ZeRO-1 are shown in the diagram as six symmetric diagonal
lines within the four 32x32 rectangular topologies. It is important to note
that DP/ZeRO-1 also involves intra-node communication traffic, which
accumulates into the same heatmap grid as TP. Due to its relatively small
communication volume, PP forms two yellow lines on the heatmap at coordinates
((32, 0), (128, 96)) and ((0, 32), (96, 128)). (In this diagram, all
communications use the ring-based collective algorithm)

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=center, font=, rectangle, draw=black,
rounded corners, align=left, text centered, minimum width=4em, edge+=black,
line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, line width=0.8pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=15em,font=,, where level=2text width=15em,font=,, [
Communication Optimizations for LLM Training, ver [ Collective Communication ,
fill=blue!10 [ Pre-Defined Algorithm, fill=yellow!10 [ Library: MPI [328, 329,
330], NCCL [331], RCCL [332]
Algorithm: Ring [333], Tree [334], Hybrid [335, 336, 337, 338, 339], leaf,
text width= 26em ] ] [ Synthesized Algorithm, fill=yellow!10 [ GC3 [340], SCCL
[341], TACCL [342], Blink [343],
$P^{2}$ [344], leaf, text width= 26em ] ] ] [ Communication Scheduling,
fill=blue!10 [ FIFO-based Scheduling, fill=yellow!10 [ Poseidon [345],
GradientFlow [346],
Pytorch DDP [142], leaf, text width= 26em ] ] [ Priority-based Scheduling,
fill=yellow!10 [ P3 [347], TicTac [348], ByteScheduler [349],
PACE [350], Lina [188], leaf, text width= 26em ] ] [ Decomposition-based
Scheduling, fill=yellow!10 [ Pipeline Stage Decomp.: Breadth-First [159],
Fold3D [351], TriRace [352]
Communication Decomp.: Wang et al. [353],
SYNDICATE [354], Centauri [355], DeAR [356]
Computation Decomp.: CoCoNet [357], T3 [358],
Oases [359], Lynx [360]
Out-of-order backpropagation (ooo) [361], leaf, text width= 26em ] ] ] [ In-
Network Aggregation, fill=blue!10 [ Ethernet-based Aggregation, fill=yellow!10
[ SwitchML [362], FPISA [363], NetReduce [364],
AllReduce-Switch [365], PANAMA [366], ATP [367], leaf, text width= 26em ] ] [
Infiniband-based Aggregation, fill=yellow!10 [ NVIDIA Mellanox’s SHARP
v1/v2/v3 [368], leaf, text width= 26em ] ] ] ]

Figure 14: Studies on communication optimizations for distributed LLM
training.

Different parallelism mechanisms introduce varying patterns of network
communication traffic. For instance, tensor parallelism requires AllReduce
operations across the tensor parallelism ranks. Data parallelism, on the other
hand, necessitates AllReduce operations for gradient synchronization across
data parallelism ranks at the end of each iteration. Pipeline parallelism
involves passing activation values to the next stage at the end of each stage.
Typically, training frameworks place tensor or sequence parallel communication
groups, which demand high bandwidth, within high-bandwidth domains (e.g., the
same node), while placing data parallel or pipeline parallel communication
groups, which have lower bandwidth requirements, between high-bandwidth
domains. Fig. 13 shows the communication heatmap of LLM training in practice
and well reflects the data traffic brought by different parallel strategies.
From this heatmap, it can be observed that the LLM training communication
traffic exhibits a clear pattern and hierarchy, with most communication
occurring within smaller scopes, and only a little fraction of the traffic
crossing the entire cluster. This insight has inspired approaches like rail-
optimized topology [62], which reduces unnecessary core switches to cut costs.

This section introduces systems and techniques for optimizing the collective
communication performance of distributed LLM training. As shown in Fig. 14, we
first discuss collective communication libraries, which utilize both pre-
defined and synthesized algorithms. Next, we explore communication scheduling
techniques designed to reorganize communication operations to overlap with
computation, thereby reducing delays and accelerating the training process.
Finally, we delve into in-network aggregation (INA), which leverages the
computational capabilities of network devices to perform aggregation
operations, such as summing gradients of deep learning models.

Compressing model parameters and gradients effectively reduces communication
overhead during distributed LLM training. Various studies explore sparse
communication and quantization approaches. For example, ZeRO++ [302] adopt
quantization on weights to shrink down each model parameter from FP16 to INT8
data type before communicating. However, these works typically involve lossy
sparsification or quantization techniques. We do not survey lossy data
compression techniques in this section, as they are beyond the scope of this
work.

###  7.1 Collective Communication

The Message Passing Interface (MPI) is a widely adopted programming model for
large-scale scientific applications on parallel computing architectures. MPI
has several implementations, including OpenMPI [328], MPICH2 [329], and
MVAPICH [330]. These libraries provide a variety of CUDA-aware primitives such
as AllReduce, AllGather, and ReduceScatter, which are essential for
distributed LLM training. In practice, current training frameworks prefer
collective communications tailored to specific AI accelerators with pre-
defined or synthesized algorithms.

####  7.1.1 Pre-Defined Collective Communication Algorithm

NVIDIA’s NCCL [331] and AMD’s RCCL [332] are highly optimized libraries that
typically outperform MPI-based collective communication libraries on their
respective AI accelerators. These libraries usually select pre-defined
algorithms to perform collectives based on conditions such as network topology
and input tensor size.

Ring Algorithm. The Ring algorithm is used for collective communications like
AllReduce to move data across all GPUs. With this algorithm, the input tensor
is split into multiple chunks and transferred one by one during the operation.
This pipeline reduces the idle time that each device spends waiting for data.
Baidu used the bandwidth-optimal ring AllReduce algorithm [333] for
distributed deep learning model training. Horovod [143] replaced the Baidu
ring-AllReduce implementation with NCCL and designed a user-friendly interface
for distributed training.

Tree Algorithm. The latency of the Ring algorithm increases with the number of
GPU devices [346]. The Double Binary Tree algorithm [334] was proposed to
solve this problem. Double binary trees rely on the fact that half or fewer
ranks in a binary tree are nodes and half or more ranks are leaves. Therefore,
a second tree can be built using leaves as nodes and vice-versa for each
binary tree. This algorithm is implemented in MPI-based libraies, NCCL and
RCCL.

Hybrid Algorithm. Several approaches propose using hybrid algorithms to handle
collective communication tasks on training clusters with heterogeneous intra-
node and inter-node communication bandwidth. Two-level AllReduce [335] divides
a single AllReduce operation into three steps: intra-node Reduce utilizing
PCIe/NVLINK, inter-node AllReduce utilizing network, and intra-node Broadcast.
2D-Torus AllReduce [336] and ACCL [337] decompose a single AllReduce operation
into three phases: intra-node ring-based ReduceScatter, inter-node tree-based
AllReduce, and intra-node ring-based AllGather. BlueConnect [338] breaks down
a single AllReduce operation into numerous parallelizable ReduceScatter and
AllGather operations. Each operation can be mapped to different network
fabrics, leveraging the best-performing pre-defined implementation for each
specific fabric. Plink [339] could probes network topology and efficiently
generates two-level hybrid communication plans, exploiting locality in
datacenter networks.

####  7.1.2 Synthesized Collective Communication Algorithm

Several approaches have emerged that synthesize collective communication
algorithms and kernels specifically tailored to the hardware topology, aiming
to outperform generic pre-defined algorithms in many cases. GC3 [340]
introduces a data-oriented domain-specific language (DSL) for designing custom
collective communication algorithms. It includes an optimizing compiler that
translates these algorithms into executable forms optimized for specific
hardware configurations. SCCL [341] encodes the collective communication
synthesis problem as an SMT (Satisfiability Modulo Theories) formula. This
approach aims to derive exact schedules for Pareto-optimal algorithms,
optimizing both latency and bandwidth utilization. TACCL [342] formulates the
problem of finding optimal communication algorithms as a mixed integer linear
program (MILP). It leverages a communication sketch abstraction to efficiently
gather essential information and reduce the search space, with the goal of
minimizing overall execution time. Blink [343] dynamically constructs a
topology with suitable link capacities by probing available link sets for each
job at runtime. Using this topology, it optimizes communication rates through
the creation of packet generation trees, and generating CUDA code. $P^{2}$
[344] utlizes parallel matrices to partition the parallel axis at the system
level, thereby generating topology-aware parallel placement and reduction
strategies. By simulating and predicting communication costs, this method
reduces the number of actual evaluations required.

###  7.2 Communication Scheduling

Communication scheduling in distributed training reorganizes communication
operations to overlap with computation, thereby reducing delays and
accelerating the training process. The key concept of communication scheduling
involves reordering communication operations based on the data dependencies of
parallel training. Hybrid parallel LLM training necessitates multidimensional
communication scheduling schemes to manage communications generated by data,
pipeline, tensor, and sequence parallelism, as well as their combinations.

####  7.2.1 FIFO-based Scheduling

During the backward phase, rather than waiting for all gradient calculations
to complete before initiating communication, communication can begin as soon
as each gradient is ready. This wait-free backpropagation approach leverages a
dependency-directed acyclic graph to manage tasks efficiently. Poseidon [345]
employs a First-In-First-Out (FIFO) queue to schedule AllReduce operators,
ensuring that each layer starts its communication once its gradients are
generated. Motivated by the efficiency of collective communications on large
tensors, GradientFlow [346] and Pytorch DDP [142] fuse multiple sequential
AllReduce communication operations into a single operation. This method avoids
transmitting a large number of small tensors over the network by waiting for a
short period of time and then combining multiple gradients into one AllReduce
operation during the backward phase.

####  7.2.2 Priority-based Scheduling

The FIFO scheme is often sub-optimal because the generated communication
sequence of in the backward phase differs from the computation sequence in the
forward phase. This mismatch can lead to communication blocking computation,
even when overlap is enabled. Consequently, many approaches employ priority
queues to schedule communication operators efficiently. P3 [347] schedules
AllReduce operations at a finer granularity, overlapping gradient
communication of the current layer with the forward computation of the next
layer. Unlike FIFO queue-based scheduling, this method divides layers into
fixed-sized slices and prioritizes synchronizing slices based on the order in
which they are processed in forward propagation. Therefore, the first layer
gets the highest priority, with priority decrementing towards the end. When
utilizing the parameter-server architecture for distributed model training,
TicTac [348] prioritizes transfers that accelerate the critical path within
the underlying computational graph.

ByteScheduler [349] and PACE [350] are proposed to generalize priority-based
communication scheduling across training frameworks. Specifically,
ByteScheduler [349] introduces a unified abstraction to facilitate
communication scheduling without disrupting the original dependencies within
framework engines. ByteScheduler achieves good performance by using Bayesian
optimization to automatically tunes two critical parameters: partition size
and credit size. PACE [350] implements preemptive communications by segmenting
primitive AllReduce operations into smaller pieces. The preempted AllReduce
operators can be resumed at a later time. This preemption strategy prevents
head-of-line blocking of large communication tensors. Additionally, PACE uses
a dynamic programming approach to fuse small communication tensors to reduce
the overhead caused by handling a large number of small tensors, thereby
achieving more efficient bandwidth utilization.

To improve bandwidth efficiency in MoE systems, Lina [188] prioritizes All-to-
All operations over AllReduce. Typically, expert-parallel (All-to-All) and
data-parallel (AllReduce) processes use separate CUDA streams, causing
potential overlap and bandwidth sharing without coordination. Lina breaks
tensors into smaller chunks, ensures All-to-All operations get full bandwidth
while allowing AllReduce micro-ops to run during idle time. In addition,
micro-ops enable overlap All-to-All operations with expert computations.

####  7.2.3 Decomposition-based Scheduling

Several advancements have focused on decomposing communication and computation
operations into fine-grained tasks, reordering these operations with greater
flexibility to maximize overlap and optimize execution efficiency.

Pipeline Stage Decomposition. When using conventional pipeline parallelism,
each GPU stores a contiguous segment of layers. Breadth-First [159] further
splits these contiguous stages into finer-grained stages distributed across
different GPUs, forming a loop by connecting the first and last GPUs, so each
GPU is assigned multiple stages. This allows the given micro-batch to reach
the end of the pipeline earlier, reducing pipeline bubbles. Breadth-First uses
a breadth-first scheduling strategy to achieve greater computation-
communication overlap. Fold3D [351] employs an all-in-all-out scheduling
strategy to overlap the pipeline’s gradient synchronization process with
computation. This involves further folding model fragments within the
pipeline, where each device contains two model fragments, allowing one
fragment’s gradient synchronization to overlap with another fragment’s forward
or backward computation.

Asynchronous pipeline parallelism relaxes data dependencies between gradients
and parameter updates. Leveraging this characteristic, TriRace [352] postpones
parameter updates to maximize computation overlap with gradient communication.
Additionally, TriRace decomposes bidirectional P2P communication between
pipeline stages into two separate unidirectional operations and prioritizes
them based on critical path analysis.

Communication Decomposition. Communication primitives could be decomposed into
fine-grained operations with high scheduling flexibility. Wang et al. [353]
decomposed communication operations (e.g., AllGather and ReduceScatter), into
a series of fine-grained peer-to-peer collections. In addition, computational
operations (e.g., Einstein Summation) were divided into fine-grained tasks,
each performing a part of the computation. This decomposition creates more
opportunities for overlapping communication with computation. SYNDICATE [354]
segments communication operations into smaller sub-operations, termed Motifs,
and employs a Central Optimizer using Markov Chain Monte Carlo search to
achieve optimal overlap execution plans. Centauri [355] adopts a different
approach by using Primitive Partition, Group Partition, and Workload Partition
to decompose communication operations into fine-grained atomic operations.
These operations are then scheduled using Workload-aware Scheduling, Backward
Scheduling, and Elastic Scheduling to maximize overlap efficiency. DeAR [356]
also decomposes communication primitives, specifically breaking down AllReduce
into AllGather and ReduceScatter. This decomposition allows subsequent
operations to overlap with the forward propagation process of the model, thus
eliminating the need to wait for the completion of both communication steps.

Computation Decomposition. When using tensor parallelism, an AllReduce
communication is required to synchronize the matrix multiplication outputs in
the forward phase. CoCoNet [357] facilitates the overlap of matrix
multiplication and AllReduce by partitioning the output into smaller blocks
and immediately initiating the AllReduce kernel after computing each result
block within the matrix multiplication kernel. To minimize waiting time for
the AllReduce kernel, the data blocks are fed into the matrix multiplication
kernel in a carefully scheduled order. T3 [358] applies a hardware-software
co-design approach, which transparently overlaps matrix multiplication with
communication while minimizing resource contention. At the hardware level, T3
introduces a track-and-trigger mechanism to orchestrate the producer’s compute
and communication activities. Additionally, it employs compute-enhanced
memories to handle the attendant compute operations required by the
communication processes.

The backward pass generates two types of gradients: the output gradient, which
is used to calculate the gradients of the preceding layer, and the weight
gradient, which is used to update the layer’s weight parameters. These weight
gradients need to be synchronized with other ranks using AllReduce.
Conventional frameworks simultaneously perform gradient computation for both
weights and outputs. Out-of-order backpropagation (ooo-backprop) [361]
decouples the gradient computations for weights and outputs, scheduling the
weight gradient computations flexibly out of their original order. This allows
more critical computations to be prioritized and scheduled accordingly.
Consequently, ooo-backprop optimizes overall performance by scheduling
communications based on this out-of-order computation strategy. This scheme is
also used by Zero Bubble [156] to reduce the bubble rate of pipeline
parallelism.

With activation checkpointing enabled, training frameworks need to recompute
activations during the backward phase. This recomputation also involves
AllReduce communication when using tensor parallelism. Oases [359] reduces
redundant communication in recomputation by always placing AllReduce
communication as the last forward communication operation of a recomputation
unit, and further splits the batch into smaller sub-batches, allowing the
communication and computation of two batches to overlap. Lynx [360] also
exploits the potential of recomputation and communication overlap, using two
recomputation scheduling algorithms, OPT and HEU, to search for the optimal or
near-optimal recomputation scheduling strategy, achieving the best overlap and
training performance.

###  7.3 In-Network Aggregation

In-network aggregation (INA) uses the computational capabilities of network
devices to perform aggregation operations like summing gradients of deep
learning models. This technique has been previously proposed to accelerate big
data processing. Notably, frameworks like NetAgg [369], SwitchAgg [370], and
CamDoop [371] have demonstrated significant performance advantages by
executing data aggregation at switch-attached high-performance middleboxes or
servers within a direct-connect topology. Many approaches have been proposed
to apply in-network aggregation to deep learning model training, aiming to
reduce the data exchanged between nodes during AllReduce operations on
gradients in the backward phase [372].

####  7.3.1 Ethernet-based Aggregation

Many Ethernet-based in-network aggregation systems depend on programmable
switches, and can be leveraged for distributed LLM training. SwitchML [362]
supports offloading collective communication operations to programmable
network switches during the backward phase of distributed training. Since
complete model updates can exceed the storage capacity of a switch, SwitchML
streams the aggregation through the switch, processing the aggregation
function on a limited number of vector elements at a time. There are two
limitations of SwitchML. First, when dealing with floating-point operations,
SwitchML cannot directly perform collective communications (such as AllReduce
) for floating-point tensors. Instead, it converts floating-point values into
32-bit integers using a block floating-point-like approach. Second, SwitchML
is primarily implemented on DPDK, and while there is an RDMA-capable
implementation, it is difficult to integrate with training frameworks.

To better facilitate distributed model training, FPISA [363] implements
floating-point computation as a P4 [373] program running directly on a
programmable switch. Therefore, training frameworks could offload collective
communication operations on FP16 tensors to switches without converting them
to 32-bit integers. NetReduce [364] supports in-network aggregation compatible
with RoCE, fully utilizing the congestion control and reliability design of
RoCE without the need for costly network protocol processing stacks in
switches. NetReduce is prototyped with an FPGA board attached to an Ethernet
switch. AllReduce-Switch [365] is closely related to NetReduce and compatible
with its network protocol. It introduces a novel switch architecture tailored
for in-network aggregation tasks and has implemented a prototype using FPGA
hardware. PANAMA [366] and ATP [367] have also contributed to the field with
their in-network aggregation frameworks designed for shared environments.
PANAMA focuses on optimizing network load by managing bandwidth allocation
among multiple active training jobs concurrently. It addresses the challenge
that traditional congestion controls may not adequately support simultaneous
training operations. ATP, on the other hand, enables multiple concurrent
tenants to run several jobs simultaneously, emphasizing the support for
diverse workloads in shared environments.

Certain works are tailored for specific training workloads, making them
unsuitable for distributed LLM training. For example, Libra [374] is designed
for sparse model training using a parameter server architecture. It offloads
the aggregation of frequently updated parameters to programmable switches,
leaving infrequently updated parameters to be handled by servers. This
approach effectively reduces server load. On the other hand, iSwitch [375] is
designed for parameter aggregation in reinforcement learning training tasks.
Although its FPGA-based implementation supports native floating point
operations, it operates at a significantly lower bandwidth. Furthermore,
iSwitch stores an entire gradient vector during aggregation, which is feasible
for reinforcement learning workloads but does not scale well for large-scale
models, especially LLMs.

####  7.3.2 Infiniband-based Aggregation

NVIDIA Mellanox’s Scalable Hierarchical Aggregation Protocol (SHARP) [368] is
a proprietary in-network aggregation scheme available in certain InfiniBand
switches and NVIDIA GPUs. Built on InfiniBand, SHARP leverages link-layer flow
control and lossless guarantees and employs dedicated on-chip FPUs for
collective offloading. SHARPv1 was introduced on InfiniBand EDR switches, and
SHARPv2 was enhanced on InfiniBand HDR switches with features such as support
for collective communications (e.g., Barrier, Reduce, AllReduce and
Broadcast), integer and floating-point operations (16/32/64 bits), and
GPUDirect RDMA. SHARPv2 also uses streaming aggregation for large vector
reductions at line rate, integrates with NCCL, and is easily usable by
existing training frameworks. Enabled on the latest InfiniBand NDR switches,
SHARP is production-ready for distributed LLM training and has been deployed
in many training clusters. In addition to Infiniband, NVIDIA’s NVSwitch-v3
[46] also integrates SHARP to speed up collective operations in GPU-based
clusters.

##  8 Fault Tolerance

LLM training typically involves extended training periods ranging from weeks
to months, utilizing clusters of tens of thousands of GPUs. The vast array of
components involved, spanning from the underlying infrastructure to training
system optimizations, necessitates robust fault tolerance mechanisms to ensure
the reliability of training processes. This is because a single point of
failure in any part of the system can result in a suspension of the training
process due to the synchronous nature of the training. In this section, we
first present a failure analysis in LLM training, then investigate the
approaches for fast failure detection and recovery.

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=center, font=, rectangle, draw=black,
rounded corners, align=left, text centered, minimum width=4em, edge+=black,
line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, line width=0.8pt,
ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center,
, where level=1text width=15em,font=,, where level=2text width=15em,font=,, [
Fault Tolerance for LLM Training, ver [ Anomaly Detection, fill=blue!10 [
Statistical Monitoring, fill=yellow!10 [ Healthd in TPUv4 [376], MegaScale
[71], C4 [377],
Vela [378], Unicron [379], Transom [380], NCCLX [9],
NCCL flight recorder [273], leaf, text width= 26em ] ] [ Proactive Validation,
fill=yellow!10 [ Lightweight Test in MegaScale [71], SuperBench [381],
Vela [378], Preflight Check in TPUv4 [376], leaf, text width= 26em ] ] ] [
Checkpointing-Based Recovery, fill=blue!10 [ Persistent Checkpoiting,
fill=yellow!10 [ Synchronous: DeepSpeed [22], Varuna [167],
JIT-Checkpointing [382], Flash-Checkpoint [383],
Universal Checkpointing [384]
Snapshot-Stall: Check-N-Run [385],
TorchSnapshot [386]
Asynchronous: DeepFreeze [387], CheckFreq[388],
LightCheck [389], DataStates-LLM [390],
FastPersist [391], leaf, text width= 26em ] ] [ In-Memory Checkpoiting,
fill=yellow!10 [ Gemini [392], REFT [393], leaf, text width= 26em ] ] ] [
Checkpointing-Free Recovery, fill=blue!10 [ Live Migration, fill=yellow!10 [
Parcae [394], Oobleck [395], leaf, text width= 26em ] ] [ Module Redundancy,
fill=yellow!10 [ Bamboo [396], SlipStream [397], SWARM [231], leaf, text
width= 26em ] ] ] ]

Figure 15: Studies on fault tolerance techniques for distributed LLM training.

###  8.1 LLM Failure Analysis

Empirical evidence from various sources underscores the frequency of failures
in LLM training. For example, the training of Bloom experiences 1-2 GPU
failures per week on average on a cluster with 384 GPUs [292]. Meta’s
comprehensive training records [398] of 175B OPT model on 992 A100 GPUs
document over 40 interruptions within a two-week period, attributed to
hardware, infrastructure, and other external factors. More recent studies
further highlight this issue. Acme [23] reported failure occurrences every 1-2
days on average in their training process using over 1,000 A100 GPUs.
ByteDance’s MegaScale project [71], utilizing 12,288 Ampere GPUs, experiences
over 100 times failures in several weeks. Meta’s LLaMA3 experiences 466 job
interruptions during a 54-day period of pre-training on a cluster of 16,384
H100 GPUs [9]. The frequent failure is primarily attributed to the immense
complexity and scale of these systems and extended training periods. The whole
training system encompasses a vast array of components as we investigated in
previous sections. Moreover, the synchronized training further exacerbates
this issue, as errors in any single node can cause the entire job to fail,
making the system particularly vulnerable to even isolated hardware or
software faults. Even a seemingly low $1.5\%$ daily failure rate for a single
node, as observed in Alibaba’s cluster [383], translates to a staggering
$84.8\%$ daily failure rate when scaled to a system with 1,000 GPUs. However,
the trend of scaling up the training system continues to grow, emphasizing the
concomitant challenges for fault tolerance mechanisms to maintain system
reliability.

The reasons behind these failures are multifaceted and stem from various
components of the LLM training system. According to Acme [23], the most severe
impact comes from hardware failures, such as issues with GPU (e.g., CUDA-
Error, ECC-Error), NVLink, and network system (e.g., NCCL-Timeout-Error,
Connection-Error). Similar observations are also delivered in Alibaba C4
[377]. C4 further observes that the majority of errors (about $82.5\%$) are
confined to specific nodes or even individual devices, although most errors
observed by users are NCCL error. LLaMA3 pre-training [9] also reports that
$78\%$ of the failures are hardware issues. Moreover, the latest generation
GPUs (A100 and H100) tend to exhibit high error rates, likely due to rapid
development, rushed delivery, and increased power consumption [377, 399].
Beyond hardware, software-related issues in distributed training frameworks,
data preprocessing pipelines, or library dependencies can lead to crashes or
unexpected behavior [23, 399, 378]. The complex nature of the models
themselves can introduce instabilities such as loss spikes, numerical overflow
or underflow, gradient explosions, or optimization difficulties [398, 400].
External factors like power outages or cooling system failures in data centers
further contribute to system instabilities. For example, the high temperature
in the cluster server room also tends to result in GPU overheating, which can
cause NVLink-Error or ECC-Error [23] or unstable training speed [9].

These high frequent and multifaceted LLM failures lead to significant waste of
GPUs. This inefficiency manifests in two primary ways: failure recovery and
performance degradation. First, LLM training jobs periodically save
checkpoints during runtime to maintain progress. Upon failure, system
maintainers must first locate and diagnose the issue before restarting the
training by rolling back to previous checkpoints. Some hardware failures,
however, can be challenging to detect proactively and often require
considerable time to diagnose and recover from, resulting in prolonged stalls
in LLM training. Second, stragglers in the cluster, caused by network link
failures [377] or abnormal computational slowdowns [71], can significantly
decrease the MFU, further compounding the overall training inefficiency. The
training of Meta’s 175B OPT model exemplifies these inefficiencies [398].
While the ideal training time was estimated at about 25 days based on the MFU,
the actual training lasted approximately 57 days. This means that a staggering
$56\%$ of the total time was wasted handling various failures, underscoring
the severe impact of system instabilities on resource utilization and training
efficiency in LLM training.

###  8.2 Anomaly Detection

Rapid detection and diagnosis of LLM failures are crucial for maintaining
training stability and efficiency. This process, known as anomaly detection,
primarily employs two approaches: statistical monitoring and proactive
validation.

####  8.2.1 Statistical Monitoring

Statistical monitoring is a systematic approach to observing and analyzing
various metrics and indicators throughout the LLM training process. This
method involves collecting, processing, and interpreting data to identify
anomalies or deviations from expected behavior. In a typical setup, each GPU
is assigned a dedicated monitoring process responsible for collecting basic
information and runtime statistics [71, 379, 378]. These statistics are then
transmitted to a central monitor node as heartbeat messages for further
analysis. Nodes that fail to send heartbeat messages are considered to have
failed. The primary objective of this monitoring system is to detect anomalies
promptly, allowing for quick recovery that minimize training interruptions and
maintain overall efficiency.

Most runtime statistics monitored in LLM training are hardware-related,
encompassing both GPU and network metrics. Recent works [71, 379, 378] collect
GPU-related statistics with NVIDIA DCGM [401], including SM block utilization,
SM occupancy, SM pipe utilization, PCIe traffic rate, NVLink traffic rate and
etc. One frequently occurring issue is GPU memory row-remapping, which
seamlessly replaces known degraded memory cells with sparse ones in hardware.
Vela [378] detects this by leveraging the DCGM_FI_DEV_ROW_REMAP_PENDING
statistics from DCGM. Megascale [71] and Transom [380] also detect errors by
analyzing errors occurred in training logs.

In addition to GPU metrics, network statistics are crucial for monitoring
distributed training performance. MegaScale [71] tracks RDMA traffic metrics
to detect potential anomalies. It also develops visualization system to
identify inefficiency GPUs manually. Unicron [379] detects errors like NCCL
timeout, TCP timeout, and task hangs with delayed notification during
training. C4 [377] gathers connection specifics such as RDMA IP and QP
numbers, along with message statistics including counts, sizes, and durations
of transfers at the transport layer to detect training slowdowns and hangs.
Collective communication activities can also be monitored with PyTorch’s
built-in NCCL flight recorder [273], which captures collective metadata and
stack traces into a ring buffer for later diagnosis. Meta further co-designs
NCCLX [9] with PyTorch, allowing PyTorch to access its internal state for fast
and accurate failure detection. NCCLX traces the kernel and network activities
of each NCCLX communication, which can help diagnose communication issues.
Vela [378] implements an enhanced Multi-NIC health checker that collects node
network bandwidth data for all 2-node pairs on every port. This information
can be utilized to detect nodes with degraded RoCE/GDR performance. Leveraging
the key characteristics of LLMs training as prior knowledge, Transom [380]
develops machine learning algorithms to do anomaly detection.

Statistical monitoring also enables the resiliency of Google’s TPUv4
supercomputer [376]. Each TPUv4 machine is equipped with a healthd daemon that
performs real-time monitoring of ICI (Inter-Chip Interconnect) links, PCIe
links and TPU ASIC. Detected severe symptoms will notify cluster scheduler for
appropriate action, such as evicting affected jobs or rescheduling them.

####  8.2.2 Proactive Validation

Proactive validation offers an alternative to reactive troubleshooting based
on online statistical monitoring, aiming to validate the training system
before failures occur. However, a trade-off exists between validation test
time and accuracy, as comprehensive validation can significantly impact
effective training time. MegaScale [71] introduces a suite of lightweight
tests, including intra-network host and NCCL tests, to diagnose a wide
spectrum of potential failures. Vela [378] employs a two-tiered strategy with
lightweight tests running periodically on every node and more intrusive tests
executed only when nodes are idle. Google’s TPUv4 supercomputer implements a
preflight check [376] before user jobs, consisting of an end-to-end check and
an intent-driven checker for hardware health. SuperBench [381] resents a
comprehensive benchmark suite for evaluating individual hardware components,
incorporating a selector to balance validation time against potential issue-
related penalties.

###  8.3 Checkpoint-Based Recovery

Periodically saving the model states, i.e., _checkpointing_ , and resuming
computation from the latest checkpoint after failures happen is the common
practice for fault tolerant LLM training. However, this presents a dilemma:
frequent checkpointing incurs high I/O overhead, while infrequent
checkpointing results in substantial progress loss when failures occur. To
address this dilemma, fast persistent and in-memory checkpointing approaches
have been designed.

####  8.3.1 Persistent Checkpointing

Persistent checkpointing involves saving model states to non-volatile storage,
e.g. SSD and remote cloud storage, ensuring data persistence across system
failures. The process typically consists of two phases: first, the _snapshot_
phase copies model states from GPU to CPU memory, and second, the _persist_
phase writes the snapshots to persistent storage devices. Despite considerable
I/O overhead due to the low bandwidth of storage devices, persistent
checkpointing remains a widely used approach for fault tolerance due to its
ease-of-use and reliability. Advanced persistent checkpointing approaches have
been proposed to reduce training stall, thereby enabling more frequent
checkpointing without significant performance penalties.

Synchronous Checkpointing. To keep consistency of model parameters,
DeepSpeed’s default synchronous checkpointing [22] and Varuna [167]
periodically stalls the training process to perform checkpointing to
persistent storage synchronously on data parallel rank 0. This approach
results in GPU idle time during both the snapshot and persist phases, leading
to resource underutilization. Recognizing that most failures are attributable
to a single GPU or network device, JIT-Checkpointing [382] proposes an
alternative strategy. It takes just-in-time checkpoints immediately after
failures occur, allowing training to resume from these JIT checkpoints. This
approach significantly reduces the cost of wasted GPU time, limiting it to at
most one mini-batch iteration of work. DLRover Flash-Checkpoint [383]
accelerates the migration efficiency utilizing a distributed caching service.
Universal Checkpointing [384] introduces a universal checkpoint representation
to decouple the distributed checkpoints storage from parallelism techniques.
Universal Checkpointing can seamlessly transform checkpoints from one
parallelization strategy to another upon demands.

Snapshot-Stall Checkpointing. To reduce LLM training stalls while maintaining
checkpoint consistency, Check-N-Run [385] decouples the snapshot and persist
phases. It achieves atomic checkpointing by stalling training only during the
snapshot phase and asynchronously persisting snapshots using dedicated
background CPU processes. TorchSnapshot [386] further optimizes this process
through tensor chunking and multi-threaded disk writing. By creating chunked
snapshots, TorchSnapshot allows the persist phase to begin earlier through
parallel writing, thereby reducing overall training stall time. MegaScale [71]
and InternEvo [18] also adopt a snapshot-stall approach for fast checkpointing
and recovery. The snapshot phase stalls training for several seconds to
capture the model states, while the persist phase asynchronously transfers
checkpoints from CPU memory to a distributed file system. MegaScale optimizes
the recovery process by designating a single worker within the data parallel
group to read from the distributed file system, thus mitigating the low
bandwidth bottleneck. This worker then broadcasts the checkpoint data to other
GPUs, enabling faster and more efficient recovery across the entire system. To
save storage space, InternEvo also asynchronously moves checkpoints from
expensive hot storage to cheaper cold storage.

Asynchronous Checkpointing. Asynchronous checkpointing aims to minimize
training stall by executing the snapshot and persist phases concurrently with
training. DeepFreeze [387] applies both lightweight (snapshot) and heavy
(persist) persistence strategies in the background, sharding checkpoints
across data-parallel GPUs to distribute I/O workload. CheckFreq [388]
carefully pipelines the snapshot and persist phases with subsequent
iteration’s forward and backward passes, ensuring snapshot completion before
the next parameter update. It also dynamically tunes checkpointing frequency
to balance recovery costs and runtime overhead. LightCheck [389] exploits
inter-iteration data dependencies, introducing layer-wise checkpointing
pipeline to reduce stall. DataStates-LLM [390] addresses slow host memory
allocation by pre-allocating pinned host memory for snapshots and utilizes
efficient computation, snapshot, and persist layer-wise pipelining.
FastPersist [391] identifies risks in fully asynchronous persist phases and
synchronizes them with the next iteration’s parameter update. It improves SSD
bandwidth utilization through double-buffering pinned memory and reduces
hardware contention by using a subset of data-parallel ranks for checkpoint
writing.

####  8.3.2 In-Memory Checkpointing

The low bandwidth of remote persistent storage severely restricts the
frequency of checkpointing, in-memory checkpointing addresses the limitations
by storing checkpoints in the memory of other compute nodes or dedicated in-
memory storage systems, significantly reducing I/O overhead and enabling
higher checkpointing frequencies. Gemini [392] proposes checkpointing to CPU
memory for faster failure recovery, along with a checkpoint placement strategy
to minimize checkpoint loss and a traffic scheduling algorithm to reduce
interference with training. REFT [393] asynchronously caches model states to
host memory and in-memory storage like Redis, bypassing checkpoint I/O and
enabling high checkpointing frequency. It also leverages erasure coding to
implement RAIM5 (inspired by RAID5 with “Disk” replaced by “Memory”) that
protects data against node failures. While these approaches significantly
advance fault tolerance for LLM training by enabling more frequent
checkpointing without performance penalties, they may not provide the same
long-term data persistence as traditional storage-based methods. Consequently,
hybrid approaches combining both in-memory and persistent checkpointing is
necessary for comprehensive fault tolerance strategies.

###  8.4 Checkpoint-Free Recovery

Checkpoint-free recovery methods aim to minimize training stalls by
eliminating the need to restart and roll back to previous checkpoints when
failures occur. These techniques depend on automatic failure detection
mechanisms to identify issues promptly. When a failure is detected,
checkpoint-free approaches automatically address the problem and allow the
training process to continue without interruption. By avoiding the time-
consuming process of loading from a checkpoint and repeating computations,
these methods can significantly reduce downtime and improve overall training
efficiency. Checkpoint-free recovery strategies can be broadly categorized
into two main approaches: live migration and module redundancy.

####  8.4.1 Live Migration

Live migration leverages the inherent redundancy present in distributed LLM
training setups, particularly the model replicas across different data
parallel pipelines, to restore model states in case of failure. When a failure
is detected, live migration approaches dynamically reconfigure the
parallelization strategy using the remaining healthy instances or by
incorporating new instances into the training cluster. The current model
states are then transferred to these reconfigured nodes, allowing the training
process to continue with minimal interruption. Parcae [394] proposes three
distinct migration mechanisms, each with different communication overheads, to
efficiently transfer model states between varying parallelization strategies.
Oobleck [395] takes a pipeline template-based approach to live migration. It
maintains a set of predefined pipeline templates and, upon detecting a
failure, swiftly instantiates new heterogeneous pipelines based on these
templates.

####  8.4.2 Module Redundancy

Module redundancy, like live migration, also leverages the redundancy of model
states. However, instead of restoring the latest model states across different
GPUs, this approach continues training by routing computation to redundant
modules. Bamboo [396] places a redundant pipeline stage in the GPU holding an
adjacent pipeline stage within the same pipeline. This redundant stage
performs redundant computations during training, utilizing pipeline bubbles,
and is activated as a normal stage upon failure. SlipStream [397] leverages
the redundancy across model replica pipelines, routing the computation of
failed nodes to nodes in different data parallel pipelines. SWARM [231]
proposes a similar solution but focuses more on poorly connected,
heterogeneous, and unreliable devices. In addition to redundant computation,
SWARM also incorporates instance migration to rebalance the pipeline,
combining aspects of both redundant computation and live migration approaches.

##  9 Conclusion and Outlooks

The rise of LLMs has transformed AI, enabling applications like personal
assistants, code generation, and scientific research. Models such as GPT,
LLaMA, and Gemini have set new standards, but training these massive models,
exemplified by LLaMA-3’s 54-day process on 16,384 GPUs, presents challenges in
scalability, efficiency, and reliability. Managing vast GPU clusters requires
innovative hardware and networking solutions. Efficient training demands
optimizing computation, communication, and memory use. Reliability involves
robust mechanisms to detect and recover from failures over long training
periods. This survey reviews recent advancements in LLM training systems and
infrastructure, highlighting approaches to enhance scalability, efficiency,
and reliability.

Traditional digital circuit-based computing systems, guided by Moore’s Law and
Dennard Scaling, are facing significant physical and economic constraints in
meeting the computational demands for training and deploying LLMs.
Consequently, the AI industry necessitates innovative solutions. One promising
approach is large-scale optoelectronic integration technology, which replaces
traditional digital circuits with integrated silicon photonics to enhance
computational and communication capabilities [402]. This optoelectronic hybrid
data center technology combines optical computing with optical networks,
increasing single-node computing power and the efficiency of large-scale
distributed computing. Several works have proposed leveraging optical networks
for LLM training. For instance, TopoOpt [67] optimizes both the optical
network topology and parallelization strategies in distributed training,
enhancing computational and communication efficiency. TPUv4 [42] uses Optical
Circuit Switches to dynamically reconfigure its 3D-Torus interconnect
topology, improving data flow for the intensive communication patterns in LLM
training. Additionally, Taichi [403] explores a distributed diffractive-
interference hybrid photonic computing architecture to effectively scale the
optical neural network to the million-neuron level with 160 tera-operations
per second per watt (TOPS/W) energy efficiency. The future may necessitate a
paradigm shift in LLM training and inference towards silicon photonics.
However, this transition will require extensive innovation across system
design and implementation.

## References

  * [1] X. L. Dong, S. Moon, Y. E. Xu, K. Malik, and Z. Yu, “Towards next-generation intelligent assistants leveraging llm techniques,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023, pp. 5792–5793.
  * [2] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [3] M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu _et al._ , “Chipnemo: Domain-adapted llms for chip design,” _arXiv preprint arXiv:2311.00176_ , 2023.
  * [4] A. Jo, “The promise and peril of generative ai,” _Nature_ , vol. 614, no. 1, pp. 214–216, 2023.
  * [5] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [6] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [7] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth _et al._ , “Gemini: a family of highly capable multimodal models,” _arXiv preprint arXiv:2312.11805_ , 2023.
  * [8] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” _arXiv preprint arXiv:2001.08361_ , 2020.
  * [9] LlamaTeam. (2024) The llama 3 herd of models. [Online]. Available: https://ai.meta.com/research/publications/the-llama-3-herd-of-models
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [11] N. Shazeer, “Fast transformer decoding: One write-head is all you need,” _arXiv preprint arXiv:1911.02150_ , 2019.
  * [12] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” _arXiv.org_ , 2023.
  * [13] DeepSeek-AI, “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,” _arXiv preprint arXiv:2405.04434_ , 2024.
  * [14] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” _Neural computation_ , vol. 3, no. 1, pp. 79–87, 1991.
  * [15] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” _arXiv preprint arXiv:2006.16668_ , 2020.
  * [16] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _3rd International Conference on Learning Representations_ , ser. ICLR ’15, 2015.
  * [17] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , 1998.
  * [18] Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu _et al._ , “Internlm2 technical report,” _arXiv preprint arXiv:2403.17297_ , 2024.
  * [19] T. Sun, X. Zhang, Z. He, P. Li, Q. Cheng, X. Liu, H. Yan, Y. Shao, Q. Tang, S. Zhang _et al._ , “Moss: An open conversational large language model,” _Machine Intelligence Research_ , pp. 1–18, 2024.
  * [20] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” _arXiv preprint arXiv:1909.08053_ , 2019.
  * [21] L. Zheng, Z. Li, H. Zhang, Y. Zhuang, Z. Chen, Y. Huang, Y. Wang, Y. Xu, D. Zhuo, E. P. Xing _et al._ , “Alpa: Automating inter-and $\\{$Intra-Operator$\\}$ parallelism for distributed deep learning,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022, pp. 559–578.
  * [22] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 3505–3506.
  * [23] Q. Hu, Z. Ye, Z. Wang, G. Wang, M. Zhang, Q. Chen, P. Sun, D. Lin, X. Wang, Y. Luo _et al._ , “Characterization of large language model development in the datacenter,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 709–729.
  * [24] C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He _et al._ , “A comprehensive survey on pretrained foundation models: A history from bert to chatgpt,” _arXiv preprint arXiv:2302.09419_ , 2023.
  * [25] Y. Huang, J. Xu, Z. Jiang, J. Lai, Z. Li, Y. Yao, T. Chen, L. Yang, Z. Xin, and X. Ma, “Advancing transformer architecture in long-context large language models: A comprehensive survey,” _arXiv preprint arXiv:2311.12351_ , 2023.
  * [26] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [27] S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang, “Instruction Tuning for Large Language Models: A Survey,” _arXiv preprint arXiv:2308.10792_ , 2023.
  * [28] Y. Wang, W. Zhong, L. Li, F. Mi, X. Zeng, W. Huang, L. Shang, X. Jiang, and Q. Liu, “Aligning large language models with human: A survey,” _arXiv preprint arXiv:2307.12966_ , 2023.
  * [29] Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury, and M. Zhang, “Efficient Large Language Models: A Survey,” _arXiv.2312.03863_ , 2023.
  * [30] Y. Liu, H. He, T. Han, X. Zhang, M. Liu, J. Tian, Y. Zhang, J. Wang, X. Gao, T. Zhong _et al._ , “Understanding llms: A comprehensive overview from training to inference,” _arXiv preprint arXiv:2401.02038_ , 2024.
  * [31] M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y. Zhao, C. Yang, S. Wang, Q. Zhang, Z. Lu, L. Zhang, S. Wang, Y. Li, Y. Liu, X. Jin, and X. Liu, “A Survey of Resource-efficient LLM and Multimodal Foundation Models,” _arXiv preprint arXiv:2401.08092_ , 2024.
  * [32] X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “A survey on model compression for large language models,” _arXiv preprint arXiv:2308.07633_ , 2023.
  * [33] Z. Han, C. Gao, J. Liu, S. Q. Zhang _et al._ , “Parameter-efficient fine-tuning for large models: A comprehensive survey,” _arXiv preprint arXiv:2403.14608_ , 2024.
  * [34] X. He, F. Xue, X. Ren, and Y. You, “Large-scale deep learning optimizations: A comprehensive survey,” _arXiv preprint arXiv:2111.00856_ , 2021.
  * [35] R. Mayer and H.-A. Jacobsen, “Scalable deep learning on distributed infrastructures: Challenges, techniques, and tools,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 1, pp. 1–37, 2020.
  * [36] P. Liang, Y. Tang, X. Zhang, Y. Bai, T. Su, Z. Lai, L. Qiao, and D. Li, “A survey on auto-parallelism of large-scale deep learning training,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 34, no. 8, pp. 2377–2390, 2023.
  * [37] NVIDIA. NVIDIA Ampere Architecture. [Online]. Available: {https://www.nvidia.com/en-us/data-center/ampere-architecture}
  * [38] ——. NVIDIA Hopper Architecture. [Online]. Available: {https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture}
  * [39] ——. NVIDIA Blackwell Architecture. [Online]. Available: {https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture}
  * [40] R. Swaminathan, M. J. Schulte, B. Wilkerson, G. H. Loh, A. Smith, and N. James, “Amd instinct tm mi250x accelerator enabled by elevated fanout bridge advanced packaging architecture,” in _2023 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits)_. IEEE, 2023, pp. 1–2.
  * [41] Habana. (2023) Gaudi training platform white paper. [Online]. Available: https://www.intel.com/content/www/us/en/content-details/784830/gaudi-training-platform-white-paper.html
  * [42] N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles _et al._ , “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , 2023, pp. 1–14.
  * [43] M. Emani, S. Foreman, V. Sastry, Z. Xie, S. Raskar, W. Arnold, R. Thakur, V. Vishwanath, and M. E. Papka, “A comprehensive performance study of large language models on novel ai accelerators,” _arXiv preprint arXiv:2310.04607_ , 2023.
  * [44] J.-P. Fricker, “The cerebras cs-2: Designing an ai accelerator around the world’s largest 2.6 trillion transistor chip,” in _Proceedings of the 2022 International Symposium on Physical Design_ , 2022, pp. 71–71.
  * [45] NVIDIA, “Nvidia dgx-1 system architecture white paper,” NVIDIA, White Paper, 2017.
  * [46] J. Choquette, E. Lee, R. Krashinsky, V. Balan, and B. Khailany, “3.2 the a100 datacenter gpu and ampere architecture,” in _2021 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 64. IEEE, 2021, pp. 48–50.
  * [47] S. Naffziger, K. Lepak, M. Paraschou, and M. Subramony, “2.2 amd chiplet architecture for high-performance server and desktop products,” in _2020 IEEE International Solid-State Circuits Conference-(ISSCC)_. IEEE, 2020, pp. 44–45.
  * [48] N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson, “A domain-specific supercomputer for training deep neural networks,” _Communications of the ACM_ , vol. 63, no. 7, pp. 67–78, 2020.
  * [49] T. Norrie, N. Patil, D. H. Yoon, G. Kurian, S. Li, J. Laudon, C. Young, N. P. Jouppi, and D. A. Patterson, “Google’s training chips revealed: Tpuv2 and tpuv3.” in _Hot Chips Symposium_ , 2020, pp. 1–70.
  * [50] G. Shainer, A. Ayoub, P. Lui, T. Liu, M. Kagan, C. R. Trott, G. Scantlen, and P. S. Crozier, “The development of mellanox/nvidia gpudirect over infiniband—a new model for gpu to gpu communications,” _Computer Science-Research and Development_ , vol. 26, pp. 267–273, 2011.
  * [51] G. F. Pfister, “An introduction to the infiniband architecture,” _High performance mass storage and parallel I/O_ , vol. 42, no. 617-632, p. 10, 2001.
  * [52] Infiniband Trade Association, “Supplement to infiniband architecture specification volume 1 release 1.2.2 annex a16,” pp. 1–17, 2010.
  * [53] ——, “Supplement to infiniband architecture specification volume 1 release 1.2.2 annex a17,” pp. 1–17, 2010.
  * [54] RDMA Consortium. Architectural Specifications for RDMA over TCP/IP . [Online]. Available: {http://www.rdmaconsortium.org}
  * [55] C. Clos, “A study of non-blocking switching networks,” _Bell System Technical Journal_ , vol. 32, no. 2, pp. 406–424, 1953.
  * [56] C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu, “Bcube: a high performance, server-centric network architecture for modular data centers,” in _Proceedings of the ACM SIGCOMM 2009 conference on Data communication_ , 2009, pp. 63–74.
  * [57] C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, “Dcell: a scalable and fault-tolerant network structure for data centers,” in _Proceedings of the ACM SIGCOMM 2008 conference on Data communication_ , 2008, pp. 75–86.
  * [58] A. Singla, C.-Y. Hong, L. Popa, and P. B. Godfrey, “Jellyfish: Networking data centers randomly,” in _9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)_ , 2012, pp. 225–238.
  * [59] J. Duato, S. Yalamanchili, and L. Ni, _Interconnection networks_. Morgan Kaufmann, 2003.
  * [60] J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” _ACM SIGARCH Computer Architecture News_ , vol. 36, no. 3, pp. 77–88, 2008.
  * [61] A. Shpiner, Z. Haramaty, S. Eliad, V. Zdornov, B. Gafni, and E. Zahavi, “Dragonfly+: Low cost topology for scaling datacenters,” in _2017 IEEE 3rd International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB)_. IEEE, 2017, pp. 1–8.
  * [62] K. Mandakolathur and S. Jeaugey. (2018) Doubling all2all performance with nvidia collective communication library 2.12. [Online]. Available: https://developer.nvidia.com/blog/nvswitch-leveraging-nvlink-to-maximum-effect/
  * [63] K. Qian, Y. Xi, J. Cao, J. Gao, Y. Xu, Y. Guan, B. Fu, X. Shi, F. Zhu, R. Miao _et al._ , “Alibaba hpn: A data center network for large language model training,” in _Proceedings of the ACM SIGCOMM 2024 Conference_ , 2024.
  * [64] W. Wang, M. Ghobadi, K. Shakeri, Y. Zhang, and N. Hasani, “Optimized network architectures for large language model training with billions of parameters,” _arXiv preprint arXiv:2307.12169_ , 2023.
  * [65] T. Hoefler, T. Bonato, D. De Sensi, S. Di Girolamo, S. Li, M. Heddes, J. Belk, D. Goel, M. Castro, and S. Scott, “Hammingmesh: a network topology for large-scale deep learning,” in _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_. IEEE, 2022, pp. 1–18.
  * [66] M. Khani, M. Ghobadi, M. Alizadeh, Z. Zhu, M. Glick, K. Bergman, A. Vahdat, B. Klenk, and E. Ebrahimi, “Sip-ml: high-bandwidth optical network interconnects for machine learning training,” in _Proceedings of the 2021 ACM SIGCOMM 2021 Conference_ , 2021, pp. 657–675.
  * [67] W. Wang, M. Khazraee, Z. Zhong, M. Ghobadi, Z. Jia, D. Mudigere, Y. Zhang, and A. Kewitsch, “$\\{$TopoOpt$\\}$: Co-optimizing network topology and parallelization strategy for distributed training jobs,” in _20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)_ , 2023, pp. 739–767.
  * [68] C. Hopps. (2000) Analysis of an equal-cost multi-path algorithm. [Online]. Available: https://www.rfc-editor.org/rfc/rfc2992.html
  * [69] A. Dixit, P. Prakash, Y. C. Hu, and R. R. Kompella, “On the impact of packet spraying in data center networks,” in _2013 proceedings ieee infocom_. IEEE, 2013, pp. 2130–2138.
  * [70] V. Addanki, P. Goyal, and I. Marinos, “Challenging the need for packet spraying in large-scale distributed training,” _arXiv preprint arXiv:2407.00550_ , 2024.
  * [71] Z. Jiang, H. Lin, Y. Zhong, Q. Huang, Y. Chen, Z. Zhang, Y. Peng, X. Li, C. Xie, S. Nong _et al._ , “$\\{$MegaScale$\\}$: Scaling large language model training to more than 10,000 $\\{$GPUs$\\}$,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 745–760.
  * [72] IEEE. 802.1qbb – priority-based flow control. [Online]. Available: https://1.ieee802.org/dcb/802-1qbb
  * [73] R. Mittal, V. T. Lam, N. Dukkipati, E. Blem, H. Wassel, M. Ghobadi, A. Vahdat, Y. Wang, D. Wetherall, and D. Zats, “Timely: Rtt-based congestion control for the datacenter,” _ACM SIGCOMM Computer Communication Review_ , vol. 45, no. 4, pp. 537–550, 2015.
  * [74] G. Kumar, N. Dukkipati, K. Jang, H. M. Wassel, X. Wu, B. Montazeri, Y. Wang, K. Springborn, C. Alfeld, M. Ryan _et al._ , “Swift: Delay is simple and effective for congestion control in the datacenter,” in _Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication_ , 2020, pp. 514–528.
  * [75] Y. Zhu, H. Eran, D. Firestone, C. Guo, M. Lipshteyn, Y. Liron, J. Padhye, S. Raindel, M. H. Yahia, and M. Zhang, “Congestion control for large-scale rdma deployments,” _ACM SIGCOMM Computer Communication Review_ , vol. 45, no. 4, pp. 523–536, 2015.
  * [76] Y. Zhu, M. Ghobadi, V. Misra, and J. Padhye, “Ecn or delay: Lessons learnt from analysis of dcqcn and timely,” in _Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies_ , 2016, pp. 313–327.
  * [77] Y. Li, R. Miao, H. H. Liu, Y. Zhuang, F. Feng, L. Tang, Z. Cao, M. Zhang, F. Kelly, M. Alizadeh _et al._ , “Hpcc: High precision congestion control,” in _Proceedings of the ACM special interest group on data communication_ , 2019, pp. 44–58.
  * [78] V. Olteanu, H. Eran, D. Dumitrescu, A. Popa, C. Baciu, M. Silberstein, G. Nikolaidis, M. Handley, and C. Raiciu, “An edge-queued datagram service for all datacenter traffic,” in _19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)_ , 2022, pp. 761–777.
  * [79] P. Taheri, D. Menikkumbura, E. Vanini, S. Fahmy, P. Eugster, and T. Edsall, “Rocc: robust congestion control for rdma,” in _Proceedings of the 16th International conference on emerging networking experiments and technologies_ , 2020, pp. 17–30.
  * [80] S. Rajasekaran, S. Narang, A. A. Zabreyko, and M. Ghobadi, “Mltcp: Congestion control for dnn training,” _arXiv preprint arXiv:2402.09589_ , 2024.
  * [81] S. Rajasekaran, M. Ghobadi, and A. Akella, “$\\{$CASSINI$\\}$:$\\{$Network-Aware$\\}$ job scheduling in machine learning clusters,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 1403–1420.
  * [82] H. Wang, H. Tian, J. Chen, X. Wan, J. Xia, G. Zeng, W. Bai, J. Jiang, Y. Wang, and K. Chen, “Towards $\\{$Domain-Specific$\\}$ network transport for distributed $\\{$DNN$\\}$ training,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 1421–1443.
  * [83] S. Pan, T. Stavrinos, Y. Zhang, A. Sikaria, P. Zakharov, A. Sharma, M. Shuey, R. Wareing, M. Gangapuram, G. Cao _et al._ , “Facebook’s tectonic filesystem: Efficiency from exascale,” in _19th USENIX Conference on File and Storage Technologies (FAST 21)_ , 2021, pp. 217–231.
  * [84] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop distributed file system,” in _2010 IEEE 26th symposium on mass storage systems and technologies (MSST)_. Ieee, 2010, pp. 1–10.
  * [85] S. Weil, S. A. Brandt, E. L. Miller, D. D. Long, and C. Maltzahn, “Ceph: A scalable, high-performance distributed file system,” in _Proceedings of the 7th Conference on Operating Systems Design and Implementation (OSDI’06)_ , 2006, pp. 307–320.
  * [86] P. Schwan _et al._ , “Lustre: Building a file system for 1000-node clusters,” in _Proceedings of the 2003 Linux symposium_ , vol. 2003, 2003, pp. 380–386.
  * [87] F. Schmuck and R. Haskin, “$\\{$GPFS$\\}$: A $\\{$Shared-Disk$\\}$ file system for large computing clusters,” in _Conference on file and storage technologies (FAST 02)_ , 2002.
  * [88] F. Chowdhury, Y. Zhu, T. Heer, S. Paredes, A. Moody, R. Goldstone, K. Mohror, and W. Yu, “I/o characterization and performance evaluation of beegfs for deep learning,” in _Proceedings of the 48th International Conference on Parallel Processing_ , 2019, pp. 1–10.
  * [89] H. Li, A. Ghodsi, M. Zaharia, S. Shenker, and I. Stoica, “Tachyon: Reliable, memory speed storage for cluster computing frameworks,” in _Proceedings of the ACM Symposium on Cloud Computing_ , 2014, pp. 1–15.
  * [90] JuiceFS. Juicefs: A High-Performance, Cloud-Native, Distributed File System . [Online]. Available: {https://github.com/juicedata/juicefs}
  * [91] A. V. Kumar and M. Sivathanu, “Quiver: An informed storage cache for deep learning,” in _18th USENIX Conference on File and Storage Technologies (FAST 20)_ , 2020, pp. 283–296.
  * [92] R. Gu, K. Zhang, Z. Xu, Y. Che, B. Fan, H. Hou, H. Dai, L. Yi, Y. Ding, G. Chen _et al._ , “Fluid: Dataset abstraction and elastic acceleration for cloud-native deep learning training jobs,” in _2022 IEEE 38th International Conference on Data Engineering (ICDE)_. IEEE, 2022, pp. 2182–2195.
  * [93] J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. Liu, and C. Guo, “Tiresias: A $\\{$GPU$\\}$ cluster manager for distributed deep learning,” in _16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)_ , 2019, pp. 485–500.
  * [94] K. Mahajan, A. Balasubramanian, A. Singhvi, S. Venkataraman, A. Akella, A. Phanishayee, and S. Chawla, “Themis: Fair and efficient $\\{$GPU$\\}$ cluster scheduling,” in _17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20)_ , 2020, pp. 289–304.
  * [95] D. Gu, Y. Zhao, Y. Zhong, Y. Xiong, Z. Han, P. Cheng, F. Yang, G. Huang, X. Jin, and X. Liu, “Elasticflow: An elastic serverless training platform for distributed deep learning,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 266–280.
  * [96] D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, and M. Zaharia, “Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads,” in _14th USENIX Symposium on Operating Systems Design and Implementation_ , ser. OSDI ’20. USENIX Association, 2020, pp. 481–498.
  * [97] S. Chaudhary, R. Ramjee, M. Sivathanu, N. Kwatra, and S. Viswanatha, “Balancing efficiency and fairness in heterogeneous gpu clusters for deep learning,” in _Proceedings of the Fifteenth European Conference on Computer Systems_ , ser. EuroSys ’20. Association for Computing Machinery, 2020.
  * [98] Q. Weng, L. Yang, Y. Yu, W. Wang, X. Tang, G. Yang, and L. Zhang, “Beware of fragmentation: Scheduling GPU-Sharing workloads with fragmentation gradient descent,” in _2023 USENIX Annual Technical Conference_ , ser. USENIX ATC ’23. USENIX Association, 2023, pp. 995–1008.
  * [99] Q. Hu, M. Zhang, P. Sun, Y. Wen, and T. Zhang, “Lucid: A non-intrusive, scalable and interpretable scheduler for deep learning training jobs,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 457–472.
  * [100] A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing, “Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning,” in _15th USENIX Symposium on Operating Systems Design and Implementation_ , ser. OSDI ’21. USENIX Association, 2021, pp. 1–18.
  * [101] S. Jayaram Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, and G. R. Ganger, “Sia: Heterogeneity-aware, goodput-optimized ml-cluster scheduling,” in _Proceedings of the 29th Symposium on Operating Systems Principles_ , 2023, pp. 642–657.
  * [102] C. Xue, W. Cui, H. Zhao, Q. Chen, S. Zhang, P. Yang, J. Yang, S. Li, and M. Guo, “A codesign of scheduling and parallelization for large model training in heterogeneous clusters,” _arXiv preprint arXiv:2403.16125_ , 2024.
  * [103] Q. Hu, Z. Ye, M. Zhang, Q. Chen, P. Sun, Y. Wen, and T. Zhang, “Hydro: Surrogate-Based hyperparameter tuning service in datacenters,” in _17th USENIX Symposium on Operating Systems Design and Implementation_ , ser. OSDI ’23. USENIX Association, 2023.
  * [104] M. Blöcher, L. Wang, P. Eugster, and M. Schmidt, “Switches for hire: Resource scheduling for data center in-network computing,” in _Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2021, pp. 268–285.
  * [105] H. Zhao, Z. Han, Z. Yang, Q. Zhang, M. Li, F. Yang, Q. Zhang, B. Li, Y. Yang, L. Qiu _et al._ , “Silod: A co-design of caching and scheduling for deep learning clusters,” in _Proceedings of the Eighteenth European Conference on Computer Systems_ , 2023, pp. 883–898.
  * [106] J. Mohan, A. Phanishayee, J. Kulkarni, and V. Chidambaram, “Looking beyond $\\{$GPUs$\\}$ for $\\{$DNN$\\}$ scheduling on $\\{$Multi-Tenant$\\}$ clusters,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022, pp. 579–596.
  * [107] S. Choi, I. Koo, J. Ahn, M. Jeon, and Y. Kwon, “$\\{$EnvPipe$\\}$: Performance-preserving $\\{$DNN$\\}$ training framework for saving energy,” in _2023 USENIX Annual Technical Conference (USENIX ATC 23)_ , 2023, pp. 851–864.
  * [108] J. You, J.-W. Chung, and M. Chowdhury, “Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training,” in _Proc. USENIX NSDI_ , 2023.
  * [109] J.-W. Chung, Y. Gu, I. Jang, L. Meng, N. Bansal, and M. Chowdhury, “Perseus: Removing energy bloat from large model training,” _arXiv preprint arXiv:2312.06902_ , 2023.
  * [110] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” _arXiv preprint arXiv:2312.15234_ , 2023.
  * [111] J. Choquette, “Nvidia hopper gpu: Scaling performance,” in _2022 IEEE Hot Chips 34 Symposium (HCS)_. IEEE Computer Society, 2022, pp. 1–46.
  * [112] D. Schneider, “The exascale era is upon us: The frontier supercomputer may be the first to reach 1,000,000,000,000,000,000 operations per second,” _IEEE spectrum_ , vol. 59, no. 1, pp. 34–35, 2022.
  * [113] S. Dash, I. R. Lyngaas, J. Yin, X. Wang, R. Egele, J. A. Ellis, M. Maiterth, G. Cong, F. Wang, and P. Balaprakash, “Optimizing distributed training on frontier for large language models,” in _ISC High Performance 2024 Research Paper Proceedings (39th International Conference)_. Prometeus GmbH, 2024, pp. 1–11.
  * [114] J. Yin, S. Dash, J. Gounley, F. Wang, and G. Tourassi, “Evaluation of pre-training large language models on leadership-class supercomputers,” _The Journal of Supercomputing_ , vol. 79, no. 18, pp. 20 747–20 768, 2023.
  * [115] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 16 344–16 359, 2022.
  * [116] T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” _arxiv:2307.08691_ , 2023.
  * [117] C. Zhang, B. Sun, X. Yu, Z. Xie, W. Zheng, K. A. Iskra, P. Beckman, and D. Tao, “Benchmarking and in-depth performance study of large language models on habana gaudi processors,” in _Proceedings of the SC’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis_ , 2023, pp. 1759–1766.
  * [118] N. Dey, G. Gosal, H. Khachane, W. Marshall, R. Pathria, M. Tom, J. Hestness _et al._ , “Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster,” _arXiv preprint arXiv:2304.03208_ , 2023.
  * [119] Z. Zhang, C. Chang, H. Lin, Y. Wang, R. Arora, and X. Jin, “Is network the bottleneck of distributed training?” in _Proceedings of the Workshop on Network Meets AI & ML_, 2020, pp. 8–13.
  * [120] L. Dai, H. Qi, W. Chen, and X. Lu, “High-speed data communication with advanced networks in large language model training,” _IEEE Micro_ , 2024.
  * [121] Y. Zhao, Y. Liu, Y. Peng, Y. Zhu, X. Liu, and X. Jin, “Multi-resource interleaving for deep learning training,” in _Proceedings of the ACM SIGCOMM 2022 Conference_ , 2022, pp. 428–440.
  * [122] K. Kong, “Using pci express® as the primary system interconnect in multiroot compute, storage, communications and embedded systems,” _White Paper, Integrated Device Technology_ , p. 12, 2008.
  * [123] A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker, “Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 31, no. 1, pp. 94–110, 2019.
  * [124] NVIDIA. (2018) Nvidia dgx-2: The world’s most powerful system for the most complex ai challenges. [Online]. Available: {https://www.nvidia.com/en-us/data-center/dgx-2/}
  * [125] A. C. Elster and T. A. Haugdahl, “Nvidia hopper gpu and grace cpu highlights,” _Computing in Science & Engineering_, vol. 24, no. 2, pp. 95–100, 2022.
  * [126] “Infiniband product guide,” 2020, accessed: 2024-07-01. [Online]. Available: https://network.nvidia.com/files/doc-2020/br-infiniband-product-guide.pdf
  * [127] “Roce vs. iwarp competitive analysis,” 2017. [Online]. Available: https://network.nvidia.com/sites/default/files/pdf/whitepapers/WP_RoCE_vs_iWARP.pdf
  * [128] Nvidia. (2024) Dgx superpod architecture. [Online]. Available: https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/dgx-superpod-architecture.html
  * [129] J. Dong, Z. Cao, T. Zhang, J. Ye, S. Wang, F. Feng, L. Zhao, X. Liu, L. Song, L. Peng _et al._ , “Eflops: Algorithm and system co-design for a high performance distributed training platform,” in _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2020, pp. 610–622.
  * [130] C. Guo, H. Wu, Z. Deng, G. Soni, J. Ye, J. Padhye, and M. Lipshteyn, “Rdma over commodity ethernet at scale,” in _Proceedings of the 2016 ACM SIGCOMM Conference_ , 2016, pp. 202–215.
  * [131] Kevin Lee, Adi Gangidi and Mathew Oldham. Building Meta’s GenAI Infrastructure. [Online]. Available: {https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure}
  * [132] J. Qiu, H. Lv, Z. Jin, R. Wang, W. Ning, J. Yu, C. Zhang, P. Chu, Y. Qu, R. Peng _et al._ , “Wanjuan-cc: A safe and high-quality open-sourced english webtext dataset,” _arXiv preprint arXiv:2402.19282_ , 2024.
  * [133] Z. Ye, W. Gao, Q. Hu, P. Sun, X. Wang, Y. Luo, T. Zhang, and Y. Wen, “Deep learning workload scheduling in gpu datacenters: A survey,” _ACM Computing Surveys_ , vol. 56, p. 1–38, 2024.
  * [134] A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “Pipedream: Fast and efficient pipeline parallel dnn training,” _arXiv preprint arXiv:1806.03377_ , 2018.
  * [135] D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in _Proceedings of the 27th ACM symposium on operating systems principles_ , 2019, pp. 1–15.
  * [136] S. Li and T. Hoefler, “Chimera: efficiently training large-scale neural networks with bidirectional pipelines,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2021, pp. 1–14.
  * [137] W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, and Y. Jia, “$\\{$AntMan$\\}$: Dynamic scaling on $\\{$GPU$\\}$ clusters for deep learning,” in _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)_ , 2020, pp. 533–548.
  * [138] W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang _et al._ , “Gandiva: Introspective cluster scheduling for deep learning,” in _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_ , 2018, pp. 595–610.
  * [139] M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, and F. Yang, “Analysis of large-scale multi-tenant GPU clusters for DNN training workloads,” in _2019 USENIX Annual Technical Conference_ , ser. USENIX ATC ’19. USENIX Association, 2019, pp. 947–960.
  * [140] Q. Hu, P. Sun, S. Yan, Y. Wen, and T. Zhang, “Characterization and prediction of deep learning workloads in large-scale gpu datacenters,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , ser. SC ’21, 2021.
  * [141] Q. Weng, W. Xiao, Y. Yu, W. Wang, C. Wang, J. He, Y. Li, L. Zhang, W. Lin, and Y. Ding, “$\\{$MLaaS$\\}$ in the wild: Workload analysis and scheduling in $\\{$Large-Scale$\\}$ heterogeneous $\\{$GPU$\\}$ clusters,” in _19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)_ , 2022, pp. 945–960.
  * [142] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania _et al._ , “Pytorch distributed: Experiences on accelerating data parallel training,” _arXiv preprint arXiv:2006.15704_ , 2020.
  * [143] A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” _arXiv preprint arXiv:1802.05799_ , 2018.
  * [144] Y. Xu, H. Lee, D. Chen, H. Choi, B. Hechtman, and S. Wang, “Automatic cross-replica sharding of weight update in data-parallel training,” _arXiv preprint arXiv:2004.13336_ , 2020.
  * [145] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models,” in _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_. IEEE, 2020, pp. 1–16.
  * [146] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer _et al._ , “Pytorch fsdp: Experiences on scaling fully sharded data parallel,” _Proceedings of the VLDB Endowment_ , vol. 16, no. 12, pp. 3848–3860, 2023.
  * [147] Z. Zhang, S. Zheng, Y. Wang, J. Chiu, G. Karypis, T. Chilimbi, M. Li, and X. Jin, “Mics: Near-linear scaling for training gigantic model on public,” _Proceedings of the VLDB Endowment_ , vol. 16, no. 1, pp. 37–50, 2022.
  * [148] Q. Xu and Y. You, “An efficient 2d method for training super-large deep learning models,” in _2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. IEEE, 2023, pp. 222–232.
  * [149] B. Wang, Q. Xu, Z. Bian, and Y. You, “Tesseract: Parallelize the tensor parallelism efficiently,” in _Proceedings of the 51st International Conference on Parallel Processing_ , 2022, pp. 1–11.
  * [150] Z. Bian, Q. Xu, B. Wang, and Y. You, “Maximizing parallelism in distributed training for huge neural networks,” _arXiv preprint arXiv:2105.14450_ , 2021.
  * [151] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu _et al._ , “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [152] S. Fan, Y. Rong, C. Meng, Z. Cao, S. Wang, Z. Zheng, C. Wu, G. Long, J. Yang, L. Xia _et al._ , “Dapple: A pipelined data parallel approach for training large models,” in _Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_ , 2021, pp. 431–445.
  * [153] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro _et al._ , “Efficient large-scale language model training on gpu clusters using megatron-lm,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2021, pp. 1–15.
  * [154] Z. Li, S. Zhuang, S. Guo, D. Zhuo, H. Zhang, D. Song, and I. Stoica, “Terapipe: Token-level pipeline parallelism for training large-scale language models,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 6543–6552.
  * [155] Z. Lin, Y. Miao, G. Xu, C. Li, O. Saarikivi, S. Maleki, and F. Yang, “Tessel: Boosting distributed execution of large dnn models via flexible schedule search,” in _2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2024, pp. 803–816.
  * [156] P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble pipeline parallelism,” in _The Twelfth International Conference on Learning Representations_ , 2023.
  * [157] Z. Liu, S. Cheng, H. Zhou, and Y. You, “Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2023, pp. 1–13.
  * [158] S. Ao, W. Zhao, X. Han, C. Yang, Z. Liu, C. Shi, and M. Sun, “Seq1f1b: Efficient sequence-level pipeline parallelism for large language model training,” _arXiv preprint arXiv:2406.03488_ , 2024.
  * [159] J. Lamy-Poirier, “Breadth-first pipeline parallelism,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [160] C. Jiang, Z. Jia, S. Zheng, Y. Wang, and C. Wu, “Dynapipe: Optimizing multi-task training through dynamic pipelines,” in _Proceedings of the Nineteenth European Conference on Computer Systems_ , 2024, pp. 542–559.
  * [161] J. Huang, Z. Zhang, S. Zheng, F. Qin, and Y. Wang, “Distmm: Accelerating distributed multimodal model training,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 1157–1171.
  * [162] B. Jeon, M. Wu, S. Cao, S. Kim, S. Park, N. Aggarwal, C. Unger, D. Arfeen, P. Liao, X. Miao _et al._ , “Graphpipe: Improving performance and scalability of dnn training with graph pipeline parallelism,” _arXiv preprint arXiv:2406.17145_ , 2024.
  * [163] T. Kim, H. Kim, G.-I. Yu, and B.-G. Chun, “Bpipe: Memory-balanced pipeline parallelism for training large language models,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 16 639–16 653.
  * [164] Q. Zhou, H. Wang, X. Yu, C. Li, Y. Bai, F. Yan, and Y. Xu, “Mpress: Democratizing billion-scale model training on multi-gpu servers via memory-saving inter-operator parallelism,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 556–569.
  * [165] H. Dreuning, H. E. Bal, and R. V. v. Nieuwpoort, “mcap: Memory-centric partitioning for large-scale pipeline-parallel dnn training,” in _European Conference on Parallel Processing_. Springer, 2022, pp. 155–170.
  * [166] P. Qi, X. Wan, N. Amar, and M. Lin, “Pipeline parallelism with controllable memory,” _arXiv preprint arXiv:2405.15362_ , 2024.
  * [167] S. Athlur, N. Saran, M. Sivathanu, R. Ramjee, and N. Kwatra, “Varuna: scalable, low-cost training of massive deep learning models,” in _Proceedings of the Seventeenth European Conference on Computer Systems_ , 2022, pp. 472–487.
  * [168] Z. Sun, H. Cao, Y. Wang, G. Feng, S. Chen, H. Wang, and W. Chen, “Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning,” in _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_ , 2024, pp. 86–100.
  * [169] S. Li, F. Xue, C. Baranwal, Y. Li, and Y. You, “Sequence parallelism: Long sequence training from system perspective,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2023, pp. 2391–2404.
  * [170] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [171] J. Fang and S. Zhao, “Usp: A unified sequence parallelism approach for long context generative ai,” _arXiv preprint arXiv:2405.07719_ , 2024.
  * [172] S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, R. Y. Aminadabi, S. L. Song, S. Rajbhandari, and Y. He, “System optimizations for enabling training of extreme long sequence transformer models,” in _Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing_ , 2024, pp. 121–130.
  * [173] NVIDIA. (2023) Megatron context parallelism. [Online]. Available: {https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html}
  * [174] D. Gu, P. Sun, Q. Hu, T. Huang, X. Chen, Y. Xiong, G. Wang, Q. Chen, S. Zhao, J. Fang _et al._ , “Loongtrain: Efficient training of long-sequence llms with head-context parallelism,” _arXiv preprint arXiv:2406.18485_ , 2024.
  * [175] H. Liu, M. Zaharia, and P. Abbeel, “Ring attention with blockwise transformers for near-infinite context,” _International Conference on Learning Representations(ICLR)_ , 2024.
  * [176] D. Li, R. Shao, A. Xie, E. P. Xing, J. E. Gonzalez, I. Stoica, X. Ma, and H. Zhang, “DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training,” _arxiv:2310.03294_ , 2023.
  * [177] X. Zhao, S. Cheng, Z. Zheng, Z. Yang, Z. Liu, and Y. You, “Dsp: Dynamic sequence parallelism for multi-dimensional transformers,” _arXiv preprint arXiv:2403.10266_ , 2024.
  * [178] W. Brandon, A. Nrusimha, K. Qian, Z. Ankner, T. Jin, Z. Song, and J. Ragan-Kelley, “Striped attention: Faster ring attention for causal transformers,” _arXiv preprint arXiv:2311.09431_ , 2023.
  * [179] S. Ao, W. Zhao, X. Han, C. Yang, Z. Liu, C. Shi, M. Sun, S. Wang, and T. Su, “Burstattention: An efficient distributed attention framework for extremely long sequences,” _arXiv preprint arXiv:2403.09347_ , 2024.
  * [180] Z. Liu, S. Wang, S. Cheng, Z. Zhao, Y. Bai, X. Zhao, J. Demmel, and Y. You, “Wallfacer: Guiding transformer model training out of the long-context dark forest with n-body problem,” _arXiv preprint arXiv:2407.00611_ , 2024.
  * [181] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” _Journal of Machine Learning Research_ , vol. 23, no. 120, pp. 1–39, 2022.
  * [182] C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram _et al._ , “Tutel: Adaptive mixture-of-experts at scale,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [183] S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” in _International conference on machine learning_. PMLR, 2022, pp. 18 332–18 346.
  * [184] S. Tan, Y. Shen, R. Panda, and A. Courville, “Scattered mixture-of-experts implementation,” _arXiv preprint arXiv:2403.08245_ , 2024.
  * [185] T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [186] S. Shi, X. Pan, X. Chu, and B. Li, “Pipemoe: Accelerating mixture-of-experts through adaptive pipelining,” in _IEEE INFOCOM 2023-IEEE Conference on Computer Communications_. IEEE, 2023, pp. 1–10.
  * [187] S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y. Yang, B. Li, and X. Chu, “Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,” in _Proceedings of the Nineteenth European Conference on Computer Systems_ , 2024, pp. 236–249.
  * [188] J. Li, Y. Jiang, Y. Zhu, C. Wang, and H. Xu, “Accelerating distributed $\\{$MoE$\\}$ training and inference with lina,” in _2023 USENIX Annual Technical Conference (USENIX ATC 23)_ , 2023, pp. 945–959.
  * [189] J. Liu, J. H. Wang, and Y. Jiang, “Janus: A unified distributed training framework for sparse mixture-of-experts models,” in _Proceedings of the ACM SIGCOMM 2023 Conference_ , 2023, pp. 486–498.
  * [190] C. Chen, M. Li, Z. Wu, D. Yu, and C. Yang, “Ta-moe: Topology-aware large scale mixture-of-expert training,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 22 173–22 186, 2022.
  * [191] J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang, “Fastmoe: A fast mixture-of-expert training system,” _arXiv preprint arXiv:2103.13262_ , 2021.
  * [192] J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,” in _Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_ , 2022, pp. 120–134.
  * [193] M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “Smartmoe: Efficiently training $\\{$Sparsely-Activated$\\}$ models through combining offline and online parallelization,” in _2023 USENIX Annual Technical Conference (USENIX ATC 23)_ , 2023, pp. 961–975.
  * [194] X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,” _Proceedings of the ACM on Management of Data_ , vol. 1, no. 1, pp. 1–19, 2023.
  * [195] W. Wang, Z. Lai, S. Li, W. Liu, K. Ge, Y. Liu, A. Shen, and D. Li, “Prophet: Fine-grained load balancing for parallel training of large-scale moe models,” in _2023 IEEE International Conference on Cluster Computing (CLUSTER)_. IEEE, 2023, pp. 82–94.
  * [196] N. Shazeer, Y. Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young _et al._ , “Mesh-tensorflow: Deep learning for supercomputers,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [197] Y. Xu, H. Lee, D. Chen, B. Hechtman, Y. Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni _et al._ , “Gspmd: general and scalable parallelization for ml computation graphs,” _arXiv preprint arXiv:2105.04663_ , 2021.
  * [198] Z. Jia, S. Lin, C. R. Qi, and A. Aiken, “Exploring hidden dimensions in parallelizing convolutional neural networks.” in _ICML_ , 2018, pp. 2279–2288.
  * [199] Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks.” _Proceedings of Machine Learning and Systems_ , vol. 1, pp. 1–13, 2019.
  * [200] M. Wang, C.-c. Huang, and J. Li, “Supporting very large models using automatic dataflow graph partitioning,” in _Proceedings of the Fourteenth EuroSys Conference 2019_ , 2019, pp. 1–17.
  * [201] L. Song, J. Mao, Y. Zhuo, X. Qian, H. Li, and Y. Chen, “Hypar: Towards hybrid parallelism for deep learning accelerator array,” in _2019 IEEE international symposium on high performance computer architecture (HPCA)_. IEEE, 2019, pp. 56–68.
  * [202] M. Schaarschmidt, D. Grewe, D. Vytiniotis, A. Paszke, G. S. Schmid, T. Norman, J. Molloy, J. Godwin, N. A. Rink, V. Nair _et al._ , “Automap: Towards ergonomic automated parallelism for ml models,” _arXiv preprint arXiv:2112.02958_ , 2021.
  * [203] H. Chen, C. H. Yu, S. Zheng, Z. Zhang, Z. Zhang, and Y. Wang, “Slapo: A schedule language for progressive optimization of large deep learning model training,” in _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2024, pp. 1095–1111.
  * [204] Z. Cai, X. Yan, K. Ma, Y. Wu, Y. Huang, J. Cheng, T. Su, and F. Yu, “Tensoropt: Exploring the tradeoffs in distributed dnn training with auto-parallelism,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 33, no. 8, pp. 1967–1981, 2021.
  * [205] W. Liu, Z. Lai, S. Li, Y. Duan, K. Ge, and D. Li, “Autopipe: A fast pipeline parallelism approach with balanced partitioning and micro-batch slicing,” in _2022 IEEE International Conference on Cluster Computing (CLUSTER)_. IEEE, 2022, pp. 301–312.
  * [206] A. Mirhoseini, H. Pham, Q. V. Le, B. Steiner, R. Larsen, Y. Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean, “Device placement optimization with reinforcement learning,” in _International conference on machine learning_. PMLR, 2017, pp. 2430–2439.
  * [207] Y. Gao, L. Chen, and B. Li, “Spotlight: Optimizing device placement for training deep neural networks,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1676–1684.
  * [208] J. M. Tarnawski, A. Phanishayee, N. Devanur, D. Mahajan, and F. Nina Paravecino, “Efficient algorithms for device placement of dnn graph operators,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 451–15 463, 2020.
  * [209] J. M. Tarnawski, D. Narayanan, and A. Phanishayee, “Piper: Multidimensional planner for dnn parallelization,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 24 829–24 840, 2021.
  * [210] C. Unger, Z. Jia, W. Wu, S. Lin, M. Baines, C. E. Q. Narvaez, V. Ramakrishnaiah, N. Prajapati, P. McCormick, J. Mohd-Yusof _et al._ , “Unity: Accelerating $\\{$DNN$\\}$ training through joint optimization of algebraic transformations and parallelization,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022, pp. 267–284.
  * [211] G. Liu, Y. Miao, Z. Lin, X. Shi, S. Maleki, F. Yang, Y. Bao, and S. Wang, “Aceso: Efficient parallel dnn training through iterative bottleneck alleviation,” in _Proceedings of the Nineteenth European Conference on Computer Systems_ , 2024, pp. 163–181.
  * [212] S. Alabed, B. Chrzaszcz, J. Franco, D. Grewe, D. Maclaurin, J. Molloy, T. Natan, T. Norman, X. Pan, A. Paszke _et al._ , “Partir: Composing spmd partitioning strategies for machine learning,” _arXiv preprint arXiv:2401.11202_ , 2024.
  * [213] Z. Lin, Y. Miao, Q. Zhang, F. Yang, Y. Zhu, C. Li, S. Maleki, X. Cao, N. Shang, Y. Yang _et al._ , “$\\{$nnScaler$\\}$:$\\{$Constraint-Guided$\\}$ parallelization plan generation for deep learning training,” in _18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)_ , 2024, pp. 347–363.
  * [214] J. Yuan, X. Li, C. Cheng, J. Liu, R. Guo, S. Cai, C. Yao, F. Yang, X. Yi, C. Wu _et al._ , “Oneflow: Redesign the distributed deep learning framework from scratch,” _arXiv preprint arXiv:2110.15032_ , 2021.
  * [215] J. Chen, S. Li, R. Guo, J. Yuan, and T. Hoefler, “Autoddl: Automatic distributed deep learning with near-optimal bandwidth cost,” _IEEE Transactions on Parallel and Distributed Systems_ , 2024.
  * [216] DeepSpeed. (2021) Deepspeed autotuning. [Online]. Available: {https://www.deepspeed.ai/tutorials/autotuning}
  * [217] X. Miao, Y. Wang, Y. Jiang, C. Shi, X. Nie, H. Zhang, and B. Cui, “Galvatron: Efficient transformer training over multiple gpus using automatic parallelism,” _Proceedings of the VLDB Endowment_ , vol. 16, no. 3, pp. 470–479, 2022.
  * [218] Z. Lai, S. Li, X. Tang, K. Ge, W. Liu, Y. Duan, L. Qiao, and D. Li, “Merak: An efficient distributed dnn training framework with automated 3d parallelism for giant foundation models,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 34, no. 5, pp. 1466–1478, 2023.
  * [219] Y. Liu, S. Li, J. Fang, Y. Shao, B. Yao, and Y. You, “Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models,” _arXiv:2302.02599_ , 2023.
  * [220] Y. Wang, Y. Jiang, X. Miao, F. Fu, S. Zhu, X. Nie, Y. Tu, and B. Cui, “Improving automatic parallel training via balanced memory workload optimization,” _IEEE Transactions on Knowledge and Data Engineering_ , 2024.
  * [221] J. H. Park, G. Yun, M. Y. Chang, N. T. Nguyen, S. Lee, J. Choi, S. H. Noh, and Y.-r. Choi, “Hetpipe: Enabling large dnn training on (whimpy) heterogeneous gpu clusters through integration of pipelined model parallelism and data parallelism,” in _2020 USENIX Annual Technical Conference (USENIX ATC 20)_ , 2020, pp. 307–321.
  * [222] L. Song, F. Chen, Y. Zhuo, X. Qian, H. Li, and Y. Chen, “Accpar: Tensor partitioning for heterogeneous deep learning accelerators,” in _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2020, pp. 342–355.
  * [223] X. Jia, L. Jiang, A. Wang, W. Xiao, Z. Shi, J. Zhang, X. Li, L. Chen, Y. Li, Z. Zheng _et al._ , “Whale: Efficient giant model training over heterogeneous $\\{$GPUs$\\}$,” in _2022 USENIX Annual Technical Conference (USENIX ATC 22)_ , 2022, pp. 673–688.
  * [224] D. Li, H. Wang, E. Xing, and H. Zhang, “Amp: Automatically finding model parallel strategies with heterogeneity awareness,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 6630–6639, 2022.
  * [225] P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy _et al._ , “Pathways: Asynchronous distributed dataflow for ml,” _Proceedings of Machine Learning and Systems_ , vol. 4, pp. 430–449, 2022.
  * [226] Y. Duan, Z. Lai, S. Li, W. Liu, K. Ge, P. Liang, and D. Li, “Hph: Hybrid parallelism on heterogeneous clusters for accelerating large-scale dnns training,” in _2022 IEEE International Conference on Cluster Computing (CLUSTER)_. IEEE, 2022, pp. 313–323.
  * [227] X. Miao, Y. Shi, Z. Yang, B. Cui, and Z. Jia, “Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training,” _Proceedings of the VLDB Endowment_ , vol. 16, no. 9, pp. 2354–2363, 2023.
  * [228] S. Zhang, L. Diao, C. Wu, Z. Cao, S. Wang, and W. Lin, “Hap: Spmd dnn training on heterogeneous gpu clusters with automated program synthesis,” in _Proceedings of the Nineteenth European Conference on Computer Systems_ , 2024, pp. 524–541.
  * [229] J. Zhang, G. Niu, Q. Dai, H. Li, Z. Wu, F. Dong, and Z. Wu, “Pipepar: Enabling fast dnn pipeline parallel training in heterogeneous gpu clusters,” _Neurocomputing_ , vol. 555, p. 126661, 2023.
  * [230] B. Yuan, Y. He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 464–25 477, 2022.
  * [231] M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “Swarm parallelism: Training large models can be surprisingly communication-efficient,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 29 416–29 440.
  * [232] Z. Tang, Y. Wang, X. He, L. Zhang, X. Pan, Q. Wang, R. Zeng, K. Zhao, S. Shi, B. He _et al._ , “Fusionai: Decentralized training and deploying llms with massive consumer-level gpus,” _arXiv preprint arXiv:2309.01172_ , 2023.
  * [233] Z. Yao, R. Y. Aminabadi, O. Ruwase, S. Rajbhandari, X. Wu, A. A. Awan, J. Rasley, M. Zhang, C. Li, C. Holmes _et al._ , “Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales,” _arXiv preprint arXiv:2308.01320_ , 2023.
  * [234] “Trl - transformer reinforcement learning.” [Online]. Available: https://github.com/huggingface/trl
  * [235] J. Hu, X. Wu, W. Wang, D. Zhang, Y. Cao _et al._ , “Openrlhf: An easy-to-use, scalable and high-performance rlhf framework,” _arXiv preprint arXiv:2405.11143_ , 2024.
  * [236] Y. Xiao, W. Wu, Z. Zhou, F. Mao, S. Zhao, L. Ju, L. Liang, X. Zhang, and J. Zhou, “An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training,” _arxiv:2312.11819_ , 2023.
  * [237] Z. Mei, W. Fu, K. Li, G. Wang, H. Zhang, and Y. Wu, “Realhf: Optimized rlhf training for large language models through parameter reallocation,” _arXiv preprint arXiv:2406.14088_ , 2024.
  * [238] K. Lei, Y. Jin, M. Zhai, K. Huang, H. Ye, and J. Zhai, “$\\{$PUZZLE$\\}$: Efficiently aligning large language models through $\\{$Light-Weight$\\}$ context switch,” in _2024 USENIX Annual Technical Conference (USENIX ATC 24)_ , 2024, pp. 127–140.
  * [239] L. Clarke, I. Glendinning, and R. Hempel, “The mpi message passing interface standard,” in _Programming Environments for Massively Parallel Distributed Systems: Working Conference of the IFIP WG 10.3, April 25–29, 1994_. Springer, 1994, pp. 213–218.
  * [240] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [241] R. A. Van De Geijn and J. Watts, “Summa: Scalable universal matrix multiplication algorithm,” _Concurrency: Practice and Experience_ , vol. 9, no. 4, pp. 255–274, 1997.
  * [242] L. E. Cannon, _A cellular computer to implement the Kalman filter algorithm_. Montana State University, 1969.
  * [243] E. Solomonik and J. Demmel, “Communication-optimal parallel 2.5 d matrix multiplication and lu factorization algorithms,” in _European Conference on Parallel Processing_. Springer, 2011, pp. 90–109.
  * [244] R. C. Agarwal, S. M. Balle, F. G. Gustavson, M. Joshi, and P. Palkar, “A three-dimensional approach to parallel matrix multiplication,” _IBM Journal of Research and Development_ , vol. 39, no. 5, pp. 575–582, 1995.
  * [245] F. Strati, P. Elvinger, T. Kerimoglu, and A. Klimovic, “Ml training with cloud gpu shortages: Is cross-region the answer?” in _Proceedings of the 4th Workshop on Machine Learning and Systems_ , 2024, pp. 107–116.
  * [246] W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 4195–4205.
  * [247] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” _arXiv preprint arXiv:1308.3432_ , 2013.
  * [248] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand _et al._ , “Mixtral of experts,” _arXiv preprint arXiv:2401.04088_ , 2024.
  * [249] S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y. He, and A. Bhatele, “A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training,” in _Proceedings of the 37th International Conference on Supercomputing_ , 2023, pp. 203–214.
  * [250] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne _et al._ , “Jax: composable transformations of python+ numpy programs,” 2018.
  * [251] G. X. team. (2017) Xla: Optimizing compiler for machine learning. [Online]. Available: https://github.com/openxla/xla
  * [252] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze _et al._ , “$\\{$TVM$\\}$: An automated $\\{$End-to-End$\\}$ optimizing compiler for deep learning,” in _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_ , 2018, pp. 578–594.
  * [253] S. Li, H. Liu, Z. Bian, J. Fang, H. Huang, Y. Liu, B. Wang, and Y. You, “Colossal-ai: A unified deep learning system for large-scale parallel training,” in _Proceedings of the 52nd International Conference on Parallel Processing_ , 2023, pp. 766–775.
  * [254] J. Wang, Y. Lu, B. Yuan, B. Chen, P. Liang, C. De Sa, C. Re, and C. Zhang, “Cocktailsgd: Fine-tuning foundation models over 500mbps networks,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 36 058–36 076.
  * [255] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” _arXiv:2203.02155_ , 2022.
  * [256] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [257] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [258] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [259] “unsloth - finetune llama 3, mistral, phi-3 & gemma 2-5x faster with 80
  * [260] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan _et al._ , “Ray: A distributed framework for emerging $\\{$AI$\\}$ applications,” in _13th USENIX symposium on operating systems design and implementation (OSDI 18)_ , 2018, pp. 561–577.
  * [261] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in _Proceedings of the 29th Symposium on Operating Systems Principles_ , 2023, pp. 611–626.
  * [262] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low-precision,” _arXiv preprint arXiv:2407.08608_ , 2024.
  * [263] H. Liu and P. Abbeel, “Blockwise parallel transformers for large context models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [264] R. Wu, X. Zhu, J. Chen, S. Liu, T. Zheng, X. Liu, and H. An, “Swattention: designing fast and memory-efficient attention for a new sunway supercomputer,” _The Journal of Supercomputing_ , pp. 1–24, 2024.
  * [265] G. Bikshandi and J. Shah, “A case study in cuda kernel fusion: Implementing flashattention-2 on nvidia hopper architecture using the cutlass library,” _arXiv preprint arXiv:2312.11918_ , 2023.
  * [266] Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “Bytetransformer: A high-performance transformer boosted for variable-length inputs,” in _2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. IEEE, 2023, pp. 344–355.
  * [267] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and S. Amarasinghe, “Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines,” _Acm Sigplan Notices_ , vol. 48, no. 6, pp. 519–530, 2013.
  * [268] H. Zhu, R. Wu, Y. Diao, S. Ke, H. Li, C. Zhang, J. Xue, L. Ma, Y. Xia, W. Cui _et al._ , “$\\{$ROLLER$\\}$: Fast and efficient tensor compilation for deep learning,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022, pp. 233–248.
  * [269] P. Tillet, H.-T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in _Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages_ , 2019, pp. 10–19.
  * [270] G. Huang, Y. Bai, L. Liu, Y. Wang, B. Yu, Y. Ding, and Y. Xie, “Alcop: Automatic load-compute pipelining in deep learning compiler for ai-gpus,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [271] S. Zheng, S. Chen, P. Song, R. Chen, X. Li, S. Yan, D. Lin, J. Leng, and Y. Liang, “Chimera: An analytical optimizing framework for effective compute-intensive operators fusion,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 1113–1126.
  * [272] Y. Shi, Z. Yang, J. Xue, L. Ma, Y. Xia, Z. Miao, Y. Guo, F. Yang, and L. Zhou, “Welder: Scheduling deep learning memory access via tile-graph,” in _17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)_ , 2023, pp. 701–718.
  * [273] J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski _et al._ , “Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation,” in _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2024, pp. 929–947.
  * [274] M. Ibrahim, S. Aga, A. Li, S. Pati, and M. Islam, “Jit-q: Just-in-time quantization with processing-in-memory for efficient ml training,” _Proceedings of Machine Learning and Systems_ , vol. 6, pp. 46–59, 2024.
  * [275] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh _et al._ , “Mixed precision training,” _arXiv preprint arXiv:1710.03740_ , 2017.
  * [276] X. He, J. Sun, H. Chen, and D. Li, “Campo:$\\{$Cost-Aware$\\}$ performance optimization for $\\{$Mixed-Precision$\\}$ neural network training,” in _2022 USENIX Annual Technical Conference (USENIX ATC 22)_ , 2022, pp. 505–518.
  * [277] D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen _et al._ , “A study of bfloat16 for deep learning training,” _arXiv preprint arXiv:1905.12322_ , 2019.
  * [278] M. Li, R. B. Basat, S. Vargaftik, C. Lao, K. Xu, M. Mitzenmacher, and M. Yu, “$\\{$THC$\\}$: Accelerating distributed deep learning using tensor homomorphic compression,” in _21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)_ , 2024, pp. 1191–1211.
  * [279] N. Wang, J. Choi, D. Brand, C.-Y. Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point numbers,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [280] X. Sun, J. Choi, C.-Y. Chen, N. Wang, S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan, “Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [281] H. Peng, K. Wu, Y. Wei, G. Zhao, Y. Yang, Z. Liu, Y. Xiong, Z. Yang, B. Ni, J. Hu _et al._ , “Fp8-lm: Training fp8 large language models,” _arXiv preprint arXiv:2310.18313_ , 2023.
  * [282] B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf _et al._ , “Microscaling data formats for deep learning,” _arXiv preprint arXiv:2310.10537_ , 2023.
  * [283] H. Xi, Y. Chen, K. Zhao, K. Zheng, J. Chen, and J. Zhu, “Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization,” _arXiv preprint arXiv:2403.12422_ , 2024.
  * [284] H. Xi, C. Li, J. Chen, and J. Zhu, “Training transformers with 4-bit integers,” _Advances in Neural Information Processing Systems_ , vol. 36, pp. 49 146–49 168, 2023.
  * [285] H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei, “Bitnet: Scaling 1-bit transformers for large language models,” _arXiv preprint arXiv:2310.11453_ , 2023.
  * [286] S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei, “The era of 1-bit llms: All large language models are in 1.58 bits,” _arXiv preprint arXiv:2402.17764_ , 2024.
  * [287] M. N. Rabe and C. Staats, “Self-attention does not need $o(n^{2})$ memory,” _arXiv preprint arXiv:2112.05682_ , 2021.
  * [288] Z. Jia, O. Padon, J. Thomas, T. Warszawski, M. Zaharia, and A. Aiken, “Taso: optimizing deep learning computation with automatic generation of graph substitutions,” in _Proceedings of the 27th ACM Symposium on Operating Systems Principles_ , 2019, pp. 47–62.
  * [289] L. Zheng, C. Jia, M. Sun, Z. Wu, C. H. Yu, A. Haj-Ali, Y. Wang, J. Yang, D. Zhuo, K. Sen _et al._ , “Ansor: Generating $\\{$High-Performance$\\}$ tensor programs for deep learning,” in _14th USENIX symposium on operating systems design and implementation (OSDI 20)_ , 2020, pp. 863–879.
  * [290] W. Niu, J. Guan, Y. Wang, G. Agrawal, and B. Ren, “Dnnfusion: accelerating deep neural networks execution with advanced operator fusion,” in _Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation_ , 2021, pp. 883–898.
  * [291] Z. Zheng, X. Yang, P. Zhao, G. Long, K. Zhu, F. Zhu, W. Zhao, X. Liu, J. Yang, J. Zhai _et al._ , “Astitch: enabling a new multi-dimensional optimization space for memory-intensive ml training and inference on modern simt architectures,” in _Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2022, pp. 359–373.
  * [292] B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon _et al._ , “Bloom: A 176b-parameter open-access multilingual language model,” _arXiv preprint arXiv:2211.05100_ , 2022.
  * [293] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2818–2829.
  * [294] K. Yang, Y.-F. Chen, G. Roumpos, C. Colby, and J. Anderson, “High performance monte carlo simulation of ising model on tpu clusters,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2019, pp. 1–15.
  * [295] K. Fischer and E. Saba, “Automatic full compilation of julia programs and ml models to cloud tpus,” _arXiv preprint arXiv:1810.09868_ , 2018.
  * [296] J. Choquette, “Nvidia hopper h100 gpu: Scaling performance,” _IEEE Micro_ , 2023.
  * [297] M. Kirisame, S. Lyubomirsky, A. Haan, J. Brennan, M. He, J. Roesch, T. Chen, and Z. Tatlock, “Dynamic tensor rematerialization,” _arXiv preprint arXiv:2006.09616_ , 2020.
  * [298] Z. Hu, J. Xiao, Z. Deng, M. Li, K. Zhang, X. Zhang, K. Meng, N. Sun, and G. Tan, “Megtaichi: Dynamic tensor-based memory management optimization for dnn training,” in _Proceedings of the 36th ACM International Conference on Supercomputing_ , 2022, pp. 1–13.
  * [299] J. Zhang, S. Ma, P. Liu, and J. Yuan, “Coop: Memory is not a commodity,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [300] P. Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, J. Gonzalez, K. Keutzer, and I. Stoica, “Checkmate: Breaking the memory wall with optimal tensor rematerialization,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 497–511, 2020.
  * [301] T. Yuan, Y. Liu, X. Ye, S. Zhang, J. Tan, B. Chen, C. Song, and D. Zhang, “Accelerating the Training of Large Language Models using Efficient Activation Rematerialization and Optimal Hybrid Parallelism,” _2024 USENIX Annual Technical Conference (USENIX ATC 24)_ , 2024.
  * [302] G. Wang, H. Qin, S. A. Jacobs, C. Holmes, S. Rajbhandari, O. Ruwase, F. Yan, L. Yang, and Y. He, “ZeRO++: Extremely Efficient Collective Communication for Giant Model Training,” _arXiv:2306.10209_ , 2023.
  * [303] C. Wu, H. Zhang, L. Ju, J. Huang, Y. Xiao, Z. Huan, S. Li, F. Meng, L. Liang, X. Zhang, and J. Zhou, “Rethinking memory and communication cost for efficient large language model training,” _arXiv preprint arXiv:2310.06003_ , 2023.
  * [304] C. Luo, T. Zhong, and G. Fox, “Rtp: Rethinking tensor parallelism with memory deduplication,” _arXiv preprint arXiv:2311.01635_ , 2023.
  * [305] Q. Chen, Q. Hu, G. Wang, Y. Xiong, T. Huang, X. Chen, Y. Gao, H. Yan, Y. Wen, T. Zhang, and P. Sun, “Amsp: Reducing communication overhead of zero for efficient llm training,” 2024.
  * [306] H. Shu, A. Wang, Z. Shi, H. Zhao, Y. Li, and L. Lu, “Roam: memory-efficient large dnn training via optimized operator ordering and memory layout,” _arXiv preprint arXiv:2310.19295_ , 2023.
  * [307] A. Imanishi and Z. Xu, “A Heuristic for Periodic Memory Allocation with Little Fragmentation to Train Neural Networks,” in _Proceedings of the 2024 ACM SIGPLAN International Symposium on Memory Management_. ACM, 2024, pp. 82–94.