#  MAD Max Beyond Single-Node: Enabling Large Machine Learning Model
Acceleration on Distributed Systems

Samuel Hsia1,2 Alicia Golden1,2 Bilge Acun1 Newsha Ardalani1 Zachary DeVito1
Gu-Yeon Wei2 David Brooks2 Carole-Jean Wu1

1FAIR  Meta  2Harvard University

<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Training and deploying large machine learning (ML) models is time-consuming
and requires significant distributed computing infrastructures. Based on real-
world large model training on datacenter-scale infrastructures, we show
14$\sim$32% of all GPU hours are spent on communication with no overlapping
computation. To minimize the outstanding communication latency, in this work,
we develop an agile performance modeling framework to guide parallelization
and hardware-software co-design strategies. Using the suite of real-world
large ML models on state-of-the-art GPU training hardware, we demonstrate
2.24$\times$ and 5.27$\times$ throughput improvement potential for pre-
training and inference scenarios, respectively.

##  1 Introduction

Billion-parameter large language models (LLMs) [6, 45, 58, 59] power
applications that have shown far-reaching impact across different domains [35,
11, 12, 44]. Similarly, trillion-parameter recommendation models [37, 70] have
demonstrated state-of-the-art user modeling and content understanding across
search [3, 8, 28, 74], social media [1, 16, 17, 69], e-commerce [76, 77], and
entertainment [18]. As these large ML models increase in size and complexity
[16, 17], the corresponding training and inference workloads become ever more
resource-intensive. Identifying better mappings between ML workloads and
distributed systems can provide significant infrastructure benefits by
reducing million-hour training times [6, 58, 59] and enabling faster
exploration of novel model architectures on new hardware systems.

As large ML models expand beyond single-node platforms, successful training
and inference solutions have to take into account the underlying distributed
systems and hardware devices [10, 25, 26, 27, 41, 39, 40, 42]. In order to
leverage advancements in compute, memory, and interconnect of data center
scale distributed systems, developers must consider how to map models and
tasks onto underlying distributed systems — parallelization strategy. Figure 1
shows the impact that parallelization strategy can have on training
performance of important large ML models: Deep Learning Recommendation Models
(DLRM) and Large Language Models (LLM). An optimal parallelization strategy
achieves 34% higher training throughput as compared to the Fully Sharded Data
Parallel (FSDP) strategy for both DLRM and LLM. Unlocking this performance
potential requires an agile performance modeling framework that considers the
interactions between model architectures, machine learning tasks, and hardware
systems.

Figure 1: For large ML model training, exploring parallelization strategy
design space can lead to strategies with higher system throughput (blue) than
common strategies such as FSDP and TP.

Generally speaking, current approaches for training and deploying large ML
models fall into three categories. The first option is to apply industry-grade
parallelization strategies that give best guarantees for feasibility while
providing adequate system performance on existing distributed systems (e.g.,
FSDP, ZeRO [73, 51]). This often comes at the expense of fully utilizing the
underlying hardware. The second option is to carefully engineer custom,
hierarchical parallelization strategies for the target model, task, and
distributed system [56]. While this likely best utilizes underlying hardware,
the engineering endeavor required for such customized parallelization strategy
is nontrivial and the strategy itself may be not transferable between tasks.
The third option is to estimate system performance via a software tool before
either training or deployment. Existing tools for predicting distributed ML
performance either require mature software implementations, are training-task
specific, or tailored to specific hardware architectures among other
limitations. or a combination of the above. To address the need for an agile
exploration tool for parallelization strategies tailored to different use-
cases, we propose a distributed ML performance model and evaluate it on a
suite of real-world, large ML models, including deep learning recommender
systems and LLMs [5, 6, 7, 9, 38, 48, 58, 59, 74].

In this work, we first characterize the suite of real-world, large ML models
at both model- and datacenter-deployment scales (Section 3). At the model
architecture level, we identify performance-critical hardware requirements
based on the models’ compute and memory characteristics. At the datacenter
scale, we quantify the communication required by conducting a fleet-wide
characterization of at-scale training, showing that 14$\sim$32% of all GPU
hours are spent on communication with no concurrent computation (i.e., exposed
communication).

To enable agile exploration of the parallelization design space, in this
paper, MAD Max Beyond Single-Node, we propose a performance model for
estimating system performance of a distributed ML workload for distributed
systems of unique characteristics. The performance model takes into account
target ML model architecture, task, parallelization scheme, and distributed
system hardware to generated per-device traces. These per-device traces can
then be pieced together to estimate the overall system performance of the
target ML model and task. Additionally, the performance model generates
detailed breakdowns of both communication collectives and computation-
communication overlap efficiency, enabling users to identify future
optimization opportunities. Our performance model is validated against
multiple real large-scale distributed training experiments, demonstrating 97%
and 91% performance prediction accuracies on serialized and overlapped
execution, respectively, for real world use-cases.

Using this performance model, we identify parallelization strategies with up
to 2.24$\times$ and 5.27$\times$ throughput improvement for pre-training and
inference, respectively across our suite of large ML models. When considering
parallelization strategies not limited by the memory capacity of current
training systems, we identify strategies that achieve 2.43$\times$ and
12.13$\times$ throughput improvement for pre-training and inference,
respectively. Using the performance model, we point out how model-level
compute and communication requirements alter optimal parallelization strategy
and increasing LLM context length calls for solutions beyond solely
parallelization exploration (Section 6). We also conduct a retrospective study
on how different generations of GPUs impact overall training efficiency and
follow up with a future technologies scaling study by showing the effects of
improving systems components like compute efficiency, memory capacity and
bandwidth, and hierarchical interconnect bandwidth (Section 6).

The main contributions of this work are as follows:

  * •

We propose a performance model that enables agile exploration of the
distributed ML training and deployment design space. Our performance model
targets both implemented and future models alike, allowing for accurate
throughput performance estimation with different model architectures, tasks,
hardware devices, and distributed systems.

  * •

We show model-level insights on how parallelization strategies interact with
DLRM and its model architecture variants. We show how asymmetric compute and
communication requirements from transformer and mixture-of-experts variants
lead to different optimal parallelization strategies. Additionally, we note
the limits of solely optimizing parallelization strategies on LLMs of
increasing context length.

  * •

We show that to improve throughput performance for both training and inference
of large ML models, hardware specifications across compute, memory, and
interconnect have to be concurrently improved.

We will open-source111Upon publication acceptance. the proposed performance
model to enable follow-on work for modeling the interaction between
parallelization strategies, models, tasks, and distributed systems on ML
system performance.

##  2 Background

In this section, we introduce a suite of model architectures across both
recommender systems and LLMs. We then outline three tasks for these models:
pre-training, fine-tuning, and inference (Section 2.1). Lastly, we discuss the
parallelization strategies currently used to map the workloads (i.e., model
and task) onto the distributed systems (Section 2.2).

Figure 2: For recommendation models, applying FSDP, TP, or DDP on an MLP layer
requires sharding or replicating parameters and communicating parameters
(orange) or partial sums (yellow). In this example, the embedding table’s
large capacity requires it to be sharded.

###  2.1 Models and Tasks

Deep learning based recommender systems and LLMs follow the general model
architecture of representing categorical inputs as embedding vectors and then
processing these embedding vectors with model-specific computation layers.
This means that there are many shared components: embedding tables, Multilayer
Perceptrons (MLPs), and more intricate processing layers like transformer
blocks that are emphasized to different degrees by each model. We focus on the
following five classes of models throughout the paper:

  1. 1.

DLRM. The canonical at-scale recommendation model takes two types of inputs:
dense and sparse features. Dense features, such as, user age and current time,
are processed by MLP layers while the sparse categorical features are
processed as lookups into large embedding tables. The processed results are
fed into a feature interaction layer, where these intermediate results are
either concatenated or multiplied with one another via dot products [61, 62].
The result of this feature interaction layer is then fed into MLP layers to
generate predictions like Click-Through Rate (CTR) [38]. For many large-scale
DLRM models, storing and communicating trillion-parameter scale embedding
tables is the primary system bottleneck [14, 16, 20, 21, 30, 31, 37, 55, 64].

  2. 2.

DLRM-Transformer. As sparse features for recommendation models have become
complex, model architectures have also evolved to better model implicit
relationships between these sparse features. Some DLRM variants replace
concatenation and dot-product based feature interactions with transformer
encoder layers that model higher-order interactions and sequential
relationship between sparse features. Others [7, 48, 70] use transformer-style
feature interaction layers to tackle challenges like behavior sequence
modeling and personalized re-ranking. From a systems perspective, transformer
layers increase both required compute and computation-communication overlap
opportunities.

  3. 3.

DLRM-MoE. In the context of DLRMs, applying Mixture-of-Experts (MoE) creates
parallel Top MLPs that are conditionally activated based on feature
interaction [74]. Because only a fraction of experts are active for each
sample, DLRM-MoE increases model capacity and (expert-to-expert) communication
while scaling computation only by the number of active experts.

  4. 4.

LLM. Large language models (LLMs) also use the “look up embeddings then
process them" architecture [6, 19, 50, 58, 72]. However, instead of using user
and content categorical features, LLMs convert tokens – character sequences –
to input embeddings. Subsequent processing layers use alternating self-
attention and feed-forward layers [60]. Unlike DLRMs, advancements in LLM
modeling have been more focused on the processing layers than embeddings,
reinforcing the importance of compute in LLM execution.

  5. 5.

LLM-MoE. In the context of LLMs, one way to apply MoE is to replace the feed-
forward layer in transformer blocks with experts. By applying this technique,
FLOPs per token grows at a slower rate than overall model capacity, leading to
more efficient training and inference. While FLOPs becomes less of a concern,
non-blocking inter-expert communication – especially during training – becomes
a larger systems concern.

For each of these model architectures, we are interested in pre-training,
fine-tuning and inference. Pre-training stresses all of compute, memory
capacity, and communication as it involves both forward and backward passes –
along with retaining intermediate activations from the forward pass. The
requirements of fine-tuning are a subset of pre-training, as the frozen
parameters of a model do not require updates, and memory capacity and
communication requirements are slightly loosened. Inference only requires the
forward pass so compute is usually proportionally larger.

###  2.2 Parallelization Strategies

A model layer can be either replicated or sharded across devices. We explore
the following parallelization strategies (Figure 2 shows forward pass
execution):

  1. 1.

Fully Sharded Data Parallelism (FSDP). Parameters are sharded across devices.
Before layer computation, missing parameter shards are gathered from other
devices via AllGather. During backward pass, weight gradients are reduced and
sharded via ReduceScatter.

  2. 2.

Tensor Parallelism (TP). Parameters are sharded across devices. During forward
pass, each device uses its local parameter shard to compute a partial sum.
Devices then communicate via AllReduce to find aggregate sum. Same principle
is applied during backward pass for gradients.

  3. 3.

Distributed Data Parallelism (DDP). Parameters are replicated across devices.
During forward pass, each device acts independently for computation. During
backward pass, devices AllReduce for aggregate weight gradients.

We apply one parallelization strategy for each layer type. Figure 2 depicts
different parallelization strategies for an MLP layer and vanilla model
parallel (MP) sharding for the embedding tables. Additionally, parallelization
strategies can be applied hierarchically for multi-node systems, creating
$N$-D parallelism strategies.

Figure 3: For large ML models, key system resource – (a) capacity, (b)
compute, (c) bandwidth – requirements vary by orders of magnitude. Figure 4:
(a) Compute and exposed communication make up the majority of observed at-
scale training cycles. (b) The degree of communication overlapped with
computation and data loading is workload dependent. (c) Breakdown of
communication collectives also varies by workload.

##  3 Characterization

In this section, we first characterize a suite of real-world large ML models
with respect to their model capacity, parameter breakdowns, FLOPs, and memory
bandwidth characteristics (Section 3.1). To get a better understanding of the
models’ communication requirements, we conduct a fleet-wide characterization
of at-scale training experiments (Section 3.2).

###  3.1 Individual Model Characterization

We first quantify the difference in compute, memory capacity, and bandwidth
requirements between six real-world recommendation models and LLMs: DLRM-{A,
B, C}, GPT-3 175B, LLaMA-65B, LLaMA 2-70B. Figure 3 quantifies this diversity
of requirements with two key observations:

O1: Parameter count – and allocation across model layers – varies by orders of
magnitude between models, impacting system capacity requirements.
Recommendation models contain significantly more parameters than LLMs (Figure
3 (a)). Despite variation in parameter count across LLMs, GPT-3 consists of
roughly 2–68$\times$ fewer parameters as compared to recommendation models.
Training and deploying these recommendation models and LLMs require multi-node
distributed systems, yet the size of the model governs how many devices (i.e.,
GPUs) are required to fit the entire model and the viable set of scale-out
parallelization strategies.

Additionally, virtually 100% of parameters in recommendation models are used
for embeddings while almost 100% of parameters in LLMs are dedicated to
compute. This reflects the transformer-heavy computation of current LLMs, in
contrast to embedding-driven recommendation models that offer at-scale
personalization.

O2: Recommendation models require fewer FLOPs per sample as compared to LLMs,
yet require >20$\times$ higher memory bandwidth for sparse lookups. Figures 3
(b, c) demonstrate how recommendation models and LLMs show opposite trends for
compute requirements as compared to sparse lookup bandwidth. Sparse lookup
bandwidth requirements for recommendation models far surpass LLMs – a fact
that is consistent with a higher proportion of parameters being dedicated to
embeddings. However, the opposite is true for compute requirements, as LLMs
require significantly higher FLOPs per sample. As discussed in Section 4,
these varying system requirements play an important role in the design of an
optimal parallelization strategy for each model.

Figure 5: Our performance model works in five stages. After workload
specifications and layer execution orders are established, traces for
individual layer execution are generated and then combined with required
communication calls to form complete computation and communication streams.

###  3.2 Fleet-wide Communication Characterization

In addition to model-level characterization, we look at fleet-wide model
training. We observe, over an extended period of time, the importance of
communication for training the latest DLRM-style models and LLMs. Figure 4
quantifies the role of communication with two key observations:

O3: Compute and exposed communication make up the majority of observable
training GPU cycles. Compute, defined as cycles with either device computation
or memory lookups (orange) and exposed communication, defined as cycles with
only inter-device communication (blue), make up >82% of all observable
training GPU cycles for both DLRM and LLMs (Figure 4 (a)). The rest of the
cycles are attributed to host-device communication – exposed memcpy (yellow) –
and inactivity due to data ingestion, kernel launch overhead, etc. – GPU idle
(grey). From this observation, we focus our performance modeling efforts on
predicting the expected behavior of compute and communication cycles.

O4: Model architecture and parallelization strategy differences impact both
the amount of computation-communication overlap and the types of communication
collectives used. When model training spans multiple devices, replicating or
sharding model components leads to communication calls for parameters,
activations and/or gradients. Being able to overlap these communication calls
with computation so that the training devices are doing “useful work" is
important for utilization. Figure 4 (b) shows that $\sim$50% of communication
calls for DLRM training are overlapped with computation, whereas >65% of
communication calls for compute-dominated LLMs are overlapped.

Figure 4 (c) shows the spread of different communication collectives during
training. For DLRM models, All2All is heavily emphasized while LLMs spend the
majority of their communication cycles on AllReduce. This is a direct result
of model architecture difference, and thus active parallelization strategy.
Since DLRMs require large amounts of sparse lookups from sharded embedding
tables, the per-device unique embedding lookups have to be distributed to each
device via All2All. On the contrary, LLMs have fewer parameters and are more
amenable to replication of compute parameters, allowing for DDP opportunities
that require AllReduce for aggregating weight gradients.

In this section, we characterize real-world large ML models from model
architecture and distributed training perspectives. From Section 3.2 we see
that model architectures and the way in we map them onto distributed systems
significantly impacts system resource utilization, and thus overall
performance. To better understand how to best map current and future large ML
models onto different distributed systems, we propose an agile, at-scale
accurate performance model.

##  4 Proposed Design

In this section, we go over the design of our performance model and walk
through how the model simulates distributed ML workloads. First, we provide an
overview of the design and key assumptions made behind its design process
(Section 4.1). Next, we explain how the performance model processes ML model
layers by the layers’ primary characteristics (Section 4.2). Finally, we cover
how individually processed layers are pieced together into complete
computation and communication streams via the required communication calls of
a given parallelization strategy (Section 4.3).

###  4.1 Design Overview.

Figure 5 highlights the five main processes of our performance model via a
DLRM-Transformer example. The performance model treats individual layers of an
ML model as core blocks for generating per-device execution traces. To
simulate the per-iteration behavior of a distributed ML workload, these
execution traces are then pieced together with the required communication
calls of the target parallelization strategy. From per-iteration behavior, the
performance model generates estimations of overall throughput and other
system-level serialized and overlapped execution breakdowns.

Users have to provide three JSON files for: 1) model architecture via layer-
specific configurations (e.g., number of MLP layers, embedding table
dimension, number of transformer layers and heads), 2) distributed system
specifications (e.g., Tensor Float (TF32) utilization, HBM peak bandwidth,
AllReduce intra-node interconnect utilization), and 3) task and
parallelization strategy (e.g., pre-training/fine-tuning/inference,
intra-/inter-node parallelization strategy, intra-/inter-node parallelization
degrees). See Section 5 for a more exhaustive list of currently supported
configuration options.

With these configurations, individual layers are first processed by their
primary system requirements. Examples include estimating embedding bag
execution by the amount of embeddings to look up and per-GPU high-bandwidth
memory (HBM) memory bandwidth and the time it takes to execute a transformer
encoder layer by TF32 compute throughput. Based on the replication and
sharding specified by the target parallelization strategy, the required
communication calls are processed by collective-specific intra- (e.g., NVLink)
and inter- (e.g., Infiniband, RDMA over Converged Ethernet (RoCE)) node
communication bandwidths.

We take into account task-level requirements (i.e., pre-training/fine-
tuning/inference) to construct per-device computation and communication
streams with data dependencies and potential computation-communication
overlap.

Assumptions:

  * •

Since we are primarily focused on large-models, underlying distributed systems
are multi-device in nature. For multi-device execution, a first-order analysis
of execution behavior and overall performance can be estimated via modeling
per-node layer execution and inter-node parallelization communication. Kernel-
level improvements (e.g., [43]), while not the focus of this work, can be
effectively modeled as increased compute and memory lookup utilization.

  * •

The performance model assumes that the entire model can be fit onto the
training/inference devices (i.e., when sharded, the model can fit onto GPUs).
Recent high-performance training platforms target this design point [37].
Design points where model parameters have to be shuffled back and forth
between CPU and device are currently unsupported.

  * •

Device-host communication (e.g., CPU-GPU data loading) is relatively a second-
order consideration and mostly overlapped and hidden between
training/inference iterations. This observation is shared in [37] and our
fleet-wide characterization in Section 3.2, Figure 4.

###  4.2 Processing Individual Model Layers

Layers are processed by their main system requirement. For example, we
illustrate how MLP and embedding bag performance are estimated differently for
our Figure 5 example.

Compute Blocks. Assuming that compute time is the main bottleneck for MLPs, we
estimate compute time per layer as:

$\sim$ (FLOPs per layer) / [(GPU peak FLOPS) * Compute utilization]

where FLOPs per layer is determined by the MLP layer’s dimensions and target
batch size. GPU peak FLOPS are heavily dependent on data type (e.g., 32-bit,
16-bit FP/TF/BF) and whether or not tensor cores are enabled. We incorporate
compute utilization – or in the case of GPUs, SM utilization/occupancy – as a
factor in [0,1]. Typical compute utilization factors for A100s on layers in
our models of interest are $\sim$70%. We adopt a similar approach for modeling
self-attention and fully-connected (FC) layers found in transformer layers,
where FLOPs per layer is estimated by additional factors such as attention
dimension and context length.

Embedding Bags. Assuming that lookup time is the main bottleneck for embedding
bags, we estimate lookup time as:

$\sim$ (Lookup bytes per GPU) / [(HBM BW) * HBM utilization]

where Lookup bytes is determined by the number of embedding tables, number of
lookups per embedding table, embedding dimension, and embedding precision.
Lookup bytes per GPU is highly parallelization strategy dependent. In this
case, we assume that the embedding table is evenly sharded across GPUs in
terms of both capacity and number of lookups. If the number of lookups are
unevenly distributed between GPUs, we can adjust the lookup bytes per GPU on a
per-GPU basis [55]. HBM utilization is a factor between [0,1] and typical
values for embedding bags of interest are $\sim$80% for A100s.

###  4.3 Piecing Together Computation and Comm. Streams

Specifying Explicit Execution Order. To generate per-device traces for
different ML tasks, an explicit execution priority must be established for the
different layers. In Figure 5, we can establish the order as such (1)
Embedding, (2) Bottom MLP, (3) Transformer, (4) Top MLP. During backward pass,
the execution order will be reversed. If the target task is fine-tuning, we
also specify frozen layers, reducing unnecessary computation and communication
of certain weight gradients.

Figure 6: Sample generated GPU compute and communication streams with labeled
exposed communication.

Generating Parallelization-Specific Streams. An explicit execution order by
itself is not enough to construct accurate traces. A target parallelization
strategy is required to specify required the communication collectives.
Explicit data dependencies, along with parallelization strategy determine the
blocking/non-blocking nature of the communication calls. In Figure 5, MLP and
transformer layers are distributed via DDP while embedding tables are
distributed via sharding.

Figure 6 illustrates generated forward pass streams from our DLRM-Transformer
example. We see that the traces are slotted into a compute stream and
communication stream. Each trace will have dependencies that come explicitly
from execution annotations and implicitly from underlying parallelization
strategies. For example, EMB has an explicit output dependency of Bot_MLP_0
and implicit output dependency of EMB_c_A2A from sharding the embedding table.
EMB_c_A2A is blocking since Transformer_Attn_0 needs EMB_c_A2A’s results.

Estimating Communication Collective Execution. For All2All, we estimate its
execution as:

$\sim$ (“SendCount” Bytes per GPU) / (Effective All2All BW)

where “SendCount” Bytes per GPU is the number of bytes sent by each GPU to
every other GPU. “SendCount” Bytes per GPU is dependent on not only “Lookup
bytes per GPU" but also the sharding degree and number of devices. Since the
All2All NCCL implementation is composed of individual point-to-point Send()
and Recv() calls, it is bound by the slowest level of interconnect. Thus,
Effective All2All BW is set as that of either Infiniband or RoCE. For other
cases, like an 8-GPU system, Effective All2All BW may be NVLink BW.

Likewise, we can generate a similar set of traces for the backward pass. Since
the MLP and transformer layers are parallelized via DDP, we have non-blocking
AllReduce communication calls during the backward pass. The AllReduce calls
are for aggregating per-layer weight gradients and are thus non-blocking
(i.e., they are not on the critical path for backpropagation). We estimate the
non-blocking AllReduce calls for weight gradient calls as:

$\sim$ (“SendBuffer” Bytes / GPU) / (Effective AllReduce BW)

where “SendBuffer” Bytes is the total number of bytes sent by each GPU and is
directly proportional to the number of parameters in each layer. Effective
AllReduce BW is a ratio of intra-node communication (e.g., NVLink) bandwidth
and inter-node communication (e.g., Infiniband or RoCE) since data is
communicated on both classes of channels for the NCCL implementation. The
exact ratio between the two communication technologies is dependent on factors
like the number of nodes and NCCL implementation version (e.g., ring vs.
tree). We use real hardware measurement data to understand what these
effective interconnect ratios and bandwidths are in practice. Large-scale
training also often exhibits non-constant bandwidth across intra- and inter-
node hierarchies. We also consider AllGather and ReduceScatter communication
calls, which are required in FSDP and TP.

Factoring In Computation-Communication Overlap. We maintain separate
computation and communication streams and overlap traces with no data
dependencies. In this performance model, we assume GPU kernels are launched
whenever data dependencies are resolved. Ideally, we want as much overlap
between computation and communication as possible. As we can see in Figure 6,
there is a segment of exposed communication for the All2All where compute and
memory units of the training device (i.e., GPU) are mostly idle and thus
underutilized.

This performance model allows us to both identify combinations of kernels and
parallelization strategies that lead to exposed communication and experiment
with different parallelization strategies to decrease exposed communication
segments. Optimizing for computation-communication overlap is an important
objective across multi-node, large-scale ML workloads. Currently, 14$\sim$32%
of GPU cycles on the training clusters come from exposed communication (Figure
4).

| Evaluation Metric |  | Measured
---
Result
| Performance
---
Model
Result
| Modeling
---
Accuracy
(%)
DLRM-A | Serialized Iteration Time (ms) | 67.40 ms | 65.30 ms | 96.89%
% Communication Exposed (%) | 82.37% | 75.46% | 91.62%
Throughput (MQPS) | 1.2 MQPS [37] | 1.21 MQPS | 99.17%
DLRM-B | 3.4 MQPS [37] | 3.06 MQPS | 90%
LLaMA-70B |  | GPU Hours for 306k steps
---
(2048 A100s)
| 1,022,361
---
Hrs
| 863,397
---
Hrs
84.66%
Days to Train 1.4T Tokens | 20.83 Days [58] | 19.21 Days | 92.27%

Table 1: Validation of various first-order execution metrics.

##  5 Experimental Methodology

This section describes our validation efforts and details the design space in
this work, e.g., variations of real-world models, hierarchical parallelization
strategies, and hardware platforms.

Performance Model Validation. Table 1 lists validation points of various
first-order execution metrics across real, measured recommendation and LLM
training experiments. For DLRM-A training [37], we validate the performance
model for first-order execution metrics of serialized iteration time, %
communication exposed, and training throughput to 96.89, 91,62, and 99.17%
modeling difference, respectively. Additionally, in Figure 7, we validate
DLRM-A training for complete serialized and overlapped execution behavior
across both 8- and 128-A100 ZionEX platforms. We validate serialized behavior
to ensure accuracy of model layers and parallelization communication,
overlapped behavior to account for at-scale latency-hiding opportunities and
systems of different number of nodes to see networking scaling effects.

In addition to DLRM-A, we also validate DLRM-B training, reporting 3.05 MQPS
from our model against a measured 3.4 MQPS. For the largest LLaMA
configuration (LLaMA-70B), our performance model estimates training time for
all 1.4T tokens to take 19.21 days as opposed to the reported 21 days in [58].
For this use-case, we use the same hardware platform as reported in [58]
(i.e., 2048 80GB HBM A100s). We also validate the aggregate GPU Hours to train
for 306k steps for 84.66% modeling accuracy. We elaborate on avenues for
further modeling accuracy in Section 7.

Figure 7: DLRM-A serialized and overlapped execution validation for 8-,
128-GPU training.

|  DLRM-A [37] | DLRM-A Transformer | DLRM-A MoE |  DLRM-B [37] | DLRM-B Transformer | DLRM-B MoE |  GPT-3 [6] |  LLaMA [58] |  LLaMA2 [59] | LLM-MoE
---|---|---|---|---|---|---|---|---|---|---
# Parameters | 793B | 795B | 332B | 333B | 175B | 65.2B | 70B | 1.8T
| FLOPs
---
per sample/token
638M | 2.6B | 957M | 60M | 2.1B | 90M | 350B | 130.4B | 140B | 550B
| Sparse Lookup Bytes
---
per sample/token
22.61 MB | 13.19 MB | 49.2 KB | 32.8 KB | 42.8 KB
Global Batch Size | 64K | 256K | 2K (4M tokens)
Context Length | N/A | 80 | N/A | 80 | N/A | 2048 | 4096 | 8192

Table 2: Target recommendation models, LLMs, and their variants by key model-
level characteristics.

|  | DLRM
---
Training System [37]
| LLM
---
Training System [58]
Base device | NVIDIA A100 40GB | NVIDIA A100 80GB
Devices per node | 8
# nodes | 16 | 256
Peak TF32 throughput | 20 PFLOPS | 319 PFLOPS
HBM capacity | 5 TB | 164 TB
HBM bandwidth | 199 TB/s | 3.96 PB/s
| Intra-node interconnect
---
bandwidth (unidirectional)
38.4 TB/s | 614.4 TB/s
Inter-node interconnect fabric | RoCE | Infiniband
| Inter-node interconnect
---
bandwidth (unidirectional)
25.6 Tbps | 409.6 Tbps

Table 3: Baseline distributed systems used in evaluation. Figure 8: We can
improve pre-training performance over FSDP baseline by applying intra- and
inter-node parallelization strategies for base dense and transformer layers
separately. Throughput-optimal parallelization strategies are listed in
(intra-, inter-) order. Black and white, underlined text refer to
recommendation base dense and transformer layers, respectively.

Model Variations. Table 2 lists the suite of large ML models explored in
Section 6. We explore transformer and MoE variants of real-world DLRM-A and
DLRM-B. The transformer feature interaction variants have 4 layers and a down-
sampled sequence length of 80. MoE variants are configured with 16 experts (2
active) per layer. For the LLM models, we follow specifications in [6, 58,
59]. For LLM-MoE, we explore a hypothetical 1.8T parameter model with 16-(2
active)way MoE for the MLPs in transformer blocks. We use fixed global batch
sizes as specified in prior studies [37, 58] to maintain target model
accuracy.

Design Space Exploration. We use FSDP [73] as the baseline due to its wide
adoption and ability to best guarantee training feasibility by minimizing
sharding memory footprint. We explore valid hierarchical parallelism
strategies at intra- and inter-node levels, considering combinations of DDP,
FSDP, and TP. For hardware, unless otherwise stated, we use training systems
from prior case studies [37, 58] (Table 3). We also explore implications of
using H100 and H100 SuperPOD systems by replacing our A100-based models with
H100 specifications [41, 42] – i.e., A100+ and A100+ (Inter+).

##  6 Evaluation Results and Analysis

When parallelization strategies are tailored to specific deep learning models
and tasks at hand, we can achieve 8$\sim$124% throughput improvement. Figure 8
overviews pre-training throughput of key large ML models (Table 2) normalized
to the baseline. We achieve, on average 65.9% pre-training throughput
improvement (blue bars) over FSDP by tuning parallelization strategies at the
layer-type granularity. The strategy that achieves optimal training throughput
is indicated in parenthesis. For example, when considering the base dense
layers of DLRM-A, applying Tensor Parallelism within a node of 8 GPUs and
Distributed Data Parallelism across nodes of GPUs (i.e., (TP, DDP)) leads to
optimal pre-training throughput. In cases like DLRM-A Transformer, where both
base dense and transformer layers are present, the optimal way to parallelize
each type of layer may differ.

Additionally, we also indicate, via the orange dotted bars, the expected
throughput improvement from optimizing parallelization strategies if model
parallelization is not constrained by the current distributed systems’ memory
capacity. The throughput-optimal parallelization strategy and its expected
improvement are determined by a multitude of factors, such as underlying model
architecture, underlying distributed system, and specific task. We highlight 7
key observations and discuss the underlying insights:

Figure 9: DLRM-A Pre-training. Considering memory capacity constraints,
applying TP and DDP for intra- and inter-node parallelism, respectively on
base dense layers achieves highest throughput. Gray bar indicates invalid
parallelism strategy due to OOM.

Insight 1: [DLRM] Trillion-parameter embedding tables in DLRMs limit
parallelization strategies for the tables to sharding, shifting overall
parallelization strategy exploration to focus on the dense components (Figure
9).

Since embedding tables of DLRM-A make up 99.96% of its 793B parameters, the
only parallelization strategy viable for DLRM embedding tables on current GPU
systems is naive model parallelism sharding. This leaves parallelization
strategy exploration on the base dense layers. Figure 9 demonstrates that,
over valid parallelization strategies of the base dense layers on the x-axis,
training throughput performance of DLRM-A can vary significantly from 0.19
((TP), (MP)) to 1.14 $\times$ ((TP, DDP), (MP)) over the FSDP baseline.
Applying TP scales communication requirements with size of partial sums and
activations. If we apply TP at the intra-node level – as opposed to globally –
we can take full of advantage of high BW NVLink to communicate the partial
sums and activations. In this case, ((DDP), (MP)) is prohibitive due to OOM
since it necessitates replicating the dense layers’ model parameters,
gradients, and optimizer states across all devices.

Insight 2: [LLMs] The billion-parameter scale of transformer layers in LLMs
makes intra-node replication for compute layers infeasible. In contrast, the
small memory footprint of word embeddings (< 2GB) allows it to be replicated
across all devices via DDP .

In contrast to DLRMs, for LLMs such as GPT-3, the FSDP baseline offers
competitive training throughput performance (Figure 8). Since the word
embeddings of LLMs are relatively small (0.37% of GPT-3), full per-device
embedding replication is a viable option via DDP. As in the DLRM cases, we
focus our parallelization strategy exploration on the compute-bound layers.
However, in the case of GPT-3, any form of layer replication across nodes
(e.g., (TP, DDP)) leads to OOM since intra-node sharding is insufficient for
meeting memory capacity requirements. Additional device memory capacity can
unlock up to 1.68$\times$ training throughput improvement.

Figure 10: Between DLRM variants, both optimal parallelization strategy and
expected throughput improvement vary.

Insight 3: [Parallelization Strategy Order] Ordering of hierarchical
parallelization strategies matter. Replication and sharding strategies must be
placed in the correct order to ensure optimal performance. (Figures 8, 9).

The “order" in which we apply hierarchical parallelization strategies matters
greatly in terms of both memory capacity footprint and expected throughput.
For example, applying ((TP), (DDP)) shards the model component by number of
devices in a node while applying ((DDP), (TP)) shards the component by number
of nodes. In Figure 9, where there are 8 GPUs within a node and 16 nodes, the
latter strategy leads to a lower per-GPU memory footprint. Additionally,
training throughput also varies from using different interconnect channels for
communication. For example, ((TP), (DDP)) leads to AllReduce of activations
over faster NVLink and weight gradients over slower RoCE/IB. On the other
hand, ((DDP), (TP)) leads to communicating activations over RoCE/IB and weight
gradients over NVLink. For LLMs, long context lengths increase the size of
activations to be communicated, so applying inter-node TP leads to significant
slowdown (0.18$\times$ for GPT-3). On the other hand, utilizing NVLink to
communicate large activations leads to 1.34$\times$ speedup for GPT-3.

Figure 11: Pareto curves of parallelization strategies for DLRM variants for
(a) pre-training and (b) inference. Each point is a different parallelization
strategy. Figure 12: Given increasing context lengths, solely altering
parallelization strategies has diminishing returns for performance benefits
over FSDP.

Insight 4: [DLRM Variants] DLRM Transformer and MoE variants introduce new
compute and communication requirements, leading to new parallelization
strategy choice and task-level implications. (Figures 10, 11).

Figure 10 shows how the same set of parallelization strategies interacts with
both DLRM-A and its variants. For DLRM-A Transformer, we apply ((TP), (DDP))
on the base dense layers since that is the optimal strategy for DLRM-A and
focus parallelization strategy exploration on transformer layers. Across the
variants, optimal strategy (yellow star) varies. These differences can be
attributed to how transformers introduce more compute and more opportunities
for communication-computation overlap while MoE increases blocking, non-
overlapping All2All communication. As models continue to evolve,
parallelization strategies will as well.

Figure 11 shows the parallelization strategy design space for DLRM-A and its
model architecture variants by per-device memory capacity requirement and
achievable pre-training/inference throughput. We denote the performance-pareto
curve with solid lines, showing how an increase in memory capacity can lead to
parallelization strategies with higher throughputs. We also observe that for
pre-training, transformer and MoE variants have lower throughput from the
additional computation and communication, respectively. For inference, MoE
variants have higher throughput than transformer variants since the newly
introduced communication calls are asymmetrically distributed in the backward
pass.

Insight 5: [Context-Length] Increasing context-lengths limits the improvements
from parallelization strategy optimizations, necessitating either changes in
model architecture or underlying distributed systems (Figure 12).

Figure 12 shows that input complexity, in terms of context length, plays a key
role in training throughput. We investigate the effectiveness of ((DDP)) and
((TP), (DDP)) across LLMs of increasing context lengths. 2K and 4K context
length examples refer to LLaMA and LLaMA2 while the 8K context length data
point comes from doubling base LLaMA2’s context length while keeping model
architecture constant.

We see that throughput gains from tuning parallelization strategy decreases
with increasing context length, indicating the limits of optimizing this
design space. To further improve throughput performance, changes have to be
made to either the underlying distributed system or ML model architecture.

Figure 13: For DLRM-A pre-training, both overall GPU improvement (A100+) and
specifically upgrading inter-node interconnect fabric lead to observable
performance benefits.

Insight 6: [GPU-Generations] Across generations of GPUs, improvements in
compute, memory, and interconnect not only improve distributed ML performance
but also unlock different viable parallelization strategies.

In Figure 13, we compare the A100 against a GPU with H100’s specifications
(denoted as “A100+”). We also consider the H100 SuperPOD configuration, where
the RoCE/IB inter-node interconnect fabric is replaced by NVLink (i.e., “A100+
(Inter+)”), leading to $\sim$4.5$\times$ inter-node interconnect bandwidth
compared to H100 DGX systems.

Compared to the A100 baselines (blue), using A100+ (orange) leads to varying
degrees of speedup for the different parallelization strategies. The exact
speedup numbers differ due to the fact that compute, memory, and networking
improve at different rates when we replace A100s with A100+s and different
strategies emphasize different system resources. For DLRM-A training,
improving inter-node bandwidth (i.e., A100+ to A100+ (Inter+)) by itself leads
to significant throughput improvement of 1.82$\times$ since the blocking
All2All embedding communication calls are directly accelerated.

Figure 14: Individually scaling different hardware capabilities for (a) DLRM-A
and (b) GPT-3 workloads leads to sub-linear speedup. Concurrently improving
all capabilities leads to super-linear speedup. Figure 15: (a, c) Serialized
execution and (b, d) communication breakdown for both DLRM-A and GPT-3
training allows us to better understand where speedups from hardware
components come from.

Insight 7: [Future Technologies Trends] For large ML workloads, improving
individual hardware components leads to limited throughput gain. Unlocking
further performance requires jointly improving hardware and systems
specifications (Figures 14, 15).

From A100 to A100+, compute, memory capacity, memory bandwidth, intra-node
interconnect bandwidth, inter-node interconnect bandwidth improve by
2.42$\times$, 2$\times$, 1.29$\times$, 1.5$\times$, 2$\times$ (9$\times$ for
SuperPOD), respectively. In Figure 14, we perform a hardware scaling study
where compute, memory capacity and bandwidth, intra- and inter-node
interconnect bandwidth are all improved by 10$\times$ separately and
concurrently. We observe the effects of these improvements on DLRM-A and GPT-3
training and inference.

For DLRM-A pre-training and inference, independently improving anything but
inter-node interconnect by 10$\times$ will only net 1.64 and 2.12$\times$
throughput improvements, respectively. For these use-cases, since blocking
All2All embedding communication is performance-critical, targeting inter-node
communication bandwidth leads to substantial performance improvement. For
GPT-3, since compute-bound layers are critical to overall throughput,
improving just compute throughput leads to more workload acceleration compared
to DLRMs.

Figure 15 details the sources of the performance changes. Serialized execution
breakdown shows execution time allocated to embedding lookups, GEMM, and
specific communication collectives, disregarding the effects of overlap.
Computation-communication overlap breakdown shows how much communication is
hidden behind embedding lookups and GEMM. These breakdowns help us better
understand the speedup results from Figure 14 since throughput improvements
can come from a variety of sources: accelerating compute-heavy layers (e.g.,
compute in GPT-3), reducing overall communication time (e.g., All2All in
recommendation models), or even unlocking new parallelization strategies with
more memory capacity (e.g., DDP for GPT-3).

For all four cases, jointly improving the hardware components leads to super-
linear performance improvement. This is because distributed ML execution is
non-serial so improving each of the performance of each trace segment can lead
to more overlap or unlock new parallelization strategies altogether.

##  7 Related Work and Discussion

Prior Work | Evaluated Large-Scale ML Workloads |  Comms-Compute Overlap |  Validation Strategy and Experiments
---|---|---|---
Transformer |  | Transformer-Variants
---
(Conformer, Transformer-MoE)
DLRM |  | DLRM-Variants
---
(DLRM-Transformer,
DLRM-MoE)
Flexflow [24] |  |  |  |  | v |  | 64-GPU
---
training
Pope. et. al. [49] | v |  |  |  | v (TPU only) |  | 64-TPU
---
inference
Pati. et. al. [46] | v |  |  |  |  | v (GPU w/ pure-DP,TP
---
only, unvalidated)
| 4-GPU GEMM,
---
AllReduce
ASTRA-Sim [52] | v |  | v |  |  | v (unvalidated,
---
see [65])
(see [65])
ASTRA-Sim 2.0 [65] | v |  | v |  | v (unvalidated) |  | 16-GPU
---
AllReduce
Mystique [33] |  |  | v |  |  |  | 64-GPU
---
training
Chakra [57] | v |  | v |  |  | v (unvalidated,
---
see [52, 65])
(see [65])
| MAD-Max
---
(This Work)
v | v | v | v | v (Figure 7) |  | 128, 2K-GPU training
---
(DLRM, Transformer,
Table 1)

Table 4: Related work for distributed AI performance modeling.

We discuss related work in two primary categories: parallelization strategy
exploration and distributed AI performance modeling (Table 4). In addition, we
share opportunities for improving our proposed performance model and its
implications on efficiently scaling large ML models.

Parallelization Strategy Exploration. [32, 68] provide compiler annotations
for identifying efficient parallelization strategies. [34, 53] focus on
optimizing communication collectives via fusion and scheduling. [75] focuses
on operator-level parallelism. [4, 24] focus on parallelization strategy
exploration but are evaluated on older and smaller ML models in Computer
Vision and NLP. [71] explores strategies to overlap compute and communication
before PyTorch. In this paper, we aim to detach parallelization strategy
exploration from existing software implementation details to enable an agile
design space exploration of potentially yet to be implemented models.
Additionally, we target latest trillion-parameters scale models and expand our
design space beyond just collectives.

Distributed AI Performance Modeling. [49] provides an analytical model for
transformer inference on TPUs. [46] projects computation-communication overlap
opportunities for future GPU-centric hardware. [52, 65] provide a simulator
for estimating distributed ML performance that is validated against AllReduce
collectives. [29] builds upon [52, 65] to introduce a design space exploration
tool, yet doesn’t focus on optimizing training throughput for specific use
cases like DLRM models. These works build upon earlier work in simulating [54,
36] and characterizing [22, 23] distributed systems. [63] emphasizes network
optimization. [33] focuses on generating replayable traces to better estimate
hardware resource utilization. [57] is an effort to standardize traces across
different software implementations for fair comparisons and generating
synthetic traces, which can potentially be integrated with our performance
model for better integration with current software implementations. We design
our performance model to be compatible with different hardware platforms,
tasks, and exploration objectives. We also focus on large ML model execution
behavior and validate accordingly.

Even though our proposed performance model successfully navigates the
parallelization strategy co-design space (Section 6), we foresee extensions in
modeling memory requirements more accurately and integrating more real-world
production characteristics.

Memory Estimation. An accurate model of peak memory consumption is critical
for identifying feasible parallelization strategies. However, estimating
memory consumption can be tricky, as operators such as convolution might
allocate temporary buffers internally that will cause a temporary rise in
active memory. Other considerations include modeling which temporaries are
saved for reverse-mode differentiation and layer-specific activation
checkpointing.

Beyond Per-Iteration Execution. Beyond first-order modeling of per-iteration
behavior, we have to consider second-order effects such as datacenter inter-
job interference, network queuing delays, and job rescheduling [2]. At the
hardware-device level, to accurately model new devices and accelerators (e.g.,
H100, TPU), we have to also take into account microarchitecture, utilization,
and software optimization differences.

Environmentally Sustainable Model Development and Deployment. Finally,
enabling higher throughput training and inference for fixed infrastructure
capacity directly improves the cost-effectiveness for AI datacenters. In the
short term, increasing system throughput decreases operational and embodied
carbon footprint [67]. In the long term, throughput/efficiency optimizations
can increase adoption of large ML models – potentially leading to larger
overall carbon footprint due to the rebound effect [66]. For sustainable AI
model development, we must keep in mind optimizing ML for efficient
development and deployment, using renewable energy in operation [47] and
further reducing negative impacts of system hardware manufacturing on the
environment [13, 15].

##  8 Conclusion

We present an agile performance modeling framework to enable large ML model
acceleration across the key model development phases: pre-training, fine-
tuning, and inference. The framework is also validated against large-scale
infrastructures used for state-of-the-art ML tasks. Using the suite of real-
world large ML models on GPU training hardware, we demonstrate 2.24$\times$
and 1.48$\times$ throughput improvement potential for pre-training and
inference, respectively.

## Acknowledgements

We thank Apostolos Kokolis, Giri Anantharaman, Kalyan Saladi and Srinivas
Sridharan for feedback on modeling effective interconnect communication at-
scale. We also thank Can Balioglu, Changhan Wang, and Kaushik Ram Sadagopan
for discussions on performance modeling and optimizations for key deep
learning applications.

## References

  * [1] Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, and Kim Hazelwood.  Understanding training efficiency of deep learning recommendation models at scale.  arXiv preprint arXiv:2011.05497, 2020.
  * [2] Hadeel Albahar, Shruti Dongare, Yanlin Du, Nannan Zhao, Arnab K. Paul, and Ali R. Butt.  Schedtune: A heterogeneity-aware gpu scheduler for deep learning.  In 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 695–705, 2022.
  * [3] Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil I. Shamir, Rakesh Shivanna, and Qiqi Yan.  On the factory floor: Ml engineering for industrial-scale ads recommendation models, 2022.
  * [4] Newsha Ardalani, Saptadeep Pal, and Puneet Gupta.  Deepflow: A cross-stack pathfinding framework for distributed ai systems, 2022.
  * [5] Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov.  Efficient large scale language modeling with mixtures of experts.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11699–11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [7] Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou.  Behavior sequence transformer for e-commerce recommendation in alibaba.  In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, DLP-KDD ’19, New York, NY, USA, 2019. Association for Computing Machinery.
  * [8] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah.  Wide & deep learning for recommender systems.  In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS 2016, page 7–10, New York, NY, USA, 2016. Association for Computing Machinery.
  * [9] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  GLaM: Efficient scaling of language models with mixture-of-experts.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 17–23 Jul 2022.
  * [10] Amin Firoozshahian, Joel Coburn, Roman Levenstein, Rakesh Nattoji, Ashwin Kamath, Olivia Wu, Gurdeepak Grewal, Harish Aepala, Bhasker Jakka, Bob Dreyer, Adam Hutchin, Utku Diril, Krishnakumar Nair, Ehsan K. Aredestani, Martin Schatz, Yuchen Hao, Rakesh Komuravelli, Kunming Ho, Sameer Abu Asal, Joe Shajrawi, Kevin Quinn, Nagesh Sreedhara, Pankaj Kansal, Willie Wei, Dheepak Jayaraman, Linda Cheng, Pritam Chopda, Eric Wang, Ajay Bikumandla, Arun Karthik Sengottuvel, Krishna Thottempudi, Ashwin Narasimha, Brian Dodds, Cao Gao, Jiyuan Zhang, Mohammed Al-Sanabani, Ana Zehtabioskuie, Jordan Fix, Hangchen Yu, Richard Li, Kaustubh Gondkar, Jack Montgomery, Mike Tsai, Saritha Dwarakapuram, Sanjay Desai, Nili Avidan, Poorvaja Ramani, Karthik Narayanan, Ajit Mathews, Sethu Gopal, Maxim Naumov, Vijay Rao, Krishna Noru, Harikrishna Reddy, Prahlad Venkatapuram, and Alexis Bjorlin.  Mtia: First generation silicon targeting meta’s recommendation systems.  In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [11] Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchan.  Chatgpt is not all you need. a state of the art review of large generative ai models, 2023.
  * [12] Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchán.  A survey of generative ai applications, 2023.
  * [13] Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu.  Act: Designing sustainable computer systems with an architectural carbon modeling tool.  In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, page 784–799, New York, NY, USA, 2022. Association for Computing Machinery.
  * [14] Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David Brooks, and Carole-Jean Wu.  Deeprecsys: A system for optimizing end-to-end at-scale neural recommendation inference.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 982–995, 2020.
  * [15] Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S. Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu.  Chasing carbon: The elusive environmental footprint of computing.  IEEE Micro, 42(4):37–47, jul 2022.
  * [16] Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Mark Hempstead, Bill Jia, et al.  The architectural implications of facebook’s dnn-based personalized recommendation.  In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 488–501. IEEE, 2020.
  * [17] Kim Hazelwood, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, et al.  Applied machine learning at facebook: A datacenter infrastructure perspective.  In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 620–629. IEEE, 2018.
  * [18] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, pages 173–182, Republic and Canton of Geneva, Switzerland, 2017. International World Wide Web Conferences Steering Committee.
  * [19] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [20] S. Hsia, U. Gupta, M. Wilkening, C. Wu, G. Wei, and D. Brooks.  Cross-stack workload characterization of deep recommendation systems.  In 2020 IEEE International Symposium on Workload Characterization (IISWC), pages 157–168, Los Alamitos, CA, USA, oct 2020. IEEE Computer Society.
  * [21] Samuel Hsia, Udit Gupta, Bilge Acun, Newsha Ardalani, Pan Zhong, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu.  Mp-rec: Hardware-software co-design to enable multi-path recommendation.  In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2023, page 449–465, New York, NY, USA, 2023. Association for Computing Machinery.
  * [22] Arpan Jain, Ammar Ahmad Awan, Quentin Anthony, Hari Subramoni, and Dhableswar K. DK Panda.  Performance characterization of dnn training using tensorflow and pytorch on modern clusters.  In 2019 IEEE International Conference on Cluster Computing (CLUSTER), pages 1–11, 2019.
  * [23] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, unjie Qian, Wencong Xiao, and Fan Yang.  Analysis of large-scale multi-tenant gpu clusters for dnn training workloads.  In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’19, page 947–960, USA, 2019. USENIX Association.
  * [24] Zhihao Jia, Matei Zaharia, and Alex Aiken.  Beyond data and model parallelism for deep neural networks, 2018.
  * [25] Norman P. Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B. Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, Thomas Norrie, Nishant Patil, Sushma Prasad, Cliff Young, Zongwei Zhou, and David Patterson.  Ten lessons from three generations shaped google’s tpuv4i : Industrial product.  In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1–14, 2021.
  * [26] Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson.  Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings, 2023.
  * [27] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson.  A domain-specific supercomputer for training deep neural networks.  Commun. ACM, 63(7):67–78, jun 2020.
  * [28] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al.  In-datacenter performance analysis of a tensor processing unit.  In Proceedings of the ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pages 1–12. IEEE, 2017.
  * [29] Divya Kiran Kadiyala, Saeed Rashidi, Taekyung Heo, Abhimanyu Rajeshkumar Bambhaniya, Tushar Krishna, and Alexandros Daglis.  Comet: A comprehensive cluster design methodology for distributed deep learning training, 2022.
  * [30] Liu Ke, Udit Gupta, Benjamin Youngjae Cho, David Brooks, Vikas Chandra, Utku Diril, Amin Firoozshahian, Kim Hazelwood, Bill Jia, Hsien-Hsin S Lee, et al.  Recnmp: Accelerating personalized recommendation with near-memory processing.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 790–803. IEEE, 2020.
  * [31] Youngeun Kwon, Yunjae Lee, and Minsoo Rhu.  Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning.  In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 740–753, 2019.
  * [32] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
  * [33] Mingyu Liang, Wenyin Fu, Louis Feng, Zhongyi Lin, Pavani Panakanti, Shengbao Zheng, Srinivas Sridharan, and Christina Delimitrou.  Mystique: Enabling accurate and scalable generation of production ai benchmarks.  In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [34] Kshiteej Mahajan, Ching-Hsiang Chu, Srinivas Sridharan, and Aditya Akella.  Better together: Jointly optimizing ML collective scheduling and execution planning using SYNDICATE.  In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 809–824, Boston, MA, April 2023. USENIX Association.
  * [35] James Manyika.  An overview of bard: an early experiment with generative ai.  https://ai.google/static/documents/google-about-bard.pdf, 2023.
  * [36] Alian Mohammad, Umur Darbaz, Gabor Dozsa, Stephan Diestelhorst, Daehoon Kim, and Nam Sung Kim.  dist-gem5: Distributed simulation of computer clusters.  In 2017 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 153–162, 2017.
  * [37] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie (Amy) Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komuravelli, Ching-Hsiang Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng, Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, KR Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Ajit Mathews, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao.  Software-hardware co-design for fast and scalable training of deep learning recommendation models.  In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, page 993–1011, New York, NY, USA, 2022. Association for Computing Machinery.
  * [38] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al.  Deep learning recommendation model for personalization and recommendation systems.  arXiv preprint arXiv:1906.00091, 2019.
  * [39] NVIDIA.  Nvidia tesla v100 gpu accelerator datasheet.  https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf, 2018\.
  * [40] NVIDIA.  Nvidia dgx-1 deep learning system datasheet.  https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/dgx-1/dgx-1-rhel-datasheet-nvidia-us-808336-r3-web.pdf, 2019\.
  * [41] NVIDIA.  Nvidia h100 tensor core gpu datasheet.  https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet, 2023\.
  * [42] NVIDIA.  Nvidia hgx h100 - ai supercomputing platform datasheet.  https://nvdam.widen.net/s/5kgbjq2v2t/hpc-hgx-h100-datasheet-nvidia-web, 2023\.
  * [43] NVIDIA.  Nvidia transformer engine version 0.10.0.  https://docs.nvidia.com/deeplearning/transformer-engine/index.html, 2023\.
  * [44] OpenAI.  Chatgpt.  https://chat.openai.com/, 2023.
  * [45] OpenAI.  Gpt-4 technical report, 2023.
  * [46] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair.  Computation vs. communication scaling for future transformers on future hardware, 2023.
  * [47] D. Patterson, J. Gonzalez, U. Holzle, Q. Le, C. Liang, L. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean.  The carbon footprint of machine learning training will plateau, then shrink.  Computer, 55(07):18–28, jul 2022.
  * [48] Changhua Pei, Yi Zhang, Yongfeng Zhang, Fei Sun, Xiao Lin, Hanxiao Sun, Jian Wu, Peng Jiang, Junfeng Ge, Wenwu Ou, and Dan Pei.  Personalized re-ranking for recommendation.  In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, page 3–11, New York, NY, USA, 2019. Association for Computing Machinery.
  * [49] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.  Efficiently scaling transformer inference, 2022.
  * [50] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.
  * [51] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models, 2020\.
  * [52] Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna.  Astra-sim: Enabling sw/hw co-design exploration for distributed dl training platforms.  In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 81–92, 2020.
  * [53] Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Sridharan, and Tushar Krishna.  Themis: A network bandwidth-aware collective scheduling policy for distributed training of dl models.  In Proceedings of the 49th Annual International Symposium on Computer Architecture, ISCA ’22, page 581–596, New York, NY, USA, 2022. Association for Computing Machinery.
  * [54] Daniel Sanchez and Christos Kozyrakis.  Zsim: Fast and accurate microarchitectural simulation of thousand-core systems.  In Proceedings of the 40th Annual International Symposium on Computer Architecture, ISCA ’13, page 475–486, New York, NY, USA, 2013. Association for Computing Machinery.
  * [55] Geet Sethi, Bilge Acun, Niket Agarwal, Christos Kozyrakis, Caroline Trippel, and Carole-Jean Wu.  Recshard: Statistical feature-based memory optimization for industry-scale neural recommendation.  In 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2022.
  * [56] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.
  * [57] Srinivas Sridharan, Taekyung Heo, Louis Feng, Zhaodong Wang, Matt Bergeron, Wenyin Fu, Shengbao Zheng, Brian Coutinho, Saeed Rashidi, Changhai Man, and Tushar Krishna.  Chakra: Advancing performance benchmarking and co-design using standardized execution traces, 2023.
  * [58] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [59] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [61] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang.  Deep & cross network for ad click predictions, 2017.
  * [62] Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi.  Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems.  In Proceedings of the Web Conference 2021, WWW ’21, page 1785–1797, New York, NY, USA, 2021. Association for Computing Machinery.
  * [63] Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch.  Topoopt: Co-optimizing network topology and parallelization strategy for distributed training jobs, 2022.
  * [64] Mark Wilkening, Udit Gupta, Samuel Hsia, Caroline Trippel, Carole-Jean Wu, David Brooks, and Gu-Yeon Wei.  Recssd: Near data processing for solid state drive based recommendation inference.  In 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2021.
  * [65] William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna.  Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale.  In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 283–294, 2023.
  * [66] Jackson Woodruff, David Schall, Michael FP O’Boyle, and Christopher Woodruff.  When does saving power save the planet?, 2023.
  * [67] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood.  Sustainable ai: Environmental implications, challenges and opportunities.  In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 795–813, 2022.
  * [68] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen.  Gspmd: General and scalable parallelization for ml computation graphs, 2021.
  * [69] Xinyang Yi, Yi-Fan Chen, Sukriti Ramesh, Vinu Rajashekhar, Lichan Hong, Noah Fiedel, Nandini Seshadri, Lukasz Heldt, Xiang Wu, and Ed H. Chi.  Factorized deep retrieval and distributed tensorflow serving.  In Proceedings of Machine Learning and Systems, SysML’18, 2018\.
  * [70] Buyun Zhang, Liang Luo, Xi Liu, Jay Li, Zeliang Chen, Weilin Zhang, Xiaohan Wei, Yuchen Hao, Michael Tsang, Wenjun Wang, Yang Liu, Huayu Li, Yasmine Badr, Jongsoo Park, Jiyan Yang, Dheevatsa Mudigere, and Ellie Wen.  Dhen: A deep and hierarchical ensemble network for large-scale click-through rate prediction, 2022.
  * [71] Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, and Eric P. Xing.  Poseidon: An efficient communication architecture for distributed deep learning on gpu clusters.  In Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’17, page 181–193, USA, 2017. USENIX Association.
  * [72] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * [73] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li.  Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023\.
  * [74] Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi.  Recommending what video to watch next: A multitask ranking system.  In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, pages 43–51, New York, NY, USA, 2019. ACM.
  * [75] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica.  Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning.  In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 559–578, Carlsbad, CA, July 2022. USENIX Association.
  * [76] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai.  Deep interest evolution network for click-through rate prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5941–5948, 2019.
  * [77] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai.  Deep interest network for click-through rate prediction.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1059–1068. ACM, 2018.