#  MLTCP: Congestion Control for DNN Training

Sudarsanan Rajasekaran Sanjoli Narang∗ Anton A. Zabreyko∗ Manya Ghobadi
Massachusetts Institute of Technology

###### Abstract.

We present MLTCP, a technique to augment today’s congestion control algorithms
to accelerate DNN training jobs in shared GPU clusters. MLTCP enables the
communication phases of jobs that compete for network bandwidth to interleave
with each other, thereby utilizing the network efficiently. At the heart of
MLTCP lies a very simple principle based on a key conceptual insight: DNN
training flows should scale their congestion window size based on the number
of bytes sent at each training iteration. We show that integrating this
principle into today’s congestion control protocols is straightforward: by
adding 30–60 lines of code to Reno, CUBIC, or DCQCN, MLTCP stabilizes flows of
different jobs into an interleaved state within a few training iterations,
regardless of the number of competing flows or the start time of each flow.
Our experiments with popular DNN training jobs demonstrate that enabling MLTCP
accelerates the average and 99th percentile training iteration time by up to
2$\times$ and 4$\times$, respectively.

∗ Equal contribution

††copyright: none††doi: ††isbn: ††price:

##  1\. Introduction

The relentless pursuit of training ever-larger DNN models on shared GPU
clusters makes efficient network congestion management crucial in sustaining
high GPU utilization. Prior papers demonstrated that network congestion often
results in longer training time (Wang et al., 2023; Mudigere et al., 2022; Cho
et al., 2019). The state-of-the-art approaches to minimize the impact of
network congestion include ($i$) reducing the size of parameters to transmit
through the network using quantization or compression techniques
(Watcharapichat et al., 2016; Alistarh et al., 2017; Lin et al., 2017;
Jayarajan et al., 2019); ($ii$) using pipelining to reduce the exposed
communication time by overlapping the communication period of a job with its
computation phase (Narayanan et al., 2019; Huang et al., 2019; Li et al.,
2021; Jiang et al., 2020a; Mohan et al., 2022; Peng et al., 2019); ($iii$)
employing job scheduling and worker placement techniques that minimize network
sharing by placing workers of the same job as close to each other as possible
(Xiao et al., 2018; Mahajan et al., 2020; Qiao et al., 2021; Gu et al., 2019;
Zheng et al., 2023; Peng et al., 2018; Hashemi et al., 2019; Zhao et al.,
2022; Jayaram Subramanya et al., 2023).

Although these techniques reduce the chances of network congestion, cross-job
network contention is inevitable in large-scale shared GPU clusters. Recent
work (Rajasekaran et al., 2022) demonstrated that introducing unfairness in
congestion control has a surprising payoff that accelerates the training time
of DNN jobs in shared GPU clusters. The authors manually adjusted DCQCN (Zhu
et al., 2015a) congestion control parameters to force a static unfair
bandwidth sharing (60%-40% instead of 50%-50%) between competing DNN training
jobs. After a few training iterations, this static unfairness created a
desirable side-effect that nudged the communication phase of one job to
interleave with the other, resulting in better overall network utilization and
accelerating the training time of both jobs.

In contrast to pipelining (Jiang et al., 2020a; Mohan et al., 2022; Peng et
al., 2019; Narayanan et al., 2019; Huang et al., 2019; Li et al., 2021), which
overlaps the compute and communication phases of the same training job, the
goal of inter-job communication interleaving is to fit the communication
phases of different jobs to share the network more efficiently (§2.1). We show
that robust communication interleaving, while promising, has several unsolved
challenges (§2.2): ($i$) robustness to stragglers; ($ii$) robustness to
partially compatible jobs; ($iii$) robustness to circular dependencies between
jobs and links.

This paper asks the following question: “Is it possible to achieve robust
inter-job communication interleaving in a completely distributed manner?” At
first blush, it appears that achieving this goal is impossible because
communication interleaving is dependent on the exact traffic pattern of the
jobs. How can two (or more) jobs interleave their communication patterns if
they are unaware of each other’s existence? Without a centralized controller,
end-hosts do not have any information about which other jobs are in the
network, making it seemingly impossible to self-adjust their communication
time to achieve interleaving.

Figure 1. Inter-job communication interleaving. Figure adapted from (Zhao et
al., 2022; Rajasekaran et al., 2022, 2024).

We propose MLTCP, a technique to augment today’s congestion control algorithms
to enable DNN training jobs to converge automatically into an interleaved
state. An attractive feature of MLTCP is its simple principle based on a key
conceptual insight: DNN training flows should dynamically adjust their
aggressiveness based on the number of bytes sent. This principle applies to
window-based and rate-based congestion control protocols. For instance, upon
the receipt of an acknowledgment packet, window-based congestion control
protocols update the congestion window, $cwnd$, as:

(1) $cwnd\leftarrow cwnd+additive\\_increase$

where $additive\\_increase$ indicates the magnitude of congestion window
increase based on the specific goals of each congestion control variant. MLTCP
updates Equation 1 to:

(2) $cwnd\leftarrow cwnd+\mathcal{F}(bytes\\_ratio)\times additive\\_increase$

where $\mathcal{F}$ is a linear increasing function and $bytes\\_ratio$ is the
number of bytes successfully sent normalized by the total number of bytes
during a training iteration (§3). Unlike traditional datacenter traffic, where
accurately estimating the size of every flow is intractable (Alizadeh et al.,
2013; Ðukić et al., 2019; Gao et al., 2015), the total number of bytes in a
training iteration is a predictable and consistent value throughout the
lifetime of most DNN training jobs as long as the hyper-parameters remain
unchanged (Jiang et al., 2020b; Wang et al., 2023; Unger et al., 2022).

The intuition behind this simple technique is that MLTCP adjusts the
aggressiveness of competing flows based on the remaining bytes of the current
iteration, thereby approximating the Shortest Remaining Processing Time (SRPT)
policy. Given that DNN training workloads are periodic, MLTCP creates a
positive cascading effect that separates the communication phases of competing
jobs into an interleaved state after a few iterations.

To evaluate MLTCP, we build a testbed with 12 servers, each equipped with one
NVIDIA A100 GPU (a10, 2021) and one 50 Gbps RDMA NIC. We then augment TCP Reno
(Jacobson, 1988a), TCP CUBIC (Ha et al., 2008), and RoCE DCQCN (Zhu et al.,
2015a) algorithms with MLTCP by adding 30–60 lines of code for each variant.
Our experiments with 7 representative DNN models show that MLTCP stabilizes
the flows into an interleaved state after $\approx$10 training iterations
while improving the average and 99th percentile tail training iteration times
of jobs by up to 2$\times$ and 4$\times$, respectively, compared to default
congestion control algorithms. We also show that MLTCP reduces the average
number of dropped or ECN marked packets by up to 14.59$\times$. Finally, we
demonstrate that MLTCP flows are robust to stragglers, partially compatible
jobs, and circular dependencies between jobs and flows. Our kernel patch and
evaluation scripts will be available online.

##  2\. Communication Interleaving

In this section, we first provide a background on inter-job communication
interleaving (§2.1). We then identify several unsolved challenges to achieving
robust communication interleaving (§2.2).

###  2.1. Background

Muri (Zhao et al., 2022) introduced the concept of inter-job multi-resource
interleaving for DNN training jobs. Muri scheduled jobs such that different
resources (GPU, CPU, network, and storage) of different jobs are interleaved
across the time dimension. To do so, Muri formulated the problem of resource
interleaving as a k-dimensional maximum weighted matching problem and
demonstrated significant improvements in job completion times. However, given
the complexity of the matching problem, Muri could only consider jobs sharing
the same set of GPUs, CPUs, memory, and network resources (Zhao et al., 2022).

Rajasekaran et al. (Rajasekaran et al., 2022, 2024) extended Muri’s footprint
by focusing on the network as the only bottlenecked resource. They proposed
interleaving the communication phases of jobs running on different GPU
servers. Figure 1(a) illustrates their approach with two jobs: Job1 (training
on GPU1 and GPU2) and Job2 (training on GPU3 and GPU4), where each GPU is
located in a different server in the cluster. Figures 1(b) and (c) show the
compute and communication patterns of Job1 and Job2 in isolation,
respectively. Each job uses pipelining to overlap its compute and
communication phases. However, pipelining cannot prevent network congestion
when the communication phases of two jobs collide. As shown in Figure 1(d),
without explicit inter-job communication interleaving, the communication
phases of jobs sharing a network link will likely collide, causing congestion.
Figure 1(e) illustrates a more favorable case where the communication phases
of Job1 and Job2 are interleaved, allowing the network link to be shared more
efficiently. The compute phases are not interleaved because they are on
different GPUs.

Figure 2. Circular dependency between jobs and links.

To achieve inter-job communication interleaving, (Rajasekaran et al., 2022)
used an unfair congestion control algorithm by manually changing the
parameters of DCQCN (Zhu et al., 2015a). For instance, the paper showed that
when two VGG19 (Simonyan and Zisserman, 2015) training jobs share a network
link, forcing a 60%-40% bandwidth share (instead of the default 50%-50%)
results in 1.23$\times$ speed up for both jobs compared to the default case
(Rajasekaran et al., 2022). The intuition behind this surprising result is
that in fair bandwidth-sharing, the jobs communicate simultaneously, causing
congestion similar to Figure 1(d). In contrast, unfair bandwidth-sharing
nudges the competing jobs from synchronization and pushes them toward an
interleaved state, similar to Figure 1(e).

In follow-up work, Cassini (Rajasekaran et al., 2024) used a centralized
scheduler that shifts the start time of jobs to fit the compute phase of one
job into the communication phase of another. Cassini consists of two parts.
First, a network-aware scheduler considers job compatibility when placing jobs
on the network. Second, an agent on each server continuously monitors these
time shifts to ensure jobs do not deviate from their intended start times.

The following section explains that both (Rajasekaran et al., 2022) and
(Rajasekaran et al., 2024) considered relatively stable conditions in testbed
experiments and ignored several practical challenges to achieve robust inter-
job communication interleaving. We then propose MLTCP, a straightforward
change to congestion control algorithms that automatically achieves robust
interleaving without manually changing the congestion control parameters or
using a centralized scheduler (§3).

###  2.2. Challenges of Comm. Interleaving

In this section, we discuss three practical challenges to achieving robust
inter-job communication interleaving in DNN training clusters: ($i$) partially
compatible jobs; ($ii$) stragglers; ($iii$) circular dependencies across
links.111Note that Muri (Zhao et al., 2022) cannot solve any of these
challenges because the jobs we consider do not share the same GPUs.

Figure 3. MLTCP high-level concept.

Challenge 1: Robustness to partially compatible jobs. Interleaving the
communication phases of jobs that share a link requires that the compute phase
of one job fit into the communication phase of the other. In other words, the
jobs must be compatible (Rajasekaran et al., 2022). However, given the
plethora of DNN models, it is nearly impossible for all jobs sharing a network
link to be perfectly compatible. Our experiments show that prior approaches
(Rajasekaran et al., 2022, 2024) cannot achieve speedups when the
compatibility ratio is below 70% (§4.6). In particular, when partially
compatible jobs share a link, prior approaches to achieve interleaving
oscillate between interleaved, partially interleaved, and not interleaved
states. Since these partially interleaved and not interleaved states repeat
throughout the training, some of the jobs’ training times become significantly
worse than fair-sharing, defeating the purpose of interleaving to accelerate
all jobs.

Challenge 2: Robustness to stragglers. Although several testbed experiments
(Khani et al., 2021; Wang et al., 2023; Xiao et al., 2018) have shown that the
communication pattern of DNN training jobs is predictable, servers in shared
clusters often experience uncorrelated latency jitters due to the presence of
stragglers (Harlap et al., 2016; Zaharia et al., 2008; Cortez et al., 2017;
Ananthanarayanan et al., 2012; Tandon et al., 2017; Cipar et al., 2013; Yang
et al., 2022). Robustness to noise becomes crucial to ensure the network
maintains an interleaved state while gracefully handling these unpredictable
factors without compromising the overall training time. Our experiments show
that when the straggler probability is above 10%, prior approaches
(Rajasekaran et al., 2022, 2024) are unable to maintain interleaving and end
up slowing down the 99th percentile training iteration times compared to
default congestion control (§4.5).

Figure 4. (a) Favoring Job1 interleaves the jobs. (b) Favoring Job2 overlaps
the jobs.

Challenge 3: Robustness to circular dependencies across links. In practical
deployments, jobs share the network with multiple jobs at different links.
Consider the example shown in Figure 2(a) where Job1 is competing with Job2 on
link $l_{1}$, Job2 is competing with Job3 on link $l_{2}$, and Job1 is
competing with Job3 on link $l_{3}$. This scenario creates a circular
dependency across the three jobs, shown in Figure 2(b). Cassini (Rajasekaran
et al., 2024) cannot find the correct schedule to interleave these jobs
because the affinity graph (shown in Figure 2(c)) has a loop.222Theorem 1 in
Cassini (Rajasekaran et al., 2024) requires a loop-free affinity graph.
Similarly, the approach proposed in (Rajasekaran et al., 2022) considers the
links independently and cannot determine an unfair bandwidth sharing between
these jobs on different links. In contrast, our experiments demonstrate that
MLTCP flows converge to optimal interleaving even in such circular
dependencies (§4.7).

##  3\. MLTCP Design

This section describes MLTCP, a congestion control feature to automatically
interleave the communication phases of DNN jobs that share a network. First,
we describe MLTCP’s goals and high-level design principles (§3.1). Then, we
describe MLTCP’s core techniques: the job favoritism policy (§3.2) and the
bandwidth aggressiveness function (§3.3). Next, we show how these techniques
apply to various congestion control implementations, including window-based
algorithms, such as Reno and CUBIC, as well as rate-based ones, such as DCQCN
(§3.4). Finally, we discuss how MLTCP updates its parameters during the
training (§3.5).

###  3.1. Goals and High-Level Concept

MLTCP’s goal is to stabilize flows belonging to different DNN training jobs
into an interleaved state without using a centralized controller, regardless
of job start times, DNN model size, parallelization strategy, or number of
jobs competing for bandwidth. At the same time, MLTCP should maintain
robustness to stragglers, partially compatible jobs, and circular dependencies
across jobs and links.

Figure 3 illustrates MLTCP’s high-level concept when two jobs Job1 and Job2
share a bottleneck. Intuitively, transforming their communication patterns
from a non-interleaved state to a fully interleaved state means the
communication phase of one job should slide to the left of the figure, while
the other simultaneously slides to the right until they are fully interleaved.
MLTCP uses two techniques to achieve this goal. First, it selects which job
should slide left by allowing it to occupy a higher portion of the link
bandwidth at every iteration until the communication patterns are interleaved.
This technique is called the job favoritism policy (§3.2). Second, to achieve
the effect of favoritism in a distributed manner, MLTCP introduces a novel
feature that dynamically adjusts the congestion window (or rate) of flows that
belong to each competing job using a scaling function called the bandwidth
aggressiveness function (§3.3).

###  3.2. Job Favoritism Policy

DNN training jobs are periodic; hence, when two or more jobs compete for
network, allocating more bandwidth to the flows that belong to one of the jobs
accelerates that job’s current training iteration and, more importantly,
affects the start times of its subsequent training iterations. In other words,
giving more bandwidth to one of the jobs slides this job’s communication
pattern to the left of the others. Eventually, this process nudges the
communication phases of the jobs to separate. The question is, which job
should be given higher bandwidth? In fact, the choice of the job matters!

For example, consider two DNN training jobs sharing a bottleneck link with 50
Gbps capacity, as shown in Figure 4(a). At time $t=0$, Job1 has a slight head
start over Job2. As a result, favoring Job1 over Job2 accelerates its training
iterations even more and causes the communication phases to further separate
in the subsequent iterations. In contrast, favoring Job2 hurts the eventual
interleaving state and leads the communication pattern of the two jobs to
overlap even more, as shown in Figure 4(b).

In principle, any favoritism ordering that follows the Shortest Remaining
Processing Time (SRPT) policy is valid. For instance, policies that favor jobs
based on earlier iteration start time, earlier iteration finish time, smallest
data remaining, or largest data sent all stabilize into an interleaved state.
The intuition is that the favoritism policies that reinforce SRPT eventually
achieve an interleaving state. However, a policy that cancels out the effect
of SRPT, such as favoring jobs based on the latest iteration start time,
latest iteration finish time, largest data remaining, or smallest data sent,
cannot achieve interleaving. In MLTCP, we use the percentage of bytes sent as
our favoritism policy because it enables us to implement MLTCP in a fully
distributed manner. Time-based policies require a centralized controller to
compare jobs’ start or finish times. An important property of MLTCP’s job
favoritism technique is that it provides native robustness to stragglers and
partially compatible jobs because it fundamentally emulates an SRPT policy. As
a result, any disturbance in communication patterns automatically triggers the
sliding mechanism toward stabilizing into an interleaved state.

###  3.3. Bandwidth Aggressiveness Function

To realize MLTCP’s favoritism policy in a distributed manner, the flow closest
to completing its iteration should automatically grab a larger bandwidth share
than others, helping MLTCP achieve the desired sliding effect. MLTCP achieves
this goal by dynamically adjusting the congestion window (cwnd) or sending
rate of flows based on a bandwidth aggressiveness function
$\mathcal{F}(bytes\\_ratio)$, where $bytes\\_ratio$ is the number of bytes
sent in the current iteration normalized by the total bytes
($\frac{bytes\\_sent}{total\\_bytes}$). MLTCP obtains these parameters from
the TCP socket (details in §3.5).

Figure 5 provides a conceptual illustration of several potential
aggressiveness functions. The x-axis represents $bytes\\_ratio$, and the
y-axis determines the range of aggressiveness. The figure shows several
function shapes. Our evaluations (§4.8) demonstrate that many aggressiveness
functions can achieve MLTCP’s interleaving goals as long as three requirements
are satisfied: ($i$) the range is large enough to absorb the noise (e.g.,
slight variations in round-trip time (RTT) or iteration times) in the network;
($ii$) the derivative of the function is non-negative; ($iii$) all flows
employ the same bandwidth aggressiveness function. Hence, to simplify the
implementation, we propose a linear increasing function:

(3) $\mathcal{F}(bytes\\_ratio)=S\times bytes\\_ratio+I$

Where $S$ and $I$ represent the linear function’s slope and intercept,
respectively. These are constant parameters tuned based on the congestion
control algorithm, link rate, and noise in the system. Our evaluations suggest
guidelines to select these parameters (§4.8).

MLTCP has two mechanisms for using $\mathcal{F}$ to augment default congestion
control algorithms. The first mechanism uses $\mathcal{F}$ as a scaling factor
for the cwnd increment made during the window (or rate) increase step. As a
result, the flow with the most bytes sent increases its window (or rate) more
aggressively, triggering MLTCP’s favoritism policy and eventually interleaving
the communication phases of different jobs. The second mechanism uses
$\mathcal{F}$ as a scaling factor during the multiplicative decrease step.
This mechanism has a similar effect: the flow with the most bytes sent reduces
its window (or rate) less aggressively, enabling it to claim a bigger share of
the bandwidth even after packet loss (or ECN) events. This again follows the
job favoritism principle and leads to interleaving. Note that $\mathcal{F}$’s
output is unitless.

###  3.4. Augmenting Reno, CUBIC, & DCQCN

Figure 5. Potential bandwidth aggressiveness functions for adjusting the cwnd
(or rate).

This section explains how we augment Reno, CUBIC, and DCQCN algorithms with
MLTCP. Following Linux’s implementation, throughout this paper, we assume cwnd
is expressed in packets (not bytes).

Reno additive increase. The TCP Reno algorithm uses a cumulative ack mechanism
to acknowledge multiple in-flight packets with a single ack. Upon the receipt
of an ack, Reno increases its congestion window as follows:

(4) $cwnd\leftarrow cwnd+\frac{\\#num\\_acks}{cwnd}$

MLTCP updates the above equation to:

(5) $cwnd\leftarrow
cwnd+{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times\cfrac{\\#num\\_acks}{cwnd}$

Reno multiplicative decrease. With every dropped packet, TCP Reno updates its
cwnd as follows:

(6) $cwnd\leftarrow 0.5\times cwnd$

We analyze the relationshp

MLTCP updates the above equation to:

(7)
$cwnd\leftarrow{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times
0.5\times cwnd$

CUBIC window growth. CUBIC’s window growth algorithm computes the target cwnd
using a cubic function based on the time gap from the last multiplicative
decrease event, as shown below:

(8) $cwnd\leftarrow CUBIC(time)$

MLTCP scales the calculated time gap by $\mathcal{F}(bytes\\_ratio)$ before
applying the cubic function:

(9) $cwnd\leftarrow
CUBIC({\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times
time)$

When different MLTCP flows compete for bandwidth, a smaller
$\mathcal{F}(bytes\\_ratio)$ value has a dilation effect on the time
calculation, causing less favored MLTCP-CUBIC flows with smaller
$\mathcal{F}(bytes\\_ratio)$ to increase their cwnd more slowly than those
with higher $\mathcal{F}(bytes\\_ratio)$ values.

CUBIC multiplicative decrease. The multiplicative decrease step of CUBIC is
similar to Reno, but instead of having a fixed value of 0.5, CUBIC specifies
the multiplicative decrease factor as a parameter called $\beta$. The cwnd
update during multiplicative decrease for CUBIC is given by:

(10) $cwnd\leftarrow\beta\times cwnd$

MLTCP updates the multiplicative decrease factor of CUBIC similar to Reno:

(11)
$cwnd\leftarrow{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times\beta\times
cwnd$

1:procedure Initialize:($total\\_bytes$, $INIT\\_COMM\\_GAP$)

2:$\triangleright$ Input Parameter $total\\_bytes$: Total bytes per iteration

3:$\triangleright$ Input Parameter $INIT\\_COMM\\_GAP$: Minimum gap in
communication for detecting iteration boundary

4: $bytes\\_ratio=0$ $\triangleright$ Current fraction of bytes sent in this
iteration

5: $bytes\\_sent=0$ $\triangleright$ Number of successfully sent bytes

6: $prev\\_ack\\_tstamp=0$ $\triangleright$ Timestamp of the previous ack

7: $iter\\_gap=INIT\\_COMM\\_GAP$ $\triangleright$ Estimated iteration gap

8: $\triangleright$ Maximum communication gap observed in the iteration

9: $max\\_gap=INIT\\_COMM\\_GAP$

10: $g=0.75$; $\triangleright$ Noise tolerance in detecting iteration gap

11: $\gamma=0.5$; $\triangleright$ EWMA factor

12: $MTU=1500$ $\triangleright$ Maximum packet size used by the system

13:procedure Update_MLTCP_params($num\\_acks$)

14:$\triangleright$ Input Parameter $num\\_acks$

15: $bytes\\_sent$ += $num\\_acks\times MTU$

16: $curr\\_ack\\_tstamp=\textsc{get\\_real\\_time}($)

17: $curr\\_gap=curr\\_ack\\_tstamp-prev\\_ack\\_tstamp$

18: $max\\_gap=\textsc{MAX}(max\\_gap,curr\\_gap)$

19: if $curr\\_gap$ ¿ ($g\times iter\\_gap)$ then

20: $\triangleright$ Start of new training iteration

21: $\triangleright$ Updating estimate for $iter\\_gap$

22: $iter\\_gap=(1-\gamma)\ iter\\_gap+(\gamma)\ max\\_gap$

23: $\triangleright$ MLTCP state reset

24: $bytes\\_ratio=0$; $bytes\\_sent$ = 0;

25: $max\\_gap=INIT\\_COMM\\_GAP$

26: else

27: $\triangleright$ Middle of training iteration

28: $bytes\\_ratio=min\left(1,\,\,\frac{bytes\\_sent}{total\\_bytes}\right)$

29: $prev\\_ack\\_tstamp=curr\\_ack\\_tstamp$

30: return

Algorithm 1 Updating MLTCP Parameters

DCQCN rate increase. DCQCN is a rate-based congestion control algorithm
commonly used in lossless RoCE networks (Zhu et al., 2015a). It periodically
increases the target rate by fixed steps $R_{AI}$ during the additive increase
stage:

(12) $target\\_rate\leftarrow target\\_rate+R_{AI}$

MLTCP updates the above equation to:

(13) $target\\_rate\leftarrow
target\\_rate+{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times
R_{AI}$

DCQCN rate decrease. DCQCN uses ECN-marked packets to detect congestion. Upon
the receipt of a Congestion Notification Packet (CNP) from the receiver, the
sender reduces its current rate using a multiplicative decrease factor:

(14) $curr\\_rate\leftarrow(1-\cfrac{\alpha}{2})\times curr\\_rate$

where $\alpha$ is the rate reduction factor of the DCQCN algorithm. MLTCP
updates the above equation to:

(15)
$curr\\_rate\leftarrow{\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}{\mathcal{F}(bytes\\_ratio)}}\times(1-\cfrac{\alpha}{2})\times
curr\\_rate$

MLTCP does not make any changes to any other parts of the congestion control
algorithm (e.g., slow start, initial congestion window). Initially, we assumed
that MLTCP must update both increase and decrease phases of congestion control
algorithms to achieve interleaving. However, our evaluations show that
updating only the increase or decrease phase is enough to achieve
interleaving, and they both have similar gains (§4.8).

###  3.5. Updating MLTCP Parameters

An essential detail in MLTCP is correctly updating
$bytes\\_ratio=\frac{bytes\\_sent}{total\\_bytes}$ during each iteration.
There are many ways to achieve this goal. This section describes our approach.

Updating bytes_sent. MLTCP uses the pluggable congestion module (plu, 2005;
Hemminger, 2005) to insert the Update_MLTCP_Params procedure, shown in
Algorithm 1, as a hook into the TCP stack. This function has two essential
goals: the first is to update the number of successfully sent bytes based on
the acked packets, and the second is to identify iteration boundaries to reset
$bytes\\_sent$ at the start of each training iteration. Update_MLTCP_Params is
called by the TCP stack whenever an ack packet is received. It uses the number
of acks to estimate the bytes that the sender sent successfully (line 15). For
training jobs with a simple on-off communication pattern (i.e., data parallel
jobs), detecting the start of the communication phase is relatively
straightforward: wait for a gap in the ack arrivals that is longer than a
typical RTT. However, to support training jobs with more complex communication
patterns with multiple peaks appearing during the same communication phase
(i.e., pipeline/tensor/model parallel jobs), we use a heuristic that estimates
the gaps between the communication phase of different iterations, called
$iter\\_gap$ (line 7). Our heuristic begins by keeping track of the current
gap and the maximum gap between subsequent acks (lines 17 and 18,
respectively). If the current gap is within an acceptable threshold of
$iter\\_gap$, it considers the current ack as the start of a new iteration’s
communication phase (line 19), updates $iter\\_gap$ using an exponentially
weighted moving average (EWMA) function (line 22), and resets MLTCP’s
parameters (lines 24–25). Otherwise, it computes $bytes\\_ratio$ based on the
current $bytes\\_sent$ (line 28).

Obtaining total_bytes. The total bytes in each iteration ($total\\_bytes$) is
a constant variable that depends on the size of the DNN model, the
parallelization strategy, and the communication collective. In our
implementation, we pre-calculate this number and store it in a file, but an
optimization of our implementation would be to automatically learn this value
by measuring the total amount of data during the first few iterations. We
leave this optimization to future work.

##  4\. Evaluation

In this section, we first describe our evaluation methodology and
implementation details of MLTCP (§4.1). Then, we benchmark MLTCP’s convergence
(§4.2) and report its training iteration speedup (§4.3). We then study its
performance with different DNN models and parallelization strategies (§4.4).
Next, we evaluate MLTCP’s robustness in the presence of stragglers (§4.5),
partially compatible jobs (§4.6), and circular dependency between jobs and
links (§4.7). Finally, we provide a set of guidelines for MLTCP’s parameters
(§4.8).

Figure 6. Our experiment topologies.

###  4.1. Evaluation Methodology

Setup. We implement MLTCP as a Linux kernel module to demonstrate its
convergence and gains in real-world settings. Our prototype includes 12 ASUS
ESC4000A-E10 servers connected via a 64 port Tofino switch. Each server has
one A100 Nvidia GPU (a10, 2021) 40 GB of HBM2 memory and one 50 Gbps Mellanox
ConnectX5 NIC. We experiment with both TCP and RoCEv2. For RoCE experiments,
we enable DCB (dcb, 2021) and PFC to support a lossless fabric for RDMA. The
servers run Ubuntu 20.04 LTS. Our experiments use PyTorch versions 1.13 and
1.8, CUDA version 11.7, and NCCL version 2.14.3.

Topologies. We use two topologies in our experiments. First, a single-
bottleneck dumbbell topology, shown in Figure 6(a), where servers are grouped
into six pairs: $\\{S_{1},S_{2}\\}$, $\\{S_{3},S_{4}\\}$, $\\{S_{5},S_{6}\\}$,
$\\{S_{7},S_{8}\\}$, $\\{S_{9},S_{10}\\}$, $\\{S_{11},S_{12}\\}$. Each pair
corresponds to a different DNN training job and communicates gradient updates
via the bottleneck link. We change the number of active server pairs
throughout the experiments. Second, we use a hierarchical topology, shown in
Figure 6(b), to capture the impact of multiple bottlenecks in the network.

DDN workloads. We use seven real-world DNN models: VGG16 (Dash, [n. d.]),
WideResNet101 (Zagoruyko and Komodakis, 2016), RoBERTa (Liu et al., 2019),
CamemBERT (Martin et al., 2019), GPT-1 (Radford et al., [n. d.]), GPT-2
(Radford et al., 2018), GPT-3 (Brown et al., 2020). Table 1 summarizes the
models and batch sizes used in our experiments. For clarity of presentation,
we start all jobs at the same time. Our takeaways hold when jobs start at
random times as well.

Compared schemes. We implement the following schemes in our testbed.

  * •

MLTCP-Reno. We experiment with two variants of MLTCP-Reno: MLTCP-Reno-WI and
MLTCP-Reno-MD, depending on whether we use MLTCP’s bandwidth aggressiveness
function during the window increase or multiplicative decrease phases of the
Reno algorithm. The default MLTCP-Reno corresponds to the WI version in all
our experiments. The default Reno protocol cannot saturate 50 Gbps bandwidth
due to kernel overheads and lack of robust zerocopy functional APIs (Belay et
al., 2014; Cai et al., 2021). To overcome this, we create eight sockets per
job on each server to parallelize the transmissions. In our experiments, we
aggregate statistics across all sockets that belong to the same job. Following
prior work (Chen et al., 2009) we set the RTO minimum to 1 $ms$. Unless
otherwise specified, we use $S=1.75$ and $I=0.25$ for MLTCP-Reno-WI and $S=1$
and $I=0.5$ for MLTCP-Reno-MD.

DNN |  Batch size/GPU |  Parallelization strategy |  Type
---|---|---|---
VGG16 (Dash, [n. d.]) |  1400 |  Data Parallel |  Vision
WideResNet101 (Zagoruyko and Komodakis, 2016) |  800 |  Data Parallel |  Vision
RoBERTa (Liu et al., 2019) |  28 |  Data Parallel |  Language
CamemBERT (Martin et al., 2019) |  28 |  Data Parallel |  Language
GPT-1 (Radford et al., [n. d.]) |  31 |  Data Parallel |  Language
GPT-2 (Radford et al., 2018) |  5-44 |  Data Parallel |  Language
GPT-3 (Brown et al., 2020) |  3 |  Hybrid Data/Pipeline/Model Parallel |  Language
Table 1. DNN models used in our experiments. Figure 7. Reno and MLTCP-Reno
performance on convergence benchmarks. Figure 8. CUBIC and MLTCP-CUBIC
performance on convergence benchmarks.

  * •

MLTCP-CUBIC. Similar to Reno, we experiment with two variants of MLTCP-CUBIC:
MLTCP-CUBIC-WI and MLTCP-CUBIC-MD, depending on whether we use MLTCP’s
bandwidth aggressiveness function during the window increase or multiplicative
decrease phases of the CUBIC algorithm. The default MLTCP-CUBIC corresponds to
the WI version in all our experiments. For CUBIC, we create four parallel
sockets to saturate the link capacity. We also set the RTO minimum to 1 $ms$
and scale the bic_scale parameter by 1010 to enable CUBIC to work on smaller
RTT scales in our testbed. We use $S=1.0$ and $I=0.5$ for MLTCP-CUBIC-WI and
$S=0.8$ and $I=0.8$ for MLTCP-CUBIC-MD.

  * •

MLQCN. Since the DCQCN algorithm is implemented inside Mellanox NICs, we do
not directly change the NIC implementation. The NICs API exposes a parameter
called $rp\\_ai\\_rate$ corresponding to DCQCN’s $R_{AI}$ rate. We use a C++
program that continuously monitors the amount of data sent by reading the
NIC’s hardware counter registers and updating $rp\\_ai\\_rate$ based on
MLQCN’s algorithm. MLQCN uses $I=0.267$ and $S=1.067$.

  * •

Cassini. We use the Cassini (Rajasekaran et al., 2024) scheduler as one of our
baselines to evaluate MLTCP’s robustness. Cassini uses a centralized scheduler
to determine a series of time-shift values to adjust the start time of jobs to
achieve interleaving. It takes each job’s expected optimal iteration time and
uses an end-host agent to enforce the time-shifts on each server.

  * •

Static. We use the approach proposed in (Rajasekaran et al., 2022) as another
baseline by manually setting the congestion window parameters such that the
bandwidth share between the competing jobs is statically configured to be
unfair. We refer to this approach as Static in our evaluations.

###  4.2. Convergence Benchmarks

We begin our evaluations by demonstrating MLTCP’s convergence when two GPT-2
training jobs share our dumbbell topology (Figure 6). Each job uses data-
parallel training on the Book Corpus dataset (Zhu et al., 2015b). We train
each job for 1000 iterations ($\approx$15 minutes) and compare link
utilization, number of dropped (or ECN-marked) packets, and Cumulative
Distribution Function (CDF) for training iteration times of MLTCP-Reno, MLTCP-
CUBIC, and MLQCN with the default Reno, CUBIC, and DCQCN algorithms,
respectively.

Figure 9. DCQCN and MLQCN performance on convergence benchmarks. Figure 10.
MLTCP’s speedup as the number of jobs competing for bandwidth increases.

MLTCP-Reno. Figure 7(a) shows that MLTCP-Reno automatically interleaves the
communication pattern of the two jobs after just six iterations. This
interleaving state remains persistent for the entire duration of the
experiment. In contrast, the default Reno algorithm continues to experience
congestion and packet loss. Figure 7(b) plots the number of packet drops per
second for the first 200 seconds of the experiment and shows that, on average,
MLTCP-Reno reduces the packet loss rate by 3.08$\times$. Consequently, MLTCP
accelerates the average and 99th percentile training iteration time of both
jobs by $1.1\times$ and $1.18\times$, respectively, as shown in Figure 7(c).
Note that the two MLTCP-Reno curves overlap on top of each other.

MLTCP-CUBIC. Despite CUBIC being a different congestion control algorithm than
Reno, we observe a similar behavior when we augment CUBIC with MLTCP. Figure
8(a) shows that MLTCP-CUBIC interleaves the training jobs after ten
iterations. Figure 8(b) demonstrates that MLTCP-CUBIC reduces the packet loss
rate by a factor of 2.25$\times$ compared to default CUBIC and Figure 8(c)
shows that MLTCP-CUBIC accelerates the average and 99th percentile training
iteration time of both jobs by $1.2\times$ and $1.23\times$, respectively.
Similar to MLTCP-Reno, the two MLTCP-CUBIC curves overlap.

MLQCN. The previous two experiments show MLTCP’s performance with the
traditional TCP/IP stack. With RoCE, MLTCP achieves even better results, as
shown in Figure 9: MLQCN jobs converge within eight iterations, experience
14.59$\times$ less ECN marks, and improve the average and 99th percentile
iteration times by 1.34$\times$ by 1.47$\times$, respectively. Note that in
Figure 9(c), the training iteration times of both jobs overlap.

Competing jobs (batch size) |  Num. servers |  Parallel-ism |  Comp-atibility
---|---|---|---
WideResNet101 (800) |  5 |  data |  0.88
VGG16 (1400) |  5 |  data |
CamemBERT (28) |  5 |  data |  0.9
RoBERTa (28) |  5 |  data |
GPT-1 (31) |  4 |  data |  1
GPT-1 (31) |  4 |  data |
GPT-2 (15) |  4 |  data |
GPT-3 (3) |  4 |  hybrid |  1
GPT-3 (3) |  4 |  hybrid |
Figure 11. Snapshots of jobs competing for bandwidth.

Figure 12. Impact of model diversity.

Figure 13. Impact of stragglers.

Figure 14. MLQCN’s robustness to partially compatible jobs compared with prior
work.

###  4.3. Training Iteration Time Speedup

To quantify MLTCP’s training iteration time speedup, we repeat the experiments
in the previous section while increasing the number of jobs that compete on
the bottleneck link. For each MLTCP variant, we compute its training iteration
time speedup as the ratio of the training iteration time using MLTCP over the
training iteration time of its congestion control counterpart. Figure 10 shows
the average and 99th percentile training iteration time speedups with MLTCP-
Reno plateau at 1.3$\times$ and 1.5$\times$, respectively. In contrast, as the
number of jobs increases, MLQCN’s average and tail speedups increase to
2$\times$ and 4$\times$, respectively. This is because the default DCQCN
algorithm suffers from extensive pause frames during congestion periods, which
further elongates the training iteration times. As shown in Figure 9, MLQCN
dramatically reduces the number of ECN-marked packets compared to DCQCN,
resulting in significant gains.

###  4.4. Impact of DNN Model & Parallelization Strategy Diversity

We evaluate MLTCP’s performance when different DNN models with different
parallelization strategies, number of servers, and job placement compete in a
hierarchical topology. To do so, we follow Cassini’s snapshot trace and
generate different combinations of several DNN training jobs sharing our clos
topology (Figure 6(b)). We make job placement decisions based on the Cassini
scheduler, which tries to place compatible jobs on network links but does not
always achieve this goal. Table 13 lists the DNN model names and details of
each snapshot in this experiment.

Figure 13 plots the CDF of the iteration times of all our snapshot
configurations. For each snapshot, we train each job for 500 iterations and
collect the training iteration times with MLQCN and DCQCN as the congestion
control algorithm. The Ideal curve in the plot captures the iteration times of
the same jobs when run in isolation in the cluster with the same number of
workers. There are three takeaways from this experiment. First, compared to
DCQCN, MLQCN improves the average and 99th percentile training iteration times
by 1.33$\times$ and 1.57$\times$, respectively. Second, MLCQN’s average and
tail iteration times are within $5\%$ and $4\%$ of the Ideal case. Third,
MLQCN achieves interleaving across various DNN models, parallelization
strategies, number of workers, job placement, and compatibility scores.

###  4.5. Impact of Stragglers

To demonstrate MLTCP’s robustness to stragglers, we train two GPT-2 jobs
sharing a bottleneck link and synthetically insert sleep commands in each
server’s training code based on a uniform straggling probability. If a server
decides to lag at a particular iteration, it sleeps for a period uniformly
randomly chosen between 5% and 10% of the isolation iteration time, where
isolation iteration time refers to the iteration time when the job is running
in isolation.

In the case of Cassini, frequent straggling events break the interleaving for
a few iterations, forcing the jobs to compete for bandwidth. Cassini’s end-
host agent tries to overcome this by forcing the straggling server to skip an
iteration, hoping that the noise in the system will be resolved in the next
iteration and the jobs will return to an interleaved state. In contrast,
MLQCN’s bandwidth adjustment function dynamically reacts to the straggling
server to maintain the interleaving. Figure 13 shows the training iteration
time speedups (normalized to default DCQCN) as the straggler probability
increases. There are two takeaways from this figure. First, MLQCN’s speedup
does not change dramatically in the presence of stragglers because both MLQCN
and DCQCN experience the same straggler event. As a result, the relative
speedup remains more or less constant. Second, Cassini’s tail iteration time
speedup drops sharply as the straggler probably goes beyond 10%.

###  4.6. Impact of Partially Compatible Jobs

We now study the impact of partial compatibility on MLTCP and our Static
baseline. We run two sets of experiments. First, we train two partially
compatible GPT-2 jobs sharing a bottleneck link and adjust the congestion
control parameters based on the suggestions in (Rajasekaran et al., 2022).
Figure 14(a) shows that when we use the Static approach, the communication
phases of the jobs constantly oscillate between interleaved, partially
interleaved, and not interleaved states. Since these partially interleaved and
not interleaved states repeat throughout the training, Job2’s training time
becomes worse than the default DCQCN, as shown in Figure 14(b). This defeats
the purpose of using Static’s unfair congestion control to accelerate both
jobs. In contrast, we find that MLQCN is robust to partial compatibility and
is able to partially interleave the jobs throughout the entire training.

In the second experiment, we increase to three GPT-2 training jobs sharing a
bottleneck link and vary their batch sizes to sweep the training iteration
time speedups across different compatibility scores between the jobs. Figure
14(c) compares the average and 99th percentile training iteration time
speedups of MLQCN and Static approaches (normalized by the training time of
DCQCN). There are three takeaways from this figure. First, as the
compatibility score increases, the speedup of both MLQCN and Static increases.
This is because the more compatible jobs are, the better their communication
patterns interleave. Second, MLQCN outperforms 99th percentile tail iteration
times of Static, even when the jobs are fully compatible. MLQCN uses its
aggressiveness function to dynamically adjust the sending rate, making it
tolerant to small noises in the system, whereas the Static approach is unable
to react to the network’s conditions. Third, when the compatibility score is
below 0.7, the Static approach performs extremely poorly where the 99th
percentile training iteration times are lower than the default DCQCN approach
(training iteration time speedup is lower than 1).

Figure 15. MLTCP’s robustness to circular dependency across jobs and links.

###  4.7. Circular Dependency Across Links

To evaluate MLTCP’s robustness to circular dependencies between jobs and
links, we experiment with the topology shown in Figure 2 using MLQCN as the
congestion control algorithm while training three GPT-2 models. There are
three congestion bottleneck links in this topology. Each job experiences two
congestion points and competes with a different job at each point. Figure 15
shows that the communication patterns on each link $l_{1}$, $l_{2}$, and
$l_{3}$ are interleaved such that jobs competing for bandwidth on all links
occupy the maximum possible link utilization. In contrast, Cassini cannot find
the correct schedule to interleave these jobs and Static cannot determine an
unfair bandwidth sharing between these jobs on different links.

###  4.8. Guidelines for Choosing Parameters

Aggressiveness function. To evaluate the impact of different bandwidth
aggressiveness functions $\mathcal{F}$ on MLTCP’s performance, we experiment
with three GPT-2 training jobs using MLTCP-Reno with six functions
$\mathcal{F}_{1},...,\mathcal{F}_{6}$:

  * •

$\mathcal{F}_{1}=1.75(bytes\\_ratio)+0.25$

  * •

$\mathcal{F}_{2}=1.75(bytes\\_ratio)^{2}+0.25$

  * •

$\mathcal{F}_{3}={1}/({-3.5(bytes\\_ratio)+4})$

  * •

$\mathcal{F}_{4}=-1.75(bytes\\_ratio)^{2}+3.5(bytes\\_ratio)+0.25$

  * •

$\mathcal{F}_{5}=-1.75(bytes\\_ratio)+2$

  * •

$\mathcal{F}_{6}=-1.75(bytes\\_ratio)^{2}+2$

All these functions have the same range (0.25 – 2) but
$\mathcal{F}_{1},...,\mathcal{F}_{4}$ are increasing and $\mathcal{F}_{5}$ and
$\mathcal{F}_{6}$ are decreasing. Figure 18 shows the average training
iteration time of different iterations as the jobs start their training
process. As shown, the iteration time of MLTCP-Reno with
$\mathcal{F}_{1},...,\mathcal{F}_{4}$ starts to decrease after $\approx$10
iterations, whereas the iteration times of MLTCP-Reno with $\mathcal{F}_{5}$
and $\mathcal{F}_{6}$ do not improve. In particular, even though different
increasing functions take slightly different numbers of iterations to
interleave the jobs, they eventually speed up the training iteration times as
long as the function increases.

Slope and Intercept ($S$ and $I$). Next, to study the impact of slope and
intercept parameters on $\mathcal{F}$, we sweep a series of possible
parameters and measure the average and 99th percentile training iteration time
speedups of MLTCP-Reno over default Reno, shown in Figure 18. The color in the
heatmap represents the speedup value. The heatmaps show that MLTCP-Reno is
able to achieve speedups for a wide range of slope and intercept parameters.
The star marks the set of parameters with the largest speedup in this
experiment.

MLTCP on window increase or multiplicative decrease? Thus far, our evaluations
augment the congestion window (or rate) increase phases of Reno, CUBIC, and
DCQCN. MLTCP’s bandwidth aggressiveness function can be applied to both the
increase and decrease phases. Figure 18 compares the training iteration time
for two jobs running GPT-2 model and sharing a single bottleneck link in a
dumbbell topology for MLTCP-Reno-WI, MLTCP-Reno-MD, MLTCP-CUBIC-WI and MLTCP-
CUBIC-MD. We find that the WI or MD variants of MLTCP-Reno and MLTCP-CUBIC
perform similarly for both average and tail iteration times.

Figure 16. Impact of different bandwidth aggressiveness functions on training
iteration time.

Figure 17. Heatmap of MLTCP-Reno’s speedup with different slopes and
intercepts.

Figure 18. Window increase vs. multiplicative decrease.

##  5\. Discussion

Complex communication patterns. Algorithm 1 uses a heuristic of detecting the
longest gap in communication peaks to identify iteration boundaries. MLTCP’s
mechanism is not tied to detecting the exact iteration boundary of the
training model; in fact, MLTCP’s approach works well even if the parameter
update happens in the middle of a training iteration, as long as its close to
a large compute gap and the the parameter updates take place consistently at
every training iteration.

Fairness goals. Traditionally, congestion control algorithms converge to fair
bandwidth sharing among competing flows. With MLTCP, flows may start with
unfair bandwidth sharing and slowly converge to an interleaved state. But
MLTCP is not an unfair congestion control algorithm for the entire duration of
a training job. It shifts the granularity of fairness from sub-RTT time scales
into training iteration (many RTTs) time scales.

Priority vs. favoritism. We intentionally avoid calling MLTCP’s favoritism
policy as a priority function to avoid confusion with conventional packet
priority concept. Packet priorities require switch support, and have limited
priority levels. In contrast, MLTCP’s favoritism is a completely distributed
function implemented in congestion control with fine-grained granularity
through adjusting the congestion window (or sending rate).

Interactions between MLTCP’s flows and legacy flows. In our implementation, we
enable MLTCP in the congestion control parameter of NCCL’s FAST socket plugin
(ncc, [n. d.]). Hence, only DNN flows using the NCCL library change cwnd based
on MLTCP, and non-DNN training flows use the default congestion algorithm.
Since MLTCP changes the aggressiveness of the congestion control, DNN jobs
running MLTCP could potentially be unfair towards non-DNN flows. But MLTCP’s
technique can be used to address this challenge by assigning a large
$\mathcal{F}$ function (larger than DNN flows) to latency-sensitive legacy
flows. As a result, flows at higher priorities can be safeguarded from the DNN
traffic.

##  6\. Related Work

Our work builds on several lines of related research.

Congestion control. There is vast literature on congestion control. Many rely
on feedback signals indicating congestion in the network, and reduce their
sending cwnd or rate. Such protocols include drop-based (Jacobson, 1988b; Ha
et al., 2008; Brakmo and Peterson, 1995; Xu et al., 2004; Gerla et al., 2001),
delay-based (Kumar et al., 2020; Mittal et al., 2015; Lee et al., 2017; Liu et
al., 2008; Baiocchi et al., [n. d.]), and ECN-based (Alizadeh et al., 2010;
Zhu et al., 2015a). We adapt TCP Reno because it is the canonical congestion
control algorithm, and CUBIC because it is the default in Linux. Further, we
augment DCQCN because it is commonly used in machine learning training today.
We believe that MLTCP can augment feedback-based protocols without significant
change. Other types of congestion control include deadline-aware (Wilson et
al., 2011; Vamanan et al., 2012), router-based (Katabi et al., 2002; Dukkipati
et al., 2005; Taheri et al., 2020), and receiver-based (Handley et al., 2017;
Montazeri et al., 2018). For deadline-based, inter-job interleaving could be
implemented by adjusting the deadlines of training jobs competing on the same
link. However, this requires redefining the concept of deadlines before flows
converge to an interleaved state. Router-assisted approaches could also
implement inter-job interleaving by enabling the router to determine the
unfairness, based on when the flows started. Similarly, receiver-based
protocols may delegate unfairness control to the receiver, which knows exactly
how many bytes have been sent. We leave the implementation and analysis of
these algorithms to future work.

Flow scheduling. There has been many workload-specific dynamic flow scheduling
techniques designed for optimizing datacenter networks. Many of these focus on
efficient utilization of network resources, achieving high bisection
bandwidths (Al-Fares et al., 2010; Chowdhury and Stoica, 2015; Chowdhury et
al., 2014; Jajoo et al., 2019; Perry et al., 2014) and minimizing the flow
completion times (Hong et al., 2012; Alizadeh et al., 2013; Bai et al., 2017).
These proposals tend to assume the scheduler has extensive knowledge about
active flows in the network. In contrast, MLTCP achieves interleaving in a
distributed manner without any information about flow arrivals/departures.

DNN job placement schedulers. Prior work demonstrated that generic flow
schedulers are not optimal for DNN training jobs (Pan et al., 2022; Mai et
al., 2015). As a result, there has been much work in creating schedulers for
DNN training jobs. Many focused on compute-optimization in how they assign
workers to jobs, and only considered the network so far as to try and schedule
workers for a job close together (Xiao et al., 2018; Mahajan et al., 2020;
Qiao et al., 2021; Gu et al., 2019; Zheng et al., 2023; Peng et al., 2018;
Jayaram Subramanya et al., 2023). MLTCP complements these schedulers by taking
advantage of incidental compatibility between jobs and interleaving them.

Network-aware DNN schedulers. There are also schedulers for the network and
communication strategy of the jobs. TACCL (Shah et al., 2023) and BytePS
(Jiang et al., 2020b) develop frameworks for optimizing the collective
communication algorithm. TicTac (Hashemi et al., 2019) optimally interleaves a
job’s computation with its own communication. ByteScheduler (Peng et al.,
2019) optimizes the partitioning and order of tensor transmissions. Syndicate
(Mahajan et al., 2023) proposes breaking down communication to smaller pieces
to perform joint optimization of scheduling and execution plan of collective
communications. All of these approaches focus on optimizing a single job,
while MLTCP further enhances these strategies by optimizing the communication
interleaving over all jobs sharing links.

Multi-resource sharing. Muri (Zhao et al., 2022) introduced the idea of multi-
resource interleaving for DNN training. It interleaves a variety of resources
together to optimize cluster utilization. However, Muri requires all jobs to
share the same set of GPUs. In practice, GPU-resource sharing is not possible
in large DNN training cluster. In contrast, MLTCP only considers communication
interleaving and is applicable to all jobs in the cluster regardless of
computation resources being used.

Benefits of unfairness in the network. Recently, there have been calls to
consider new notions of fairness in the network. Zaplatel et al. (Zaplatel and
Kuipers, 2023) argue that flow completion time rather than bandwidth fairness
should be the metric for fairness. Arslan et al. (Arslan et al., 2023) follow
a similar line of reasoning to conclude that unfairness can also lead to
higher energy efficiency.

##  7\. Conclusion

MLTCP is a technique to augment a family of congestion control algorithms for
shared DNN training clusters. The key idea in MLTCP is to reinforce a shortest
remaining processing time first policy by dynamically adjusting the congestion
window (or sending rate) of flows based on the number of successfully sent
bytes. We show that this straightforward change in congestion control
algorithms results in automatic inter-job communication interleaving and
accelerates the average and 99th percentile of training iteration times of
jobs by up to 2$\times$ and 4$\times$, respectively. This work does not raise
any ethical issues.

## References

  * (1)
  * ncc ([n. d.]) [n. d.].  NCCL Fast Socket.  ([n. d.]).  https://github.com/google/nccl-fastsocket
  * plu (2005) 2005\.  Pluggable congestion avoidance modules.  (2005).  https://lwn.net/Articles/128681/
  * dcb (2021) 2021\.  Data Center Bridging eXchange (DCBX).  (2021).  https://man7.org/linux/man-pages/man8/dcb-dcbx.8.html.
  * a10 (2021) 2021\.  NVIDIA A100 Tensor Core GPU.  (2021).  https://www.nvidia.com/en-us/data-center/a100/.
  * Al-Fares et al. (2010) Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, and Amin Vahdat. 2010.  Hedera: dynamic flow scheduling for data center networks. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI’10). USENIX Association, USA, 19.
  * Alistarh et al. (2017) Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017.  QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc., 1709–1720.  https://proceedings.neurips.cc/paper/2017/file/6c340f25839e6acdc73414517203f5f0-Paper.pdf
  * Alizadeh et al. (2010) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2010.  Data center TCP (DCTCP).  SIGCOMM Comput. Commun. Rev. 40, 4 (aug 2010), 63–74.  https://doi.org/10.1145/1851275.1851192
  * Alizadeh et al. (2013) Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013.  pFabric: minimal near-optimal datacenter transport. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM ’13). Association for Computing Machinery, New York, NY, USA, 435–446.  https://doi.org/10.1145/2486001.2486031
  * Ananthanarayanan et al. (2012) Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, and Ion Stoica. 2012.  Why let resources idle? Aggressive Cloning of Jobs with Dolly. In USENIX HotCloud (usenix hotcloud ed.).  https://www.microsoft.com/en-us/research/publication/let-resources-idle-aggressive-cloning-jobs-dolly/
  * Arslan et al. (2023) Serhat Arslan, Sundarajan Reneganathan, and Bruce Spang. 2023.  Green With Envy: Unfair Congestion Control Algorithms Can Be More Energy Efficient. In Proceedings of the 22nd ACM Workshop on Hot Topics in Networks (HotNets ’23). 8.
  * Bai et al. (2017) Wei Bai, Li Chen, Kai Chen, Dongsu Han, Chen Tian, and Hao Wang. 2017.  PIAS: Practical Information-Agnostic Flow Scheduling for Commodity Data Centers.  IEEE/ACM Trans. Netw. 25, 4 (aug 2017), 1954–1967.  https://doi.org/10.1109/TNET.2017.2669216
  * Baiocchi et al. ([n. d.]) Andrea Baiocchi, Angelo Castellani, and Francesco Vacirca. [n. d.].  YeAH-TCP: Yet another highspeed TCP.  ([n. d.]).
  * Belay et al. (2014) Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christos Kozyrakis, and Edouard Bugnion. 2014.  IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). USENIX Association, Broomfield, CO, 49–65.  https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay
  * Brakmo and Peterson (1995) L.S. Brakmo and L.L. Peterson. 1995.  TCP Vegas: end to end congestion avoidance on a global Internet.  IEEE Journal on Selected Areas in Communications 13, 8 (1995), 1465–1480.  https://doi.org/10.1109/49.464716
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  CoRR abs/2005.14165 (2020).  arXiv:2005.14165 https://arxiv.org/abs/2005.14165
  * Cai et al. (2021) Qizhe Cai, Shubham Chaudhary, Midhul Vuppalapati, Jaehyun Hwang, and Rachit Agarwal. 2021.  Understanding host network stack overheads. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (SIGCOMM ’21). Association for Computing Machinery, New York, NY, USA, 65–77.  https://doi.org/10.1145/3452296.3472888
  * Chen et al. (2009) Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, and Anthony D. Joseph. 2009.  Understanding TCP incast throughput collapse in datacenter networks. In Proceedings of the 1st ACM Workshop on Research on Enterprise Networking (WREN ’09). Association for Computing Machinery, New York, NY, USA, 73–82.  https://doi.org/10.1145/1592681.1592693
  * Cho et al. (2019) M. Cho, U. Finkler, M. Serrano, D. Kung, and H. Hunter. 2019.  BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy.  IBM Journal of Research and Development 63, 6 (2019), 1:1–1:11.  https://doi.org/10.1147/JRD.2019.2947013
  * Chowdhury and Stoica (2015) Mosharaf Chowdhury and Ion Stoica. 2015.  Efficient Coflow Scheduling Without Prior Knowledge.  SIGCOMM Comput. Commun. Rev. 45, 4 (aug 2015), 393–406.  https://doi.org/10.1145/2829988.2787480
  * Chowdhury et al. (2014) Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica. 2014.  Efficient coflow scheduling with Varys.  SIGCOMM Comput. Commun. Rev. 44, 4 (aug 2014), 443–454.  https://doi.org/10.1145/2740070.2626315
  * Cipar et al. (2013) James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Gregory R. Ganger, Garth Gibson, Kimberly Keeton, and Eric Xing. 2013.  Solving the Straggler Problem with Bounded Staleness. In 14th Workshop on Hot Topics in Operating Systems (HotOS XIV). USENIX Association, Santa Ana Pueblo, NM.  https://www.usenix.org/conference/hotos13/session/cipar
  * Cortez et al. (2017) Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. 2017.  Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP ’17). ACM, New York, NY, USA, 153–167.  https://doi.org/10.1145/3132747.3132772
  * Dash ([n. d.]) Abhipraya Kumar Dash. [n. d.].  VGG-16 Architecture.  ([n. d.]).  https://iq.opengenus.org/vgg16/.
  * Dukkipati et al. (2005) Nandita Dukkipati, Masayoshi Kobayashi, Rui Zhang-Shen, and Nick McKeown. 2005.  Processor Sharing Flows in the Internet. In Quality of Service – IWQoS 2005, Hermann de Meer and Nina Bhatti (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 271–285.
  * Gao et al. (2015) Peter X. Gao, Akshay Narayan, Gautam Kumar, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2015.  pHost: distributed near-optimal datacenter transport over commodity network fabric. In Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies (CoNEXT ’15). Association for Computing Machinery, New York, NY, USA, Article 1, 12 pages.  https://doi.org/10.1145/2716281.2836086
  * Gerla et al. (2001) M. Gerla, M.Y. Sanadidi, Ren Wang, A. Zanella, C. Casetti, and S. Mascolo. 2001.  TCP Westwood: congestion window control using bandwidth estimation. In GLOBECOM’01. IEEE Global Telecommunications Conference (Cat. No.01CH37270), Vol. 3. 1698–1702 vol.3.  https://doi.org/10.1109/GLOCOM.2001.965869
  * Gu et al. (2019) Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. 2019.  Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 485–500.  https://www.usenix.org/conference/nsdi19/presentation/gu
  * Ha et al. (2008) Sangtae Ha, Injong Rhee, and Lisong Xu. 2008.  CUBIC: a new TCP-friendly high-speed TCP variant.  SIGOPS Oper. Syst. Rev. 42, 5 (jul 2008), 64–74.  https://doi.org/10.1145/1400097.1400105
  * Handley et al. (2017) Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W. Moore, Gianni Antichi, and Marcin Wójcik. 2017.  Re-architecting datacenter networks and stacks for low latency and high performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM ’17). Association for Computing Machinery, New York, NY, USA, 29–42.  https://doi.org/10.1145/3098822.3098825
  * Harlap et al. (2016) Aaron Harlap, Henggang Cui, Wei Dai, Jinliang Wei, Gregory R. Ganger, Phillip B. Gibbons, Garth A. Gibson, and Eric P. Xing. 2016.  Addressing the Straggler Problem for Iterative Convergent Parallel ML. In Proceedings of the Seventh ACM Symposium on Cloud Computing (SoCC ’16). ACM, New York, NY, USA, 98–111.  https://doi.org/10.1145/2987550.2987554
  * Hashemi et al. (2019) Sayed Hadi Hashemi, Sangeetha Abdu Jyothi, and Roy Campbell. 2019.  TicTac: Accelerating Distributed Deep Learning with Communication Scheduling. In Proceedings of Machine Learning and Systems, A. Talwalkar, V. Smith, and M. Zaharia (Eds.), Vol. 1. 418–430.  https://proceedings.mlsys.org/paper/2019/file/84d9ee44e457ddef7f2c4f25dc8fa865-Paper.pdf
  * Hemminger (2005) Stephen Hemminger. 2005.  TCP infrastructure split out.  (2005).  http://lwn.net/Articles/128626/
  * Hong et al. (2012) Chi-Yao Hong, Matthew Caesar, and P. Brighten Godfrey. 2012.  Finishing flows quickly with preemptive scheduling.  SIGCOMM Comput. Commun. Rev. 42, 4 (aug 2012), 127–138.  https://doi.org/10.1145/2377677.2377710
  * Huang et al. (2019) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019.  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2019/file/093f65e080a295f8076b1c5722a46aa2-Paper.pdf
  * Jacobson (1988a) V. Jacobson. 1988a.  Congestion avoidance and control. In Symposium Proceedings on Communications Architectures and Protocols (SIGCOMM ’88). Association for Computing Machinery, New York, NY, USA, 314–329.  https://doi.org/10.1145/52324.52356
  * Jacobson (1988b) V. Jacobson. 1988b.  Congestion avoidance and control.  SIGCOMM Comput. Commun. Rev. 18, 4 (aug 1988), 314–329.  https://doi.org/10.1145/52325.52356
  * Jajoo et al. (2019) Akshay Jajoo, Y. Charlie Hu, and Xiaojun Lin. 2019.  Your Coflow has Many Flows: Sampling them for Fun and Speed. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, Renton, WA, 833–848.  https://www.usenix.org/conference/atc19/presentation/jajoo
  * Jayarajan et al. (2019) Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, and Gennady Pekhimenko. 2019.  Priority-based Parameter Propagation for Distributed DNN Training.  CoRR abs/1905.03960 (2019).  arXiv:1905.03960 http://arxiv.org/abs/1905.03960
  * Jayaram Subramanya et al. (2023) Suhas Jayaram Subramanya, Daiyaan Arfeen, Shouxu Lin, Aurick Qiao, Zhihao Jia, and Gregory R. Ganger. 2023.  Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 642–657.  https://doi.org/10.1145/3600006.3613175
  * Jiang et al. (2020a) Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020a.  A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). USENIX Association, USA, Article 26, 17 pages.
  * Jiang et al. (2020b) Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020b.  A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, 463–479.  https://www.usenix.org/conference/osdi20/presentation/jiang
  * Katabi et al. (2002) Dina Katabi, Mark Handley, and Charlie Rohrs. 2002.  Congestion control for high bandwidth-delay product networks.  SIGCOMM Comput. Commun. Rev. 32, 4 (aug 2002), 89–102.  https://doi.org/10.1145/964725.633035
  * Khani et al. (2021) Mehrdad Khani, Manya Ghobadi, Mohammad Alizadeh, Ziyi Zhu, Madeleine Glick, Keren Bergman, Amin Vahdat, Benjamin Klenk, and Eiman Ebrahimi. 2021.  SiP-ML: High-Bandwidth Optical Network Interconnects for Machine Learning Training. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (SIGCOMM ’21). Association for Computing Machinery, New York, NY, USA, 657–675.  https://doi.org/10.1145/3452296.3472900
  * Kumar et al. (2020) Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat. 2020.  Swift: Delay is Simple and Effective for Congestion Control in the Datacenter. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM ’20). Association for Computing Machinery, New York, NY, USA, 514–528.  https://doi.org/10.1145/3387514.3406591
  * Lee et al. (2017) Changhyun Lee, Chunjong Park, Keon Jang, Sue Moon, and Dongsu Han. 2017.  DX: Latency-Based Congestion Control for Datacenters.  IEEE/ACM Transactions on Networking 25, 1 (2017), 335–348.  https://doi.org/10.1109/TNET.2016.2587286
  * Li et al. (2021) Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021.  TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models.  (2021).  arXiv:cs.LG/2102.07988
  * Lin et al. (2017) Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2017.  Deep gradient compression: Reducing the communication bandwidth for distributed training.  arXiv preprint arXiv:1712.01887 (2017).
  * Liu et al. (2008) Shao Liu, Tamer Başar, and R. Srikant. 2008.  TCP-Illinois: A loss- and delay-based congestion control algorithm for high-speed networks.  Performance Evaluation 65, 6 (2008), 417–440.  https://doi.org/10.1016/j.peva.2007.12.007 Innovative Performance Evaluation Methodologies and Tools: Selected Papers from ValueTools 2006.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  (2019).  https://doi.org/10.48550/ARXIV.1907.11692
  * Mahajan et al. (2020) Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020.  Themis: Fair and Efficient GPU Cluster Scheduling. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 289–304.  https://www.usenix.org/conference/nsdi20/presentation/mahajan
  * Mahajan et al. (2023) Kshiteej Mahajan, Ching-Hsiang Chu, Srinivas Sridharan, and Aditya Akella. 2023.  Better Together: Jointly Optimizing ML Collective Scheduling and Execution Planning using SYNDICATE. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 809–824.  https://www.usenix.org/conference/nsdi23/presentation/mahajan
  * Mai et al. (2015) Luo Mai, Chuntao Hong, and Paolo Costa. 2015.  Optimizing Network Performance in Distributed Machine Learning. In 7th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 15). USENIX Association, Santa Clara, CA.  https://www.usenix.org/conference/hotcloud15/workshop-program/presentation/mai
  * Martin et al. (2019) Louis Martin, Benjamin Müller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019.  CamemBERT: a Tasty French Language Model.  CoRR abs/1911.03894 (2019).  arXiv:1911.03894 http://arxiv.org/abs/1911.03894
  * Mittal et al. (2015) Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. 2015.  TIMELY: RTT-based Congestion Control for the Datacenter.  SIGCOMM Comput. Commun. Rev. 45, 4 (aug 2015), 537–550.  https://doi.org/10.1145/2829988.2787510
  * Mohan et al. (2022) Jayashree Mohan, Amar Phanishayee, Janardhan Kulkarni, and Vijay Chidambaram. 2022.  Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 579–596.  https://www.usenix.org/conference/osdi22/presentation/mohan
  * Montazeri et al. (2018) Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018.  Homa: a receiver-driven low-latency transport protocol using network priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (SIGCOMM ’18). Association for Computing Machinery, New York, NY, USA, 221–235.  https://doi.org/10.1145/3230543.3230564
  * Mudigere et al. (2022) Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie (Amy) Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komuravelli, Ching-Hsiang Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng, Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, KR Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Ajit Mathews, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao. 2022.  Software-Hardware Co-Design for Fast and Scalable Training of Deep Learning Recommendation Models. In Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA ’22). Association for Computing Machinery, New York, NY, USA, 993–1011.  https://doi.org/10.1145/3470496.3533727
  * Narayanan et al. (2019) Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019.  PipeDream: Generalized Pipeline Parallelism for DNN Training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP’19). Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3341301.3359646
  * Pan et al. (2022) Rui Pan, Yiming Lei, Jialong Li, Zhiqiang Xie, Binhang Yuan, and Yiting Xia. 2022.  Efficient flow scheduling in distributed deep learning training with echelon formation. In Proceedings of the 21st ACM Workshop on Hot Topics in Networks (HotNets ’22). Association for Computing Machinery, New York, NY, USA, 93–100.  https://doi.org/10.1145/3563766.3564096
  * Peng et al. (2018) Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. 2018.  Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters. In Proceedings of the Thirteenth EuroSys Conference (EuroSys ’18). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages.  https://doi.org/10.1145/3190508.3190517
  * Peng et al. (2019) Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. 2019.  A Generic Communication Scheduler for Distributed DNN Training Acceleration. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 16–29.  https://doi.org/10.1145/3341301.3359642
  * Perry et al. (2014) Jonathan Perry, Amy Ousterhout, Hari Balakrishnan, Devavrat Shah, and Hans Fugal. 2014.  Fastpass: a centralized ”zero-queue” datacenter network. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM ’14). Association for Computing Machinery, New York, NY, USA, 307–318.  https://doi.org/10.1145/2619239.2626309
  * Qiao et al. (2021) Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021.  Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 1–18.  https://www.usenix.org/conference/osdi21/presentation/qiao
  * Radford et al. ([n. d.]) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. [n. d.].  Improving Language Understanding by Generative Pre-Training.  ([n. d.]).  https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
  * Radford et al. (2018) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018.  Language Models are Unsupervised Multitask Learners.  (2018).  https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf.
  * Rajasekaran et al. (2024) Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. 2024.  Cassini: Network-Aware Job Scheduling in Machine Learning Clusters.  To appear in Proceedings of 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (Apr. 2024).
  * Rajasekaran et al. (2022) Sudarsanan Rajasekaran, Manya Ghobadi, Gautam Kumar, and Aditya Akella. 2022.  Congestion Control in Machine Learning Clusters. In Proceedings of the 21st ACM Workshop on Hot Topics in Networks (HotNets ’22). 235–242.
  * Shah et al. (2023) Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023.  TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 593–612.  https://www.usenix.org/conference/nsdi23/presentation/shah
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  (2015).  arXiv:cs.CV/1409.1556
  * Taheri et al. (2020) Parvin Taheri, Danushka Menikkumbura, Erico Vanini, Sonia Fahmy, Patrick Eugster, and Tom Edsall. 2020.  RoCC: robust congestion control for RDMA. In Proceedings of the 16th International Conference on Emerging Networking EXperiments and Technologies (CoNEXT ’20). Association for Computing Machinery, New York, NY, USA, 17–30.  https://doi.org/10.1145/3386367.3431316
  * Tandon et al. (2017) Rashish Tandon, Qi Lei, Alexandros G. Dimakis, and Nikos Karampatziakis. 2017.  Gradient Coding: Avoiding Stragglers in Distributed Learning. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, Sydney, Australia, 3368–3376.  http://proceedings.mlr.press/v70/tandon17a.html
  * Ðukić et al. (2019) Vojislav Ðukić, Sangeetha Abdu Jyothi, Bojan Karlas, Muhsen Owaida, Ce Zhang, and Ankit Singla. 2019.  Is advance knowledge of flow sizes a plausible assumption?. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 565–580.  https://www.usenix.org/conference/nsdi19/presentation/dukic
  * Unger et al. (2022) Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. 2022.  Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 267–284.  https://www.usenix.org/conference/osdi22/presentation/unger
  * Vamanan et al. (2012) Balajee Vamanan, Jahangir Hasan, and T.N. Vijaykumar. 2012.  Deadline-aware datacenter tcp (D2TCP). In Proceedings of the ACM SIGCOMM 2012 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM ’12). Association for Computing Machinery, New York, NY, USA, 115–126.  https://doi.org/10.1145/2342356.2342388
  * Wang et al. (2023) Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023.  TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 739–767.  https://www.usenix.org/conference/nsdi23/presentation/wang-weiyang
  * Watcharapichat et al. (2016) Pijika Watcharapichat, Victoria Lopez Morales, Raul Castro Fernandez, and Peter Pietzuch. 2016.  Ako: Decentralised Deep Learning with Partial Gradient Exchange.  SoCC ’16 (2016).
  * Wilson et al. (2011) Christo Wilson, Hitesh Ballani, Thomas Karagiannis, and Ant Rowtron. 2011.  Better never than late: meeting deadlines in datacenter networks. In Proceedings of the ACM SIGCOMM 2011 Conference (SIGCOMM ’11). Association for Computing Machinery, New York, NY, USA, 50–61.  https://doi.org/10.1145/2018436.2018443
  * Xiao et al. (2018) Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018.  Gandiva: Introspective Cluster Scheduling for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 595–610.  https://www.usenix.org/conference/osdi18/presentation/xiao
  * Xu et al. (2004) Lisong Xu, K. Harfoush, and Injong Rhee. 2004.  Binary increase congestion control (BIC) for fast long-distance networks. In IEEE INFOCOM 2004, Vol. 4. 2514–2524 vol.4.  https://doi.org/10.1109/INFCOM.2004.1354672
  * Yang et al. (2022) Mingran Yang, Alex Baban, Valery Kugel, Jeff Libby, Scott Mackie, Swamy Sadashivaiah Renu Kananda, Chang-Hong Wu, and Manya Ghobadi. 2022.  Using trio: juniper networks’ programmable chipset - for emerging in-network applications. In Proceedings of the ACM SIGCOMM 2022 Conference (SIGCOMM ’22). Association for Computing Machinery, New York, NY, USA, 633–648.  https://doi.org/10.1145/3544216.3544262
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide Residual Networks.  (2016).  https://doi.org/10.48550/ARXIV.1605.07146
  * Zaharia et al. (2008) Matei Zaharia, Andy Konwinski, Anthony D. Joseph, Randy Katz, and Ion Stoica. 2008.  Improving MapReduce Performance in Heterogeneous Environments. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, Berkeley, CA, USA, 29–42.
  * Zaplatel and Kuipers (2023) Adrian Zaplatel and Fernando Kuipers. 2023.  Slowdown as a Metric for Congestion Control Fairness. In Proceedings of the 22nd ACM Workshop on Hot Topics in Networks (HotNets ’23). 8.
  * Zhao et al. (2022) Yihao Zhao, Yuanqiang Liu, Yanghua Peng, Yibo Zhu, Xuanzhe Liu, and Xin Jin. 2022.  Multi-Resource Interleaving for Deep Learning Training. In Proceedings of the ACM SIGCOMM 2022 Conference (SIGCOMM ’22). Association for Computing Machinery, New York, NY, USA, 428–440.  https://doi.org/10.1145/3544216.3544224
  * Zheng et al. (2023) Pengfei Zheng, Rui Pan, Tarannum Khan, Shivaram Venkataraman, and Aditya Akella. 2023.  Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 703–723.  https://www.usenix.org/conference/nsdi23/presentation/zheng
  * Zhu et al. (2015a) Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015a.  Congestion Control for Large-Scale RDMA Deployments. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (SIGCOMM ’15). Association for Computing Machinery, New York, NY, USA, 523–536.  https://doi.org/10.1145/2785956.2787484
  * Zhu et al. (2015b) Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. 2015b.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 19–27.  https://doi.org/10.1109/ICCV.2015.11