# Characterizing TCP’s Performance for Low-Priority Flows Inside a Cloud

Hafiz Mohsin Bashir
Tufts University  Abdullah Bin Faisal
Tufts University  Fahad R. Dogar
Tufts University

###### Abstract

Many cloud systems utilize low-priority flows to achieve various performance
objectives (e.g., low latency, high utilization), relying on TCP as their
preferred transport protocol. However, the suitability of TCP for such low-
priority flows – specifically, how _prioritization-induced_ delays in packet
transmission can cause spurious timeouts and low utilization – is relatively
unexplored. In this paper, we conduct an empirical study to investigate the
performance of TCP for low-priority flows under a wide range of realistic
scenarios: use-cases (with accompanying workloads) where the performance of
low-priority flows is crucial to the functioning of the overall system as well
as various network loads and other network parameters. Our findings yield two
key insights: 1) for several popular use-cases (e.g., network scheduling),
TCP’s performance for low-priority flows is within $2\times$ of a near-optimal
scheme, 2) for emerging workloads that exhibit an _on-off_ behavior in the
high priority queue (e.g., distributed ML model training), TCP’s performance
for low-priority flows is poor. Finally, we discuss and conduct preliminary
evaluation to show that two simple strategies – weighted fair queuing (WFQ)
and _cross-queue_ congestion notification – can substantially improve TCP’s
performance for low-priority flows.

##  1 Introduction

In this measurement paper, we investigate the interplay of two important
aspects of today’s cloud systems: (1) the widespread adoption of network
prioritization and (2) the prevalent use of TCP as the primary transport
protocol. Our specific focus is on understanding the suitability of TCP for
low-priority flows, and we frame our inquiry as: Is TCP good enough for low-
priority flows under a cloud setting?

Modern cloud systems host a multitude of applications with varying performance
objectives. To support these needs, there has been a widespread move towards
adopting network prioritization (i.e., using priority queues inside switches).
Network prioritization guarantees isolation across different types of
applications, enabling various use-cases such as network flow scheduling
(e.g., DAS [11], pFabric [6], Homa [52], Baraat [20]) and workload co-location
(e.g., perfIso [36], Heracles [48], SmartHarvest [69]). For example, EBB [19],
Meta’s backbone network, services several traffic classes in a strictly
prioritized manner. A new use-case of prioritization is in the context of
RDMA-based stacks where network operators can use the TCP/IP traffic as a
reliable and robust low-priority backup, with RDMA-based traffic as the higher
priority traffic in the system [25]. For many of the above use-cases, the
performance of low-priority flows – albeit less important than the high
priority traffic – is still important. For example, in systems like DAS [11]
which uses duplication to reduce tail latency, a lower priority flow may
determine the application performance if the corresponding high priority flow
becomes a straggler.

On the other hand, despite a large body of work highlighting the performance
implications of using TCP for latency-critical applications [33, 24, 6, 63,
5], it remains the defacto protocol in today’s cloud systems for important
reasons such as backward compatibility and stability [38, 23, 25, 70]. For
instance, a recent work from Alibaba [70] emphasized that their preference for
TCP stems from the lack of specialized in-network support and the presence of
a fleet of servers lacking support for emerging technologies (e.g., DPDK).

This motivates an investigation into potential issues that may arise when TCP
is used for low-priority flows under network prioritization. To the best of
our knowledge, there has been no prior research dedicated to investigating
this specific inquiry. Intuitively, the use of TCP for low-priority flows can
introduce a range of issues. This is primarily because low-priority flows are
susceptible to extended queuing delays, which, in the worst-case scenario, can
be unbounded [16]. These delays impact TCP’s feedback loop, potentially
resulting in unnecessary backoffs, and spurious retransmissions, where packets
are mistakenly assumed to be lost while they are stuck within network
switches.

Building on this intuition, our study evaluates the performance of TCP for
low-priority flows across various important use-cases, encompassing a wide
range of settings, including characteristics of high and low-priority traffic,
the load and workload profiles, and specific TCP and switch configurations
(e.g., RTOmin, buffer size, etc.). Our study has two primary objectives:
First, to shed light on the cases where TCP is a suitable choice for low-
priority flows and identify scenarios where it is not. Second, to explore
minor adjustments that can help address any limitations observed in TCP’s use
for low-priority flows. Our methodology involves conducting experiments on a
small-scale testbed and NS3 simulations [59]. We use a subset of experiments
to cross-validate the results on both these platforms and then use simulations
to explore a wide range of scenarios.

To quantify the impact of network prioritization and to uncover the room for
improving performance of low-priority flows beyond what TCP offers, we
consider a _near-optimal_ (Near-Opt) fair transport protocol as a baseline.
The Near-Opt protocol has the advantage of instantaneous access to network
state information, including available bandwidth, queue lengths, and packet
loss details.

Consequently, it represents an upper-bound that an efficient _fair_ transport
protocol can achieve for low-priority flows. Our key measurement insights are
as follows:

  * •

For several use-cases (e.g., network scheduling), TCP’s performance for low-
priority flows is within $2\times{}$ of Near-Opt. This is because, for such
use-cases, low-priority flows frequently get some fraction of network share
allowing TCP to continually adapt to the network conditions.

  * •

Low-priority TCP flows suffer the most — more than $12\times{}$ compared to
Near-Opt — when they coexist with high priority traffic that exhibits an _on-
off_ behavior (typical of distributed ML model training workloads [24, 30]).
The _on_ period can completely starve the low-priority queue, causing TCP’s
feedback loop to stall and experience severe performance degradation. During
the _off_ period, unnecessary congestion backoffs limit TCP from efficiently
utilizing the link bandwidth.

  * •

Emerging technology trends such as increased link speeds and relatively
smaller switch buffer sizes [9] significantly affect TCP’s performance.
Specifically when available buffer is equally divided between all the queues,
TCP’s performance is degraded by up to $5\times{}$ compared to when the entire
buffer is available for the low-priority queue.

  * •

Two simple strategies – weighted fair queuing (WFQ), and _cross-queue_
congestion notification – effectively alleviate stalls in TCP’s feedback loop,
resulting in up to 9$\times$ reduction in completion times for low-priority
flows.

The paper is organized as follows: We first present popular and emerging use-
cases of network prioritization and highlight their performance implications
for TCP when used for low-priority flows (§2). In §3, we measure TCP’s
performance for low-priority flows across popular use-cases. In §4, we
investigate the role of several configurations crucial to the performance of
TCP such as emerging technology trends (e.g., smaller buffer sizes) and
important network and transport parameters (e.g., RTOmin). Finally, in §5, we
discuss two simple strategies (and do the preliminary evaluation) to show
substantial improvement in TCP’s performance for low-priority flows.

##  2 Preliminaries and Motivation

We begin by highlighting important use-cases of network prioritization in
modern cloud systems, emphasizing on the importance of low-priority traffic
(§2.1). We then discuss the performance implications of using TCP for low-
priority flows (§2.2).

###  2.1 Common Use-Cases of Network Prioritization

#### Network Scheduling:

Network prioritization is a cornerstone in the design of network scheduling
schemes (e.g., flow/co-flow/task scheduling) [20, 66, 33, 11, 53, 8, 6, 63].
For example, PIAS [8] and pFabric [6] use priority queues at network switches
to minimize average flow completion times (FCTs). Similarly, duplication based
schemes [11, 26, 65] leverage low-priority flows to minimize tail latency. For
example, DAS [11] duplicates incoming request at a lower priority to mask
_stragglers_ 111Uncertain performance issues that commonly arise in large
scale systems. These are notoriously hard to predict or mitigate at run time
[55] that contribute significantly to the tail latency. Recently, a growing
body of research highlights the benefits of incorporating network
prioritization in the distributed training of machine learning models [24, 32,
57, 54]. This line of work further highlights the importance and relevance of
this particular use-case.

#### Workload Co-location:

Cloud systems, particularly those hosting user-facing applications such as web
search, are typically designed to handle peak loads. However, during the
periods of low demand, this over-provisioning results in under-utilization of
cloud resources (e.g., network, compute etc.,) [62, 58, 18, 10, 46]. To
improve the utilization of such systems and to reduce the total cost of
ownership (TCO), several proposals advocate to co-locate best-effort workloads
alongside latency-critical applications [48, 27, 46, 7, 36]. Co-location
enables best-effort workloads to harvest spare system resources. To ensure
that the performance of latency critical applications is not negatively
affected by the best-effort workloads, operators employ prioritization (i.e.,
giving latency critical workloads priority over best effort workloads) at
potential bottleneck resources (e.g., network). For example, _PerfIso_ [36]
leverages prioritization to safely scavenge spare system resources in
_Microsoft Bing’s_ cluster, enabling the operators to simultaneously maintain
low _response time_ for latency critical services and high utilization of the
overall system.

#### Hybrid RDMA/TCP-IP Services:

A key requirement in a modern cloud system is the high availability of
critical services (e.g., key-value stores) that provide support for many other
applications. The recent trend of cloud operators turning to RDMA [44, 71, 51,
25, 49] has raised concerns about the availability of these critical services
because of several outstanding challenges with the adoption of RDMA (e.g.,
pause frame storm) [40, 25, 56, 35]. An effective strategy is the joint-
deployment of RDMA and TCP such that TCP runs in the background at very low
load as a fail-safe option. Should RDMA experience an anomaly (e.g., PFC
storm), TCP can take over, ensuring high availability [25, 23, 45]. For
example, Gao et. al., integrated TCP and RDMA in Pangu [25] – a cloud storage
system – and provided support for fine grained switching between RDMA and TCP
to ensure high availability. To keep TCP from interfering with RDMA, TCP
traffic can be mapped to a lower priority queue at the switches.

Use-Case | Objective(s) |  | Example
---
High / Low-Priority
Proposal(s)
Scheduling |  | Policy Specific
---
(e.g., Improved FCTs)
| Small
---
Flows
| Long
---
Flows
| PIAS[8], Baraat[20],
---
DAS[11], pFabric[6]
| Workload
---
Co-location
| Increased
---
Utilization
| Latency
---
Critical
| Batched
---
Workload
| perfIso[36], Hercales[48],
---
Bistro[27], SmartHarvest[69]
| Hybrid
---
Services
| High
---
Availability
RDMA | TCP | Pangu[25]

Table 1:  Synthesizes important use-cases of network prioritization and
highlights that the performance of low-priority flows can be crucial in
achieving a certain objective in a cloud system.

#### Synthesis:

Table 1 summarizes the above use-cases, highlighting how network
prioritization helps achieve a diverse set of objectives by providing
isolation across various abstractions (i.e., flows, workloads, services).
Despite the pivotal role of low-priority flows in the success of these use-
cases, their performance is generally overlooked in the existing work. In
section §3, we show that TCP’s performance for low-priority flows varies
across each of these use-cases.

(a)

(b)

Figure 1: TCP assumes fair-scheduling inside the network. This assumption
fails under priority scheduling. (a) depicts TCP’s view of the network (i.e.,
all packets from low-priority (Blue) and high priority (Red) senders share the
same queue). (b) shows that in reality packets from different priority classes
are stored in their respective priority queue and they are serviced in the
order of their priority.

###  2.2 TCP Under Network Prioritization

Fair-share assumption of TCP: TCP’s congestion control assumes that data
packets receive a “fair” (non-discriminatory) treatment inside the network
(i.e., packets are assumed to be serviced in the order of their arrival). In
reality, under network prioritization, the priority of a packet can override
the service order. Figure 1 depicts that the difference between TCP’s assumed
view of the network (1(a)) and the actual view (1(b)) results in longer
queuing delays for low-priority packets – unbounded in the worst case [16].
For example, the low-priority packet (shown in _blue_) can only get the
service once all the high priority packets (shown in _red_) have been
serviced.

(a)

(b)

Figure 2: Shows substantial impact of priority queuing on TCP’s performance
for low-priority flows. (a) highlights a significant increase in the
retransmission rate experienced by low-priority flows under priority queues
compared to fairshare. (b) highlights the impact of priority queuing on flow
completion times (FCT) of low-priority flows between TCP and Near-Opt (§3).

#### Implications for low-priority Flows:

TCP’s congestion control relies on an end-to-end feedback loop (_ack-
clocking_) to estimate the available network bandwidth and to take congestion
control decisions. Under network prioritization, high priority flows can delay
the transmission of low-priority packets. This causes _long_ and highly
_variable_ stalls in TCP’s feedback loop which can be detrimental to the
performance of low-priority flows. For example, the increased queuing delay
can trigger timeouts which result in spurious retransmissions and unnecessary
congestion back-offs.

To quantify these observations, we conduct a small experiment on a 9 node
cluster. We co-locate a background storage workload with a latency-critical
workload. The latency-critical workload _emulates_ the network footprint of
distributed ML training [24, 30] and periodically generates a 8:1 incast that
exhibits an _on-off_ communication pattern (more details in §3.2). Both
applications share a common bottleneck link. We run this experiment with and
without network prioritization. In the former case, flows from the storage
workload are mapped to a lower priority queue compared to the flows from
latency-critical application, whereas in the latter case, flows from both the
applications share a single queue.

Figure 2(a) shows that the introduction of priority queues significantly
increases the retransmission rate for the long flows (§3). For example, at the
80th percentile (p80), the retransmission rate is approximately 8% when
network prioritization is enabled (PriorityQueue) while it is negligible in
the case of no prioritization (FairShare). This happens because of two main
reasons:

Spurious Timeouts: Prioritization results in an increased queuing delay for
low-priority flows. Consequently, TCP faces challenges in distinguishing
between _prioritization-induced_ (caused by network prioritization) and the
actual packet losses, which makes it susceptible to experiencing spurious
timeouts (i.e., an in-flight packet is mistakenly assumed to be lost). TCP’s
current approach of estimating the timeout value as a long term moving average
of round trip times proves limiting as it fails to adapt to _transient_ spikes
in the round trip time caused by large bursts of high priority packets.

Convergence issues: Bursty arrivals of high priority packets create large
fluctuation in the available bandwidth for low-priority flows which leads to
the following challenges:

  * •

_Overshoot:_ The presence of high priority traffic results in long feedback
delays for low-priority flows. This slows down TCP’s reaction to congestion
and leads to the buildup of low-priority queues, causing a large number of
packet losses.

  * •

_Undershoot:_ The absence of high priority traffic yields the full link
capacity to the low-priority flows. However, the congestion backoffs in the
previous epoch of low bandwidth may result in an under-utilization of the
network links.

To assess the impact of the above discussed issues on flow completion times,
we repeat the experiment and substitute TCP with a Near-Opt scheme which is
designed to estimate near-optimal flow completion times of low-priority flows
(§3). Figure 2(b) captures the flow completion times of low-priority long
flows. It clearly highlights the substantial opportunity to enhance TCP’s
performance.

While slow convergence is a well researched problem for TCP, the frequent
stalls in the feedback loop coupled with high fluctuation in the available
network bandwidth exacerbates this issue for low-priority flows.

#### Conclusion:

The above discussed use-cases underscore the importance of low-priority flows
in achieving a wide range of objectives in modern cloud systems (e.g.,
mitigating tail latency, ensuring high availability etc.,). TCP’s performance
for low-priority flows, however, can be adversely affected by challenges such
as spurious timeouts and convergence issues. This motivates the need to
systematically study TCP’s performance for low-priority flows under a diverse
set of conditions.

##  3 A Closer Look at TCP’s Performance for Low-Priority Flows

We now study TCP’s performance for low-priority flows under a broad spectrum
of use-cases (§2.2), load distribution across priority classes, choice of
transport protocol for high priority traffic, and workloads. Table 2
synthesizes various settings across high and low-priority classes (e.g., load,
flow size distribution) and highlights our key insights.

We conduct experiments on both a small scale testbed and the NS3 simulator.
The testbed experiments establish the credibility of the key insights of our
evaluation, under realistic settings, with a real end-host stack and a network
switch with prioritization support. It also helps us cross validate the
simulator setup, which we use to explore a wider range of network and
transport parameters. We present the testbed results in §3.4; all others
results are from our NS3 evaluation.

Use-Case |  | Load
---
Distribution
| Flow Size
---
Distribution
Key Insights
| Duplicate-Aware
---
Scheduling (DAS) [11]
Same | Same |  | Equal load across queues yields ample bandwidth for
---
low-priority flows to make progress. Consequently,
TCP does not suffer from issues discussed in 2
| Shortest-Job-First
---
(SJF)
Different | Different |  | The fraction of load induced by high priority flows determines
---
the impact of prioritization. The skewness in a workload’s flow
size distribution plays a crucial role: greater skewness results
in higher performance penalty.
| Workload Colocation
---
Different | Different |  |  _On-Off_ network behavior of high priority workload
---
severely affects low-priority flows. This is primarily due to
the rapid fluctuations between very high and very low load,
which leads to high retransmissions and frequent backoffs.
| Hybrid Services
---
Different | Same |  | The choice of high priority transport protocol has little impact.
---
Rather, the choice of workload is a dominant factor.

Table 2: Compares high and low priority traffic classes in our evaluation
scenarios across different axes and highlights key insights derived from each
use-case. For example, in the context of DAS, both the high and low priority
traffic experience the same load, and service the same workload (i.e., flow
size distribution).

#### Experimental Setup:

For the NS3 [59] evaluation, we use the following settings as default unless
specified otherwise. We assume a single cluster setup where 40 nodes are
interconnected via 10Gbps link to a single switch. We provision strict
priority queues within both the end hosts and the network, comprising a high
priority queue and a low-priority queue. The switch buffer is statically
shared among ports. According to the settings discussed in an earlier work
[9], each switch port is allocated a 192KB buffer size, which is further
partitioned in a 2:1 ratio between high and low-priority queues, respectively.
In the absence of any traffic between servers, the average round trip time
(RTT) is estimated to be 100us.

We configure TCP-Cubic as the default congestion control algorithm and enable
the selective acknowledgment (SACK) option. The initial congestion window is
set to 2 packets, and the minimum retransmission timeout value (RTOmin) is
configured to 1ms. Additionally, we allocate 1MB for the send and receive
buffers for TCP. To prevent priority-inversion within the driver queue, we
limit the driver queue to 100 packets. We observed maximum throughput for TCP
(when used for low-priority flows) under these settings. Finally, to generate
traffic between servers, we use the publicly available DAS setup [1] and
tailored it for our use-cases. We consider three load settings in our
evaluation, _low_ , _medium_ , and _high_ which refer to network loads of
$<20\%$, $20-40\%$ and $>40\%$ respectively. Similarly, we classify flows as
_small_ , _medium_ and _long_ which refer to flow sizes of $<50KB$,
$50KB-1MB$, and $>1MB$ respectively.

#### Baseline and Evaluation Criteria:

To quantify the impact of network prioritization and to uncover the room for
improvement for low-priority flows using TCP, we consider an RCP [21] style
near-optimal fair transport scheme (Near-Opt) for low-priority traffic as our
baseline. Near-Opt relies on two important pieces of information to provide a
strong baseline: _Loss Information (LI)_ and _Rate Information (RI)_ 222Note
that Near-Opt is _only_ used for comparison and may not be practically
realizable..

(a)

(b)

(c)

Figure 3: TCP’s performance for low-priority flows under network scheduling
scenario. (a) Shows that for DAS [11] workload, TCP’s performance for flows is
insensitive to the flow sizes. (b) and (c) shows TCP’s performance for long
flows under shortest job first policy (SJF) for the _web-search_ and _data-
mining_ workloads. Under SJF, we assign low-priority to long flows (size
greater than 1 MB).

  * •

Loss Information: Knowing whether a packet was lost in the network allows
Near-Opt to avoid spurious retransmissions. Near-Opt makes use of a global
data structure that records packet losses. When a timeout event occur, Near-
Opt looks up the global data structure to validate the loss allowing it to
isolate spurious timeouts from true losses. The Near-Opt can access this
information in O(1) time.

  * •

Rate Information: By knowing the fair-share rate of a low-priority flow, Near-
Opt can avoid _undershoot_ and _overshoot_ issues (§2.2). Near-Opt estimates
the fair-share rate for each flow for the next RTT using the updated load and
queue length information on a per link basis. Specifically, it first
calculates the high priority load by dividing the high priority bytes received
in previous round (_t-1_) — sum of total bytes sent (_B_) and the current
queue length Q(_t_) — by the round trip time (_T_). It then discounts the high
priority load from the total link rate (_L_) to estimate the available
capacity for low-priority traffic and divides the result by the low-priority
flow count on that link (_F_) to keep track of the available flow rate333The
rate estimation is done on the basis of load in the last round. The actual
load in the next round can differ from the estimated load hence, Near-Opt
carries a small error. We choose _T_ as the link delay instead of RTT to
minimize this error on per link basis as shown in the equation below. Finally,
it picks the minimum flow rate on all the links in the flow’s path.

$\displaystyle r_{i}(t)$ $\displaystyle=$
$\displaystyle\left(R-\sum_{c=0}^{i-1}B_{c}(t-1)/T-\sum_{c=0}^{i}Q_{c}(t)/T\right)\div|F|$

Note that both the _loss information_ and parameters needed to compute a
flow’s rate are available to Near-Opt in _O(1)_ time.

We use performance gap between TCP and Near-Opt as our primary evaluation
criteria. For this, we run each experiment using TCP and Near-Opt for low-
priority flows. To highlight the performance gap, we normalize TCP’s FCTs by
the FCTs achieved using Near-Opt. A lower (resp. higher) value of normalized
FCT indicates superior (resp. poor) performance of TCP. Since our focus is on
the performance of low-priority flows, we only look at the FCTs of low-
priority flows unless noted otherwise.

We now evaluate TCP for three distinct use-cases (discussed in §2) under a
broad range of settings.

###  3.1 Network Scheduling

We first look at how TCP performs for low-priority flows under two different
network scheduling schemes:

#### Duplicate Aware Scheduling (DAS)

DAS is a request duplication scheme which strives to minimize the tail latency
of a cloud applications [11]. DAS creates a duplicate copy of each incoming
request at a lower priority and attempts to fetch both the responses (the
primary and the duplicate) from two different servers. Upon retriving one
copy, DAS purges the other one, a feature which we disable to simplify our
experiment. A distinguishing property of this scenario is that both the
workload and load across high and low-priority queues is the same.

#### Setup:

We consider a uniform flow size distribution with different mean values:
128KB, 1MB and 5MB. We generate requests in a poisson process at medium load
(30%) which translates to a high aggregate load (60% – 30% in each priority
class) on the bottleneck link.

#### Observation:

Figure 3(a) shows that under the DAS workload, TCP’s performance is within
2$\times$ of Near-Opt, across different flow sizes. This is because a
relatively low load in the high priority queue provides ample bandwidth for
low-priority flows to make progress. As a result, TCP is largely unaffected by
the challenges posed by network prioritization.

#### Shortest Job First (SJF)

Shortest job first (SJF) is well known to be optimal in minimizing the flow
completion times [22]. Many popular schemes such as PIAS [8], PASE [53],
pFabric [6], and PDQ [33] approximate this policy which makes it a good
candidate in our evaluation 444Note that our goal here is not to argue about
the efficacy of a scheduling policy, rather to investigate its performance
implications on TCP for low-priority flows.. The key difference between our
last experiment and this one lies in the variation of flow sizes and the
distribution of load across priority queues.

(a)

(b)

(c)

Figure 4: TCP’s performance under workload co-location scenario across
different loads, flow sizes and ML-model gradient update sizes in the high
priority queue: (a) shows the performance of TCP for low-priority flows when
high priority traffic exhibit an _on-off_ behavior caused by the gradient
update of size 4MB across different loads. (b) evaluates TCP across different
flow size for the same workload at high load. (c) compares TCP’s performance
across different gradient update sizes.

#### Setup:

For this experiment, we consider the _web-search_ [5] and _data-mining_ [29]
workloads. Following the guidelines in [5], we classify all flows of size
greater or equal to 1MB as long flows. We assign low-priority to the long
flows and high priority to the rest of the flows (flow size less than 1MB).
Flows are generated at _low_ , _medium_ and _high_ loads in a poisson process.

#### Observation:

For the _web-search_ workload, the high priority flows are relatively large
compared to the _data mining_ workload hence, for a given load setting, the
flow arrivals in the high priority queue tend to be less bursty. Contrarily,
the _data-mining_ workload exhibits extreme skewness, with over 80% of the
flows being less than 10KB in size (hence, the arrival process tends to be
bursty). Approx. 3% of the flows are larger than 35MB and contribute to about
95% to the overall load [6].

Figure 3(b) shows that TCP’s performance is within 2$\times$ to the Near-Opt
for web-search workload whereas, for the data-mining workload this difference
can be up to 3$\times$. For instance, under high load, at 99.9th percentile,
the performance gap between TCP and Near-Opt is 44% larger for the data-mining
workload compared to the _web-search_ workload.

Overall, under both web-search and data-mining workloads, the low-priority
queues experience the bulk of the total load (low load in the high priority
queue). Consequently, similar to the DAS experiment, this yields enough
bandwidth for low-priority flows, shielding TCP from the challenges of network
prioritization.

###  3.2 Workload Co-Location

We now consider the scenario where two different applications are co-located
to share cloud resources. We broadly focus on the storage harvesting case – a
common scenario to utilize the spare storage capacity in compute intensive
clusters [7].

#### Experiment Setup:

We consider a storage workload from Alibaba’s cluster [3] to generate low-
priority flows. Since storage applications are often bottlenecked by the
storage media, they tend to have a smaller network footprint; thus, we
consider low load in the low-priority queue for this workload. For high
priority traffic, we consider a distributed machine learning (ML) training
workload. Distributed ML training typically involves utilizing a parameter
server [43] to periodically aggregate (disseminate) gradient updates from (to)
multiple workers – a process that takes place after each training iteration
[30] and often leads to an incast on the parameter server link, when updates
from multiple workers converge. The combination of incast and the periodicity
in the arrival process of these updates gives rise to an _on-off_ network
behavior. The specific properties of the _on-off_ behavior (e.g., the
aggregate load, the size and frequency of _on_ periods) is a function of
various factors such as the machine learning models and the underlying
hardware and the software stack [30, 37, 39]. By default, we consider a
gradient update size of 500KB from each worker (also referred as flow size),
resulting in an aggregate update of 4MB (8:1 incast) on the parameter server’s
link. By adjusting the frequency of these gradient updates, we can control the
load (low, medium, and high) on the bottleneck link. Additionally, we will
assess the impact of different gradient update sizes in subsequent
evaluations.

#### Observation:

Figure 4(a) exhibits TCP’s performance across different load settings.
Notably, TCP experience more than 4$\times$ performance difference compared to
the Near-Opt under high load while this gap is within 2$\times$ at low and
medium loads. This outcome is expected because of the continued availability
of link capacity for low-priority flows at low and medium loads which keeps
the feedback loop of low-priority flows active.

Zooming in on the high load scenario, in figure 4(b) we examine the
performance implications across different flow sizes. We observe that TCP
experience performance degradation regardless the flow size. The medium flows
suffers the most, up-to $4.9\times$. To further investigate this phenomenon,
we repeat the experiment across different update sizes. Figure 4(c) shows that
as the size of gradient updates increases, the performance gap between TCP and
Near-Opt continues to widen. Specifically, when the update size reaches 12MB,
the performance gap surpasses $10\times$. For updates of larger sizes, the
long duration of _on_ period in the high priority queue results in the
starvation of the low-priority flows. Consequently, the duration of _on_
period becomes a dominant factor in determining the flow completion times
which masks the protocol’s inefficiencies and makes the choice of protocol
irrelevant resulting in a significant reduction in the performance gap between
TCP and Near-Opt.

###  3.3 Hybrid Services

We now consider the use-case of hybrid services. Recall that a key aspect of
this use-case is the co-existence of a high performance transport protocol
(e.g., HPCC [44]) for the high-priority traffic and TCP for low-priority
traffic. Unlike TCP – which requires multiple round-trip times to fully
utilize the link – a high performance transport protocol can quickly saturate
the link hence, the presence of a long flow in the high priority queue can
completely halt the feedback loop of a low-priority flow.

#### Setup:

For the high priority traffic, we use Near-Opt as our high performance
transport due to its ability to quickly saturate link bandwidth. We use one
node as a storage client while other nodes in the cluster act as storage
servers. The client fetches storage objects using two independent poisson
processes (one for each priority class). We use fixed flow sizes of 32KB,
64KB,128KB, and 1MB and keep the aggregate load (high priority+low-priority)
at 80% while changing its distribution across the two priority classes.

#### Observation:

Fig 5(a) shows TCP’s performance across different flow sizes at medium load in
the low-priority queue (50% load in high priority queue - 30% load in the low-
priority). In this particular use-case, we observe that TCP exhibits superior
performance for short flows compared to the long flows. This can be attributed
to the high efficiency of Near-Opt in the high priority queue, which can
rapidly saturate the link, consequently having a greater impact on the
feedback loop of low-priority flows. As long flows remain in the network for
an extended duration, they become more susceptible to repeated stalls, leading
to performance issues. Fig 5(b) shows TCP’s performance across different loads
in the low-priority queue. We notice that the load settings matter less and
TCP experience similar performance across different load configurations.

(a)

(b)

Figure 5: Shows TCP’s performance for Hybrid Services use-case at 80% total
load (High + low-priority). (a) Shows TCP’s performance at 30% load in the
low-priority queue across different flow sizes. In this case, TCP’s
performance improves as the flow size decreases. (b) Shows TCP’s performance
for 128KB flow sizes across different load settings for low-priority queue.
With increased load in the low-priority queue the performance gap between TCP
and Near-Opt increases.

In conclusion, for this use-case, we find that the flow size (workload) is the
dominant factor in determining the performance implications for TCP.

(a) _On-Off_ \- Small

(b) _On-Off_ \- Med

(c) _On-Off_ \- Long

(d) DAS - 128KB

(e) DAS - 1MB

Figure 6: Compares performance of low-priority flows from testbed results and
NS3 based simulation results under network scheduling §3.1 (a-c) and workload
co-location §3.2 (d-e) use-cases to establish the fidelity of our simulations.

###  3.4 Testbed Evaluation

To establish the fidelity of our simulation results, we now repeat a set of
experiment on a real testbed and compare the results with the simulation
results.

#### Testbed Setup:

For our testbed evaluations we provisioned 9 Dell servers on CloudLab [2] that
are connected with a Mellanox SN2410 switch via 10Gbps links. Each server is
powered by 10$\times$core Intel E5-2640v4 processor and 64GB RAM and runs on a
linux based operating system. To enable network prioritization, we leverage
linux queuing discipline at the end hosts – a common practice employed in
cloud systems – whereas, inside the network, we enable strict priority queuing
at the switch. We focus on two use-cases of workload co-location §3.2 and
network scheduling §3.1 to draw the comparison between simulation and testbed
results. Note that our testbed experiments lacks coverage of _hybrid services_
use-case because of practical constraints in realizing Near-Opt in a real
testbed. We configure the network switch and linux stack at the end-host using
the same settings as used in our simulation with the exception of adjusting
the _RTO min_ to 5ms – the smallest value we could configure in our setup
without using _hrtimers_ [64]. We also repeat the NS3 experiments with the
updated value of _RTO min_ for a fair comparison. We thoroughly examine the
impact of this modification in §4.

Figure 6 shows a general alignment between our testbed results and NS3
simulation results. However, a closer look reveals noticeable differences
(e.g., at the tail for long flows in figure 6(c)). We attribute these
differences to fundamental disparities between NS3 and the Linux stack,
particularly highlighting NS3’s superior capability in enforcing strict
prioritization at the end hosts.

##  4 Beyond Default Settings

We now revisit default configurations that we used so far in our study.
Throughout this section, we focus on the workload co-location scenario (§3.2)
at a high load setting and fix the gradient update size to 4MB.

Figure 7: Shows that a smaller buffer size, a common case with low-priority
queue, contributes to the performance degradation of TCP.

Figure 8: Shows that a too small or too high value of _RTO min_ severely
degrades the performance of TCP for low-priority flows.

#### Role of Buffer Sizes:

Data center network link speeds are increasing from 10Gbps to 40Gbps and
100Gbps however, the relative switch buffer sizes are shrinking. Studies show
that per port per Gbps buffer size has shrunk from 192KB for 10Gbps to less
than 10KB for 40Gbps switches [9]. Moreover, the available buffer size for
low-priority queues is even smaller [12]. These settings can have detrimental
impact on the performance of TCP for low-priority flows. Following the
Broadcom Tridant+ switch configuration [9] – commonly deployed in data center
networks, we assume per port buffer size of 192KB. We evaluate 4 different
configuration of buffer sharing: 192KB indicates the availability of the
entire port buffer for low-priority flow, 96KB represents the equal
partitioning of port buffer between high and low-priority queues, 64KB refers
to the default case in our experiments discuss in §3, and 24KB represents the
equal sharing of port buffer across 8 queues (typically 8 queues per port are
available in switches).

Figure 8 shows as the available buffer for low-priority queue shrinks, the
performance of TCP deteriorates. For example, at p99 TCP’s performance is
5.7$\times$ worse for 24KB case compared to 192KB case when entire port buffer
is available to low-priority queue. While a low buffer size is generally bad
for TCP (i.e., fair-queuing case), this issue becomes even more pronounced
when dealing with low-priority flows.

#### Role of RTOmin:

The retransmission timeout (_RTO_) indicates packet loss and triggers the
packet retransmission. This value is estimated by continuously monitoring a
flows RTT and incorporates _RTO min_ as a lower bound for on the _RTO_ value.
Thus, it plays an important role in TCP’s loss recovery process. Existing
literature, on the other hand, argues in the favor of choosing a smaller value
of RTOmin for data center network in fairshare settings [64, 6, 67, 61, 31].
Since, low-priority flows are prone to spurious timeouts (discussed in §2),
one can argue that a lower value of _RTO min_ can exacerbate this issue.
Overall, coming up with a right timeout value is a challenge in priority queue
settings [34].

To investigate this in detail, we choose four different values of _RTO min_
commonly used in the literature: 500$\mu$s (following the 3$\times$ Avg RTT),
1ms (our default case), 5ms, and 10ms [5, 61, 31]. Figure 8 shows that a too
small and too high value of RTOmin severely affects the flow completion times.
This is primarily due to the fact that a larger value significantly
contributes to latency in loss recovery, which becomes a major concern,
particularly when dealing with tail losses where fast recovery is not an
option. On the other hand, an excessively small value notably increases the
occurrence of spurious timeouts.

#### Impact of TCP Configurations:

Our study focuses on TCP-Cubic with the SACK (Selective Acknowledgment) option
enabled. TCP-Cubic is designed to maintain a plateau around _WMAX_ , which is
determined after a packet loss event. However, in the presence of network
prioritization which can cause excessive packet loss, TCP-Cubic, in theory,
can act conservatively. To evaluate this, we modify the TCP configurations by
considering both TCP-Cubic and TCP-NewReno and comparing their performance
with and without the SACK option enabled. Figure 10 shows that changing these
configurations has minimal to no impact on TCP’s performance.

Figure 9: Compares the performance of various configurations of TCP. The
choice of these configurations have negligible impact on the performance of
TCP

Figure 10: Shows that low-priority transport protocols designed for WAN
settings are misfit for cloud environment.

#### Low-Priority Transports for WAN:

Several proposals have looked at the need to optimize transport protocols for
low-priority traffic under wide area network (WAN) settings [50, 60, 42].
These proposals implement congestion control mechanisms specifically tailored
for background traffic, which tend to be conservative to avoid disrupting
higher priority traffic when network prioritization is absent. However, due to
their end-to-end nature, these protocols are susceptible to frequent stalls in
their feedback loop. The challenges associated with such frequent stalls,
combined with their conservative rate control, make them sub-optimal choices
for low-priority flows under our scenario.

To evaluate this, we consider TCP-LP [42] and LEDBAT [60]. Both of these are
delay based transport protocols designed for low-priority traffic for WAN.
LEDBAT aims to maintain the bottleneck queuing delay around a specified
_target_. Estimating this bottleneck queuing delay for low-priority packets is
challenging because it is influenced by both the queue lengths and the
_priority-induced_ delay. To simplify this choice, we made an observation in
our experiments that a low-priority packet, once enqueued, experiences the
_priority-induced_ delay due to a single gradient update. This observation
allowed us to approximate the _target_ value, which we set to 3.2ms. Figure 10
shows that both these protocols experience a significant increase in the FCTs.
Specifically, compared to TCP, we observed up to 2.4$\times$ and 6.67$\times$
increase in FCTs on avg and at p99 respectively.

##  5 Improving TCP’s Performance for Low-Priority Flows

Our evaluation points out that for certain use-cases, TCP can experience
severe performance degradation for low-priority flows. We discuss that this is
primarily because of the transient starvation of low-priority queue. We now
discuss (and do a preliminary evaluation) of two simple strategies for
addressing these issues.

#### Weighted Fair Queuing instead of Priority Queues:

One approach to address this issue is to replace strict priority queuing (SPQ)
with weighted fair queuing (WFQ). WFQ allows for differential service while
avoiding starvation in low-priority queues. Recent work [68] has highlighted
the performance benefits of WFQ over SPQ, particularly in achieving service-
level objectives (SLO) for remote procedural calls (RPC). By assigning a small
weight to the lower priority class, WFQ ensures that low-priority flows
receive a minimal fraction of the link share, thereby preventing stalls in
their feedback loop while maintaining desired isolation across priority
queues.

To assess the advantages of using WFQ for low-priority flows and examine the
impact on the high priority workload, we repeat workload co-location
experiment from section §3.2. We choose the gradient update size of 12MB – the
case where TCP observed maximum performance degradation – and replace SPQ with
WFQ at the switches. We choose a weight distribution of 99:1 for high and low-
priority classes respectively. Figure 11 shows that WFQ significantly improves
TCP’s performance. Specifically, at p99.9, we observe up to 9$\times$ and
5.4$\times$ reduction in the FCTs of small and medium flows respectively.
However, our analysis revealed that WFQ resulted in an increase of 6% in FCTs
for high priority flows at p99.9.

While weighted fair queuing (WFQ) shows promise in enhancing TCP’s performance
for low-priority flows, its use presents a challenge in selecting the
appropriate weight value for lower priority classes – using a small value may
render WFQ ineffective, while a large value can potentially interfere with the
performance of high priority traffic. Moreover, when dealing with multiple
lower priority classes [14, 8], additional caution is necessary in assigning
minimum weights, warranting further investigation.

(a)

(b)

(c)

Figure 11: Shows that efficacy of weighted fair queuing (WFQ) and TCP+. Both
these approaches have the potential to limit the performance implications of
network prioritization.

#### Cross-Queue Congestion Notification (CQCN)

For certain systems that necessitate strict isolation across priority classes
(e.g., DAS [11]) or in scenarios where changing the switch scheduling
algorithm may not be an option, we discuss an end-host based approach that we
term as _Cross-Queue Congestion Notification_. Drawing inspiration from DCTCP
[5], we make a simple yet an important observation: employing Explicit
Congestion Notification (ECN) in a _cross-queue_ manner enables TCP to discern
_priority-induced_ congestion at the end-host. Such a cross-queue signal can
enable TCP to make an informed decision to stop packet transmission and halt
congestion control for low-priority flows upon experiencing _priority-induced_
congestion. Consequently, CQCN can avoid overwhelming low-priority queues,
effectively limiting packet losses and spurious congestion backoffs for low-
priority flows.

To enable CQCN, low-priority flows can leverage _low-overhead_ yet _high-
priority_ probe packets that act to create a secondary, _non-blocking_ ,
feedback loop between end-hosts. Similar to DCTCP, probe packets get marked if
the length of the high priority queue exceeds a certain threshold – indicating
_priority-induced_ congestion. However, unlike DCTCP – which may encounter
feedback stalls when employed for low-priority flows – the secondary feedback
loop continues to provide a low-priority TCP sender with updates about the
state of high priority traffic. Cross-queue can leverage existing ECN support
in network switches without requiring any modification inside the network.

To evaluate the potential benefits of such an approach we repeat the workload
co-location experiment from section §3.2 with gradient update size of 12MB.
Figure 11 shows that such an approach, shown as TCP+, can enhance TCP’s
performance. Specifically, we observe up to 6$\times$ decrease in FCTs for
medium flows at p99.

While our preliminary evaluation indicates the promise of this approach, it
also present several challenges. For instance, despite the probe packets being
_low-overhead_ , per-flow probing can interfere with high priority traffic,
thereby violating the isolation constraint. While probing overhead is
inevitable with this approach, its impact can be minimized by multiplexing
probing feedback across all flows between a pair of end-hosts. Similarly,
another challenge lies in selecting the appropriate probe marking threshold. A
low threshold may inaccurately indicate excessive congestion in the high
priority queue, while a high threshold may render the probing feedback
ineffective, causing TCP to fail in responding to congestion in the high
priority queue. Overall, these challenges present intriguing avenues for
future research on this problem.

##  6 Discussion

#### Emerging Data Center Transport Protocols:

In our study we focused on the performance evaluation of TCP for low-priority
flows. But the challenges introduced by network prioritization can affect
_fair-transports_ 555End-host based transport protocols that are designed to
fairly share the network bandwidth in general. Majority of the existing data
center transport protocols follow the fairshare policy and rely on an end-end
feedback loop between the sender and the receiver [5, 41, 51, 71, 44]. We
hypothesize that these proposals are susceptible to performance issues when
used for low-priority flows however, the sensitivity of these protocols to
network prioritization may vary depending on their choice of congestion signal
(e.g., delay, ECN etc.,) and the congestion control algorithm [72, 47]. For
example, a recent work [28] shows that despite HPCC [44] leveraging a rich
network state (e.g., queue size and link utilization information), its
performance suffers under network prioritization. The increasing interest in
the large-scale deployment of these proposals [44, 71] may require a separate
study on their performance evaluation for low-priority flows.

#### Dual Congestion Control for Data Center Applications:

Annulus [61], propose a dual congestion control loop to improve the co-
location of WAN and data center network traffic. Annulus relies on a direct
congestion signal from network switches to the end host. Their findings align
with our observations, indicating that augmenting TCP with additional
congestion information can improve TCP’s performance. A recent work, On-Ramp
[47], advocates for the modularization of congestion control into two
independent components to handle transient and persistent congestion
separately. On-Ramp leverages one-way delay estimates to classify congestion.
Similar to On-Ramp, we discuss that the use of a dual congestion control loop
to enhance TCP’s performance in the presence of network prioritization.

####  Reconfigurable Data Center Networks (RDCN):

Under RDCNs, the path between the end hosts may alternate rapidly between slow
electrical network and fast optical network resulting in an order of magnitude
difference in latency and throughput. Chen et. al. [15] shows that existing
TCP variants are unable to take full advantage of the additional capacity
provided by RDCNs. Our findings (see §4) also highlight that TCP suffers
significantly under high fluctuation in the available network bandwidth.

#### Switch Buffer Sharing Algorithms:

In §4, we evaluate the impact of buffer size on the performance of TCP for
low-priority flows. We consider static partitioning of the available buffer
between the queues. However, the switch buffer is often dynamically shared
across queues [17, 4]. Buffer sharing algorithms provides control over queue
lengths across priority classes via a configurable parameter $\alpha$.
Typically, a high (resp. low) value of $\alpha$ is selected for high (resp.
low) priority classes. A recent work [4] shows that under a large number of
priority classes (typically 8), the dynamic sharing can lead to starvation for
lower priority classes. We observe similar behavior when the available buffer
size for low-priority queues is set to a low value.

#### End-host Stack Delays:

Feedback delay for an end-host based transport (e.g., TCP) can broadly be
divided in two categories: 1) end-host stack delays, 2) fabric delays. In this
study, we focused on the fabric delays caused by network prioritization. A
recent work [13] showed that stack induced delays can be very large especially
for networks with high link speeds. We believe that stack induced delays will
further deteriorate TCP’s problem for the scenarios considered in this study.

##  7 Conclusion

In this study we show that the use of network prioritization in a cloud system
can pose serious challenges to the use of TCP for low-priority flows. Our
study thoroughly investigates the resulting performance implications under a
range of workloads, load settings and a variety of configurations. We conclude
that TCP’s performance is severely impacted under workloads that exhibits _on-
off_ behavior on the network. We demonstrate that TCP’s performance can be
improved by incorporating the state of high priority traffic in the congestion
control of low-priority flows which opens up exciting an avenue for future
research.

## References

  * [1] https://github.com/hmmohsin/DAS
  * [2] CloudLab. https://www.cloudlab.us/
  * [3] Alibaba storage workload. https://github.com/alibaba-edu/High-Precision-Congestion-Control (2019)
  * [4] Addanki, V., Apostolaki, M., Ghobadi, M., Schmid, S., Vanbever, L.: Abm: active buffer management in datacenters. In: Proc. ACM SIGCOMM (2022)
  * [5] Alizadeh, M., Greenberg, A., Maltz, D.A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., Sridharan, M.: Data center tcp (dctcp). In: Proc. ACM SIGCOMM (2010)
  * [6] Alizadeh, M., Yang, S., Sharif, M., Katti, S., McKeown, N., Prabhakar, B., Shenker, S.: pfabric: Minimal near-optimal datacenter transport. In: Proc. ACM SIGCOMM (2013)
  * [7] Ambati, P., Goiri, I., Frujeri, F., Gun, A., Wang, K., Dolan, B., Corell, B., Pasupuleti, S., Moscibroda, T., Elnikety, S., et al.: Providing slos for resource-harvesting in cloud platforms. In: Proce. USENIX OSDI (2020)
  * [8] Bai, W., Chen, L., Chen, K., Han, D., Tian, C., Wang, H.: Information-agnostic flow scheduling for commodity data centers. In: Proc. Usenix NSDI (2015)
  * [9] Bai, W., Hu, S., Chen, K., Tan, K., Xiong, Y.: One more config is enough: Saving (dc) tcp for high-speed extremely shallow-buffered datacenters. IEEE/ACM Transactions on Networking (2020)
  * [10] Barroso, L.A., Clidaras, J., Hölzle, U.: The datacenter as a computer: An introduction to the design of warehouse-scale machines. Synthesis lectures on computer architecture (2013)
  * [11] Bashir, H.M., Faisal, A.B., Jamshed, M.A., Vondras, P., Iftikhar, A.M., Qazi, I.A., Dogar, F.R.: Reducing tail latency using duplication: A multi-layered approach. In: Proc. ACM CoNEXT (2019)
  * [12] Beheshti, N., Lapukhov, P., Ganjali, Y.: Buffer sizing experiments at facebook. In: Proc. Workshop on Buffer Sizing (2019)
  * [13] Cai, Q., Chaudhary, S., Vuppalapati, M., Hwang, J., Agarwal, R.: Understanding host network stack overheads. In: Proc. ACM SIGCOMM (2021)
  * [14] Chen, L., Lingys, J., Chen, K., Liu, F.: Auto: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization. SIGCOMM ’18, Association for Computing Machinery (2018)
  * [15] Chen, S.S., Wang, W., Canel, C., Seshan, S., Snoeren, A.C., Steenkiste, P.: Time-division tcp for reconfigurable data center networks. In: Proc. ACM SIGCOMM (2022)
  * [16] Chen, X., Feibish, S.L., Koral, Y., Rexford, J., Rottenstreich, O., Monetti, S.A., Wang, T.Y.: Fine-grained queue measurement in the data plane. In: Proc. ACM CoNEXT (2019)
  * [17] Choudhury, A.K., Hahne, E.L.: Dynamic queue length thresholds for shared-memory packet switches. IEEE/ACM Transactions On Networking (1998)
  * [18] Delimitrou, C., Kozyrakis, C.: Quasar: Resource-efficient and qos-aware cluster management. ACM SIGPLAN (2014)
  * [19] Denis, M., Yao, Y., Hatch, A., Zhang, Q., Lim, C.L., Zhang, S., Sugrue, K., Kwok, H., Fernandez, M.J., Lapukhov, P., et al.: Ebb: Reliable and evolvable express backbone network in meta. In: Proc. ACM SIGCOMM
  * [20] Dogar, F.R., Karagiannis, T., Ballani, H., Rowstron, A.: Decentralized Task-aware Scheduling for Data Center Networks. In: Proc. ACM SIGCOMM (2014)
  * [21] Dukkipati, N.: Rate Control Protocol (Rcp): Congestion Control to Make Flows Complete Quickly. Ph.D. thesis (2008)
  * [22] Faisal, A.B., Bashir, H.M., Qazi, I.A., Uzmi, Z., Dogar, F.R.: Workload adaptive flow scheduling. In: Proc. ACM CoNEXT (2018)
  * [23] Firestone, D., Putnam, A., Mundkur, S., Chiou, D., Dabagh, A., Andrewartha, M., Angepat, H., Bhanu, V., Caulfield, A., Chung, E., et al.: Azure accelerated networking:$\\{$SmartNICs$\\}$ in the public cloud. In: Proc. NSDI 18 (2018)
  * [24] Gandhi, S., Iyer, A.P.: P3: Distributed deep graph learning at scale. In: 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) (2021)
  * [25] Gao, Y., Li, Q., Tang, L., Xi, Y., Zhang, P., Peng, W., Li, B., Wu, Y., Liu, S., Yan, L., Feng, F., Zhuang, Y., Liu, F., Liu, P., Liu, X., Wu, Z., Wu, J., Cao, Z., Tian, C., Wu, J., Zhu, J., Wang, H., Cai, D., Wu, J.: When cloud storage meets RDMA. In: 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association (2021)
  * [26] Gardner, K.: Modeling and Analyzing Systems with Redundancy. Ph.D. thesis (2017)
  * [27] Goder, A., Spiridonov, A., Wang, Y.: Bistro: Scheduling $\\{$Data-Parallel$\\}$ jobs against live production systems. In: Proc. USENIX ATC (2015)
  * [28] Goyal, P., Shah, P., Zhao, K., Nikolaidis, G., Alizadeh, M., Anderson, T.E.: Backpressure flow control. In: Proc. USENIX NSDI (2022)
  * [29] Greenberg, A., Hamilton, J.R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D.A., Patel, P., Sengupta, S.: Vl2: a scalable and flexible data center network. In: Proc. ACM SIGCOMM. ACM (2009)
  * [30] Gu, J., Chowdhury, M., Shin, K.G., Zhu, Y., Jeon, M., Qian, J., Liu, H., Guo, C.: Tiresias: A GPU cluster manager for distributed deep learning. In: 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19) (2019)
  * [31] Handley, M., Raiciu, C., Agache, A., Voinescu, A., Moore, A.W., Antichi, G., Wójcik, M.: Re-architecting datacenter networks and stacks for low latency and high performance. In: Proc. ACM SIGCOMM (2017)
  * [32] Hashemi, S.H., Abdu Jyothi, S., Campbell, R.: Tictac: Accelerating distributed deep learning with communication scheduling. Proc. SysML (2019)
  * [33] Hong, C.Y., Caesar, M., Godfrey, P.B.: Finishing flows quickly with preemptive scheduling. ACM SIGCOMM Computer Communication Review (2012)
  * [34] Hu, S., Bai, W., Zeng, G., Wang, Z., Qiao, B., Chen, K., Tan, K., Wang, Y.: Aeolus: A building block for proactive transport in datacenters. In: Proc. ACM SIGCOMM (2020)
  * [35] Hu, S., Zhu, Y., Cheng, P., Guo, C., Tan, K., Padhye, J., Chen, K.: Deadlocks in datacenter networks: Why do they form, and how to avoid them. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks. Association for Computing Machinery (2016)
  * [36] Iorgulescu, C., Azimi, R., Kwon, Y., Elnikety, S., Syamala, M., Narasayya, V., Herodotou, H., Tomita, P., Chen, A., Zhang, J., Wang, J.: PerfIso: Performance isolation for commercial Latency-Sensitive services. In: 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association (2018)
  * [37] Jiang, Y., Zhu, Y., Lan, C., Yi, B., Cui, Y., Guo, C.: A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters. In: Proc. USENIX OSDI
  * [38] Judd, G.: Attaining the promise and avoiding the pitfalls of TCP in the datacenter. In: USENIX NSDI (2015)
  * [39] Keuper, J., Pfreundt, F.J.: Distributed training of deep neural networks: Theoretical and practical limits of parallel scalability (2016)
  * [40] Kong, X., Zhu, Y., Zhou, H., Jiang, Z., Ye, J., Guo, C., Zhuo, D.: Collie: Finding performance anomalies in RDMA subsystems. In: 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association (2022)
  * [41] Kumar, G., Dukkipati, N., Jang, K., Wassel, H.M., Wu, X., Montazeri, B., Wang, Y., Springborn, K., Alfeld, C., Ryan, M., et al.: Swift: Delay is simple and effective for congestion control in the datacenter. In: Proc. ACM SIGCOMM (2020)
  * [42] Kuzmanovic, A., Knightly, E.W.: Tcp-lp: A distributed algorithm for low priority data transfer. In: Proc. IEEE INFOCOM (2003)
  * [43] Li, M., Andersen, D.G., Park, J.W., Smola, A.J., Ahmed, A., Josifovski, V., Long, J., Shekita, E.J., Su, B.Y.: Scaling distributed machine learning with the parameter server. In: Proc. Usenix OSDI (2014)
  * [44] Li, Y., Miao, R., Liu, H.H., Zhuang, Y., Feng, F., Tang, L., Cao, Z., Zhang, M., Kelly, F., Alizadeh, M., et al.: Hpcc: High precision congestion control. In: Proc. ACM SIGCOMM (2019)
  * [45] Li, Z., Liu, N., Wu, J.: Toward a production-ready general-purpose rdma-enabled rpc. In: Proc. ACM SIGCOMM 2019 Conference Posters and Demos (2019)
  * [46] Liu, Q., Yu, Z.: The elasticity and plasticity in semi-containerized co-locating cloud workload: a view from alibaba trace. In: Proc. ACM SoCC (2018)
  * [47] Liu, S., Ghalayini, A., Alizadeh, M., Prabhakar, B., Rosenblum, M., Sivaraman, A.: Breaking the Transience-Equilibrium nexus: A new approach to datacenter packet transport. In: 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21) (2021)
  * [48] Lo, D., Cheng, L., Govindaraju, R., Ranganathan, P., Kozyrakis, C.: Heracles: Improving resource efficiency at scale. In: Proc. ISCA (2015)
  * [49] Lu, Y., Chen, G., Li, B., Tan, K., Xiong, Y., Cheng, P., Zhang, J., Chen, E., Moscibroda, T.: Multi-Path transport for RDMA in datacenters. In: 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). USENIX Association (2018)
  * [50] Meng, T., Schiff, N.R., Godfrey, P.B., Schapira, M.: Pcc proteus: Scavenger transport and beyond. In: Proc. ACM SIGCOMM (2020)
  * [51] Mittal, R., Lam, V.T., Dukkipati, N., Blem, E., Wassel, H., Ghobadi, M., Vahdat, A., Wang, Y., Wetherall, D., Zats, D.: Timely: Rtt-based congestion control for the datacenter. In: Proc. ACM SIGCOMM (2015)
  * [52] Montazeri, B., Li, Y., Alizadeh, M., Ousterhout, J.: Homa: A receiver-driven low-latency transport protocol using network priorities. In: Proc. ACM SIGCOMM (2018)
  * [53] Munir, A., Baig, G., Irteza, S.M., Qazi, I.A., Liu, A.X., Dogar, F.R.: Friends, not Foes: Synthesizing Existing Transport Strategies for Data Center Networks. In: Proc. ACM SIGCOMM (2014)
  * [54] Peng, Y., Zhu, Y., Chen, Y., Bao, Y., Yi, B., Lan, C., Wu, C., Guo, C.: A generic communication scheduler for distributed dnn training acceleration. In: Proc. ACM SOSP (2019)
  * [55] Primorac, M., Argyraki, K., Bugnion, E.: When to hedge in interactive services. In: Proc. USENIX NSDI (2021)
  * [56] Qian, K., Cheng, W., Zhang, T., Ren, F.: Gentle flow control: Avoiding deadlock in lossless networks. In: Proceedings of the ACM Special Interest Group on Data Communication. Association for Computing Machinery (2019)
  * [57] Rajasekaran, S., Ghobadi, M., Kumar, G., Akella, A.: Congestion control in machine learning clusters. In: Proc. ACM HotNets (2022)
  * [58] Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In: Proc. ACM SoCC (2012)
  * [59] Riley, G.F., Henderson, T.R.: The ns-3 network simulator. In: Modeling and Tools for Network Simulation. Springer (2010)
  * [60] Rossi, D., Testa, C., Valenti, S., Muscariello, L.: Ledbat: the new bittorrent congestion control protocol. In: Proc. IEEE Computer Communications and Networks (2010)
  * [61] Saeed, A., Gupta, V., Goyal, P., Sharif, M., Pan, R., Ammar, M., Zegura, E., Jang, K., Alizadeh, M., Kabbani, A., et al.: Annulus: A dual congestion control loop for datacenter and wan traffic aggregates. In: Proc. ACM SIGCOMM (2020)
  * [62] Tirmazi, M., Barker, A., Deng, N., Haque, M.E., Qin, Z.G., Hand, S., Harchol-Balter, M., Wilkes, J.: Borg: the next generation. In: Proc. ACM EuroSys (2020)
  * [63] Vamanan, B., Hasan, J., Vijaykumar, T.: Deadline-aware datacenter tcp (d2tcp). In: Proc. ACM SIGCOMM (2012)
  * [64] Vasudevan, V., Phanishayee, A., Shah, H., Krevat, E., Andersen, D.G., Ganger, G.R., Gibson, G.A., Mueller, B.: Safe and effective fine-grained tcp retransmissions for datacenter communication. ACM SIGCOMM computer communication review (2009)
  * [65] Vulimiri, A., Godfrey, P.B., Mittal, R., Sherry, J., Ratnasamy, S., Shenker, S.: Low latency via redundancy. In: Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies. CoNEXT ’13, Association for Computing Machinery (2013)
  * [66] Wilson, C., Ballani, H., Karagiannis, T., Rowtron, A.: Better never than late: Meeting deadlines in datacenter networks. ACM SIGCOMM Computer Communication Review (2011)
  * [67] Zats, D., Iyer, A.P., Ananthanarayanan, G., Agarwal, R., Katz, R., Stoica, I., Vahdat, A.: Fastlane: making short flows shorter with agile drop notification. In: Proc. ACM SoCC (2015)
  * [68] Zhang, Y., Kumar, G., Dukkipati, N., Wu, X., Jha, P., Chowdhury, M., Vahdat, A.: Aequitas: admission control for performance-critical rpcs in datacenters. In: Proc. ACM SIGCOMM (2022)
  * [69] Zhang, Y., Prekas, G., Fumarola, G.M., Fontoura, M., Goiri, Í., Bianchini, R.: $\\{$History-Based$\\}$ harvesting of spare cycles and storage in $\\{$Large-Scale$\\}$ datacenters. In: Proc. USENIX OSDI) (2016)
  * [70] Zhu, L., Shen, Y., Xu, E., Shi, B., Fu, T., Ma, S., Chen, S., Wang, Z., Wu, H., Liao, X., et al.: Deploying user-space $\\{$TCP$\\}$ at cloud scale with $\\{$LUNA$\\}$. In: Proc. USENIX ATC
  * [71] Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M.H., Zhang, M.: Congestion control for large-scale rdma deployments. In: Proc. ACM SIGCOMM (2015)
  * [72] Zhu, Y., Ghobadi, M., Misra, V., Padhye, J.: Ecn or delay: Lessons learnt from analysis of dcqcn and timely. In: Proc. ACM CoNEXT (2016)