# GraphCC: A Practical Graph Learning-based Approach to Congestion Control in
Datacenters

Guillermo Bernárdez, José Suárez-Varela, Xiang Shi, Shihan Xiao, Xiangle
Cheng, Pere Barlet-Ros and Albert Cabellos-Aparicio

###### Abstract.

Congestion Control (CC) plays a fundamental role in optimizing traffic in Data
Center Networks (DCN). Currently, DCNs mainly implement two main CC protocols:
DCTCP and DCQCN. Both protocols — and their main variants — are based on
Explicit Congestion Notification (ECN), where intermediate switches mark
packets when they detect congestion. The ECN configuration is thus a crucial
aspect on the performance of CC protocols. Nowadays, network experts set
static ECN parameters carefully selected to optimize the average network
performance. However, today’s high-speed DCNs experience quick and abrupt
changes that severely change the network state (e.g., dynamic traffic
workloads, incast events, failures). This leads to under-utilization and sub-
optimal performance. This paper presents GraphCC, a novel Machine Learning-
based framework for in-network CC optimization. Our distributed solution
relies on a novel combination of Multi-agent Reinforcement Learning (MARL) and
Graph Neural Networks (GNN), and it is compatible with widely deployed ECN-
based CC protocols. GraphCC deploys distributed agents on switches that
communicate with their neighbors to cooperate and optimize the global ECN
configuration. In our evaluation, we test the performance of GraphCC under a
wide variety of scenarios, focusing on the capability of this solution to
adapt to new scenarios unseen during training (e.g., new traffic workloads,
failures, upgrades). We compare GraphCC with a state-of-the-art MARL-based
solution for ECN tuning — (_ACC_) —, and observe that our proposed solution
outperforms the state-of-the-art baseline in all of the evaluation scenarios,
showing improvements up to 20% in Flow Completion Time as well as significant
reductions in buffer occupancy (38.0-85.7%).

Graph Neural Networks, Multi-Agent Reinforcement Learning, Congestion Control,
Datacenters

G. Bernárdez, P. Barlet-Ros and A. Cabellos-Aparicio are with Barcelona Neural
Networking Center, Universitat Politècnica de Catalunya, Barcelona, Spain.
Contact<EMAIL_ADDRESS>
J. Suárez-Varela is with Telefonica Research, Madrid, Spain.

X. Shi, S. Xiao and X. Cheng are with the Network Technology Lab., Huawei
Technologies Co., Ltd., Beijing, China.

##  1\. Introduction

The last decade has witnessed an ever-growing interest on optimizing Data
Center Networks (DCN) given the critical services they run and the high CAPEX
and OPEX these infrastructures entail. Among the large spectrum of traffic
optimization mechanisms (Li et al., 2021), Congestion Control (CC) has been
especially explored in the literature (Alizadeh et al., 2010; Zhu et al.,
2015; Mittal et al., 2015; Montazeri et al., 2018; Li et al., 2019; Kumar et
al., 2020; Yan et al., 2021). Nowadays, most production DCNs run two main CC
protocols: $(i)$ DCTCP (Alizadeh et al., 2010), for datacenters based on the
traditional TCP/IP stack, and $(ii)$ DCQCN (Zhu et al., 2015), for emerging
RDMA-based DCNs. Both CC schemes rely on congestion notifications raised by
intermediate switches in the network — known as the Explicit Congestion
Notification (ECN) mechanism (Ramakrishnan et al., 2001). This is the only
feedback that end-hosts receive to dynamically adapt the flow rate. The ECN
configuration thus become a crucial aspect for optimizing traffic in today’s
datacenters (Shan and Ren, 2018; Yan et al., 2021), and finding the optimal
ECN parameters is a complex and time consuming task. Nowadays, network experts
end up selecting configurations that can work well on average, while being
conservative enough to absorb traffic microbursts and avoid queue buildup
(Shan and Ren, 2018).

However, traffic in modern high-speed DCNs is more and more dynamic. For
example, in emerging applications —such as distributed cloud storage— it is
very frequent to find incast events (synchronous many-to-one connections),
which put great pressure over specific switch ports for short time spans. At
the same time, production DCNs experience failures frequently (Zhou et al.,
2014; Liu et al., 2013). This means topology changes that break a main design
principle of today’s DCNs: network symmetry. Failures cause network imbalance,
which may lead to severe performance degradation (up to 40% throughput
reduction in real-world networks (Gill et al., 2011)).

In this vein, Machine Learning (ML) has raised a special interest as a
suitable technique to dynamically optimize CC in DCNs. Nowadays, we can attest
some pioneering ML-based CC proposals, such as AuTO (Chen et al., 2018),
Aurora (Jay et al., 2019), or Orca (Abbasloo et al., 2020). However, these
solutions are not compatible with widely deployed equipment in datacenters, as
they propose to re-implement the network stack. A more recent solution, ACC
(Yan et al., 2021), proposes to perform in-network optimization by dynamically
adapting the ECN configuration on switches. This solution has shown
outstanding performance in production environments and it is compatible with
current datacenter equipment running widely deployed ECN-based CC protocols
(e.g., DCTCP, DCQCN). Nevertheless, ACC is designed for online training; i.e.
it gradually learns how to adapt to the current network conditions. As a
result, it may suffer from critical transient performance degradation when
traffic changes. In general, online training is not always appropriate in
production environments, as: (i) it carries an implicit uncertainty on what
would be the resulting performance of agents after re-training, (ii) the
training adds an extra execution cost, and (iii) it may be not compatible with
legacy hardware —or simply training takes too much time due to the
computational requirements for training models there.

This paper presents GraphCC, a ML-based framework for in-network CC
optimization that does not need further training once deployed; by design, our
solution is able to adapt to varying DCN conditions despite it being trained
in a fully offline manner. Similar to ACC (Yan et al., 2021), GraphCC
dynamically optimizes the ECN configuration on switches, and it is compatible
with widely deployed CC protocols (e.g., DCTCP, DCQCN). However, our method is
based on a novel combination of Multi-Agent Reinforcement Learning (MARL) and
Graph Neural Networks (GNN) that, after training, produces a single agent
implementation that can be deployed in a distributed way on switches to
optimize the ECN configuration at the interface level. In contrast to previous
proposals, deployed GraphCC agents do communicate with adjacent agents to get
local context, and they actually learn how to cooperate to optimize the global
Flow Completion Time (FCT).

We evaluate GraphCC under a diverse spectrum of scenarios with DCQCN,
including different real-world traffic workloads unseen during the training
phase. Also, we test how this solution behaves under aggressive incast events
and drastic topology changes, such as link failures or network upgrades. We
compare the performance with respect to: $(i)$ a static ECN setting used in
Alibaba’s production networks (Li et al., 2019), and $(ii)$ ACC (Yan et al.,
2021), the previously mentioned state-of-the-art ML-based solution for dynamic
ECN tuning. In our evaluation, we use the average FCT slowdown as a reference
(Alizadeh et al., 2013; Bai et al., 2015; Gao et al., 2015). Our experimental
results show that GraphCC achieves improvements of up to 20% in the avg. FCT
slowdown w.r.t. ACC without re-training. Likewise, by tuning the ECN
configuration GraphCC learns to optimize flow-level performance while keeping
short queue lengths (reduction of 38.0-85.7% w.r.t. ACC). This may be
beneficial to achieve stability under unpredictable traffic microbursts (Shan
and Ren, 2018), and it is a trend already seen in other near-optimal state-of-
the-art CC mechanisms relying on advanced telemetry, such as HPCC (Li et al.,
2019).

Congestion Control (CC) has been extensively studied in the past. As a result,
there exists a plethora of pioneering solutions for DCNs tackling the problem
from different angles, such as RTT-based (Mittal et al., 2015; Kumar et al.,
2020), credit-based (Cho et al., 2017; Montazeri et al., 2018), or telemetry-
based (Li et al., 2019; Ben Basat et al., 2020) mechanisms. Nowadays most
production DCNs implement two main well-established CC protocols: DCTCP
(Alizadeh et al., 2010), and DCQCN (Zhu et al., 2015). The former is the main
standard in traditional networks based on the TCP/IP stack, while the latter
is the _de facto_ standard in modern RDMA-based networks111RDMA stands for
Remote Direct Memory Access. It is a link-level technology that optimizes
memory access across distributed nodes in DCNs.. Both state-of-the-art
mechanisms — as well as their enhanced schemes (Zhang et al., 2019; Mittal et
al., 2018) — rely on Explicit Congestion Notification (ECN) (Ramakrishnan et
al., 2001), so that switches mark packets when they experience congestion, and
end-hosts dynamically adapt their transmission rate accordingly.

This paper focuses on in-network optimization of widely deployed ECN-based CC
protocols (e.g., DCTCP, DCQCN). GraphCC attempts to optimize the handling of
congestion notifications in switches, which is a crucial component of CC
protocols to efficiently optimize traffic (Yan et al., 2021). In this context,
both DCTCP and DCQCN implement a similar approach: switches mark the
Congestion Experienced (CE) bit of packets in case the queue length exceeds
some predefined thresholds. DCTCP adopts a hard cutoff, i.e., all packets are
marked when the queue length exceeds a certain value $k$. Instead, DCQCN
implements a softer RED-like probabilistic approach based on three ECN
configuration parameters $\\{k_{min},k_{max},p_{max}\\}$ (Floyd and Jacobson,
1993). These parameters have a significant impact on the resulting network
performance (e.g., FCTs), and their optimal values are highly dependent on the
current traffic conditions (Yan et al., 2021). This poses a great challenge on
how to dynamically adapt these values to better exploit network resources, as
further discussed in Section 2.

Figure 1. Schematic representation of GraphCC: Distributed in-network CC
optimization.

In this context, GraphCC deploys a set of distributed agents in switches that
communicate between them to jointly optimize the ECN configuration on NICs.
The operational workflow —Figure 1 shows a schematic representation— is as
follows:

1) Measurement collection on NICs (Fig. 1; step 1 ): First, the distributed
agents of GraphCC retrieve basic measurements from their local NICs.
Specifically, they collect the bytes transmitted by the NIC ($tx\\_rate$), the
queue length ($q\\_len$), and the number of packets marked by ECN in the past
($ECN\\_marks$). These measurements are commonly supported by commercial
switches (Broadcom, 2022b, a; Intel, 2022), and can be locally obtained with
low computational overhead at microsecond timescales (Yan et al., 2021).

2) GraphCC agents optimize the ECN configuration (Fig. 1; step 2 ): Once
agents collect NIC measurements, they start a communication with other agents
deployed in adjacent switches to gain a local context. This communication is
done through a novel NN-driven message passing, where agents exchange messages
directly encoded by NN modules in order to find the best ECN settings in their
associated NICs. This communication only requires to exchange few bytes with
neighboring switches and can take few $\mu$s (the base link propagation delay
in production DCNs is typically 1-2$\mu$s (Li et al., 2019; Yan et al.,
2021)). To set the new ECN parameters — e.g., $\\{k_{min},k_{max},p_{max}\\}$
in DCQCN — agents can directly interface with forwarding chips using their
API, which is typically vendor-specific. At this point, switches start to mark
packets according to the new ECN settings. More details about the
architecture, inner workings, communication overhead and performance of
GraphCC are described in Sections 3 and 4.

3) End-hosts adjust the flow rate (Fig. 1; step 3 ): Lastly, the CC protocol
executed at end-hosts (e.g., DCTCP, DCQCN) adjusts the flow transmission rate
based on the ECN feedback. The process is as follows: if a host receives an
ECN-marked packet, it notifies it to the sender in the corresponding ACK. When
the sender receives the ACK, it re-computes the flow rate according to the
protocol-specific algorithm (e.g., Additive-Increase/Multiplicative-Decrease).
This CC mechanism thus enables to gradually react at one-RTT timescales
($\approx$10$\mu$s in high-speed DCNs (Kumar et al., 2020; Li et al., 2019;
Yan et al., 2021)). Note that the in-network optimization mechanism of GraphCC
is orthogonal and complementary to the selection of the flow rate control
algorithm (e.g., DCTCP, DCQCN). GraphCC is compatible with any ECN-based CC
protocol and can be deployed along with any other well-established traffic
optimization techniques, such as flow scheduling (Alizadeh et al., 2013; Bai
et al., 2015).

##  2\. Main Challenges in CC Optimization

This section discusses some key aspects to consider when optimizing CC
protocols in high-speed DCNs, which have driven the design of GraphCC.

###  2.1. Performance tradeoffs in Congestion Control

Nowadays, storage and computation speeds are some orders of magnitude faster
than networking operations (Kumar et al., 2020). Hence, the network becomes
the main performance bottleneck in today’s datacenters. The main operational
goal in DCNs is to maximize the throughput, while keeping low latency at the
flow level. At the same time, current DCNs carry heavy-tailed traffic
distributions, where a large amount of flows are short and time-sensitive, and
a small portion of flows are long and throughput-sensitive (Roy et al., 2015;
Zhu et al., 2015; Alizadeh et al., 2010). We discuss below the main tradeoffs
to consider when optimizing CC in DCNs.

####  2.1.1. Thoughput vs. latency

A main challenge when operating DCNs is to keep a good compromise between the
throughput for long flows and the latency for short flows. The main logic
behind ECN-based schemes is to estimate network congestion based on the queue
occupation on NICs; small queue length thresholds (e.g., [$k_{min}$,
$k_{max}$] in DCQCN) lead to aggressively dropping packets and keeping short
latency on queues, while large thresholds lead to higher bandwidth utilization
at the expense of increased latency. In this context, the Flow Completion Time
(FCT) metric has been widely accepted as the main performance indicator in
datacenters nowadays (Dukkipati and McKeown, 2006), as it unifies throughput
and latency in a single metric. In particular, the _FCT slowdown_ metric
—which computes the ratio between the actual FCTs and the baseline FCTs if
flows were sent at line-rate— further introduces fairness across flows, and it
is used in most state-of-the-art works to quantify the performance impact on
applications (Alizadeh et al., 2013; Bai et al., 2015; Gao et al., 2015)
Finally, we note that applications in modern DCNs are mainly based on
partition/aggregate design patterns (Alizadeh et al., 2010; Li et al., 2019)
where jobs are broken into small tasks and farmed out across servers, so then
partial results are aggregated to produce the final output. This poses a
special interest in minimizing the tail latency experienced by flows (e.g.,
95/99-pct of FCT slowdown), as it often dominates the overall application
performance (Alizadeh et al., 2010). Based on this, in the design and
evaluation of GraphCC we consider the 95-pct and 99-pct of the FCT slowdown as
central performance metrics.

####  2.1.2. Throughput vs stability

Nowadays, DCNs are exposed to highly dynamic traffic patterns, such as incast
events, where a large number of servers send traffic to a specific host. These
patterns are very frequent in modern applications based on the
partition/aggregate principle, as in every aggregation phase distributed
workers send partial results in a synchronized way to aggregator nodes. Thus,
beyond the canonical optimization goals of DCNs (i.e., throughput and
latency), CC mechanisms need to account for stability. That is, to be prepared
for rapid traffic variations (e.g., incast events, workload changes). Hence, a
new tradeoff arises: conservative ECN settings often underutilize the network
(i.e., less thoughput) while avoid fast queue buildup during transient incast
events; on the contrary, maximizing network utilization leaves scarce headroom
in buffers to absorb traffic microbursts and may lead to severe performance
degradation (Shan and Ren, 2018). In GraphCC, we introduce this tradeoff by
explicitly including the queue length and the throughput ($tx_{rate}$) in the
agents’ reward function.

###  2.2. ECN parameters are hard to optimize

ECN settings have a large impact on network performance, and they are highly
sensitive to traffic conditions (Yan et al., 2021). As an example, Figure 2
shows some experimental results on a small 2-layer Clos network under three
different public real-world traffic workloads, considering in each of them
three different DCQCN configurations; for clarity, values are normalized by
the median FCT slowdown of configuration ECN #3. As we can see, FCT values
vary considerably depending on the workload. For example, in the case of
Workload #1, ECN #1 and ECN #2 far outperform ECN #3 ($\approx$ 25% better on
median), while in Workload #3, ECN #3 outperforms the two other
configurations.

Figure 2. Normalized FCTs under various static ECN settings.

As a result of the multiple optimization tradeoffs described in the previous
subsection and the sensitivity of ECN settings, finding the optimal
operational point in DCNs is a cumbersome task. Nowadays, network operators
struggle to find good ECN parameters that can perform well on average, while
at the same time guaranteeing stability under drastic events (e.g., incast,
failures). This often leads to quite conservative ECN settings. The process to
refine ECN parameters can typically take weeks to months (Yan et al., 2021),
as network administrators need to make accurate workload characterizations and
carefully evaluate alternative configurations under a broad casuistry (e.g.,
stress testing, check traffic variations, failure scenarios). More
importantly, traffic workloads change drastically along the day (Yan et al.,
2021) and networks experience large daily variations on RTT (beyond 3x in real
networks (Zhang et al., 2019)). Also, link failures occur very frequently
(Zhou et al., 2014; Alizadeh et al., 2014), which generate network asymmetries
that are highly disruptive for DCNs (Gill et al., 2011; Liu et al., 2013). For
example, the study in (Gill et al., 2011) states that failures may cause up to
40% throughput reduction despite the typical link redundancy.

All this motivates the need for automatic ECN tuning techniques, such as
GraphCC, that can efficiently adapt the ECN configuration to the fast traffic
dynamics of nowadays DCNs. In this context ML is promising, especially
Reinforcement Learning-based models —such as GraphCC— that can autonomously
learn just by interacting with the environment. Section 5 revisits some
pioneering ML-based solutions for traffic optimization in DCNs, and further
discusses their current limitations (as well as their differences w.r.t
GraphCC).

###  2.3. Dealing with decreasing buffer size

There is a fundamental aspect that DCNs are increasingly witnessing as new
generations of forwarding devices come to the market. During the last decade,
switch capacity and link speeds have grown dramatically ($\approx$x10 in six
years (Goyal et al., 2022)). However, buffer size on switches have not been
scaled at the same pace. This is mainly due to the high cost and technological
barriers to scale switch memories to the ever-increasing link speeds. As a
result, the ratio buffer size vs. switch capacity has been considerably
decreased in the last years. As an example, high-end datacenter switches from
Broadcom have reduced this ratio by approximately a factor of 2 in six years
(from 2012 to 2018) (Goyal et al., 2022). This means that buffers now fill up
faster, and this trend is expected to continue exacerbating in future
datacenter generations. This poses the need for efficient CC mechanisms that
can quickly react to the fast traffic dynamics (e.g., incasts). Also, keeping
reduced queue length may be more important than ever to achieve stability,
e.g., to have sufficient headroom to absorb traffic microbursts. In this vein,
GraphCC intrinsically accounts for the minimization of the buffer occupancy
(i.e., queue lengths) during the optimization process.

##  3\. GraphCC

GraphCC was carefully designed to tackle the challenges previously exposed in
Section 2. This section describes the design and implementation of the
proposed solution. We first introduce some background on its key technologies
(MARL and GNN). Then, we present a mathematical formulation of GraphCC, and
fully contextualize its application to the considered CC optimization scenario
(see Section LABEL:sec:scenario). Lastly, we describe some details about the
deployment of this solution.

###  3.1. Background on RL and GNN

GraphCC relies on two main ML technologies: Multi-Agent Reinforcement Learning
and Graph Neural Networks. We provide an overview of key aspects of these two
technologies in order to better understand the inner architecture and design
choices of the proposed solution.

####  3.1.1. (Multi-Agent) Reinforcement Learning

According to the regular setting of RL (Bertsekas and Tsitsiklis, 1996), an
agent interacts with the environment in the following way: at each step $t$,
the agent selects an action $a_{t}$ based on its current state $s_{t}$, to
which the environment responds with a reward $r_{t}$ and then moves to the
next state $s_{t+1}$. This interaction is modeled as an episodic, time-
homogeneous Markov Decision Process (MDP)
$(\operatorname*{\mathcal{S}},\operatorname*{\mathcal{A}},r,P,\gamma)$, where
$\operatorname*{\mathcal{S}}$ and $\operatorname*{\mathcal{A}}$ are
respectively the state and action spaces; $P$ is the transition kernel,
$s_{t+1}\sim P(\cdot|s_{t},a_{t})$; $r_{t}$ represents the immediate reward
given by the environment after taking action $a_{t}$ from state $s_{t}$; and
$\gamma\in(0,1]$ is the discount factor used to compute the return $G_{t}$,
defined as the –discounted– cumulative reward from a certain time-step $t$ to
the end of the episode $T$: $G_{t}=\sum_{t=0}^{T}\gamma^{t}r_{t}$. The
behavior of the agent is described by a policy
$\pi:\mathcal{S}\to\mathcal{A}$, which maps each state to a probability
distribution over the action space, and the goal of an RL agent is to find the
optimal policy in the sense that the actions it takes maximize the expected
return $\hat{G}_{t}$. From among the wide variety of existing RL approaches
and algortihms to tackle this problem formulation, we implemented Q-learning
(Sutton and Barto, 2018), an action-value method where the policy $\pi$ is
indirectly defined from the learned estimates of the action value function
$Q^{\pi}(s,a)\leavevmode\nobreak\ =\leavevmode\nobreak\
\operatorname*{\mathbb{E}}_{\pi}\left[G_{t}|s_{0}=s,a_{0}=a\right]$. We also
considered several extensions, such as Double DQN (Hasselt, 2010) and
Prioritized Replay (Schaul et al., 2015), that enhance the basic Q-learning
algorithm.

In a MARL framework there is a set of agents $\mathcal{V}$ interacting with a
common environment that have to learn how to cooperate to pursue a common
goal. In contrast to single-agent RL, such a setting is generally formulated
as a Decentralized Partially Observable MDP (Dec-POMDP) (Foerster, 2018)
where, besides the global state space $\operatorname*{\mathcal{S}}$ and action
space $\operatorname*{\mathcal{A}}$, it distinguishes local state and action
spaces for every agent –i.e., $\operatorname*{\mathcal{S}}_{v}$ and
$\operatorname*{\mathcal{A}}_{v}$ for $v\in\mathcal{V}$. At each time step $t$
of an episode, each agent may choose an action
$a_{v}^{t}\in\operatorname*{\mathcal{A}}_{v}$ based on local observations of
the environment encoded in its current state
$s_{v}^{t}\in\operatorname*{\mathcal{S}}_{v}$. Then, the environment produces
individual rewards $r_{v}^{t}$ (and/or a global reward $r^{t}$), and it
evolves to a next global state $s^{t+1}\in\operatorname*{\mathcal{S}}$ –i.e.,
each agent $v$ transitions to the following state
$s_{v}^{t+1}\in\operatorname*{\mathcal{S}}_{v}$. Typically, a MARL system
seeks for the optimal global policy by learning a set of local policies
$\\{\pi_{\theta_{v}}\\}_{v\in\mathcal{V}}$. For doing so, most state-of-the-
art MARL solutions implement traditional (single-agent) RL algorithms on each
distributed agent, while incorporating some kind of cooperation mechanism
between them (Foerster, 2018). The standard approach for obtaining a robust
decentralized execution, however, is based on a centralized training where
extra information can be used to guide agents’ learning (Oliehoek et al.,
2008).

####  3.1.2. Graph Neural Networks

These models are a recent family of neural networks specifically conceived to
operate over graph-structured data (Scarselli et al., 2008; Battaglia et al.,
2018). In their basic form, they consist in associating some initial states to
the different elements of an input graph, and combine them considering how
these elements are connected in that graph. The resulting state
representations, which now may encode some topological awareness, are then
used to produce the final output of the GNN, which can be at the level of
graph elements, or at a global graph level.

Among the numerous GNN variants developed to date (Wu et al., 2020), we focus
on Message Passing Neural Networks (MPNN) (Gilmer et al., 2017), which is a
well-known type of GNN whose operation is based on an iterative message-
passing algorithm that propagates information between elements in a graph
$\mathcal{G}=(\operatorname*{\mathcal{N}},\mathcal{E})$. Focusing on the set
of nodes, the process is as follows: first, each node
$v\in\operatorname*{\mathcal{N}}$ initializes its hidden state $h_{v}^{0}$
using some initial features already included in the input graph. At every
message-passing step $k$, each node $v$ receives via messages the current
hidden state of all the nodes in its neighborhood
$\mathcal{B}(v)=\\{u\in\operatorname*{\mathcal{N}}|\exists
e\in\mathcal{E},e=(u,v)\lor e=(v,u)\\}$, and processes them individually by
applying a message function m(·) together with its own internal state
$h_{v}^{k}$. Then, the processed messages are combined by an aggregation
function a(·),
$M_{v}^{k}=a(\\{m(h_{v}^{k},h_{i}^{k})\\}_{i\in\mathcal{B}(v)})$. Finally, an
update function u(·) is applied to each node $v$; taking as input the
aggregated messages $M_{v}^{k}$ and its current hidden state $h_{v}^{k}$, it
outputs a new hidden state for the next step ($k+1$),
$h_{v}^{k+1}=u(h_{v}^{k},M_{v}^{k})$. After a certain number of message
passing steps $K$, a readout function r(·) takes as input the final node
states $h_{v}^{K}$ to produce the final output of the GNN model. This readout
function can predict either features of individual elements (e.g., a node’s
class) or global properties of the graph. Note that a MPNN model generates a
single set of message, aggregation, update, and readout functions that are
replicated at each selected graph element.

###  3.2. Framework Formalization

Figure 3. Operational workflow of a GraphCC agent.

GraphCC models a DCN as a graph
$\mathcal{G}=(\operatorname*{\mathcal{N}},\mathcal{E},\mathcal{V})$, with
$\operatorname*{\mathcal{N}}$ and $\mathcal{E}$ being the set of nodes and
edges, respectively, and $\mathcal{V}$ being the set of agents that can
control some of the graph entities (nodes or edges). Let
$\operatorname*{\mathcal{S}}$ and $\operatorname*{\mathcal{A}}$ represent the
global state and action spaces, respectively, defined as the joint and union
of the respective agents’ local spaces,
$\operatorname*{\mathcal{S}}=\prod_{v\in\mathcal{V}}\operatorname*{\mathcal{S}}_{v}$
and
$\operatorname*{\mathcal{A}}=\bigcup_{v\in\mathcal{V}}\operatorname*{\mathcal{A}}_{v}$.
By implementing Q-learning, our model aims to learn the state-action value
function $Q_{\theta}$ for every agent $v\in\mathcal{V}$.

GraphCC makes agents learn that state-value function approximator in a fully
distributed fashion –i.e., all agents end up constructing and using the very
same $Q_{\theta}$. In fact, each agent $v\in\mathcal{V}$ computes this
function mainly through message communications with their direct neighboring
agents $\mathcal{B}(v)$ and their local computations, no longer needing a
centralized entity responsible for collecting and processing all the global
information together. Such a decentralized, message-based generation of the
global function is achieved by modeling $Q_{\theta}$ with a MPNN (see Sec.
3.1.2). Thus, all agents $v\in\mathcal{V}$ deployed in the network are
actually replicas of the MPNN modules (message, aggregation, update and
readout functions) that perform regular message exchanges with their neighbors
$\mathcal{B}(v)$ following the message passing iteration procedure of MPNNs.
Note that such parameter sharing implies that all agents share as well the
same local state and action spaces. This reinterpretation of a MPNN as a set
of copies of its internal modules is especially important due to the fact that
in our approach we directly map the graph $\mathcal{G}$ to the real DCN
scenario, deploying copies of MPNN modules along switches and making message
communications through the network infrastructure. Hence, GraphCC naturally
distributes the execution of the MPNN, and consequently is able to fully
decentralize the execution pipeline.

We formalize the optimization problem as a Dec-POMDP of standard MARL
(Foerster, 2018), where all agents can act simultaneously at each step of the
optimization process. More in detail, at each time-step $t$ of an episode of
length $T$, the MPNN-driven procedure of approximating
$Q_{\theta}(s^{t}_{v},a^{t}_{v})$ for each agent $v\in\mathcal{V}$ –where
$s^{t}_{v}\in\operatorname*{\mathcal{S}}_{v}$ and
$a^{t}_{v}\in\operatorname*{\mathcal{A}}_{v}$ refer to the local state and
action at $t$– first constructs a hidden state $h_{v}$ encoding meaningful
information for each agent $v\in\mathcal{V}$. Each hidden state $h_{v}$ is
updated based on the hidden representations of the neighboring agents
$\mathcal{B}(v)$ — i.e., the node’s local context —, and its initialization
$h_{v}^{0}$ is a function of the current agent state
$s^{t}_{v}\in\operatorname*{\mathcal{S}}_{v}$, which is in turn based on some
predefined internal agent features $x_{v}^{t}$ (e.g. port utilization , queue
length, etc…; more details on how agents construct their corresponding input
feature vector in next section). Those hidden states are shaped during $K$
message-passing steps, where they are iteratively propagated through the graph
via messages between direct neighbors. In particular, successive hidden states
$h_{v}^{k}$, where $k$ accounts for the message-passing step, are computed by
the message, aggregation and update functions of the MPNN, as previously
described in Section 3.1.2.

Once agents generate their final hidden representation, the readout function
–following the MPNN nomenclature– is applied separately to each agent to
finally obtain its corresponding state-value function. For each agent
$v\in\mathcal{V}$, the readout function takes the final hidden state
$h^{K}_{v}$ as input, and produces the q-value estimates
$Q_{\theta}(s^{t}_{v},a_{v})$ for every possible action
$a_{v}\in\operatorname*{\mathcal{A}}_{v}$. Finally, following the standard
procedure of Q-learning algorithms, at that time-step $t$ each agent selects
the action with the maximum associated q-value, $a_{v}^{t}=\text{arg
max}_{a_{v}}Q_{\theta}(s^{t}_{v},a_{v})$.

###  3.3. Applying GraphCC to Congestion Control in DCNs

A DCN can be described in terms of its hardware devices
$\mathcal{W}=\\{\mathcal{W}_{s},\mathcal{W}_{h}\\}$ —where $\mathcal{W}_{s}$
and $\mathcal{W}_{h}$ denote the sets of switches and hosts, respectively— and
the link connections between them,
$\mathcal{L}=\\{l=(w^{src},w^{dst}),w^{src}\in\mathcal{W}\text{ and
}w^{dst}\in\mathcal{W}\text{ connected}\\}$. Therefore, in our graph-based
model $\mathcal{G}=(\operatorname*{\mathcal{N}},\mathcal{E},\mathcal{V})$ of a
DCN, we can directly relate the set of nodes $\operatorname*{\mathcal{N}}$
with hardware devices $\mathcal{W}$, and the set of edges $\mathcal{E}$ with
the actual set of network interfaces $\mathcal{L}$. GraphCC identifies each
egress port of a switch as an agent, i.e.
$\mathcal{V}=\\{(n^{src},n^{dst})\in\mathcal{E},n^{src}\in\mathcal{W}_{s}\\}$,
which in particular allows to define the ECN marking thresholds at a link
granularity (i.e., at the interface level)222Note that, in practice, port-
based agents are deployed and run in their adjacent switches.. In this
context, we can differentiate two different neighborhoods for each link-based
agent $v=(n^{src}_{v},n^{dst}_{v})\in\mathcal{V}$:

  * •

Ingress neighborhood $\mathcal{B}_{i}(v)$, defined as the set of links
$e=(n^{src}_{e},n^{dst}_{e})\in\mathcal{E}$ that can potentially inject
traffic into $v$,
$\mathcal{B}_{i}(v)=\\{e\in\mathcal{E}|n^{dst}_{e}=n^{src}_{v}\\}$.

  * •

Egress neighborhood $\mathcal{B}_{e}(v)$, consisting in the links that can
potentially receive traffic from $v$,
$\mathcal{B}_{e}(v)=\\{e\in\mathcal{E}|n^{src}_{e}=n^{dst}_{v}\\}$.

Let us now fully describe the operation of GraphCC for CC in DCNs (Figure 3
shows a schematic operational workflow). We assume that our solution interacts
with the environment (i.e. the network) every time interval $\Delta t$, which
is pre-defined. This enables to make the time evolution discrete, and hence
facilitates the episodic formulation of RL. At each step $t$ of an episode,
each agent $v\in\mathcal{V}$ first gathers three relevant NIC-level metrics
available at the switch: ($i$) the port utilization $u_{v}^{t}$ (computed as
the $tx_{rate}$ normalized by link capacity), ($ii$) the instantaneous queue
length $q_{v}^{t}$, and ($iii$) the ECN marking rate $ECN_{v}^{t}$ (normalized
by the link capacity). Then, agents construct their corresponding input
feature vector $x_{v}^{t}$ based on the current values of these metrics as
well as the values of the previous $p$ steps, i.e.

$x_{v}^{t}=(u_{v}^{t},q_{v}^{t},ECN_{v}^{t},u_{v}^{t-1},q_{v}^{t-1},ECN_{v}^{t-1},\dots,u_{v}^{t-p},q_{v}^{t-p},ECN_{v}^{t-p}).$

Empirically we found that $p=2$ works best for providing some temporal context
while keeping a low input dimension.

Agents initialize their initial MPNN-based hidden state $h_{v}^{0}$ with their
input feature vector $x_{v}^{t}$; since the hidden state vector dimension is
typically equal or higher than that of the input feature vector, remaining
components are simply 0-padded. GraphCC then executes $K$ message-passing
steps in which distributed communications between agents are involved. More in
detail, at each message-passing step $k$ each agent $v\in\mathcal{V}$ first
sends its current hidden representation $h^{k}_{v}$ to its egress neighbors
$\mathcal{B}_{e}(v)$, and consequently they receive a set of messages from
their ingress adjacent agents $\mathcal{B}_{i}(v)$. At that point, agents
individually combine each of the received hidden states with their own through
the message function –in our case, a feed-forward neural network–, and all the
resulting outputs are in turn merged into a fixed-size representation
$M_{v}^{k}$ via the aggregation function. For doing so, we implement element-
wise operations, such as min and max. Finally, each agent applies the update
function –another fully-connected neural network– to the aggregated
information $M_{v}^{k}$ and its own hidden state $h^{k}_{v}$, which outputs
its new hidden representation $h^{k+1}_{v}$.

After concluding the $K$ message-passing steps, all agents end up with a final
hidden state $h_{v}^{K}$ for that specific time-step $t$. This final
representation is then fed by each agent into their readout module, which
provides with the final values that are used to define their individual
policies. In particular, since GraphCC implements a Q-learning based pipeline,
each agent’s readout directly outputs the q-value estimates of all possible
actions for the current state of the DCN, and as described in previous
subsection 3.2 each agent takes the action with the highest value. This lead
us to the actual definition of the action space. GraphCC faces the CC problem
by optimizing the ECN marking thresholds, and by design it is able to adapt
the ECN parameters $\\{k_{min},k_{max},p_{max}\\}$ for each individual switch
egress port. In our implementation, we discretize the values of these
parameters into some pre-defined values; we provide more details in Section
4.1.

Regarding the learning process, all MPNN internal modules (message, update and
readout functions, which are replicated among all agents) are trained based on
the rewards computed at each step $t$ of a training episode. More in detail,
in our implementation we define the reward $r_{v}^{t}$ of agent
$v\in\mathcal{V}$ at step $t$ as

$r_{v}^{t}=w_{1}\cdot f(q_{v}^{t})+w_{2}\cdot u_{v}^{t},$

where $f(\cdot)$ is a descreasing function with respect to the queue length
$q_{v}^{t}$ of the associated port ($\text{dom}(f)=[0,1]$, based on (Yan et
al., 2021)), $u_{v}^{t}$ accounts for the utilization of that port, and
$w_{1},w_{2}\in[0,1]$ are the corresponding weights, with $w_{1}+w_{2}=1$
($w_{1}=0.7$ and $w_{2}=0.3$ worked best in our experiments). At each time
step $t$ of a training episode, GraphCC gathers the global RL-based sequence
$(\\{x_{v}^{t}\\}_{v\in\mathcal{V}},\\{r_{v}^{t}\\}_{v\in\mathcal{V}},\\{x_{v}^{t+1}\\}_{v\in\mathcal{V}})$
and stores it as a single sample in a replay buffer. Then our model randomly
selects a batch of these samples from the buffer and performs the training and
update of $Q_{\theta}$ accordingly (Sutton and Barto, 2018).

###  3.4. Discussion on Deployability

In this section we discuss some relevant practical implications when deploying
the proposed solution in DCNs. GraphCC naturally distributes the modules of
the MPNN among the switches of the DCN, which enables to parallellize all the
node-level computations and communications on the local neighborhood. By
construction, the GNN-based modules behind GraphCC –interpreted as agents– can
be replicated and deployed in any switch, to optimize the ECN configuration on
a particular NIC, regardless of the size and shape of the DCN topology
considered. The process of GraphCC to exchange messages between neighbors
–i.e. share the agent’s hidden states– is in fact generic and scale-invariant
(Ruiz et al., 2020). This provides GraphCC with excellent scalability and
generalization capabilities.

GraphCC applies parameter sharing over all the NIC-based agents; thus during
training the MPNN modules jointly learn from the individual perspective of all
NICs in the network, and also in this process agents learn what information to
exchange with neighbors in order to achieve effective coordination between
agents at different levels of granularity –following the topology structure
from the local context (i.e., direct neighbors) to a more global context
within the network. Note, however, that no matter all agents are in fact
replicas, at execution time each of them is able to specifically adjust its
behaviour based on its local state and the information received from its
neighbors.

Finally, the multi-agent formulation of GraphCC facilitates exploring the
large solution space. By enabling the adjustment of ECN parameters
$\\{k_{min},k_{max},p_{max}\\}$ at the interface level, the combinatorial of
all the possible actions would explode from a single-agent perspective
–specially taking into account that modern datacenters may have up to tens of
thousands of servers (Roy et al., 2015; Singh et al., 2015). In this sense,
GraphCC approach of defining a policy for each of these NIC instances
separately allows to effectively deal with such complexity and fully
distribute the agent’s decision making.

##  4\. Evaluation

(a) _FB_Hadoop_ with 16:1 incasts

(c) _WebSearch_ with 16:1 incasts

(e) _AliStorage_ with 16:1 incasts

(b) _FB_Hadoop_ without incasts

(d) _WebSearch_ without incasts

(f) _AliStorage_ without incasts

Figure 4. Normalized FCT median, p95 and p99 of the DCQCN and ACC baselines
–with respect to GraphCC– when considering different traffic workloads
(_FB_Hadoop_ , _WebSearch_ , _AliStorage_) with 16:1 incasts (top) or without
(bottom). Our GraphCC model is exclusively trained on traffic from _FB_Hadoop_
with incasts. The measured flow density can be visualized to facilitate the
interpretation of the results.

This section comprises an extensive performance evaluation of GraphCC,
including a direct comparison against ACC (Yan et al., 2021), a state-of-the-
art ML-based solution for in-network optimization of ECN parameters. In
particular, our evaluation seeks to answer the following questions:

  * •

Can GraphCC accommodate different traffic patterns (e.g., incasts) and
workloads unseen during training? (Sections 4.3 and 4.4.1)

  * •

How does GraphCC behave under increasing network loads? E.g., up to 80%
(Section 4.4.2)

  * •

How does GraphCC perform under topology changes? E.g., failures, upgrades
(Section 4.5)

###  4.1. Experimental setup

We use the ns-3 simulator (Riley and Henderson, 2010). Experiments are done in
a 2-layer Clos Network similar to the testbed used in ACC (Yan et al., 2021),
with 24 hosts, 4 leaf switches, and 2 spine switches. Switch-to-switch links
have a capacity of 100Gbps, and host-to-switch links have 25Gbps. All links
have a propagation delay of 1$\mu$s, consequently the maximum base RTT is
8$\mu$s. Switches have a shared-memory buffer of 32MB, which is derived from
real devices (Broadcom, 2022b). The network comprises a single RDMA domain,
and PFC is enabled on switches (PFC, 2021). End-hosts implement a DCQCN (Zhu
et al., 2015) distribution mimicking the implementation of Mellanox ConnectX-4
cards (Li et al., 2019), with a fixed window that limits the inflight bytes to
the maximum Bandwidth-Delay Product (BDP) in the network. This configuration
is known to perform better than the original one, as it avoids PFC storms (Li
et al., 2019). During our simulations, GraphCC agents act every 100$\mu$s, and
each episode lasts 25ms. We then leave sufficient time to ensure the same set
of flows finishes in all experiments, including the longest flows.

We note that we will always consider the same GraphCC model along all our
evaluation, which was exclusively trained using a real-world _FB_Hadoop_
workload (Roy et al., 2015) trace with a normal traffic load of 60% (Li et
al., 2019) and considering periodic 16:1 incasts events (Yan et al., 2021). In
our implementation of $Q_{\theta}$, message, update and readout functions are
2-layer feed forward NNs, and the aggregation function combines an element-
wise min and max. Regarding the tuning of some important hyper-parameters of
the model333A detailed description of the rest of parameters involved will be
provided with the public release of our implementation., the dimension of the
agents’ hidden states $h_{v}$ is set to $24$, and $K=2$ message passing steps
are considered —experimentally, we have found that more iterations do not lead
to better performance, since the diameter of DCNs is typically very limited
(2-3 hops). In addition, the possible ECN configurations are discretized
according to the sets $k_{min}=\\{2,4,8,16,32\\}$,
$k_{max}=\\{16,32,64,128,256\\}$ and $p_{max}=\\{0.01,0.25,0.5,0.75,1.0\\}$,
resulting in an agent action space $\mathcal{A}_{v}$ with $120$ effective
combinations.

|  | Mean FCT Slowdown |  | Mean Throughput (Mbps) |  | Mean queue length (kB) |
---|---|---|---|---|---|---|---
|  |  DCQCN |  ACC |  GraphCC |  |  DCQCN |  ACC |  GraphCC |  |  DCQCN |  ACC |  GraphCC |
Incasts |  _FB_Hadoop_ |  5.76 |  3.25 |  2.84 |  $\rightarrow$ -12.6% |  401 |  399 |  398 |  $\rightarrow$ -0.25% |  42.4 |  12.6 |  6.00 |  $\rightarrow$ -52.4%
16:1 |  _WebSearch_ |  6.48 |  5.38 |  5.30 |  $\rightarrow$ -1.48% |  395 |  394 |  390 |  $\rightarrow$ -1.02% |  37.3 |  9.64 |  1.65 |  $\rightarrow$ -82,9%
|  _AliStorage_ |  5.95 |  3.49 |  3.10 |  $\rightarrow$ -11.2% |  399 |  400 |  399 |  $\rightarrow$ -0.25% |  49.2 |  16.7 |  9.70 |  $\rightarrow$ -41,9%
No |  _FB_Hadoop_ |  3.94 |  2.59 |  2.16 |  $\rightarrow$ -16.6% |  362 |  361 |  356 |  $\rightarrow$ -1.38% |  26.6 |  10.0 |  3.83 |  $\rightarrow$ -61.7%
Incasts |  _WebSearch_ |  4.37 |  3.70 |  3.65 |  $\rightarrow$ -1.35% |  345 |  346 |  343 |  $\rightarrow$ -0.86% |  24.2 |  7.95 |  1.14 |  $\rightarrow$ -85,7%
|  _AliStorage_ |  3.72 |  2.71 |  2.38 |  $\rightarrow$ -12.2% |  355 |  355 |  354 |  $\rightarrow$ -0.28% |  27.0 |  13.2 |  6.73 |  $\rightarrow$ -49.0%
Big |  High (70%) |  5.81 |  3.41 |  2.80 |  $\rightarrow$ -17.8% |  416 |  415 |  411 |  $\rightarrow$ -0.96% |  42.0 |  12.5 |  5.34 |  $\rightarrow$ -57.2%
loads |  Extreme (80%) |  7.01 |  4.27 |  3.41 |  $\rightarrow$ -20.1% |  444 |  443 |  438 |  $\rightarrow$ -1.13% |  50.8 |  15.4 |  6.38 |  $\rightarrow$ -58.6%
Table 1. Mean FCT slowdown, throughput, and queue length obtained by GraphCC
and the baselines for different traffic workloads (_FB_Hadoop_ , _WebSearch_ ,
_AliStorage_) —with and without incasts—, and with high traffic loads. It also
includes the relative difference (in %) of GraphCC results w.r.t. those of
ACC.

###  4.2. Benchmarks and Performance Metrics

Before presenting our experimental results, in this section we aim to define
how we actually quantify the performance of GraphCC. First of all, we
considered as baselines the following two state-of-the-art methodologies
—revisited in Section 5— compatible with widely-deployed ECN protocols:

  * •

DCQCN (Zhu et al., 2015): This benchmark represents the most widely used
approach in today’s DCNs, i.e., careful selection of static ECN parameters. It
implements the static ECN configuration used in Alibaba’s production DCNs (Li
et al., 2019):
$k_{min}=100KB$ × $\frac{Bw}{25Gbps}$; $k_{max}=400KB$ × $\frac{Bw}{25Gbps}$.

  * •

ACC (Yan et al., 2021): State-of-the-art solution for ECN tuning based on MARL
(not GNN). We have implemented it based on the description in (Yan et al.,
2021). This solution is designed for online training and it does not perform
parameter sharing across agent implementations (i.e., agents are trained
independently). For fairness, we re-train the solution on each evaluation
scenario selecting the best set of agents after hyper-parameter tuning.

Our goal is to evaluate our trained model in varying DCN scenarios —most of
them different than those seen in training— and compare the results against
these benchmarks. As stated in Section 2, the Flow Completion Time (FCT) is
broadly accepted the most complete performance metric in DCNs, and we pay
special attention to the 95 and 99 percentiles of this metric due to the
aforementioned impact on the partition/aggregate design patterns dominant in
today’s applications (Alizadeh et al., 2010; Li et al., 2019). Hence, for each
scenario, we compute the FCT median, 95-pct and 99-pct obtained by our
solution against those of the benchmarks. In particular, we designed a
detailed visualization where i results are aggregated and shown by flow size;
ii the FCT values of the baselines are normalized with respect to those of our
GraphCC model; and iii the flow size distribution is shown in parallel to
properly contextualize the relevance of the obtained results. Finally, these
plots are complemented with a direct table comparison of the mean absolute
values of FCT slowdown, throughput and queue length metrics.

###  4.3. Direct Performance Comparison

In our first set of experiments, we aim to evaluate our solution in the same
scenario considered in training, with traffic generated from the _FB_Hadoop_
workload. We generate two different traffic traces to test the performance of
our method: with periodic incasts –similar to those seen during training– and
without them. Respectively, Figures 4(a) and 4(b) compare the per-flow size
FCT median, percentile 95 (p95) and percentile 99 (p99) achieved by GraphCC
against the defined baselines.

In both cases, we can see that GraphCC clearly outperforms the static DCQCN
setting and improves the state-of-the-art ACC solution, especially on short
and medium sized flows –which represent the vast majority of flows– in median
and both p95 and p99 tails. An aspect that we can observe in this case is that
GraphCC learns to slightly sacrifice latency for long flows, as they account
for a small percentage of the total counting. Overall, and as shown in Table
1, GraphCC achieves a notable reduction in the mean FCT slowdown (up to
$16.6\%$ reduction w.r.t. ACC without incasts, and $12.6\%$ reduction without)
while keeping equivalent mean throughput to that of ACC ($1.38\%$ of
difference in the worst case).

Moreover, Table 1 also shows an interesting behaviour of our solution: it
achieves a significant reduction in queue length –more than $50\%$ in both
_FB_Hadoop_ experiments–, which directly relates to significantly lower buffer
occupancies in switches. These results suggest that GraphCC is able to attain
good flow-level latency (i.e., FCT) by learning how to efficiently manage
queues so as to achieve stability and be prepared for potential microbursts
and incast events. This near-zero queue behavior is a trend seen in other
state-of-the-art solutions, such as HPCC (Li et al., 2019) (based on advanced
telemetry), and it especially helps achieve ultra-low latency on short flows.
As previously discussed in Section 2, this behavior is also very beneficial in
modern DCNs, given the ongoing trend on decreasing the ratio between buffer
size and switch capacity.

###  4.4. Evaluation under Traffic Changes

Our next goal is to demonstrate that our trained GraphCC model can adapt to
unseen traffic scenarios without requiring any further re-training. In this
regard, we design two different challenging scenarios: different workload
distributions, and very high traffic loads.

####  4.4.1. Different Traffic Workloads

Different applications may have completely different traffic distributions, so
we are interested in testing how our trained GraphCC model performs over
workloads not previously seen in training. Hence, in this set of experiments
we generate traffic traces from two different real-world traffic
distributions, _WebSearch_ and _AliStorage_ , with and without periodic
incasts. Figure 4 and Table 1 summarize the experimental results. We first
notice that the flow size distributions of these workloads greatly differ from
the one seen in training –i.e. _FB_Hadoop_. In particular, _WebSearch_ -based
traffic (see Figures 4(c) and 4(d)) involves dealing with a considerable
higher amount of medium and long-sized flows. However, we can observe that
GraphCC improves its FCT metrics for the longer flows on this scenario,
suggesting that it is properly prioritizing them. In fact, as shown in Table
1, GraphCC still gets a slightly better mean FCT slowdown than ACC in this
case, with and without incast bursts. On the other hand, GraphCC model again
sacrifices a bit of long flows’ latency for AliStorage traffic traces (Figures
4(e) and 4(f)), since in this scenario those flows have even less density than
in _FB_Hadoop_ traces. By doing so, our model is able to obtain better FCT
median, p95 and p99 metrics than baselines for the major number of flows –with
and without incasts–, and reducing the mean FCT slowdown of ACC by more than
$11\%$ (Table 1). As we can also see in Table 1, GraphCC achieves such good
FCT-based results in all these scenarios only at the expense of a small
reduction in the mean throughput ($1\%$ of difference w.r.t. ACC at worst).
However, GraphCC provides as well with significant improvements in terms of
buffer occupancies, lowering ACC’s mean queue length from $40-50\%$ for
AliStorage traffic, up to more than $80\%$ for _WebSearch_ traces.

####  4.4.2. Higher Traffic Loads

Next, we analyze whether our trained model can successfully operate over
higher traffic loads. For doing so, we evaluate our GraphCC model over new
traces from the _FB_Hadoop_ workload with an average network load of 70% and
80% –instead of the average 60% experienced in training. Figure 5 presents the
normalized FCT median, p95 and p99 achieved by ACC with respect to our
solution in both cases. We can see both in Figure 5 and in Table 2 that the
results of both loads are alike, and not very different to those previously
observed with a 60% load: compared to ACC, GraphCC improves all three FCT
metrics for short and medium flows –which represent almost 90% of the total
flows–, and achieves equivalent mean throughput –within $1.35\%$. GraphCC
manages to reduce the mean queue length by more than 57%. These results
suggest that GraphCC is robust against varying traffic loads.

(a) Very high load - 70% load

(b) Extreme load - 80% load

Figure 5. Evaluation of GraphCC against ACC when considering higher loads than
those seen in training (_FB_Hadoop_ workload).

| FCT Slowdown | Throughput (Mbps) | Queue Length (kB)
---|---|---|---
| ACC |  GraphCC | ACC |  GraphCC | ACC |  GraphCC
1 failure | 2.91 |  2.66 (-8.59%) | 343 |  338 (-1.46%) | 9.10 |  3.89 (-57.3%)
2 failures | 5.80 |  5.72 (-1.38%) | 275 |  275 (=) | 7.92 |  4.91 (-38.0%)
Extra branch | 1.76 |  1.56 (-11.3%) | 543 |  541 (-0.37%) | 5.73 |  2.94 (-48.7%)
32 hosts | 2.74 |  2.20 (-19.7%) | 475 |  468 (-1.47%) | 10.7 |  3.80 (-64.5%)
40 hosts | 2.70 |  2.27 (-15.9%) | 564 |  560 (-0.71%) | 8.60 |  3.65 (-57.6%)

Table 2. Mean FCT slowdown, throughput, and queue length of GraphCC and ACC
under various topological changes: link failures (top), adding an extra branch
(middle), and adding extra hosts (bottom). In all cases, traffic is generated
from the _FB_Hadoop_ workload.

###  4.5. Evaluation under Topology Changes

Lastly, we define a set of experiments involving topological changes to
further evaluate the robustness of a trained GraphCC model. In particular, we
analyze two different scenarios: random link failures, and flatter networks
–i.e. adding extra hosts connected to leaf switches.

####  4.5.1. Link failures

In production DCNs, link failures occur frequently (Zhou et al., 2014; Liu et
al., 2013), and often lead to network asymmetries and severe performance
degradation (see Section 2.2). In these experiments, we simulate how GraphCC
responds to critical failures in links between leaf and spine switches.
Figures 6(a) and 6(b) show, respectively, a comparison of FCT metrics between
GraphCC and ACC, for 1 and 2 link failures. In the case of a single failure,
GraphCC improves ACC on these metrics for most of the flows. On the other
hand, the 2-link failure scenario presents less performance gap between our
method and ACC, which was individually trained on each target network
scenario. Here, GraphCC gets a small improvement in median FCT for small and
medium flows, but the p95 and p99 tails are equivalent to those obtained by
ACC for all sizes. This exposes how challenging this scenario is, especially
for models pre-trained offline, as it is the case of GraphCC. Overall, in
terms of absolute values (Table 2), even with 2 failures the proposed solution
still provides with a slightly better mean FCT slowdown serving exactly the
same throughput, and does so while reducing the mean queue length up to a 38%.

(a) 1 link failure

(b) 2 link failures

Figure 6. Evaluation of GraphCC against ACC when the DCN suffers from link
failures (_FB_Hadoop_ workload).

####  4.5.2. Network upgrade

Datacenters are periodically upgraded to increase their processing and switch
capacity, for example by adding new Points-of-Delivery to the network (Yan et
al., 2021). This can change drastically the overall network state. In these
experiments we aim to simulate a topology upgrade. In particular, we add a new
branch to the 2-layer Clos topology considered in our experiments, which
includes an additional core switch, two spine switches, and six hosts. In
Table 2 we observe that GraphCC achieves significant improvements in terms of
FCT w.r.t. ACC (11.3%), while keeping the same throughput and considerable
queue length reduction (48.7%).

####  4.5.3. Adding extra hosts

Datacenters are periodically upgraded to increase their processing and switch
capacity, for example by adding new Points-of-Delivery to the network (Yan et
al., 2021). This can change drastically the overall network state. In this
section, we aim to evaluate how GraphCC operates when it is deployed in
network topologies with different properties. For this purpose, we
increasingly connect new hosts to leaf switches in the original network where
GraphCC was trained (with 24 hosts, i.e. 6:1 host/switch ratio). Figure 7
gathers a comparison of the FCT between GraphCC and ACC for two new scenarios
(with 32 and 40 hosts). Likewise, Table 2 shows the corresponding aggregated
mean values. The overall performance comparison w.r.t ACC –FCT, throughput and
queue length metrics– is similar to that obtained for the original topology
with 24 hosts. We observe an increment on the mean throughput –due to the
increase of hosts–, as well as a very slight behaviour difference on the p95
and p99 tails of the FCTs for small flows, if we compare it with the analogous
results in the original network (Figure 4(b)). Overall, we see that GraphCC
improves ACC a bit further on these metrics, which suggests that our solution
can effectively handle flatter networks with higher congestion levels on the
core.

(a) 32 hosts (8:1 ratio)

(b) 40 hosts (10:1 ratio)

Figure 7. Evaluation of GraphCC against ACC on flatter networks (_FB_Hadoop_
workload).

##  5\. Related Work

CC has been largely studied in the past, with a rich body of proposals
especially focused on high-speed DCNs. This section comprises an overview of
relevant works related to GraphCC.

Advanced CC mechanisms for DCNs: Some recent pioneering works have proposed
sophisticated CC mechanisms showing outstanding performance in DCNs (Alizadeh
et al., 2010; Zhu et al., 2015; Mittal et al., 2015; Montazeri et al., 2018;
Li et al., 2019; Kumar et al., 2020). For example, HPCC (Li et al., 2019)
achieves remarkably short FCTs while offering good throughput and stability to
traffic changes (e.g., incast events). To this end, it leverages accurate
fine-grained measurements produced by modern In-band Network Telemetry (INT)
mechanisms (Tan et al., 2021; Ben Basat et al., 2020). TIMELY (Mittal et al.,
2015) and Swift (Kumar et al., 2020) rely on accurate delay measurements on
NICs to control flow rates. Other well-known works, such as pHost (Gao et al.,
2015) or Homa (Montazeri et al., 2018) are credit-based solutions, where
receivers control the flow rate by sending credit packets to senders. All the
previous CC mechanisms rely on novel network architectures and/or protocol
stacks that unfortunately are not supported by most legacy switches deployed
in DCNs nowadays. At the time of this writing, the most widely deployed CC
standards are DCTCP (Alizadeh et al., 2010) in networks running the TCP/IP
stack, and DCQCN (Zhu et al., 2015) in RDMA-based networks. Both are ECN-based
mechanisms, where switches mark packets in case they detect congestion (i.e.,
queue length above certain thresholds). GraphCC is designed to inter-operate
with traditional ECN-based mechanisms in an efficient and distributed way.
Although in this paper we test GraphCC only in RDMA networks with DCQCN (Zhu
et al., 2015), our solution can be easily adapted to optimize any other ECN-
based CC mechanism, such as the aforementioned DCTCP (Alizadeh et al., 2010),
or TCP-Bolt (Stephens et al., 2014)).

Machine Learning-based mechanisms for CC: Recent works posit the use of ML
techniques to produce data-driven solutions that can efficiently adapt to the
network dynamics. Among the most popular solutions, works such as Aurora (Jay
et al., 2019), or Orca (Abbasloo et al., 2020), propose to use RL to adapt
flow rates at end-hosts. These solutions are focused on adapting the flow rate
at end-hosts according to the congestion feedback received, and they require
to re-implement the network stack. Likewise, AuTO (Chen et al., 2018) proposes
a novel two-level mechanism that accurately controls routing for long flows
and queue scheduling for optimizing the latency of short flows. Instead,
GraphCC is focused on distributed in-network optimization of widely deployed
ECN-based CC mechanisms. The closest work to GraphCC is arguably ACC (Yan et
al., 2021), where the authors propose a MARL-based mechanism to optimize ECN-
based schemes. However, that work — and all previous ones — contemplate online
training to dynamically learn how to adapt to the network state. As a result,
these ML-based solutions may suffer from transient performance degradations
when changes occur in the network (e.g., new traffic workload, incast events,
failures). Also, online training entails an intrinsic uncertainty on what
would be the resulting performance after re-training. This would require
strong supervision mechanisms to check the evolution of agents and be ready to
deploy backup mechanisms.

On the other hand, GraphCC naturally exhibits high robustness to generalize
across traffic and topology changes unseen during the training phase. This is
thanks to the GNN architecture that it internally implements, which naturally
induces a cooperation mechanism among agents —via message exchanges between
neighbours– that contrasts with the greedy behaviour of ACC agents —which act
based on its local measurements, as no contextual data is provided to them. On
top of that, our GNN design implements _parameter sharing_ to finally produce
a general agent implementation jointly learned from the individual perspective
of each node in the network. This property can be especially interesting from
a practical standpoint, as it enables to train the solution offline (e.g., in
a controlled testbed), and then be able to deploy it directly in production
networks, without the need for (re)training it on premises. Also, it permits
extensive testing prior to deployment, giving vendors the possibility to issue
certifications with the safe operational ranges that the solution would safely
support once deployed (e.g., link capacities, max. network size). This process
would be better aligned with the standard way network products are
commercialized nowadays.

##  6\. Conclusion

This paper has introduced GraphCC, a distributed solution for in-network CC
optimization in DCNs. GraphCC is compatible with any network running widely
deployed ECN-based CC mechanisms, such as DCQCN, or DCTCP. In GraphCC, agents
are deployed in switches; they cooperate and exchange information to
dynamically adapt the ECN configuration and optimize the global flow-level
performance. In our evaluation, we have benchmarked GraphCC against two
baselines: $(i)$ a static ECN configuration used in Alibaba’s production DCNs,
and $(ii)$ ACC, a state-of-the-art ML-based solution for dynamic ECN tuning.
The experimental results show that our solution significantly outperforms the
two previous baselines in terms of Flow Completion Time. At the same time, we
have observed that our solution learns to keep small queue lengths (up to
85.7% w.r.t. ACC), thus being especially interesting for next-generation DCNs,
where buffer size is expected to continue shrinking w.r.t. the skyrocketing
switch capacities.

As discussed earlier, an important feature of ML-based solutions for
networking is their capability to generalize to different scenarios to those
seen during training, as it avoids the need for online training. However,
existing ML-based CC optimization solutions are designed to be trained online
and gradually learn how to adapt to the current network conditions. In this
vein, GraphCC exhibits good behavior when operating in new scenarios never
seen during training, such as shifts on the traffic workload, or topology
changes. This is thanks to its internal GNN-based framework, which leverages
two main features: $(i)$ during training GraphCC produces a single agent
implementation jointly learned from the individual perspective of each agent
in the network, using _parameter sharing_ , and $(ii)$ agents run a topology-
aware message passing mechanism to get local context and cooperate with each
other. As a result, GraphCC produces more robust and general agent
implementations that can successfully operate on significantly different
network scenarios to those seen during training.

Acknowledgments: This publication is part of the Spanish I+D+i project
TRAINER-A (ref. PID2020-118011GB-C21), funded by MCIN/ AEI/10.13039
/501100011033. This work is also partially funded by the Catalan Institution
for Research and Advanced Studies (ICREA), the Secretariat for Universities
and Research of the Ministry of Business and Knowledge of the Government of
Catalonia, and the European Social Fund.

## References

  * (1)
  * Abbasloo et al. (2020) Soheil Abbasloo, Chen-Yu Yen, and H Jonathan Chao. 2020\.  Classic meets modern: A pragmatic learning-based congestion control for the Internet. In _ACM SIGCOMM_.
  * Alizadeh et al. (2014) Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, et al. 2014\.  CONGA: Distributed congestion-aware load balancing for datacenters. In _ACM SIGCOMM_. 503–514.
  * Alizadeh et al. (2010) Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2010.  Data center TCP (DCTCP). In _ACM SIGCOMM_.
  * Alizadeh et al. (2013) Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013.  pFabric: Minimal near-optimal datacenter transport.  _ACM SIGCOMM_ (2013).
  * Bai et al. (2015) Wei Bai, Li Chen, Kai Chen, Dongsu Han, Chen Tian, and Hao Wang. 2015\.  Information-agnostic flow scheduling for commodity data centers. In _USENIX Symposium on Networked Systems Design and Implementation (NSDI)_. 455–468.
  * Battaglia et al. (2018) Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018\.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_ (2018).
  * Ben Basat et al. (2020) Ran Ben Basat, Sivaramakrishnan Ramanathan, Yuliang Li, Gianni Antichi, Minian Yu, and Michael Mitzenmacher. 2020.  Pint: Probabilistic in-band network telemetry. In _ACM SIGCOMM_.
  * Bertsekas and Tsitsiklis (1996) Dimitri P Bertsekas and John N Tsitsiklis. 1996.  _Neuro-dynamic programming_. Vol. 5.  Athena Scientific Belmont, MA.
  * Broadcom (2022a) Broadcom. 2022a.  Broadcom Product News Releases.  https://www.broadcom.com/company/news/product-releases.  Accessed: 2022-28-01.
  * Broadcom (2022b) Broadcom. 2022b.  Broadcom StrataXGS Switch Solutions.  https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs.  Accessed: 2022-28-01.
  * Chen et al. (2018) Li Chen, Justinas Lingys, Kai Chen, and Feng Liu. 2018\.  AuTO: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization. In _ACM SIGCOMM_.
  * Cho et al. (2017) Inho Cho, Keon Jang, and Dongsu Han. 2017.  Credit-scheduled delay-bounded congestion control for datacenters. In _ACM SIGCOMM_. 239–252.
  * Dukkipati and McKeown (2006) Nandita Dukkipati and Nick McKeown. 2006.  Why flow-completion time is the right metric for congestion control.  _ACM SIGCOMM Computer Communication Review_ 36, 1 (2006), 59–62.
  * Floyd and Jacobson (1993) Sally Floyd and Van Jacobson. 1993.  Random early detection gateways for congestion avoidance.  _IEEE/ACM Transactions on networking_ 1, 4 (1993), 397–413.
  * Foerster (2018) Jakob N Foerster. 2018\.  _Deep multi-agent reinforcement learning_.  Ph. D. Dissertation. University of Oxford.
  * Gao et al. (2015) Peter X Gao, Akshay Narayan, Gautam Kumar, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2015\.  pHost: Distributed near-optimal datacenter transport over commodity network fabric. In _ACM CoNEXT_.
  * Gill et al. (2011) Phillipa Gill, Navendu Jain, and Nachiappan Nagappan. 2011\.  Understanding network failures in data centers: measurement, analysis, and implications. In _ACM SIGCOMM_. 350–361.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017\.  Neural message passing for quantum chemistry. In _International Conference on Machine Learning (ICML) \- Volume 70_. 1263–1272.
  * Goyal et al. (2022) Prateesh Goyal, Preey Shah, Kevin Zhao, Georgios Nikolaidis, Mohammad Alizadeh, and Thomas E Anderson. 2022.  Backpressure flow control. In _USENIX Symposium on Networked Systems Design and Implementation (NSDI)_. 779–805.
  * Hasselt (2010) Hado V Hasselt. 2010\.  Double Q-learning. In _Advances in Neural Information Processing Systems_. 2613–2621.
  * Intel (2022) Intel. 2022.  Intel Tofino series Programmable Ethernet Switch ASIC.  https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch/tofino-series.html.  Accessed: 2022-28-01.
  * Jay et al. (2019) Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar. 2019.  A deep reinforcement learning perspective on internet congestion control. In _International Conference on Machine Learning_. 3050–3059.
  * Kumar et al. (2020) Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan MG Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, et al. 2020\.  Swift: Delay is simple and effective for congestion control in the datacenter. 514–528.
  * Li et al. (2021) Weihe Li, Jingling Liu, Shiqi Wang, Tao Zhang, Shaojun Zou, Jinbin Hu, Wanchun Jiang, and Jiawei Huang. 2021\.  Survey on Traffic Management in Data Center Network: From Link Layer to Application Layer.  _IEEE Access_ 9 (2021), 38427–38456.
  * Li et al. (2019) Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019\.  HPCC: High precision congestion control.  In _ACM SIGCOMM_. 44–58.
  * Liu et al. (2013) Vincent Liu, Daniel Halperin, Arvind Krishnamurthy, and Thomas Anderson. 2013.  F10: A fault-tolerant engineered network. In _USENIX Symposium on Networked Systems Design and Implementation (NSDI)_. 399–412.
  * Mittal et al. (2015) Radhika Mittal, Vinh The Lam, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, and David Zats. 2015\.  TIMELY: RTT-based congestion control for the datacenter.  _ACM SIGCOMM_ (2015).
  * Mittal et al. (2018) Radhika Mittal, Alexander Shpiner, Aurojit Panda, Eitan Zahavi, Arvind Krishnamurthy, Sylvia Ratnasamy, and Scott Shenker. 2018.  Revisiting network support for RDMA. In _ACM SIGCOMM_. 313–326.
  * Montazeri et al. (2018) Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018.  Homa: A receiver-driven low-latency transport protocol using network priorities. In _ACM SIGCOMM_. 221–235.
  * Oliehoek et al. (2008) Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. 2008.  Optimal and Approximate Q-Value Functions for Decentralized POMDPs.  _J. Artif. Int. Res._ 32, 1 (May 2008), 289–353.
  * PFC (2021) PFC. 2021.  IEEE 802.11Qbb. Priority based flow control.  2011\.
  * Ramakrishnan et al. (2001) K Ramakrishnan, Sally Floyd, and D Black. 2001.  RFC3168: The addition of explicit congestion notification (ECN) to IP.
  * Riley and Henderson (2010) George F Riley and Thomas R Henderson. 2010.  The ns-3 network simulator.  In _Modeling and tools for network simulation_. 15–34.
  * Roy et al. (2015) Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015.  Inside the social network’s (datacenter) network. In _ACM SIGCOMM_.
  * Ruiz et al. (2020) Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. 2020.  Graph neural networks and the transferability of graph neural networks.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008\.  The graph neural network model.  _IEEE Transactions on Neural Networks_ 20, 1 (2008), 61–80.
  * Schaul et al. (2015) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015\.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ (2015).
  * Shan and Ren (2018) Danfeng Shan and Fengyuan Ren. 2018.  ECN marking with micro-burst traffic: Problem, analysis, and improvement.  _IEEE/ACM Transactions on Networking_ 26, 4 (2018), 1533–1546.
  * Singh et al. (2015) Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon, Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, et al. 2015\.  Jupiter rising: A decade of clos topologies and centralized control in google’s datacenter network.  _ACM SIGCOMM_ (2015).
  * Stephens et al. (2014) Brent Stephens, Alan L Cox, Ankit Singla, John Carter, Colin Dixon, and Wesley Felter. 2014\.  Practical DCB for improved data center networks. In _IEEE INFOCOM_.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Tan et al. (2021) Lizhuang Tan, Wei Su, Wei Zhang, Jianhui Lv, Zhenyi Zhang, Jingying Miao, Xiaoxi Liu, and Na Li. 2021\.  In-band network telemetry: a survey.  _Computer Networks_ 186 (2021), 107763.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020\.  A comprehensive survey on graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ (2020).
  * Yan et al. (2021) Siyu Yan, Xiaoliang Wang, Xiaolong Zheng, Yinben Xia, Derui Liu, and Weishan Deng. 2021\.  ACC: Automatic ECN tuning for high-speed datacenter networks. In _ACM SIGCOMM_.
  * Zhang et al. (2019) Junxue Zhang, Wei Bai, and Kai Chen. 2019.  Enabling ECN for datacenter networks with RTT variations. In _ACM CoNEXT_. 233–245.
  * Zhou et al. (2014) Junlan Zhou, Malveeka Tewari, Min Zhu, Abdul Kabbani, Leon Poutievski, Arjun Singh, and Amin Vahdat. 2014.  WCMP: Weighted cost multipathing for improved fairness in data centers. In _European Conference on Computer Systems_. 1–14.
  * Zhu et al. (2015) Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015\.  Congestion control for large-scale RDMA deployments.  _ACM SIGCOMM_ (2015).