# TCP Slice: A semi-distributed TCP algorithm for Delay-constrained
Applications

Dibbendu Roy  Department of Electrical Engineering
IIT Indore
<EMAIL_ADDRESS>Goutam Das  G.S. Sanyal School of Telecommunications
IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>
###### Abstract

The TCP congestion control protocol serves as the cornerstone of reliable
internet communication. However, as new applications require more specific
guarantees regarding data rate and delay, network management must adapt. Thus,
service providers are shifting from decentralized to centralized control of
the network using a software-defined network controller (SDN). The SDN
classifies applications and allocates logically separate resources called
“slices”, over the physical network. We propose “TCP Slice”, a congestion
control algorithm that meets specific delay and bandwidth guarantees.
Obtaining closed-form delay bounds for a client is challenging due to
dependencies on other clients and their traffic stochasticity. We use network
calculus to derive the client’s delay bound and incorporate it as a constraint
in the Network Utility Maximization problem. We solve the resulting
optimization using dual decomposition and obtain a semi-distributed TCP
protocol that can be implemented with the help of SDN controller and the use
of Explicit Congestion Notification (ECN) bit. Additionally, we also propose a
proactive approach for congestion control using digital twin. TCP Slice
represents a significant step towards accommodating evolving internet traffic
patterns and the need for better network management in the face of increasing
application diversity.

###### Index Terms:

TCP, Network slicing, SDN, Network calculus, semi-distributed, congestion
control

##  I Introduction

The TCP congestion control protocol is certainly the backbone of reliable
internet communication and has been the de facto standard ever since its
inception in the 1980s [1]. The success of TCP can be attributed to its
capability of being deployed in a distributed manner without necessitating
complete knowledge of the network state. Thus, each client can send data over
TCP without being concerned about the network’s other clients, leading to a
highly scalable system. However, over the years, the nature of internet
traffic has changed significantly due to increased accessibility and the
introduction of new applications and services [2]. The days of being satisfied
with basic data delivery assurances are long gone. As an example, Table I
shows the diverse quality of service (QoS) requirements of 5G use cases [3].
It is anticipated that this diversity will continue to expand with the
emergence of newer application scenarios, and future networks, including 6G
[4, 5], and therefore the future internet will be required to accommodate
these diverse demands.

TABLE I: Application, Slice and Requirement Mapping in 5G [3] Use Cases |  Slice |  QoS
---|---|---
Augmented and Virtual reality |  eMBB |  $\text{Downstream}\geq 100\text{Mbps},\text{Uplink}\geq 50\text{Mbps},\text{Latency}\leq 4\text{ms}$
Autonomous Cars, Remote Surgery |  uRLLC |  $\text{Latency}\leq 1\text{ms},\text{Reliability}=99.999\%$
IoT, Smart Factories, Platooning |  mMTC |  $\text{Connection Density}=1\times 10^{6}\text{devices/Km}^{2}$

To address the aforementioned needs, better management of networks is
necessary. Thus, a more centralized control of the network through software-
defined networks (SDN) has gained popularity [6, 7] (see Figure1).
Specifically, to meet the various QoS requirements of emerging new
applications, the notion of network slicing is introduced [8]. The idea behind
network slicing is to divide a single physical network infrastructure into
multiple virtual networks, each tailored to serve different use cases with
distinct performance, security, and resource allocation requirements. Each
network slice operates independently and can be managed and optimized
separately. Evidently, the implementation of slices requires global network
state information, making SDN a key enabler for the same. The SDN can run
network operations such as routing, network address translation, and network
slice management by implementing network functions on top of its control plane
(known as Network function virtualization - NFV, see Figure1). As shown in
Table I, based on QoS requirements, 5G defines three main categories of
slices, enhanced Mobile Broadband (eMBB), ultra Reliable Low Latency
Communications (uRLLC), and massive Machine Type Communications (mMTC).
Keeping these developments in mind, it is important to look into the
possibilities of extending the capabilities of traditional TCP algorithms.

Figure 1: SDN-based network with proposed TCP Slice as a Network Slice
Management Function (NSMF) implemented with Network Function Virtualization
(NFV).

###  I-A Related Works and Challenges

TCP, being the fundamental technology for moving internet traffic, is a well-
researched topic with vast literature [1, 9, 10, 11, 12, 13]. In this paper,
we propose am analytical TCP algorithm that specifically meets delay
guarantees which can be employed for both eMBB and uRLLC slice use cases. We
resort to the optimization-based approach for congestion control [14, 15, 16]
that maximizes the total network utility of all users/clients in the network
subject to network capacity constraints. The conventional method to yield a
distributed control algorithm using such an optimization formulation is to use
the dual decomposition technique [17, 18]. Since the total utility and
capacity constraints turn out to be easily separable in the Lagrangian form,
the resulting algorithm is simple and exhibits desirable properties. Further,
it could be shown that for certain choices of utility functions and
parameters, the algorithm follows the widely popular additive increase
multiplicative decrease (AIMD) [15] scheme. Variants such as TCP Reno [1], TCP
Vegas [10] etc., can also be analyzed using the mentioned framework [16].

Although there have been attempts to minimize delay in TCP [19], to the best
of our knowledge, none of them can guarantee delay bounds, specifically due to
the following reason: It is evident that the end-to-end delay experienced by a
client is influenced not only by its own sending rate but also by the sending
rates of other clients sharing network resources. Moreover, the stochastic
nature of queuing delay in the network poses challenges in providing strict
delay guarantees. Due to these complexities, attempting to incorporate delay
constraints directly into the optimization framework using dual decomposition
techniques may not be feasible.

To address this issue, we propose to use Network Calculus which offers a
suitable toolset to analyze the worst-case bounds on delays and buffer
requirements in a network [20]. By incorporating the derived delay bounds as
constraints in the optimization problem, we can better manage the allocation
of network resources to meet the desired delay requirements for each client.

Although Network calculus helps in computing delay bounds, it turns out that
such computations require knowledge of global network parameters that are not
available at the client site. One may then use an SDN controller to obtain
network parameters. However, the controller can only manage the
switches/routers in its domain and does not have access to influence sending
rates on the client side. Hence, feasible implementation of the derived
algorithm requires the design of a suitable mechanism such that requisite
information is disseminated without introducing overheads. We propose to use
Explicit Congestion Notification (ECN) which is already a part of the TCP
header.

###  I-B Contributions

Based on the delineated challenges, we present the contributions of the paper.

  * •

We develop a model based on network calculus that helps in characterizing the
delay bound of a TCP client/source. To the best of our knowledge, this is the
first proposal that incorporates the use of network calculus to compute delay
bounds and derives an analytical TCP algorithm using distributed optimization
techniques.

  * •

Using the derived bound, we formulate a Network Utility Maximization (NUM)
problem with the objective of maximizing the network utility subject to
network capacity and delay bound requirements. A dual descent-based solution
method is developed which requires the knowledge of global network parameters
for implementation. We state convergence results and also show how to account
for packetization effects in networks.

  * •

We propose a semi-distributed TCP implementation with the help of SDN and
OpenFlow switches by marking the ECN bit. We call our algorithm ”TCP Slice”
since it is dedicated to being used for slices with delay constraints and
their management.

  * •

We present the steady state and transient performance of TCP Slice and present
some important discussions in contrast to fairness-based schemes which are the
state of art in the literature. Additionally, we discuss the possibility of a
proactive approach without spending any time in the transients that cause
significant delays.

The rest of the paper is organized as follows: Section II describes the system
model and the problem definition. As described, we append an additional
constraint to restrict the delay bound. The relevant background for deriving
the proposed bounds is provided in Section III-A, followed by our proposed
bounds in Section III-B. Thereafter, we reformulate and present the final
problem forms in Section IV. Our derived algorithm is proposed and its
implementation steps are presented in Section V-B followed by relevant results
and discussions in Section VI. Finally, we make some important concluding
remarks with future directions in Section VII.

##  II Model Description and Problem Statement

We consider a network modeled as a Graph with $G=(V,E)$, where $V$ denotes the
set of vertices and $E(\subseteq V^{2})$ denotes the set of edges. Some of
these vertices are associated with end devices like a mobile, a PC, AR/VR
headset/equipment, etc., that run applications. We denote these vertices as
the source vertices $V_{S}\subset V$. For each edge $e$, let $S(e)$ denote the
set of sources that use the link $e$. Similarly, let $E(s)$ denote the set of
edges or links used by the source $s$. Let $x_{s}$ denote the transmission
rate for source $s$ and $U_{s}(x_{s})$ denotes the utility obtained by the
source by transmitting at the rate $x_{s}$. Let $D_{s}(x_{s})$ denote the
delay experienced by the source $s$. Let $x_{s}\in I_{s}=[m_{s},M_{s}]$ which
denotes the range of feasible rates that the source can transmit. Note that
one can treat $m_{s}$ as a minimum bandwidth constraint on source rate
$x_{s}$. Hence, consideration of minimum bandwidth constraints is implicit in
the model. Vector notations are written in bold, sets are denoted in capitals
and we use $|A|$ to denote the cardinality or size of the set $A$. The
subscript $s$ is used for a source $s$ while $e$ for an edge. The frequently
used notations in the paper are summarized in Table II.

TABLE II: Notations Notation |  Description
---|---
$s$ |  A source/client $s\in V_{S}$
$x_{s}$ |  Rate of source $s$
$U_{s}(x_{s})$ |  Utility gained by source $s$ by sending at rate $x_{s}$
$n$ |  Number of sources $=|V_{S}|$
$\bm{x}$ |  A vector of source rates $(x_{1},x_{2},\dots,x_{n})$
$\bm{x}_{-s}$ |  Vector of rates except rate of source $s$
$D_{s}(\bm{x})$ |  Delay experienced by source $s$ due to $x_{s}$ and $\bm{x}_{-s}$
$d_{s}$ |  Delay bound to be satisfied for source $s$
$c_{e}$ |  Capacity of an edge $e$
$E(s)$ |  Set of edges used by source $s$
$S(e)$ |  Set of sources used by edge $e$
$C_{s}$ |  Set of capacities of edges used by $s$
$c_{s}^{m}$ |  Minimum capacity in $C_{s},~{}(c_{s}^{m}=\min C_{s})$
$\beta_{s}(t)$ |  Effective service curve for source $s$
$D_{s}^{NC}$ |  Delay bound for source $s$, calculated using network calculus
$x_{-s}$ |  Sum of rates of sources other than $s$
$l_{max}$ |  Maximum packet size
$\sigma$ |  Maximum burst size of leaky bucket
$p_{e}$ |  Price computed for a link $e$
$p_{j}$ |  Price computed for a source $j$
$p_{-s}$ |  Sum of prices of sources other than $s$
$p^{s}$ |  Sum of prices of links used by source $s$
$m_{s}^{e}$ |  Probability of marking the ECN bit

The following optimization comes into effect where the objective is to
maximize the utility subject to delay and physical network constraints:

###  II-A Primal Problem

$\displaystyle\max_{x_{s}\in I_{s}}\sum_{s}U_{s}(x_{s})$ (1a) $\displaystyle
s.t.~{}~{}D_{s}(\bm{x})=D_{s}(x_{s},\bm{x}_{-s})\leq d_{s}~{}~{}\forall s$
(1b) $\displaystyle\sum_{s\in S(e)}x_{s}\leq c_{e}~{}~{}\forall e$ (1c)

It is understood that the delay function in (1b) is dependent on both the
source rate and the rate of other sources (that are in the path of the
source). This is a complicating non-separable constraint that cannot be easily
handled by means of the dual decomposition method, which is commonly used for
distributed optimization in case of separable objectives and constraints. To
circumvent this issue, we need to find an approximate expression for the delay
bound, which would help in obtaining a separable solution. To this purpose, we
intend to use network calculus which is a well-known tool for obtaining delay-
bound expressions in networks. However, this treatment requires some
background, and the relevant results are presented in the section below.

##  III Network Calculus

We will present the relevant results in the theory which helps us to obtain an
expression for the delay bound in a network.

###  III-A Background

Evidently, ensuring delay and backlog bounds require restrictions on the
arrival and service processes. Consider the description of the generated
traffic at a node by the cumulative process $A(t)$ which provides the total
number of bits/bytes/packets arriving at the node until time $t$. Typically,
the arrivals are restricted using arrival curves with the following
definition.

######  Definition 1 (Arrival Curve [20, 21]).

An cumulative arrival process $A(t)$ is said to have an arrival curve
$\alpha(t)$ iff $A(t)-A(s)\leq\alpha(t-s),~{}\forall~{}0\leq s\leq
t\Leftrightarrow A(t)\leq\inf_{s\in[0,t]}[A(s)+\alpha(t-s)]$

From the definition of an arrival curve, it is clear that an arrival curve
restricts the number of packets generated in a given interval. A typical and
practical example of an arrival curve is the output of a leaky-bucket
implementation at the source. If a source implements a leaky bucket to the
arriving traffic, that operates at the rate of $x$ and bucket length (or
buffer length) $\sigma$, the arrival curve is given by :

$\alpha(t)=\sigma+xt$

A leaky or token bucket operates in the following way: The arriving packets
are stored in a buffer (theoretically of infinite size). The bucket is a
separate buffer with capacity $\sigma$, in which tokens are generated at rate
$x$. Each packet that arrives at a leaky bucket, finds a token in the token
bucket, takes the token, and leaves the bucket immediately. Since the bucket
size is limited by $\sigma$ and the number of tokens that are generated in an
interval of $(t-s)$ is $x(t-s)$, the maximum number of packets that can go out
of this system in an interval of $(t-s)$ is $\sigma+x(t-s)$. Hence,
$\alpha(t)=\sigma+xt$. Often this arrival curve is termed to be
$(\sigma,x)$-upper constrained.

######  Lemma 1 (Aggregate Multiplexing [20, 21]).

If two flows $A_{1}$ and $A_{2}$ are aggregated at a node, and $\alpha_{1}$,
$\alpha_{2}$ are their arrival curves, then $\alpha_{1}+\alpha_{2}$ is an
arrival curve of the aggregated flow $A_{1}+A_{2}$.

Similar to restricting the arrivals, the service process can also be defined
in terms of service curves. If $t_{0}$ denotes the last time instant from $t$
when the server becomes busy (or backlog starts), we must have
$D(t)-D(t_{0})=c(t-t_{0})$. By definition of backlog, it must be that
$D(t_{0})=A(t_{0})$ and hence $D(t)-A(t_{0})=c(t-t_{0})\Rightarrow
D(t)=A(t_{0})+c(t-t_{0})\geq\inf_{s\in[0,t]}[A(s)+c(t-s)]$.

######  Definition 2 (Service Curve [20]).

If $B(t)$ be the cumulative departure and $A(t)$ the cumulative arrival, then
$\beta(t)$ is said to be a service curve iff
$D(t)\geq\inf_{s\in[0,t]}A(s)+\beta(t-s),~{}\forall~{}t$

######  Definition 3 (Strict Service Curve [20]).

We say that system S offers a strict service curve $\beta$ to a flow if,
during any backlogged period of duration $u$, the output of the flow is at
least equal to $\beta(u)$. Every strict service curve is a service curve.

The service curve ensures that a minimum number of bits/bytes/packets are
served in an interval. For a work conserving (server is in operation whenever
the buffer is non-empty) constant rate server or a link with serving rate $c$,
the strict service curve is $\beta(t)=ct$. In network calculus, often we
represent these types of service curves with a generalized form of

$\beta_{R,T}(t)=R[t-T]^{+}$

where $[x]^{+}=\max\\{0,x\\}$. These are also termed as rate-latency servers.

######  Definition 4 (Min-Plus Convolution ($\otimes$) [20]).

We observe that both arrival and service curves involve calculating
$\inf_{s\in[0,t]}[A(s)+h(t-s)]$. Analogous to convolution in filtering theory,
this operation can also be perceived as convolution, where the integral or sum
is replaced by an infimum (minimum), and the product is replaced by addition
or plus. Thus, this operation is termed as min-plus convolution denoted by the
operator $\otimes$.

$A\otimes h=\inf_{s\in[0,t]}[A(s)+h(t-s)]$

We present a result termed as concatenation theorem which simply states that
the concatenation of network elements (like servers) leads to the convolution
of their service curves similar to cascading of filters in signal processing.

######  Theorem 1 (Concatenation of Nodes [20]).

Assume a flow traverses systems $S_{1}$ and $S_{2}$ in sequence. Assume that
$S_{1}$ offers a service curve of $\beta_{i}$, $i=1,2$ to the flow. Then the
concatenation of the two systems offers a service curve of
$\beta_{1}\otimes\beta_{2}$ to the flow.

In addition to the mentioned theorem, we also present a result related to
finding the convolution of piecewise linear convex functions.

######  Lemma 2 (Convolution for Piecewise Linear Convex Functions [20]).

If $f$ and $g$ are convex and piecewise linear $f\otimes g$ is obtained by
putting end-to-end the different linear pieces of $f$ and $g$, sorted by
increasing slopes.

######  Corollary 2.1 (Concatenation of Rate-latency servers [20]).

If $\beta_{1}(t)=\beta_{R_{1},T_{1}}(t)=R_{1}[t-T_{1}]^{+}$ and
$\beta_{2}(t)=\beta_{R_{2},T_{2}}(t)=R_{2}[t-T_{2}]^{+}$, then
$\beta_{1}(t)\otimes\beta_{2}(t)$ is given by:

$\displaystyle\begin{split}\beta_{1}\otimes\beta_{2}&=\min\\{R_{1},R_{2}\\}\left[t-(T_{1}+T_{2})\right]^{+}\\\
&=\beta_{\min\\{R_{1},R_{2}\\},(T_{1}+T_{2})}(t)\end{split}$

The proof is a straightforward application of Lemma 2. The result can be
easily generalized to concatenate $n$ such rate-latency servers. Our objective
is to find a worst-case delay bound for a source in a network.

Assumption: Each source implements a token/leaky bucket to ensure an arrival
curve.111This is a fair assumption since TCP flow control is implemented with
the help of leaky buckets and traffic shaping is an integral part of rate
control. Further, while admitting slices, usually there are service level
agreements that have information regarding traffic characteristics. Once the
characteristic is known, it is customary to design a flow controller that
allows the smooth operation of networks.

This assumption allows us to use deterministic network calculus to compute
delay bounds using the following theorems [20].

######  Theorem 2 (Blind Multiplexing [20]).

Consider a node serving two flows, 1 and 2, with some unknown arbitration
(scheduling policy) between the two flows. Assume that the node guarantees a
strict service curve $\beta$ to the aggregate of the two flows. Assume that
$\alpha_{2}$ is an arrival curve for flow 2. Define
$\beta_{1}(t):=[\beta(t)-\alpha_{2}(t)]^{+}$ . If $\beta_{1}$ is wide-sense
increasing, then it is a service curve for flow 1.

If $\beta(t)=ct$, and $\alpha_{2}(t)=\sigma_{2}+x_{2}t$, we have,
$\beta_{1}(t)=[ct-\sigma_{2}-x_{2}t]^{+}=(c-x_{2})\left[t-\frac{\sigma_{2}}{c-x_{2}}\right]^{+}$
(assuming $c>x_{2}$ for stability). Service curves of the form
$\beta(t)=R[t-T]^{+}$ are known as rate-latency curves with rate $R$ and
latency $T$.

Typically, wired communication links are work-conserving servers implying that
the links are operational at full capacity whenever the buffer is non-empty,
or else the server is non-operational. The following theorem states that the
output of work-conserving servers are $(\sigma,x)$ constrained if the input is
so.

######  Lemma 3 ([21]).

Let $A$ and $B$ be the input and the output of a network element. Suppose that
$A$ is $(\sigma,x)$ \- upper constrained. If the network element is a work-
conserving link, then B is also $(\sigma,x)$-upper constrained.

We are now ready with the requisite theory to derive the delay bound in a
network and use it to perform the optimization (1).

###  III-B Computing Delay Bounds

Given the graph model under consideration, we intend to obtain a delay
constraint of a source. With the given graph model, the routing information is
deemed to be available and hence we can assume that at any given time, the
path used from source to destination is known. We consider that a source $s\in
S$ obeys an arrival curve $(\sigma_{s},x_{s})$. Let $C_{s}$ be the set of link
capacities used by the source $s$ to reach its destination and hence the link
capacities may be enumerated as $C_{s}=\\{c_{s}^{1},\dots,c_{s}^{|C_{s}|}\\}$.
There may be flows arriving and departing from each of these links. There may
be two ways to compute the overall delay of a source through a path. The first
way is to over-estimate the worst-case, requiring no knowledge about the exact
sources whose flows are going through a link. We assume that all links in the
path of source $s$ are subject to all flows. The overall arrival curve of the
remaining is $(\sum_{j\neq s,j\in S}\sigma_{j},\sum_{j\neq s,j\in
S}x_{j})$-upper-constrained. We may denote this as $(\sigma_{-s},x_{-s})$. By,
Theorem 2, we may compute the effective service at a link with capacity
$c_{s}^{k}$ to be
$\beta_{s}^{k}(t)=(c_{s}^{k}-x_{-s})\left[t-\frac{\sigma_{-s}}{c_{s}^{k}-x_{-s}}\right]$.
By Theorem 1, we may find the effective service curve as

$\beta_{s}(t)=\beta_{s}^{1}(t)\otimes\dots\otimes\beta_{s}^{|C_{s}|}(t)$

Considering that all sources are identical w.r.t the amount of burst it can
handle i.e. $\sigma_{s}=\sigma~{}\forall s$, we have the following proposition

######  Proposition 1.

For sources with identical token bucket size $(\sigma_{s}=\sigma~{}\forall
s)$, the effective service rate for a flow from source $s$ is given by

$\beta_{s}(t)=(c_{s}^{m}-x_{-s})\left[t-\sum_{k=1}^{|C_{s}|}\frac{(n-1)\sigma}{c_{s}^{k}-x_{-s}}\right]^{+},~{}c_{s}^{m}=\min
C_{s}$ (2)

###### Proof.

The proof is an application of Corollary 2.1. ∎

######  Proposition 2.

The worst-case delay for source $s$, $D_{s}$ can be obtained as

$D_{s}^{NC}=\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{(c_{s}^{m}-x_{-s})},\quad
c_{s}^{m}=\min C_{s}$ (3)

###### Proof.

To find the delay bound, we would compute the largest possible horizontal
distance between the arrival curve and the service curve.

$\displaystyle\sigma$
$\displaystyle=(c_{s}^{m}-x_{-s})\left[D_{s}^{NC}-\sum_{k=1}^{|C_{s}|}\frac{(n-1)\sigma}{c_{s}^{k}-x_{-s}}\right]^{+}$
(4)
$\displaystyle\begin{split}D_{s}^{NC}&=\frac{\sigma}{(c_{s}^{m}-x_{-s})}+\sum_{k=1}^{|C_{s}|}\frac{(n-1)\sigma}{c_{s}^{k}-x_{-s}}\\\
&\leq\frac{\sigma}{(c_{s}^{m}-x_{-s})}+\sum_{k=1}^{|C_{s}|}\frac{(n-1)\sigma}{c_{s}^{m}-x_{-s}}~{}(\because
c_{s}^{k}\geq c_{s}^{m},\forall k)\\\
&=\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{(c_{s}^{m}-x_{-s})}\end{split}$ (5)

∎

An alternate approach would be to have the information regarding exact flows
through the links and evaluate the effective service curve. An SDN controller
can calculate the exact effective service curve and compute the tightest
possible delay bound. However, to keep the model simple and intuitive, a
better bound can be obtained by using (3) itself.

######  Proposition 3.

A tighter delay bound for source $s$, $(D_{s}^{NC})$ can be obtained as

$D_{s}^{NC}=\frac{\left(|C_{s}|\left(\max_{k\in
E(s)}|S(k)|-1\right)+1\right)\sigma}{\min_{k\in E(s)}(c_{s}^{k}-\sum_{j\neq
s,j\in S(k)}x_{j})}$ (6)

###### Proof.

We look to replace $n$ in the numerator by the maximum number of flows that
$s$ interacts with on a link and the denominator by the minimum possible
residual capacity. Recall that $E(s)$ denotes the set of edges used by source
$s$ and $S(e)$ denotes the set of sources that use the link $e$. In the
numerator we replace $n$ (in (3)), the total number of sources by the maximum
number of sources that use a link $e\in E(s)$. This is the maximum possible
burst that the arrival curve of $s$ would face in its path. In the
denominator, we replace the overestimated $(c_{s}^{m}-x_{-s})$ by the residual
capacity of the most congested link among the links used by $s$. At any given
moment, this is the worst possible left-behind capacity for flow $x_{s}$ (the
overestimate is even worse and might not even occur in practice as we consider
all flows are mixed with others). Hence, the resulting expression will always
be an upper bound on the delay. However, this may not be the suprema and hence
is not the least upper bound. ∎

###  III-C Effect of Packetization

The Network calculus results used for obtaining the proposed distributed
algorithm hold true for a fluid-based model. However, in a network, one is
typically interested in per-packet delays. Packetization has an important
implication in terms of network calculus as the results can no longer be
directly applied and a packetizer [20] must be introduced at every server (or
link in this case). Considering a packetizer changes the service curves for
each link. As shown in [20], the service curve for a server operating at a
rate $R$ followed by a packetizer is $\beta(t)=R[t-\frac{l_{max}}{R}]^{+}$,
where the maximum length of a packet $l_{max}$. Thus, in this case,
$\beta(t)=c[t-\frac{l_{max}}{c}]^{+}$, $\alpha_{2}(t)=\sigma_{2}+x_{2}t$, we
have
$\beta_{1}(t)=\left[ct-\sigma_{2}-x_{2}t-\l_{max}\right]^{+}=(c-x_{2})\left[t-\frac{\sigma_{2}+l_{max}}{c-x_{2}}\right]^{+}$.
Thus, the effective service curve (2) changes to

$\beta_{s}(t)=(c_{s}^{m}-x_{-s})\left[t-\sum_{k=1}^{|C_{s}|}\frac{(n-1)\sigma+l_{max}}{c_{s}^{m}-x_{-s}}\right]^{+}$
(7)

where $c_{s}^{m}=\min C_{s}$. Thus, (6) should be modified as

$D_{s}^{NC}=\frac{\left(|C_{s}|\left(\max_{k\in
E(s)}|S(k)|-1\right)+1\right)\sigma+|C_{s}|l_{max}}{\min_{k\in
E(s)}(c_{s}^{k}-\sum_{j\neq s,j\in S(k)}x_{j})}$ (8)

For simplicity, we proceed with (3), however the presented ideas can be
extended for (8) as well and is used for our simulations.

##  IV Network Utility Maximization with Delay Constraints

Based on the developed model, we now concentrate on the problem of our
concern. Using (3), we may obtain a constraint for each source as follows: Let
$d_{s}$ denote a delay constraint that needs to be satisfied.

$\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{(c_{s}^{k}-x_{-s})}\leq
d_{s}\Rightarrow x_{-s}\leq\min
C_{s}-\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}$ Hence, the primal
problem changes to

###  IV-A Primal Problem

$\displaystyle\max_{x_{s}\in I_{s}}\sum_{s}U_{s}(x_{s})$ (9a) $\displaystyle
s.t.~{}~{}\sum_{j\neq s}x_{j}\leq\min
C_{s}-\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}~{}~{}\forall s$ (9b)
$\displaystyle\sum_{s\in S(e)}x_{s}\leq c_{e}~{}~{}\forall e$ (9c)

As we can observe now, by using Network calculus-based delay bound, we are
able to write (1b) in terms of the sum of source rates (9b). We are now in a
position to use dual decomposition technique in order to obtain a distributed
solution.

###  IV-B Dual Problem

The dual problem can be formulated as:

$\displaystyle\begin{split}L(\bm{x,p})&=\sum_{s}\Bigg{[}U_{s}(x_{s})-x_{s}\sum_{j\neq
s}p_{j}+p_{s}\min C_{s}-\\\ &\quad
p_{s}\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}-x_{s}\sum_{e\in
E(s)}p_{e}\Bigg{]}+\sum_{e}p_{e}c_{e}\end{split}$

where $p_{s}$ and $p_{e}$ correspond to prices or Lagrange multipliers for
each source and link. We may define,

$\begin{split}B_{s}(p_{s},p_{e})&=\max_{x_{s}\in
I_{s}}U_{s}(x_{s})-x_{s}\left(\sum_{j\neq s}p_{j}+\sum_{e\in
E(s)}p_{e}\right)\\\ &=\max_{x_{s}\in
I_{s}}U_{s}(x_{s})-x_{s}(p_{-s}+p^{s})\end{split}$ (10)

where $p_{-s}=\sum_{j\neq s}p_{j}$ and $p^{s}=\sum_{e\in E(s)}p_{e}$.
$B_{s}(p_{s},p_{e})$ can be interpreted as the effective utility of a source
which is the difference of its utility and prices associated with violation of
delay and capacity constraints. The optimal rate would try to maximize this
difference. Hence,

$\begin{split}\max_{\bm{x}}L(\bm{x},\bm{p})&=Y(\bm{p})=\sum_{s}B_{s}(p_{s},p^{s})\\\
&+\sum_{s}p_{s}\left(\min
C_{s}-\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}\right)+\sum_{e}p_{e}c_{e}\end{split}$

Thus, the dual problem is:

$\min_{p_{e},p_{s}\geq 0,~{}\forall e,s}Y(\bm{p})$ (11)

In this model, each source pays two kinds of prices $p_{-s}$ and $p^{s}$.
While $p_{-s}$ is associated with the price paid in regards to delay, $p^{s}$
is the price paid for using each link per unit bandwidth.

Assumption: The function $-U_{s}(x_{s})$ is continuous, differentiable and
strongly convex in $x_{s}$.

This assumption allows us to solve (10) uniquely. We have:

$x_{s}^{*}(\bm{p})=U_{s}^{\prime-1}(p_{-s}+p^{s})$ (12)

##  V Semi-distributed TCP Slice Algorithm

We solve the dual problem (11) using gradient projection method for source
prices and link prices respectively.

###  V-A Solution using Dual Descent

For a given price vector $\bm{p}$, each source obtains $x_{s}^{*}(\bm{p})$.

$\displaystyle p_{e}(t+1)$ $\displaystyle=\left[p_{e}(t)-\gamma\frac{\partial
Y(\bm{p}(t))}{\partial p_{e}}\right]^{+}~{}\forall e\in E$ (13) $\displaystyle
p_{s}(t+1)$ $\displaystyle=\left[p_{s}(t)-\gamma\frac{\partial
Y(\bm{p}(t))}{\partial p_{s}}\right]^{+}~{}~{}~{}\forall s\in S$ (14)

We have:

$\displaystyle Y(\bm{p})$
$\displaystyle=\sum_{s}U_{s}(x_{s}^{*})-x_{s}^{*}(p_{-s}+p^{s})+$
$\displaystyle\quad\sum_{s}p_{s}\left(\min
C_{s}-\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}\right)+\sum_{e}p_{e}c_{e}$

where $x_{s}^{*}$ is the maximizing $x_{s}$. This must satisfy:

$\frac{\partial U_{s}(x_{s}^{*})}{\partial x_{s}^{*}}-(p_{-s}+p^{s})=0$ (15)

We need to differentiate $Y$ w.r.t the price variables for each source and
link.

Differentiating $Y$ w.r.t to other source variable say $p_{j},j\neq s$

$\displaystyle\frac{\partial Y(\bm{p})}{\partial p_{j}}$
$\displaystyle=-\sum_{s\neq j}x_{s}^{*}+\left(\min
C_{j}-\frac{\left(|C_{j}|(n-1)+1\right)\sigma}{d_{j}}\right)$

Differentiating $Y$ w.r.t to link price $p_{e}$

$\displaystyle\frac{\partial Y(\bm{p})}{\partial p_{e}}$
$\displaystyle=c_{e}-\sum_{s\in S(e)}x_{s}^{*}$

Thus, we have:

$\displaystyle\begin{split}p_{j\neq
s}(t+1)&=\Bigg{[}p_{j}(t)-\gamma_{j}\bigg{(}\min C_{j}-\\\
&\quad\frac{\left(|C_{j}|(n-1)+1\right)\sigma}{d_{j}}-\sum_{s\neq
j}x_{s}^{*}\bigg{)}\Bigg{]}^{+}\end{split}$ (16) $\displaystyle p_{e}(t+1)$
$\displaystyle=\left[p_{e}(t)-\gamma_{e}\left(c_{e}-\sum_{s\in
S(e)}x_{s}^{*}\right)\right]^{+}$ (17)

Each source computes $x_{s}(\bm{p})$ for a given $\bm{p}$ using (12). Links
$e$ will update their prices $p_{e}$ following (17). Then, sources should
update the prices $p_{j}$ using (16). Note that in (16), each time the price
is decremented by $\gamma_{j}$ times the difference of the denominator and
numerator of (3). The same principle is followed while implementing (8).

From (16), it is evident that each source requires the knowledge of the
minimum capacity edges of other sources and also the sum rates of other users,
to compute its own rate. However, this information is not readily available to
a source node. Thus, a mechanism has to be developed that facilitates a source
node to update its prices. To facilitate this, a random exponential marking
(REM) scheme is designed similar to the one described in [22]. Although the
queues/buffers on the edges can compute $p_{e}(t)$ independently and implement
(17) as described in [22], now the same must be aided with additional
adjustments so that (16) is not neglected. We propose to perform this with
help of SDN controller, which controls the intermediate openflow switches.

###  V-B Implementation with SDN controller

We now devise a scheme that may be implemented with help of an SDN controller
that has global information regarding the entire network and routing states.
Specifically, we propose SDN to compute (16) and use the computed prices to
update the marking probability at the OpenFlow switches.

For a given source $s$, it needs to update its rate according to (12) which
requires computing $p_{-s}+p^{s}$. To do this, we impose an exponential
marking probability distribution for each edge by which a packet from source
$s$ traverses. We define $m_{s}^{e}(t)$ to be the probability of marking a
packet at edge $e$. Then, the probability of a packet being marked after
traveling through a path (say for source $s$) can be computed by considering
the probability that the packet is marked in any one of the links used by
source $s$. The marking information is fed back to the source through the TCP
acknowledgment facility which should allow the source to compute (12). Hence
we want the end-to-end marking probability to capture the sum $p_{-s}+p^{s}$.
This leads to the following equations

$\displaystyle 1-\prod_{e\in
E(s)}\left(1-m_{s}^{e}(t)\right)=1-e^{-\left(p_{-s}+p^{s}\right)}$
$\displaystyle\Rightarrow m_{s}^{e}(t)=1-e^{-\left(\frac{\sum\limits_{j\neq
s}p_{j}}{|E(s)|}+p_{e}\right)}$ (18)

We present our semi-distributed algorithm. We break our algorithm into two
parts. A source starts sending packets in the network with its minimum
capacity $m_{s}$. Once the packets reach SDN switches, they communicate with
the SDN regarding flow setup. Since we are concerned with slices, the a
network function named network slice management function is run at the SDN to
monitor the slices. We propose that TCP slice can be a part of this function.
Over time, the switches collect statistics regarding the data rate of users
and send them to the SDN for management purposes. The SDN then computes the
prices $p_{j}$ for each source using (16) and sends it to the switches for
marking the ECN bit. The price $p_{e}$ (17) can either be computed at the
switches or; the SDN, being a central node, can also compute the same. The
probability of marking a packet from source $s$ is computed using (18) and
switches use this probability to set the ECN bit. Algorithm 2 describes these
operations. On receiving the packets, the source can keep computing the
probability of marked packets and compute the price by an inverse
transformation. Once the prices are computed by the source, it uses (12) to
find its rate. These steps are demonstrated by Algorithm 1.

Input : $x_{s}(t)$

Output : $x_{s}(t+1)$

Compute Probability of marked packets using received ECN bit

$P_{M}(t)=\frac{\text{\\# marked packets received till t+1}}{\text{\\# packets
received till t+1}}$;

Compute the sum of link prices $p_{-s}(t)+p^{s}(t)=-\ln{(1-P_{M}(t))}$;

Compute the source rate
$x_{s}(t+1)=U_{s}^{{}^{\prime}-1}\left[-\ln{\left(1-P_{M}(t)\right)}\right]$;

Algorithm 1 Algorithm to calculate rates for a source $s$

Input : $p_{e}(t)$, $p_{j}(t)$ and $x_{s}(t)$

Output : $p_{e}(t+1)$ and $p_{j}(t+1)$

@SDN (Update Source prices): $p_{j}(t+1)=\left[p_{j}(t)-\gamma_{j}\left(\min
C_{j}-\frac{\left(|C_{s}|(n-1)+1\right)\sigma}{d_{s}}-\sum_{s\neq
j}x_{s}(t)\right)\right]^{+}$;

@Links/Edges (Update Edge prices):
$p_{e}(t+1)=\left[p_{e}(t)-\gamma_{e}\left(c_{e}-\sum_{s\in
S(e)}x_{s}(t)\right)\right]^{+}$;

@Switches:  Sends estimated data rates of users to SDN;

Gets the updated source and edge prices for a source;

Marks packets with probability $m_{s}^{e}(t)$ computed using (18)

Algorithm 2 Algorithm to calculate prices $p_{j}$ and $p_{e}$

###  V-C Convergence

It is easy to see that the primal problem (9) is convex (considering
minimization of $-U(.)$ as the objective), with linear constraints. In [18,
17], it is shown that under the assumptions of strong convexity of the
objective, the dual ascent algorithm (or descent in our case of maximization),
converges linearly i.e. $|f(x)-f(x^{*})|\leq\epsilon$ in $O(\log(1/\epsilon))$
iterations.

##  VI Results and Discussions

Figure 2: Simulation Setup in OMNeT++. Two hosts are connected to a bottleneck
link with 100 Mbps datarate while one host is connected to a 128 Mbps link
which is shared by the other two as well. Host $i$ communicates with Server
$i$. Other links are 1Gbps with negligible effect on delay bounds.

In this section, we discuss three important aspects of our designed algorithm.
First, we will present the steady state and transient performance of our
algorithm. It is important to observe that in any delay sensitive networks,
transients will eventually lead to queues which brings us to our discussion on
an SDN specific digital-twin based implementation of our algorithm. Further,
we discuss how our algorithm differs from the state of art schemes.

###  VI-A Parameters and setup

To evaluate the performance of the proposed TCP algorithm, we simulate a
simple network as shown in Figure2 using OMNeT++. The sending rates are
bounded for each source $[m_{s}=1Mbps,M_{s}=100Mbps]$. We choose the utility
function to be $U_{s}(x_{s})=a_{s}log(1+x_{s})$ with $a_{s}=10^{5}$. Note that
this is strongly concave as $U^{\prime\prime}(x_{s})\leq-10^{5}/(1+10^{6})<0$
using the fact that $x_{s}\geq 10^{6}$. We consider a maximum packet size of
1518 bytes which is the standard for Ethernet. The delay bound is set to $1$
ms which is required for satisfying eMBB slice.

Each host is equipped with a token bucket with $\sigma=1$ packet (1518 bytes)
and as mentioned in Section III-A, we regulate the bucket rate with help of
our algorithm. The step sizes for switches are set as $\gamma_{e}=10^{-6}$
while that for SDN is set as $\gamma_{j}=10^{-7}$. We did not pursue for an
optimal choice of step size, which can also be performed by using techniques
in optimization theory [23]. All prices are set to zero at start. Price
updates are performed every $5$ms. The implementation is made available at
https://github.com/dibbend8/tcpexperiments/tree/main/tcpSlice_Latest.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3:  (a), (b), (c) Sending rates for Host 0, 1 and 2. The hosts join and
leave the network as follows: Host 0 is already in the network from the start
for 10 seconds, then Host 1 joins. After 40 seconds, Host 2 joins the network.
The three hosts are active till 100 seconds when Host 0 leaves the network,
and then Host 1 leaves the network at 120 seconds while Host 2 leaves at 140
seconds. The figures clearly show that the steady-state rates reach the
bottleneck capacity of 100 Mbps while meeting the obtained bounds. (d),(e),(f)
Achieved Delay bounds. From the two figures, an important conclusion comes
out: TCP slice is NOT a fair allocation scheme, rather it focuses more on
delay-bound satisfaction while maximizing the network utilization.

###  VI-B Steady state and Transients

The designed experiment setup can give us a good idea about how the network
behaves over time, how fast it reaches the steady state (optima in our case),
and its behavior in the transient phases. Figure3 shows the life cycle of
hosts entering and leaving the network. Host 0 sends data at a 1Mbps rate. The
switches and SDN on receiving new flow, send zero prices which leads Host 0 to
boost its sending rate to the maximum. Note that we did not explicitly
implement the ECN mechanism and Host 0 would spend some update cycles figuring
out the price as it is implemented using probabilistic marking. Since, there
are no flows from other sources, and the computed delay bound is minimal
($\sim$ 0.00023 seconds), Host 0 enjoys the full capacity for the first 10s.

When Host 1 joins, initial prices being zero, Host 1 sends data and SDN
computes the price for Host 1 where, the delay bound shoots (recall, it is
$\approx 1/(100-x_{0})$ where $x_{0}$ is nearly 100 Mbps) resulting in a high
price for Host 0, forcing it to reduce its rate (see Figure3d). As shown in
Figure3e, after a few price exchanges, delay bounds reach within limits of
$1$ms. In this case, for both the hosts, the 100 Mbps link is the one that
produces minimum residual capacity resulting in both sharing their bandwidths
around $50$ Mbps where the delay bounds are around $0.8$ms. Note that our
objective is always to maximize the link utilization subject to delay
constraints.

When Host 2 joins, all three compete for the link capacity while satisfying
the delay constraint. Since prices are initially zero, the first price update
tells it to send with 100 Mbps. In this case, the bound for Host 2 is of the
order $1/(128-100)$ while for others it is $1/[128-150]^{+}$, which makes the
delay bounds shoot for Host 0 and Host 1. It is to be noted that the delay
bound calculated for Host 2 is much lower than other hosts since it traverses
a single link instead of two. Since it has more room for delay bound to play
with, our algorithm will penalize this host more than others to achieve the
delay bounds. Thus, this algorithm does not achieve fairness as most TCP
algorithms do. In its present form, our algorithm chooses bound satisfaction
over fairness (Section VI-D for a detailed description). Also, by looking at
Figure3e, one might be curious regarding why Host 1’s bound shoots after Host
0, whereas they must have started from the same bound value, as achieved
before Host 2 joins. This is purely probabilistic. We indeed found that both
started from the same delay bound value, but Host 0 got an opportunity to send
its traffic a bit earlier leading to a higher delay bound calculation at Host
1. The situation could easily have been reversed and we would have observed
reversed outcomes. The utilization is around 100 Mbps which is the full
bandwidth of the bottleneck link.

As shown in Figure3f, when Host 0 leaves the network, Host 2 is further
penalized and delay bounds are adjusted between Host 1 and Host 2 so as to
achieve the maximum possible throughput of 128 Mbps (see Figure3c and
Figure3b).

###  VI-C SDN enabled digital-twin Implementation: Proactive vs Reactive
approach

The results show transients and their effects on delay bounds. The effects can
be accounted for due to the $(1/x)$ relationship that is severely detrimental
and there is no way to avoid this. Largely, TCP protocols have been reactive
in nature, i.e. it reacts when something bad happens in the network whereas
such a scheme would be significantly damaging when it comes to delays. The
main goal of any control algorithm is to reach a steady state as soon as
possible and in this case, even more so.

When a new flow or packet arrives at an OpenFlow switch, it is forwarded to
the SDN controller. The controller then directs it to the relevant application
(NFV) for creating new flow tables. This allows the SDN to have visibility
over new joining hosts and decide the requested level/type of service/slice
mapping. By implementing a live replica of the network (a digital twin [24]),
the SDN can pre-calculate steady states before data communication begins. The
SDN can compute required prices such that the host directly starts from the
steady state. Nevertheless, the ECN and relevant control mechanisms should
always be in place for maintenance purposes. This is a proactive approach that
should lead to a much better solution for time-sensitive networks.

###  VI-D Comparison with Conventional TCP schemes

An important aspect of the TCP is the fairness associated with the steady
state. For example, both TCP BBR [12, 13] and the regular TCP [14] (all
variants such as RENO, VEGAS, etc., come in this category), would try to
achieve a point where the bottleneck bandwidth is shared equally among the
multiple flows using the bottleneck. This steady state is not ideal when there
are sources with different delay constraints over the bottleneck link.
Consider a situation with a shared link with a link rate of 40 Mbps and let
there be applications within the uRLLC slice, one demanding a delay constraint
of 1ms and the other, demanding a constraint of 0.5ms. Let us see the
resulting delay bounds if their rates are equally divided. Consider the
largest packet size and burst size of 1512 bytes. Table III shows how a fair
allocation exceeds the delay bound of a 0.5ms app, while such a requirement
could be satisfied by reducing the rates of the app with a 1ms bound.

TABLE III: Fairness vs Delay bound satisfaction $x_{1}$ ($d_{1}=1$ms) |  $x_{2}$ ($d_{2}=0.5$ms) |  $D_{1}^{NC}$ (ms) |  $D_{2}^{NC}$ (ms)
---|---|---|---
20 |  20 | 0.6048 | 0.6048
15 |  25 | 0.806 | 0.483

As more applications evolve with such diverse requirements [5], it would
reduce the sending rate of the sources with relaxed delay constraints and
increase that of others so as to maintain the delays. However, it should be
noted that our approach is not fair and there may be fairer solutions in the
feasible range of delay bounds. To look for fair solutions, the optimization
may be reformulated to include $\sum_{s}\sum_{j|j\in S(e)}\sum_{e\in
E(s)}|x_{s}-x_{j}|$ to be minimized in addition to maximization of $U_{s}$.
This is separable for sources and can be implemented by each source. However,
gradient computations may need to be replaced by subgradient computations due
to non-differentiability of the modulus operation.

##  VII Conclusion

In this paper, we propose TCP Slice, an algorithm focusing on meeting delay
constraints in a network using an optimization framework. We use network
calculus to compute delay bounds and achieve a semi-distributed algorithm via
dual decomposition. It involves penalizing sources for exceeding capacity and
violating delays. We present the requisite technique to implement the same
with help of a SDN controller and ECN feedback. We observe the steady state
and transient performance of our algorithm which highlights the aspect of
fairness vs bound satisfaction in our algorithm. Furthermore, we present a
proactive approach so that hosts do not encounter unnecessary delays in the
transient phases. We compare our approach with existing TCP algorithms and
find it to be lacking in fairness which can be rectified by changing the
objective. We suggest exploring stochastic network calculus for broader
applications, especially in wireless scenarios.

## References

  * [1] V. Jacobson, “Congestion avoidance and control,” _ACM SIGCOMM computer communication review_ , vol. 18, no. 4, pp. 314–329, 1988.
  * [2] L. Han, Y. Qu, L. Dong, and R. Li, “A framework for bandwidth and latency guaranteed service in new ip network,” in _IEEE INFOCOM 2020-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)_. IEEE, 2020, pp. 85–90.
  * [3] “5g kpis — 5g key performance indicators,” https://www.rfwireless-world.com/Terminology/5G-KPIs-Key-Performance-Indicators.html, (Accessed on 07/26/2023).
  * [4] M. Giordani, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Toward 6g networks: Use cases and technologies,” _IEEE Communications Magazine_ , vol. 58, no. 3, pp. 55–61, 2020.
  * [5] “6g – connecting a cyber-physical world - ericsson,” https://www.ericsson.com/en/reports-and-papers/white-papers/a-research-outlook-towards-6g, (Accessed on 07/26/2023).
  * [6] K. Benzekki, A. El Fergougui, and A. Elbelrhiti Elalaoui, “Software-defined networking (sdn): a survey,” _Security and communication networks_ , vol. 9, no. 18, pp. 5803–5833, 2016.
  * [7] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner, “Openflow: enabling innovation in campus networks,” _ACM SIGCOMM computer communication review_ , vol. 38, no. 2, pp. 69–74, 2008.
  * [8] X. Foukas, G. Patounas, A. Elmokashfi, and M. K. Marina, “Network slicing in 5g: Survey and challenges,” _IEEE communications magazine_ , vol. 55, no. 5, pp. 94–100, 2017.
  * [9] V. Jacobson, R. Braden, and D. Borman, “Tcp extensions for high performance,” Tech. Rep., 1992.
  * [10] L. S. Brakmo, S. W. O’malley, and L. L. Peterson, “Tcp vegas: New techniques for congestion detection and avoidance,” in _Proceedings of the conference on Communications architectures, protocols and applications_ , 1994, pp. 24–35.
  * [11] F. Sally, “Highspeed tcp for large congestion windows,” _RFC3649_ , 2003.
  * [12] N. Cardwell, Y. Cheng, C. S. Gunn, S. H. Yeganeh, and V. Jacobson, “Bbr: Congestion-based congestion control: Measuring bottleneck bandwidth and round-trip propagation time,” _Queue_ , vol. 14, no. 5, pp. 20–53, 2016\.
  * [13] ——, “Bbr: Congestion-based congestion control,” _Communications of the ACM_ , vol. 60, no. 2, pp. 58–66, 2017.
  * [14] S. H. Low and D. E. Lapsley, “Optimization flow control. i. basic algorithm and convergence,” _IEEE/ACM Transactions on networking_ , vol. 7, no. 6, pp. 861–874, 1999.
  * [15] F. P. Kelly, A. K. Maulloo, and D. K. H. Tan, “Rate control for communication networks: shadow prices, proportional fairness and stability,” _Journal of the Operational Research society_ , vol. 49, pp. 237–252, 1998.
  * [16] S. H. Low, _Analytical methods for network congestion control_. Springer Nature, 2022.
  * [17] Z.-Q. Luo and P. Tseng, “On the convergence rate of dual ascent methods for linearly constrained convex minimization,” _Mathematics of Operations Research_ , vol. 18, no. 4, pp. 846–867, 1993.
  * [18] S. A. Alghunaim and A. H. Sayed, “Linear convergence of primal–dual gradient methods and their performance in distributed optimization,” _Automatica_ , vol. 117, p. 109003, 2020.
  * [19] J. Luo, J. Jin, and F. Shan, “Standardization of low-latency tcp with explicit congestion notification: A survey,” _IEEE Internet Computing_ , vol. 21, no. 1, pp. 48–55, 2017.
  * [20] J.-Y. Le Boudec and P. Thiran, _Network calculus: a theory of deterministic queuing systems for the internet_. Springer, 2001.
  * [21] C.-S. Chang, _Performance guarantees in communication networks_. Springer Science & Business Media, 2000.
  * [22] S. Athuraliya, V. H. Li, S. H. Low, and Q. Yin, “Rem: Active queue management,” in _Teletraffic Science and Engineering_. Elsevier, 2001, vol. 4, pp. 817–828.
  * [23] N. Jorge and J. W. Stephen, _Numerical optimization_. Spinger, 2006.
  * [24] Z. Wang, R. Gupta, K. Han, H. Wang, A. Ganlath, N. Ammar, and P. Tiwari, “Mobility digital twin: Concept, architecture, case study, and future challenges,” _IEEE Internet of Things Journal_ , vol. 9, no. 18, pp. 17 452–17 467, 2022.