# Network-Aided Intelligent Traffic Steering in 6G O-RAN: A Multi-Layer
Optimization Framework

Van-Dinh Nguyen, Thang X. Vu, Nhan Thanh Nguyen, Dinh C. Nguyen, Markku
Juntti,
Nguyen Cong Luong, Dinh Thai Hoang, Diep N. Nguyen and Symeon Chatzinotas
V.-D. Nguyen is with the College of Engineering and Computer Science,
VinUniversity, Vinhomes Ocean Park, Hanoi 100000, Vietnam (e-mail:
dinh.nv2@vinuni.edu.vn).T. X. Vu and S. Chatzinotas are with the
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University
of Luxembourg, L-1855 Luxembourg City, Luxembourg (e-mail: {thang.vu,
symeon.chatzinotas}@uni.lu).N. T. Nguyen and M. Juntti are with Centre for
Wireless Communications, University of Oulu, P.O.Box 4500, FI-90014, Finland,
(email: {nhan.nguyen, markku.juntti}@oulu.fi).N. C. Luong is with the Faculty
of Computer Science, PHENIKAA University, Hanoi 12116, Vietnam (e-mail:
luong.nguyencong@phenikaa-uni.edu.vn).Dinh C. Nguyen is with the Elmore Family
School of Electrical and Computer Engineering, Purdue University, USA (e-mail:
nguye772@purdue.edu).D. T. Hoang and D. N. Nguyen are with the School of
Electrical and Data Engineering, University of Technology Sydney, Sydney, NSW
2007, Australia (e-mail: {hoang.dinh<EMAIL_ADDRESS>
###### Abstract

To enable an intelligent, programmable and multi-vendor radio access network
(RAN) for 6G networks, considerable efforts have been made in standardization
and development of open RAN (O-RAN). So far, however, the applicability of
O-RAN in controlling and optimizing RAN functions has not been widely
investigated. In this paper, we jointly optimize the flow-split distribution,
congestion control and scheduling (JFCS) to enable an intelligent traffic
steering application in O-RAN. Combining tools from network utility
maximization and stochastic optimization, we introduce a multi-layer
optimization framework that provides fast convergence, long-term utility-
optimality and significant delay reduction compared to the state-of-the-art
and baseline RAN approaches. Our main contributions are three-fold: $i$) we
propose the novel JFCS framework to efficiently and adaptively direct traffic
to appropriate radio units; $ii$) we develop low-complexity algorithms based
on the reinforcement learning, inner approximation and bisection search
methods to effectively solve the JFCS problem in different time scales; and
$iii)$ the rigorous theoretical performance results are analyzed to show that
there exists a scaling factor to improve the tradeoff between delay and
utility-optimization. Collectively, the insights in this work will open the
door towards fully automated networks with enhanced control and flexibility.
Numerical results are provided to demonstrate the effectiveness of the
proposed algorithms in terms of the convergence rate, long-term utility-
optimality and delay reduction.

###### Index Terms:

Open radio access network, intelligent resource management, traffic steering,
reinforcement learning, resource sharing.

##  I Introduction

With the great success of mobile Internet, fifth generation (5G) cellular
networks have been standardized to meet competing demands (e.g. extremely high
data rate, low-latency and massive connectivity) and proliferation of
heterogeneous devices. However, the existing “one-size-fits-all” 5G
architecture lacks sufficient intelligence and flexibility to enable the
coexistence of these demands [1]. As we move towards 6G, the latest frontier
in this endeavor is open radio access network (O-RAN) by disaggregating RAN
components and opening up interfaces, which is considered today the most
promising approach to revolutionize the wireless technology from “connected
things” to “connected intelligence” [2, 3, 4]. O-RAN is expected to fully
enable programmable, intelligent, interoperable and multi-vendor RAN [5].

Figure 1: O-RAN Alliance reference architecture and workflow [3], with Non-RT
and Near-RT RICs. A base station is disaggregated to CU, DU and RU.

Fig. 1 illustrates the high-level O-RAN Alliance reference architecture [3],
where the “black” parts and interfaces are defined by the 3rd Generation
Partnership Project (3GPP), while the “orange” parts and interfaces are
defined by O-RAN Alliance. O-RAN initiatives were developed to split the RAN
into the radio unit (RU), distributed unit (DU) and centralized unit (CU),
allowing for the interoperability of open hardware (HW), software (SW) and
interfaces (e.g. O1, A1 and E2) [3, 4]. The O-RAN architecture typically has
three main layers (or loops), including the management, control and function
layers as illustrated in Fig. 1. In particular, the management layer takes
place in non-real-time (Non-RT) over 1 s (second) with orchestration,
automation functions and trained artificial intelligence (AI) and machine
learning (ML) models. The control layer is executed in near real-time (Near-
RT) between 10 ms and 1 s to provide functions like radio resource management
(RRM), quality-of-service (QoS) management and interference management.
Finally, the function layer provides the RAN optimization of a timescale below
10 ms, such as scheduling, power control and radio-frequency assignment, etc.
The function layer (CU, DU and RU) is also connected to the Non-RT RIC through
the O1 interface for periodic feedback, aiming to fully enable autonomous and
self-optimizing networks. Two important parts introduced in O-RAN are Non-RT
RAN intelligent controller (Non-RT RIC) and Near-RT RIC that allow to access
RRM functions. The former enables AI/ML workflow for RAN components and RRM
like traffic steering (TS) as well as policy-based guidance of applications in
Near-RT RIC, while the latter is embedded with the control/optimization
algorithms of RAN and radio resources [6, 7, 5, 8].

###  I-A Motivation

Yet the existing research efforts on O-RAN in the academic community are
isolated, providing only tailored solutions to problems at either the physical
or higher layers [9, 10, 11, 12]. On the other hand, ML-based approaches (e.g.
[9, 13, 10, 12, 14, 15]) often ignored periodic feedback loops and assumed
that the RAN information is available at SMO to perform resource allocation
and RAN management, making a fully automated network impractical. The
understanding of how O-RAN could help improve network performance by
controlling data traffic and optimizing RAN functions remains rather limited
in the literature. In this paper, we aim to fill this gap by conducting an in-
depth analysis of the multi-layer design between the physical and higher
layers and developing low-complexity algorithms for network control,
scheduling and resource allocation in different time scales. We also analyze
their impact on the throughput and delay performances in the 6G O-RAN context.

In light of the above discussions, this paper focuses on designing the TS
control to intelligently direct the user traffic through a group of RUs,
taking into account available resources and users’ service requirements. To
fully realize the potential performance of the TS scheme, O-RAN allows
customization of user-centric strategies, multi-path routing and multi-
connectivity as well as proactive optimization of network parameters through
RICs. However, the problem becomes more challenging in the O-RAN setting due
to several complicating factors: $i)$ the traffic demand of user equipements
(UEs) often varies over time, and the complete information of the RAN layer is
indeterminate at the time of optimization algorithm execution. Hence, the
policies and control decisions at the service management and orchestration
(SMO) must be adapted to the variation of data traffic; $ii)$ the total data
traffic is distributed unevenly to RUs due to different downlink (DL)
throughput capabilities, causing high queueing delay; and $iii)$ the strong
correlation between congestion control and scheduling optimization influences
the optimal choice of flow-split distribution of data traffic across all RUs.
In addition, the deployment of fully automated networks is an intricate
problem in O-RAN that calls for intelligent, scalable and self-organizing
strategies for a holistic multi-layer optimization framework. In this regard,
reinforcement learning (RL) plays an important role in achieving long-term
utility optimization. To the best of our knowledge, the TS optimization
problem for O-RAN as outlined above has not been thoroughly addressed in the
literature.

###  I-B Main Contributions

In this paper, we consider a practical scenario where the complete information
of the RAN layer is not available at the beginning of each time-frame.
Instead, we assume that only their expected values are available to
approximately measure queueing delay. An interesting question naturally
arises: How does the incomplete information of user traffic demands affect the
optimal choices of the TS scheme? To answer this question and address the
challenges above, we introduce a holistic multi-layer optimization framework
that jointly optimizes the flow-split distribution, congestion control and
scheduling (called JFCS). The proposed framework effectively characterizes the
complex interactions between layers (e.g. flow-split selection, congestion
control rate and power allocation). In summary, we make the following three
key contributions:

  * •

We propose a novel JFCS framework to efficiently and adaptively direct traffic
to appropriate RUs. Our framework not only generalizes the classical queue-
length-based congestion control and scheduling (QCS) method [16], but also
provides a synergy between RL, QCS and updated network state information, and
thus enabling a closed-loop control of the TS in the O-RAN context.

  * •

To ensure the practicality, we identify inherent properties of the JFCS
problem and propose an intelligent resource management algorithm to solve it
effectively by leveraging the stochastic optimization framework [17]. In
particular, by exploiting the historical system information accumulated from
the previous time-slots, an RL process is developed to build the smoothed best
response while maximizing the long-term utility for each data-flow under
arbitrary changes in traffic demands. Given the updated queue-length vector
and the optimal flow-split distribution, two low-complexity algorithms are
developed to effectively solve the short-term power control optimization
subproblem in an iterative fashion.

  * •

Given a scaling factor $\varphi$ to minimize the Lyapunov drift [18], the
theoretical performance results are analyzed to show that the queueing network
is stable. In addition, the expected divergence in queue-length and the
optimality gap of congestion control rate still scale as
$\mathcal{O}(\sqrt{\varphi})$ and $\mathcal{O}(1/\sqrt{\varphi})$,
respectively. Thus, there always exists a scaling factor to balance utility-
optimality and latency.

We numerically evaluate the performance of the proposed framework. Results
show that the proposed framework can improve network resource utilization
significantly while achieving fast convergence and long-term utility-
optimality, compared to state-of-the-art approaches.

###  I-C Paper Organization and Mathematical Notation

The remainder of this paper is organized as follows. The related work is
discussed in Section II. In Section III, we first introduce the network model
and then present the problem formulation. The proposed JFCS framework and its
solutions are provided in Sections IV and V, respectively. Section VI presents
the key theoretical performance results of the JFCS framework. Numerical
results are given in Section VII, while Section VIII concludes the paper.

Mathematical notation: Throughout this paper, matrices and vectors are written
as bold uppercase and lowercase letters, respectively, while the scalar number
is denoted in lowercase. $\mathbf{h}^{\mathsf{H}}$ is the Hermitian transpose
of vector $\mathbf{h}$. The notation
$x\sim\mathcal{C}\mathcal{N}(0,\sigma^{2})$ implies that $x$ is a circularly-
symmetric complex Gaussian random variable with zero mean and variance
$\sigma^{2}$. $\|\cdot\|$ stands for the vector’s Euclidean norm. $\mathbb{C}$
and $\mathbb{R}$ denote the sets of all complex and real numbers,
respectively. Finally, $\mathbb{E}\\{\cdot\\}$ denotes the expectation of a
random variable.

##  II Related Work

Multi-layer (a.k.a. cross-layer) optimization for traditional cellular RAN
architectures has been extensively studied in the literature (see e.g., [19]
and references therein). For example, Tang et al. [20] studied a multi-layer
resource allocation problem to minimize the overall system power consumption
in a cloud-RAN (C-RAN), which jointly optimizes the service scaling, remote
radio head selection, and beamforming. In [21], a joint design of virtual
computing and radio resource allocation was proposed. It was shown that this
approach can efficiently allocate the virtual computing of the baseband unit
(BBU) pool to achieve load balancing among users with significantly reduced
power consumption. These problems are often solved by the difference of the
convex algorithm due to the combinatorial nature and strong coupling between
optimization variables. To address this challenge, graph theory techniques
were introduced in [22] and [23] to effectively solve the jointly coordinated
scheduling and power optimization problem in C-RAN. Recently, the multi-layer
network coding was also investigated in [24, 25, 26], taking into account the
rate heterogeneity of different users to remote radio heads. In general, these
existing works only optimized radio resources, while other factors at higher
layers (e.g. congestion control and routing) were overlooked, making
guaranteed multi-layer QoS for O-RAN infeasible. In addition, the non-causal
statistical knowledge of traffic demands is required to model queue states,
which is again impractical.

So far, there have been only a few attempts to study the applicability of the
O-RAN architecture. Kumar et al. [9] proposed an automatic relation (ANR)
approach to manage neighbour cell relationships by leveraging ML techniques,
hence improving gNodeB (gNB) handovers. The work in [13] introduced an
intelligent user access control algorithm based on deep reinforcement
learning, aiming to maximize the overall throughput and avoid frequent
handovers. The authors in [10] developed an RL-based dynamic function
splitting which is shown to be able to effectively decide the O-RAN’s function
splits and reduce operating costs. Based on the Working Group (WG)-2 AI/ML
specifications of the O-RAN Alliance, Acumos framework and open network
automation platform were introduced in [11] to generate AI/ML models to be
deployed in RIC modules and monitor the designed workflow, respectively.
Motalleb et al. [14] developed an iterative algorithm to jointly optimize
service-aware baseband resource allocation and virtual network function
activation, thus achieving better data rate and lower end-to-end delay. Very
recently, a deep reinforcement learning-based intelligent session management
for ultra-reliable and low latency communications (URLLC) was proposed in [15]
to allocate resources for serving current and new sessions more efficiently.
However, these studies did not reveal any observable information about the RAN
layer to SMO via periodic feedback loops. Thus, RICs in these studies were
unable to monitor RAN in a timely manner to enable their management automation
within O-RAN.

In traditional RAN architectures, the TS solutions are typically determined by
users’ radio conditions of a serving cell while treating signals from
neighboring cells as interference [27]. The authors in [28] proposed a
distributed TS scheme through edge servers, where the matrix-based shortest
path selection and matrix-based multipath searching algorithms were developed
to dynamically determine the optimal paths for traffic steering. Very
recently, Kavehmadavani et al. [29] showed that a dynamic multi-connectivity
(MC)-based TS scheme can help steer traffic flows towards the most suitable
cells based on user-centric conditions. In addition, the flow split for each
user was purely determined by the RUs’ capacity in delivering user traffic
demands, resulting in a very suboptimal solution. However, this work did not
embed AI/ML solutions in Non-RT RIC within the O-RAN architecture and assumed
that all network information is available at Near-RT RIC to optimize radio
resource allocation.

Different from all the above works and others in the literature that focus on
a single layer, we propose a fully multi-layer optimization framework that
captures interplays between the physical and higher layers, enabling proactive
optimization of network parameters through RICs with periodic feedback loops.
This holistic multi-layer optimization framework guarantees the long-term
utility-optimality with far less latency than state-of-the-art approaches,
opening the door towards fully automated networks with enhanced control and
flexibility.

##  III Network Model and Problem Formulation

###  III-A Network Model

Figure 2: Illustration of the O-RAN-based system model enabling TS where each
DU connects to multiple RUs towards cost-effective deployment.  Figure 3:
Illustration of frame structure with each time-frame $t$ (corresponding to one
large-scale coherence time) consisting of $T_{f}$ time-slots.

As shown in Fig. 2, we consider an O-RAN architecture with one CU, $I$ DUs and
$J$ RUs, where each DU connects to multiple RUs for cost-effective deployment.
Let us denote by $\mathcal{I}\triangleq\\{1,2,\cdots,I\\}$ the set of DUs. We
consider a downlink multi-user multiple-input single-output (MU-MISO) system,
where $J$ RUs simultaneously serve the set
$\mathcal{K}\triangleq\\{1,2,\cdots,K\\}$ of $K=|\mathcal{K}|$ single-antenna
UEs. The $j$-th RU served by the $i$-th DU is referred to as RU $(i,j)$, which
is equipped with $M_{i,j}$ antennas. The total number of RUs’ antennas is thus
$M_{\Sigma}=\sum_{\forall(i,j)}M_{i,j}$. The set of RUs served by DU $i$ is
denoted by $\mathcal{J}_{i}\triangleq\\{(i,1),\cdots,(i,J_{i})\\}$ with
$|\mathcal{J}_{i}|=J_{i}$ and $\sum_{i\in\mathcal{I}}J_{i}=J$. The total set
of RUs is denoted as
$\mathcal{J}\triangleq\cup_{i\in\mathcal{I}}\mathcal{J}_{i}$. We assume that
the midhaul (MH) link between the CU and DU and fronthaul link between the DU
and RU have sufficient capacity (i.e., high-speed optical ones), so that the
transmission latency from CU to RUs and queueing latency at CU and DUs are
negligible.

We consider that the system operates in a discrete time-frame indexed by
$t\in[1,2,\cdots,T]$, which corresponds to one large-scale coherence time with
a duration of $T_{c}$, as illustrated in Fig. 3. Each frame is divided into
$T_{f}$ time-slots of equal duration $\tau=T_{c}/T_{f}$, where the time-slot
is indexed by $t_{s}=tT_{f}+s$ with $s\in\\{1,2,\cdots,T_{f}\\}$. At CU, there
exist $K$ independent data-flows, each of which is intended for one UE. The CU
splits the data-flow of UE $k$, say flow $k$, into multiple sub-flows which
are possibly transmitted through the set of paths and then aggregated at this
UE [30, 31], so-called “traffic steering”. For data-flow $k$, we denote by
$\mathcal{P}_{k}\triangleq\\{(i,j)\\}_{\forall(i,j)\in\mathcal{J}}$ the set of
path states, including queue states and routing tables. To improve the system
throughput, a subset of separate paths in the set $\mathcal{P}_{k}$ (i.e., via
neighboring RUs indexed by ($i,j$)) should be appropriately selected. Let us
denote by
$\mathbf{c}_{k}[t]\triangleq\bigl{[}c_{k}^{i,j}[t]\bigr{]}_{(i,j)\in\mathcal{P}_{k}}$
the flow-split selection (action) vector for data-flow $k$ in time-frame $t$,
i.e., $c_{k}^{i,j}[t]=1$ if path ($i,j$) $\in\mathcal{P}_{k}$ (i.e., via RU
$(i,j)$) is selected to transmit data of flow $k$; otherwise,
$c_{k}^{i,j}[t]=0$. We let $\beta_{k}^{i,j}[t]\in[0,1]$ be the fraction of
data-flow $k$ which is routed via path $(i,j)$ in time-frame (state) $t$ by
selecting action $c_{k}^{i,j}[t]$, where
$\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]=1$. The global flow-split
decision is denoted by ${B}[t]\triangleq\\{\boldsymbol{\beta}_{k}[t],\forall
k\bigl{|}\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]=1,\forall k\\}$,
where each column flow-split vector
$\boldsymbol{\beta}_{k}[t]\triangleq\bigr{[}\beta_{k}^{i,j}[t]\bigl{]}_{(i,j)\in\mathcal{P}_{k}}^{\mathsf{T}}\in\mathbb{R}^{J}$
corresponds to the flow-split vector of data-flow $k$.

####  III-A1 Wireless Channel Model and Downlink Throughput

The large-scale fading coefficients are assumed to be invariant within one
frame $T_{c}$, while the small-scale fading components with a low degree of
mobility are assumed to be unchanged during time-slot $t_{s}$ with duration of
$\tau$ and vary independently in the next time-slot. For example, the large-
scale fading coefficients may stay invariant for a period of at least 40
small-scale fading coherence intervals for indoor scenarios [32]. The channel
vector between RU $(i,j)$ and UE $k\in\mathcal{K}$ in time-slot $t_{s}$ is
denoted by $\mathbf{h}^{i,j}_{k}[t_{s}]\in\mathbb{C}^{M_{i,j}\times 1}$, which
follows the Rician fading model with the Rician factor $\kappa^{i,j}_{k}[t]$.
In particular, $\mathbf{h}_{k}^{i,j}[t_{s}]$ is modeled as
$\mathbf{h}_{k}^{i,j}[t_{s}]=\sqrt{\xi_{k}^{i,j}}[t]\Bigl{(}\sqrt{\kappa^{i,j}_{k}[t]/(\kappa^{i,j}_{k}[t]+1)}\bar{\mathbf{h}}_{k}^{i,j}[t]+\sqrt{1/(\kappa^{i,j}_{k}[t]+1)}\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}]\Bigr{)}$
where $\xi_{k}^{i,j}[t]$ represents the large-scale fading;
$\bar{\mathbf{h}}_{k}^{i,j}[t]$ and
$\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}]\sim\mathcal{CN}(0,\mathbf{I})$ are the
line-of-sight (LoS) and non-LoS (NLoS) components, which follow a
deterministic channel and Rayleigh fading models, respectively. We let
$\mathbf{H}[t_{s}]\triangleq\bigr{[}\mathbf{h}_{1}[t_{s}]\cdots\mathbf{h}_{K}[t_{s}]\bigr{]}\in\mathbb{C}^{M\times
K}$ denote the channel matrix between all RUs and UEs in time-slot $t_{s}$
where
$\mathbf{h}_{k}[t_{s}]\triangleq\bigl{[}(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\bigr{]}^{\mathsf{H}}_{\forall
i,j}\in\mathbb{C}^{M\times 1}$ corresponds to the channel vector between RUs
and UE $k$.

Let us denote by $x_{k}^{i,j}[t_{s}]$ and
$\mathbf{w}_{k}^{i,j}[t_{s}]\in\mathbb{C}^{M_{i,j}\times 1}$ a unit-power data
symbol and a linear beamforming vector transmitted from RU $(i,j)$ to UE $k$
in time-slot $t_{s}$, respectively. The received signal at UE $k$ in time-slot
$t_{s}$ can be written as

$\displaystyle
y_{k}[t_{s}]=\sum_{(i,j)\in\mathcal{P}_{k}}(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{w}_{k}^{i,j}[t_{s}]x_{k}^{i,j}[t_{s}]$
$\displaystyle+\sum_{k^{\prime}\in\mathcal{K}\setminus\\{k\\}}\sum_{(i,j)\in\mathcal{P}_{k^{\prime}}}(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{w}_{{k^{\prime}}}^{i,j}[t_{s}]d_{k^{\prime}}^{i,j}[t_{s}]+\omega_{k}[t_{s}]$
(1)

where $\omega_{k}[t_{s}]$ is the additive white Gaussian background noise
(AWGN) with power $N_{0}$. The downlink achievable rate (bits/s) of UE $k$
from RU $(i,j)$ in time-slot $t_{s}$ can be written as
$r_{k}^{i,j}(\mathbf{w}[t_{s}])\triangleq
W\log_{2}\bigl{(}1+\gamma_{k}^{i,j}(\mathbf{w}[t_{s}])\bigr{)}$, where $W$ is
the system bandwidth and the signal-to-interference-plus-noise ratio (SINR)
$\gamma_{k}^{i,j}(\mathbf{w}[t_{s}])$ is given by
$\gamma_{k}^{i,j}(\mathbf{w}[t_{s}])=|(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{w}_{k}^{i,j}[t_{s}]|^{2}/\Phi_{k}^{i,j}(\mathbf{w}[t_{s}])$
with

$\displaystyle\Phi_{k}^{i,j}(\mathbf{w}[t_{s}])\triangleq\underbrace{\sum_{(i^{\prime},j^{\prime})\in\mathcal{P}_{k}\setminus\\{(i,j)\\}}|(\mathbf{h}_{k}^{i^{\prime},j^{\prime}}[t_{s}])^{\mathsf{H}}\mathbf{w}_{k}^{i^{\prime},j^{\prime}}[t_{s}]|^{2}}_{\text{Intra-
user interference}}$
$\displaystyle\qquad\quad+\underbrace{\sum_{k^{\prime}\in\mathcal{K}\setminus\\{k\\}}\sum_{(i,j)\in\mathcal{P}_{k^{\prime}}}|(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{w}_{k^{\prime}}^{i,j}[t_{s}]|^{2}}_{\text{Inter-
user interference}}+N_{0}$ (2)

and
$\mathbf{w}[t_{s}]\triangleq\bigl{[}(\mathbf{w}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\bigr{]}^{\mathsf{H}}_{k\in\mathcal{K},(i,j)\in\mathcal{P}_{k}}$
being the vector embracing all the beamformers. The overall effective data
rate of data-flow $k$ (or UE $k$) can be computed as
$r_{k}(\mathbf{w}[t_{s}])=\sum_{(i,j)\in\mathcal{P}_{k}}$
$r_{k}^{i,j}(\mathbf{w}[t_{s}])$. Then, for each $\mathbf{H}[t_{s}]$ and a
given $\boldsymbol{\beta}_{k}[t]$, we define the instantaneous achievable rate
region under beamformer $\mathbf{w}[t_{s}]$ as

$\displaystyle{C}_{\mathbf{H}[t_{s}]}\triangleq\Bigg{\\{}r_{k}(\mathbf{w}[t_{s}]),\forall
k\Biggl{|}{\begin{matrix}r_{k}(\mathbf{w}[t_{s}])=\sum\limits_{(i,j)\in\mathcal{P}_{k}}r_{k}^{i,j}(\mathbf{w}[t_{s}])\\\
\sum\limits_{k\in\mathcal{K}}\|\mathbf{w}_{k}^{i,j}[t_{s}]\|_{2}^{2}\leq
P_{\max}^{i,j},\forall(i,j)\end{matrix}}\Bigg{\\}}$

where $P_{\max}^{i,j}$ denotes the transmit power budget of RU $(i,j)$. We
note that the achievable rate $r_{k}^{i,j}(\mathbf{w}[t_{s}])$ is upper
bounded by $r_{k}^{i,j}(\mathbf{w}[t_{s}])\leq
W\log_{2}\bigl{(}1+P_{\max}^{i,j}\bigl{\|}\mathbf{h}_{k}^{i,j}[t_{s}]\bigr{\|}_{2}^{2}/N_{0}\bigr{)}$
for a limited transmit power budget $P_{\max}^{i,j}$, leading to
$r_{k}(\mathbf{w}[t_{s}])<\infty,\forall k,t$.

####  III-A2 Queueing Model

As illustrated in Fig. 2, each RU maintains a separate queue for each UE. Let
$A_{k}[t]$ (bits/s) be the total rate of data arriving at RU destined for UE
$k$ in time-frame $t$ with mean $\mathbb{E}\\{A_{k}\\}=\bar{A}_{k}$. We assume
that $A_{k}[t]$ is random and upper bounded by a finite constant $A^{\max}$,
such as $A_{k}[t]\leq A^{\max}<\infty,\forall k,t,$ and unknown at the
beginning of time-frame $t$. As a result, the queue-length of data-flow $k$ at
RU $(i,j)$ in time-slot $t_{s}$ evolves as follows:
$q_{k}^{i,j}[t_{s+1}]=\Bigl{[}q_{k}^{i,j}[t_{s}]+\beta_{k}^{i,j}[t]A_{k}[t]\tau-
r_{k}^{i,j}(\mathbf{w}[t_{s}])\tau\Bigr{]}^{+}$, where
$[x]^{+}\triangleq\max\\{0,x\\}$. By
$\mathbf{q}[t_{s}]\triangleq\bigl{[}q_{k}^{i,j}[t_{s}]\bigr{]}^{\mathsf{T}}_{k,(i,j)}$
and following [18], a queueing network is stable if the steady-state total
queue-length remains finite, such as

$\displaystyle\underset{t_{s}\to\infty}{\limsup}\
\mathbb{E}\\{\|\mathbf{q}[t_{s}]\|_{1}\\}<\infty.$ (4)

###  III-B Problem Formulation

Let
$\bar{r}_{k}\triangleq\underset{t_{s}\to\infty}{\lim}\frac{1}{t_{s}}\sum_{\ell=1}^{t_{s}}r_{k}(\mathbf{w}[\ell])$
denote the long-term average rate of data-flow $k$. Each UE $k$ is associated
with a utility function, denoted by $U_{k}(\bar{r}_{k})$. To facilitate the
analysis presented later, we make the following assumption to the utility
function [30, 18, 33, 34].

######  Assumption 1.

The utility function $U_{k}(\cdot)$ is assumed to satisfy the following
conditions

  * •

$U_{k}(\cdot)$ is twice continuously differentiable, increasing, and strictly
concave.

  * •

There exist positive constants $0<\psi<\Psi<\infty$, such as $\psi\leq-
U_{k}^{{}^{\prime\prime}}(\bar{r}_{k})\leq\Psi,\forall\bar{r}_{k}\in[0,\
\bar{r}^{\max}]$, with $\bar{r}^{\max}$ being the maximum long-term average
rate of any data flow.

Our goal is to maximize the network utility function
$\sum_{k\in\mathcal{K}}U_{k}(\bar{r}_{k})$, subject to the probabilistic delay
constraint, achievable rate region and queue-stability constraint. Based on
the network utility maximization (NUM) framework, the joint flow-split
distribution, congestion control and scheduling optimization problem (JFCS)
can be mathematically formulated as

$\displaystyle\textbf{JFCS}:\ $
$\displaystyle\underset{\boldsymbol{\beta},\bar{\mathbf{r}},\mathbf{w}}{\mathrm{max}}\
\sum_{k\in\mathcal{K}}U_{k}(\bar{r}_{k})$ (5a) $\displaystyle{\mathrm{s.t.}}\
\underset{t_{s}\to\infty}{\limsup}\
\mathbb{E}\\{\|\mathbf{q}[t_{s}]\|_{1}\\}<\infty$ (5b) $\displaystyle\qquad
r_{k}(\mathbf{w}[t_{s}])\in{C}_{\mathbf{H}[t_{s}]},\forall
t_{s},k\in\mathcal{K}$ (5c)
$\displaystyle\qquad\boldsymbol{\beta}_{k}[t]\in{B}[t],\forall
t,k\in\mathcal{K}$ (5d)
$\displaystyle\qquad\mathsf{Prob}\Bigl{(}\frac{q_{k}^{i,j}[t_{s}]}{\bar{A}_{k}}\leq\bar{d}_{k}\Bigr{)}\geq\epsilon_{k},\
\forall t_{s},k,(i,j)\quad$ (5e)

where
$\boldsymbol{\beta}\triangleq\bigl{[}\boldsymbol{\beta}_{k}^{\mathsf{T}}\bigr{]}^{\mathsf{T}}_{k\in\mathcal{K}}$
and
$\bar{\mathbf{r}}\triangleq\bigr{[}\bar{r}_{k}\bigr{]}^{\mathsf{T}}_{k\in\mathcal{K}}$.
Constraint (5e) ensures different minimum outage delay requirements for sub-
flows, where $\bar{d}_{k}$ and $\epsilon_{k}$ ($0\ll\epsilon_{k}\leq 1$) are
the maximum allowable average delay and the required reliable communication
for each UE, respectively. It is stated that the probability of
$\frac{q_{k}^{i,j}[t_{s}]}{\bar{A}_{k}}\leq\bar{d}_{k}$ (i.e. UEs’ maximum
allowable delay) should be greater than or equal to a certain positive
constant $\epsilon_{k}$. This probabilistic constraint is used to tackle the
randomness and variability of the arrival rate while maintaining a certain
level of performance.

######  Remark 1.

It is clear that problem (5) needs to be executed in different time scales
(i.e., over the long-term scale $t$ at Non-RT RIC and the short-term scale
$t_{s}$ at Near-RT RIC), as shown in Fig. 3. In particular, the global flow-
split vector $\boldsymbol{\beta}[t]$ is only updated once per time-frame $t$
to reduce computational complexity and information exchange, as well as to
ensure a stable queueing system. On the other hand, the beamforming vector
$\mathbf{w}[t_{s}]$ and the instantaneous achievable rate $\mathbf{r}[t_{s}]$
are optimized based on the real-time effective CSI $\mathbf{H}[t_{s}]$ in
time-slot $t_{s}$, adapting to dynamic environments.

##  IV JFCS-based Network Utility Optimization

###  IV-A Tractable Form of the JFCS Problem (5)

Challenges of Solving JFCS Problem (5): We can observe that constraint (5c) is
nonconvex while (5e) is a nonconvex probabilistic constraint, generally making
problem (5) NP-hard. In addition, the expectations in the constraints cause
the stochastic nature of the problem, which cannot be solved directly. The
classical optimization approaches, such as successive convex approximation
(SCA) [35], are often applied to solve the optimization problems of nonconvex
and deterministic constraints. However, the stochastic SCA-based algorithms
can no longer guarantee a feasible and (sub)-optimal solution of all
subsequent time intervals (TTIs) due to the dynamics of the physical layer at
small timescales. The flow-split decisions mainly rely on the previous states
updated by the RAN layer. Towards practical applications, an efficient and
adaptive solution to the long-term subproblem of (5) is necessary to achieve
high QoE for all UEs in every TTI.

Let us start by transforming problem (5) into a more tractable form. Towards a
safe design, we consider the replacement of constraint (5e) by its
deterministic constraint. From the basic property of probability, we can
rewrite (5e) as
$\mathsf{Prob}\bigl{(}q_{k}^{i,j}[t_{s}]\geq\bar{A}_{k}\bar{d}_{k}\bigr{)}\leq
1-\epsilon_{k}$. It follows from the well-known Markov inequality [36] that
$\mathsf{Prob}\bigl{(}q_{k}^{i,j}[t_{s}]\geq\bar{A}_{k}\bar{d}_{k}\bigr{)}\leq\mathbb{E}\\{q_{k}^{i,j}[t_{s}]\\}/\bar{A}_{k}\bar{d}_{k}$,
yielding

$\displaystyle\sum\nolimits_{\ell=1}^{t}\beta_{k}^{i,j}[\ell]\bar{A}_{k}\tau-(1-\epsilon_{k})\bar{A}_{k}\bar{d}_{k}-\sum\nolimits_{\ell=1}^{t_{s-1}}r_{k}^{i,j}(\mathbf{w}[\ell])\tau$
$\displaystyle\leq r_{k}^{i,j}(\mathbf{w}[t_{s}])\tau,\ \forall
t_{s},k\in\mathcal{K},(i,j)\in\mathcal{P}_{k}$ (6)

where each queue-length is always non-negative. We note that (IV-A) is a
relaxed constraint of (5e), which implies that any feasible of the former is
also feasible for the latter but not vice versa due to the Markov upper bound
on the outage probabilities.

To facilitate the following optimization, we introduce congestion control
variables
$\boldsymbol{a}[t_{s}]\triangleq\bigl{[}a_{k}[t_{s}]\bigr{]}_{k\in\mathcal{K}}^{\mathsf{T}}$,
satisfying $\bar{a}_{k}-\bar{r}_{k}\leq 0,\forall k$, where
$\bar{a}_{k}\triangleq\underset{t_{s}\to\infty}{\lim}\frac{1}{t_{s}}\sum_{\ell=1}^{t_{s}}a_{k}[\ell]$.
Problem (5) is then rewritten as

$\displaystyle\underset{\boldsymbol{\beta},\bar{\boldsymbol{a}},\bar{\mathbf{r}},\mathbf{w}}{\mathrm{max}}\
\sum_{k\in\mathcal{K}}U_{k}(\bar{a}_{k})$ (7a)
$\displaystyle{\mathrm{s.t.}}\quad\
\eqref{eq:JFCS1b},\eqref{eq:JFCS1c},\eqref{eq:JFCS1d},\eqref{eq:JFCS1e_Relaxed}$
(7b) $\displaystyle\qquad\quad\bar{a}_{k}-\bar{r}_{k}\leq 0,\forall k.$ (7c)

We also introduce a new auxiliary queue-length vector
$\hat{\mathbf{q}}[t_{s}]\triangleq\bigl{[}\hat{q}_{k}[t_{s}]\bigr{]}^{\mathsf{T}}_{k\in\mathcal{K}}$,
where $\hat{q}_{k}[t_{s+1}]=\bigl{[}\hat{q}_{k}[t_{s}]+a_{k}[t_{s}]\tau-
r_{k}(\mathbf{w}[t_{s}])\tau\bigr{]}^{+}$ to associate constraint (7c) with a
penalty function and $a_{k}[t_{s}]\in[0,A^{\max}]$. We define the total queue
backlog of all UEs in time-slot $t_{s}$ as
$L[t_{s}]=\frac{1}{2}\bigl{(}\sum_{k\in\mathcal{K}}$
$\sum_{(i,j)\in\mathcal{P}_{k}}\frac{q_{k}^{i,j}[t_{s}]^{2}}{\tau^{2}}+\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]^{2}}{\tau^{2}}\bigr{)}$,
which is the quadratic Lyapunov function [17, 37]. For given
$(\mathbf{q}[t_{s}],\hat{\mathbf{q}}[t_{s}])$, the Lyapunov drift from time-
slot $t_{s}$ to $t_{s+1}$ is given as $\Delta L[t_{s}]=L[t_{s+1}]-L[t_{s}]$.
To guarantee joint network stability and penalty minimization (i.e., (5b) and
(7c) hold true), we adopt the drift-plus-penalty procedure [17] to minimize
the drift of a quadratic Lyapunov function and rewrite (7) as

$\displaystyle\underset{\boldsymbol{\beta},\bar{\boldsymbol{a}},\bar{\mathbf{r}},\mathbf{w}}{\mathrm{max}}\quad\varphi\sum_{k\in\mathcal{K}}\mathbb{E}\\{U_{k}(a_{k}[t_{s}])\\}-\mathbb{E}\\{\Delta
L[t_{s}]\\}$ (8a) $\displaystyle{\mathrm{s.t.}}\quad\
\eqref{eq:JFCS1c},\eqref{eq:JFCS1d},\eqref{eq:JFCS1e_Relaxed}$ (8b)

where $\varphi$ is a scaling factor to balance two objective functions. We now
show that constraint (7c) holds with equality at optimum by introducing the
following lemma.

######  Lemma 1.

For each data-flow of UE $k$, the optimal congestion control rate is equal to
the optimal long-term average service rate, i.e.,
$\bar{a}_{k}^{*}-\bar{r}_{k}^{*}=0,\forall k.$

The proof Lemma 1 is straightforward by examining the Karush–Kuhn–Tucker (KKT)
complementary slackness condition over the increasing and strictly concave
objective function $U_{k}(\cdot),\forall k$.

###  IV-B Overall Intelligent Resource Management Algorithm

0: Set $t=1$ and select a positive scaling factor $\varphi$. Initialize
$\boldsymbol{\beta}_{k}[1]=\frac{1}{|\mathcal{P}_{k}|}[1,\cdots,1]$ and all
queues are set to be empty: $q_{k}^{i,j}[1_{1}]=0$ and
$\hat{q}_{k}[1_{1}]=0,\forall(i,j),k$.

0:

1: for each frame $t=1,2,\cdots,T$ do {/*Long-term scale $t$*/}

2: Flow-Split Distribution: Given $\\{\mathbf{q}[t-1],\mathbf{A}[t-1]\\}$, CU
splits data-flows of all UEs based on the optimal flow-split decisions
$\boldsymbol{\beta}^{*}[t]$ by solving L-SP at Non-RT RIC:
$\underset{\boldsymbol{\beta}_{k}[t]\in{B}[t],\forall k}{\mathrm{max}}\
\sum_{k\in\mathcal{K}}{L}_{k}[t].$

3: for each time-slot $t_{s}=tT_{f}+s$ with $s\in\\{1,\cdots,T_{f}\\}$ do
{/*Short-term scale $t_{s}$*/}

4: Congestion Controller: Given the queue-length vector
$\hat{\mathbf{q}}[t_{s}]$, solve S-SP1 (12) to obtain the optimal congestion
control variables:
$a_{k}^{*}[t_{s}]=\min\Bigl{\\{}U_{k}^{{}^{\prime}-1}\bigl{(}\frac{\hat{q}_{k}[t_{s}]}{\varphi\tau}\bigr{)},A^{\max}\Bigr{\\}},\forall
k.$

5: Weighted Queue-Length-Based Scheduler: Given the queue-length vector
$\hat{\mathbf{q}}[t_{s}]$ and the flow-split distribution
$\boldsymbol{\beta}^{*}[t]$, each RU $(i,j)\in\mathcal{P}_{k}$ schedules the
service rate $r_{k}^{i,j}(\mathbf{w}[t_{s}])$ for UE $k\in\mathcal{K}$ by
solving S-SP2:
$\underset{\mathbf{r}[t_{s}],\mathbf{w}[t_{s}]}{\mathrm{max}}\quad\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}r_{k}(\mathbf{w}[t_{s}]),\
{\mathrm{s.t.}}\ \eqref{eq:JFCS1c},\eqref{eq:JFCS1e_Relaxed}.$

6: Queue-Length Updates: Queue-Lengths are updated as  $\displaystyle
q_{k}^{i,j}[t_{s+1}]$
$\displaystyle=\bigl{[}q_{k}^{i,j}[t_{s}]+\beta_{k}^{i,j}[t]A_{k}[t]\tau$
$\displaystyle\qquad\qquad-r_{k}^{i,j}(\mathbf{w}[t_{s}])\tau\bigr{]}^{+},\
\forall k,(i,j)$ $\displaystyle\hat{q}_{k}[t_{s+1}]$
$\displaystyle=\bigl{[}\hat{q}_{k}[t_{s}]+a_{k}[t_{s}]\tau-
r_{k}(\mathbf{w}[t_{s}])\tau\bigr{]}^{+},\ \forall k.$

7: Set $s=s+1$

8: end for

9: Update
$\\{\mathbf{q}[t],\mathbf{A}[t]\\}:=\\{q_{k}^{i,j}[t],A_{k}[t]\\}_{k,(i,j)}$
to Non-RT RIC.

10: Set $t=t+1$

11: end for

Algorithm 1 Intelligent Resource Management Algorithm for Solving JFCS Problem
(5), compliant with O-RAN

To solve problem (8) in different time scales, we now decompose it into three
subproblems. To do so, we consider a worst-case design by developing an upper
bound of $\Delta L[t_{s}]$ for given
$(\mathbf{q}[t_{s}],\hat{\mathbf{q}}[t_{s}])$. From the inequality
$([x]^{+})^{2}\leq x^{2}$ and $(x+y)^{2}-x^{2}=2xy+y^{2}$, we have

$\displaystyle\Delta
L^{\mathtt{UB}}[t_{s}]\triangleq\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\frac{q_{k}^{i,j}[t_{s}]}{\tau}\bigl{(}\beta_{k}^{i,j}[t]A_{k}[t]-r_{k}^{i,j}(\mathbf{w}[t_{s}])\bigr{)}$
$\displaystyle\quad+\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}\bigl{(}a_{k}[t_{s}]-r_{k}(\mathbf{w}[t_{s}])\bigr{)}+B[t_{s}]\geq\Delta
L[t_{s}]$ (9)

where
$B[t_{s}]\triangleq\frac{1}{2}\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\bigl{(}\beta_{k}^{i,j}[t]A_{k}[t]-r_{k}^{i,j}(\mathbf{w}[t_{s}])\bigr{)}^{2}+\frac{1}{2}\sum_{k\in\mathcal{K}}\bigl{(}a_{k}[t_{s}]-r_{k}(\mathbf{w}[t_{s}])\bigr{)}^{2}$
is the summation of the second moments of the arrival and service processes.
Following [17] and [30], we consider that $B[t_{s}]$ is finite and bounded by
$\bar{B}$ for all $t_{s}$, i.e.,
$\mathbb{E}\\{B[t_{s}]\big{|}\mathbf{q}[t_{s}],\hat{\mathbf{q}}[t_{s}]\\}$
$\leq\bar{B}$. As a result, problem (8) is simplified to

$\displaystyle\underset{\boldsymbol{\beta},\bar{\boldsymbol{a}},\bar{\mathbf{r}},\mathbf{w}}{\mathrm{max}}\quad\varphi\sum_{k\in\mathcal{K}}\mathbb{E}\\{U_{k}(a_{k}[t_{s}])\\}-\mathbb{E}\\{\Delta
L^{\mathtt{UB}}[t_{s}]\\}$ (10a) $\displaystyle{\mathrm{s.t.}}\quad\
\eqref{eq:JFCS1c},\eqref{eq:JFCS1d},\eqref{eq:JFCS1e_Relaxed}.$ (10b)

Long-term subproblem (L-SP): The flow-split distribution subproblem at time-
frame $t$ is given as

$\displaystyle\textbf{L-SP}:\
\underset{\boldsymbol{\beta}_{k}[t]\in{B}[t],\forall
k}{\mathrm{max}}\quad\sum_{k\in\mathcal{K}}{L}_{k}[t]$ (11)

where
${L}_{k}[t]=\sum_{(i,j)\in\mathcal{P}_{k}}\frac{q_{k}^{i,j}[t_{s}]}{\tau}\bigl{(}r_{k}^{i,j}(\mathbf{w}[t_{s}])-\beta_{k}^{i,j}[t]A_{k}[t]\bigr{)}$.
Although problem (11) is a linear program in $\boldsymbol{\beta}$, it cannot
be solved directly by standard optimization techniques because
$A_{k}[t],\forall k$ are incompletely known at the beginning of time-frame
$t$.

Short-term subproblems (S-SPs): The congestion control subproblem at time-slot
$t_{s}$ is

$\displaystyle\textbf{S-SP1}:\ \underset{\boldsymbol{a}[t_{s}]\geq
0}{\mathrm{max}}\quad\sum_{k\in\mathcal{K}}\bigl{(}\varphi
U_{k}(a_{k}[t_{s}])-\frac{\hat{q}_{k}[t_{s}]}{\tau}a_{k}[t_{s}]\bigr{)}$ (12)

which is an unconstrained convex problem. The optimal solution of (12) exists
and is unique that is
$a_{k}^{*}[t_{s}]=U_{k}^{{}^{\prime}-1}\bigl{(}\frac{\hat{q}_{k}[t_{s}]}{\varphi\tau}\bigr{)},\forall
k$, where $U_{k}^{{}^{\prime}-1}(\cdot)$ denotes the inverse function of the
first derivation of $U_{k}(\cdot)$. Given the optimal solution
$\boldsymbol{\beta}^{*}[t]$, the short-term power control optimization
subproblem (i.e., the weighted queue-length-based scheduling) at time-slot
$t_{s}$ is given as

$\displaystyle\textbf{S-SP2}:\underset{\mathbf{r}[t_{s}],\mathbf{w}[t_{s}]}{\mathrm{max}}\quad\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}r_{k}(\mathbf{w}[t_{s}]),\
{\mathrm{s.t.}}\ \eqref{eq:JFCS1c},\eqref{eq:JFCS1e_Relaxed}.$ (13)

The overall intelligent resource management algorithm for solving the JFCS
problem (5) is summarized in Algorithm 1, where the solutions of subproblems
will be provided next.

##  V Proposed Algorithms for Solving Subproblems

We are now in a position to solve L-SP (11) and S-SP2 (13) in different time
scales. The optimality of the latter depends heavily on the optimal flow-split
decisions, which often require a prior knowledge of the statistical
information of all possible paths at Non-RT RIC. However, the assumption of
complete information is unrealistic due to the dynamic environment and the
data collected from the RAN layer being only updated to Non-RT RIC only on the
long-term scale. In this work, at time-frame $t$ we aim to exploit historical
system information accumulated from the previous time-slot, which can be used
to build the smoothed best response in maximizing the long-term utility for
each data flow.

###  V-A Reinforcement Learning Algorithm for Solving L-SP (11)

The flow-split decision $\boldsymbol{\beta}_{k}[t]$ in problem (11) can be
estimated separably by minimizing ${L}_{k}[t]$. This implies that the larger
the queue-length $q_{k}^{i,j}[t_{s}]$, the lower the flow-split decision value
$\beta_{k}^{i,j}[t]$ to guarantee fairness among all RUs
$(i,j)\in\mathcal{P}_{k}$ (i.e., to avoid large queue-lengths $q_{k}^{i,j}$ at
some RUs in the next time-slot $t_{s+1}$). Let us denote

$u_{k}^{i,j}[t]\triangleq\frac{q_{k}^{i,j}[t_{s}]}{\tau}\bigl{(}r_{k}^{i,j}(\mathbf{w}[t_{s}])-\beta_{k}^{i,j}[t]A_{k}[t]\bigr{)}$

as the instantaneous utility observation of data-flow $k$ at time-frame $t$
when selecting path $(i,j)\in\mathcal{P}_{k}$. The total utility observation
of data-flow $k$, denoted by $u_{k}[t]$, is thus

$u_{k}[t]=\sum_{(i,j)\in\mathcal{P}_{k}}u_{k}^{i,j}[t].$ (14)

However, it is unable to build a smoothed best response based on
$u_{k}^{i,j}[t]$ as it is not revealed at the beginning of time-frame $t$.
Inspired by [38], we denote $\hat{u}_{k}^{i,j}[t]$ as the estimated utility of
data-flow $k$ at time-frame $t$ when selecting path $(i,j)$. In addition, the
actual utility observed by data-flow $k$ at time-frame $t$, denoted by
$\bar{u}_{k}[t]$, is given as $\bar{u}_{k}[t]=u_{k}[t-1]$, which is based on
feedback from Near-RT RIC at time $t-1$. By initializing
$\hat{u}_{k}^{i,j}[1]=0$, the estimated utility of data-flow $k$ is updated
for action $\mathbf{c}_{k}[t]=c_{k}^{i,j}[t]$ as follows:

$\displaystyle\hat{u}_{k}^{i,j}[t]=$ $\displaystyle\
\hat{u}_{k}^{i,j}[t-1]+\eta_{u}[t]\mathbbm{1}_{\\{\mathbf{c}_{k}[t]=c_{k}^{i,j}[t]\\}}\bigl{(}\bar{u}_{k}[t]$
$\displaystyle\qquad\qquad\qquad\ -\hat{u}_{k}^{i,j}[t-1]\bigr{)},\ \forall
t>1$ (15)

where $\eta_{u}>0$ is the decreasing step size (i.e. the learning rate), which
is often decreased over time to guarantee convergence. Naturally,
$\hat{u}_{k}^{i,j}[1]$ is initialized as $\hat{u}_{k}^{i,j}[1]=0$ for $t=1$.
The indicator function $\mathbbm{1}_{\\{x=y\\}}=1$ (resp. 0) if the condition
$x=y$ is true (resp. false).

Next, we denote
$\hat{\boldsymbol{\theta}}_{k}[t]\triangleq[\hat{\theta}_{k}^{i,j}[t]]_{(i,j)\in\mathcal{P}_{k}}$
as the estimated regret vector of data-flow $k$, where each element is updated
for action $\mathbf{c}_{k}[t]=c_{k}^{i,j}[t]$ as

$\displaystyle\hat{\theta}_{k}^{i,j}[t]=$
$\displaystyle\hat{\theta}_{k}^{i,j}[t-1]+\eta_{\theta}[t]\mathbbm{1}_{\\{\mathbf{c}_{k}[t]=c_{k}^{i,j}[t]\\}}\bigl{(}\bar{u}_{k}[t]$
$\displaystyle\qquad\qquad-\hat{u}_{k}^{i,j}[t]-\hat{\theta}_{k}^{i,j}[t-1]\bigr{)},\
\forall t>1$ (16)

with $\hat{\theta}_{k}^{i,j}[1]=0$ and $\eta_{\theta}[t]$ being the learning
rate. In order to achieve high performance in the long term, the L-SP must
balance exploration and exploitation processes. We note that trying all
possible actions to choose the best paths (e.g. the exhaustive exploration)
can offer the highest payoff, but with the cost of slow convergence and even
computationally prohibitive. During the exploitation process, playing an
action associated with the highest estimated utility in (V-A) will likely
result in a very sub-optimal solution. To make this tradeoff more efficient,
let us define the best response function
$\hat{\boldsymbol{\beta}}[t]=f(\hat{\boldsymbol{\theta}}[t])$ as

$\displaystyle\small
f(\hat{\boldsymbol{\theta}}[t]):=\underset{\boldsymbol{\beta}_{k}[t]\in{B}[t]}{\operatornamewithlimits{argmin}}\Bigr{\\{}h\bigr{(}\boldsymbol{\beta}[t]\bigl{)}-\lambda\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]\hat{\theta}_{k}^{i,j}[t]\Bigl{\\}}.\qquad$
(17)

Here $\lambda$ is the so-called trade-off factor (a.k.a. Boltzmann
temperature) and $h\bigr{(}\boldsymbol{\beta}[t]\bigl{)}$ denotes the
regularization function. We note that when $\lambda\rightarrow 0$, it leads to
uniform probabilities of all actions, i.e.
$\beta_{k}^{i,j}[t]=1/|\mathcal{P}_{k}|,\forall(i,j)\in\mathcal{P}_{k}$. For
$\lambda\rightarrow\infty$, the second term in (17) will dominate the best
response function and then the actions associated with the highest estimated
regret will be selected [38].

Regularization function: The regularization function allows to learn the best
paths that maximize its own performance and stabilize the flow-split
decisions. The solutions to problem (11) lie in the unit simplex for each
data-flow. Therefore, we adopt the Gibbs-Shannon entropy as the regularization
function, i.e.
$h\bigr{(}\boldsymbol{\beta}[t]\bigl{)}=\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]\ln\bigl{(}\beta_{k}^{i,j}[t]\bigr{)}$,
which is $K$-strongly convex. Substituting
$h\bigr{(}\boldsymbol{\beta}[t]\bigl{)}$ into (17), we have

$\displaystyle f(\hat{\boldsymbol{\theta}}[t]):=$
$\displaystyle\underset{\boldsymbol{\beta}_{k}[t]\in{B}[t],\forall
k}{\operatornamewithlimits{argmin}}\Bigr{\\{}\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]\ln\bigl{(}\beta_{k}^{i,j}[t]\bigr{)}$
$\displaystyle\qquad\qquad\quad-\lambda\sum_{k\in\mathcal{K}}\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]\hat{\theta}_{k}^{i,j}[t]\Bigl{\\}}.$
(18)

The function $f(\hat{\boldsymbol{\theta}}[t])$ is convex and separable for
each $\beta_{k}^{i,j}[t]$. By solving the following equation

$\partial
f(\hat{\boldsymbol{\theta}}[t])/\partial\beta_{k}^{i,j}[t]=\ln\bigl{(}\beta_{k}^{i,j}[t]\bigr{)}+1-\lambda\hat{\theta}_{k}^{i,j}[t]=0$

we have

$\beta_{k}^{i,j}[t]=f(\hat{\theta}_{k}^{i,j}[t])=\exp{\bigl{(}\lambda\hat{\theta}_{k}^{i,j}[t]-1}\bigr{)}.$

To ensure $\sum_{(i,j)\in\mathcal{P}_{k}}\beta_{k}^{i,j}[t]=1,\forall k$ (i.e.
the unit simplex for data-flow $k$), we normalize
$f_{k}^{i,j}(\hat{\boldsymbol{\theta}}_{k}[t])$ through the exponentiated
mirror function as

$\displaystyle f_{k}^{i,j}(\hat{\boldsymbol{\theta}}_{k}[t])$
$\displaystyle=\frac{\exp{\bigl{(}\bigr{[}\lambda\hat{\theta}_{k}^{i,j}[t]-1\bigl{]}^{+}}\bigr{)}}{\sum_{(i^{\prime},j^{\prime})\in\mathcal{P}_{k}}\exp{\bigl{(}\bigr{[}\lambda\hat{\theta}_{k}^{i^{\prime},j^{\prime}}[t]-1\bigl{]}^{+}}\bigr{)}}$
$\displaystyle=\frac{\exp{\bigl{(}\lambda\bigr{[}\hat{\theta}_{k}^{i,j}[t]\bigl{]}^{+}}\bigr{)}}{\sum_{(i^{\prime},j^{\prime})\in\mathcal{P}_{k}}\exp{\bigl{(}\lambda\bigr{[}\hat{\theta}_{k}^{i^{\prime},j^{\prime}}[t]\bigl{]}^{+}}\bigr{)}}.$
(19)

As a result, the estimated value of each element of flow-split vector
$\boldsymbol{\beta}_{k}[t]$ is updated for all actions with the regret as

$\displaystyle\beta_{k}^{i,j}[t]=\beta_{k}^{i,j}[t-1]+\eta_{\beta}[t]\bigl{(}f_{k}^{i,j}(\hat{\boldsymbol{\theta}}_{k}[t])-\beta_{k}^{i,j}[t-1]\bigr{)}$
(20)

for $t>1$, where
$\boldsymbol{\beta}_{k}[1]=\frac{1}{|\mathcal{P}_{k}|}[1,\cdots,1]$ and
$\eta_{\beta}[t]$ is the learning rate. The three-step reinforcement learning
procedure includes (V-A), (V-A) and (20), which do not require expensive
computations and projection to the feasible space.

Convergence properties: The convergence conditions for the three-step
reinforcement learning procedure are given as follows:

$\displaystyle\underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{u}[t]=+\infty\
\&\ \underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{u}^{2}[t]<+\infty$
$\displaystyle\underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{\theta}[t]=+\infty\
\&\
\underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{\theta}^{2}[t]<+\infty$
$\displaystyle\underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{\beta}[t]=+\infty\
\&\
\underset{T\to\infty}{\lim}\sum\nolimits_{t=1}^{T}\eta_{\beta}^{2}[t]<+\infty$
$\displaystyle\underset{t\to\infty}{\lim}\frac{\eta_{\theta}[t]}{\eta_{u}[t]}=0\
\&\ \underset{t\to\infty}{\lim}\frac{\eta_{\beta}[t]}{\eta_{\theta}[t]}=0.$
(21)

This implies that the learning rates must be decreased over time to guarantee
the convergence of the proposed three-step RL procedure. The detailed proof of
a multiple-timescales RL algorithm can be found in [38, 39]. Following the
same arguments as those in [38] and the conditions in (V-A), the three-step RL
procedure in (V-A), (V-A) and (20) converges to the optimal solution with the
positive trade-off factor $\lambda>0$, satisfying
$\underset{t\to\infty}{\lim}\boldsymbol{\beta}_{k}[t]=\boldsymbol{\beta}_{k}^{*},\
\forall k\in\mathcal{K}$.

###  V-B Proposed Algorithm for Solving S-SP2 (13)

Given the optimal flow-split distribution of data-flow $k$,
$\boldsymbol{\beta}_{k}^{*}[t]$, we denote by $\mathcal{P}_{k}^{*}[t]$ the set
of selected path states in time-frame $t$, which only includes
$c_{k}^{i,j}[t]=1$ with $(i,j)\in\mathcal{P}_{k}$. In this section, we present
two low-complexity transmission designs for $\mathbf{w}$, namely maximum ratio
transmission (MRT) and zero-forcing beamforming (ZFBF), and then develop low-
complexity iterative algorithms for their solution.

####  V-B1 MRT-Based Transmission Design

Each RU $(i,j)$ performs MRT beamforming (a.k.a. channel-mathched beamforming)
using local CSI as
$\mathbf{w}_{k}^{i,j}[t_{s}]=\frac{\sqrt{p_{k}^{i,j}[t_{s}]}}{\sqrt{\nu_{k}^{i,j}[t_{s}]}}\mathbf{h}_{k}^{i,j}[t_{s}],\
\forall(i,j)\in\mathcal{P}_{k}[t]$ and $k\in\mathcal{K}$, where
$\nu_{k}^{i,j}[t_{s}]\triangleq\|\mathbf{h}_{k}^{i,j}[t_{s}]\|_{2}^{2}$ and
$p_{k}^{i,j}[t_{s}]$ is the transmit power coefficient allocated to UE $k$
from RU $(i,j)$ in time-slot $t_{s}$. The corresponding SINR is rewritten as

$\displaystyle\gamma_{k}^{i,j}(\mathbf{p}[t_{s}])=\frac{p_{k}^{i,j}[t_{s}]\nu_{k}^{i,j}[t_{s}]}{\Phi_{k}^{i,j}(\mathbf{p}[t_{s}])}$
(22)

where
$\Phi_{k}^{i,j}(\mathbf{p}[t_{s}])\triangleq\sum_{(i^{\prime},j^{\prime})\in\mathcal{P}_{k}[t]\setminus\\{(i,j)\\}}p_{k}^{i^{\prime},j^{\prime}}[t_{s}]\nu_{k}^{i^{\prime},j^{\prime}}[t_{s}]+\sum_{k^{\prime}\in\mathcal{K}\setminus\\{k\\}}\sum_{(i,j)\in\mathcal{P}_{k^{\prime}}[t]}p_{k^{\prime}}^{i,j}[t_{s}]\frac{|(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{h}_{k^{\prime}}^{i,j}[t_{s}]|^{2}}{\nu_{k^{\prime}}^{i,j}[t_{s}]}+N_{0}$
is linear in
$\mathbf{p}[t_{s}]\triangleq\bigl{[}p_{k}^{i,j}[t_{s}]\bigr{]}_{k\in\mathcal{K},(i,j)\in\mathcal{P}_{k}}$.
As a result, the short-term power optimization problem (13) with MRT reduces
to the following problem:

$\displaystyle\underset{\mathbf{p}[t_{s}]}{\mathrm{max}}$
$\displaystyle\quad\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}r_{k}(\mathbf{p}[t_{s}])$
(23a) $\displaystyle{\mathrm{s.t.}}$
$\displaystyle\quad\bar{R}_{k}^{i,j}[t_{s}]\leq
r_{k}^{i,j}(\mathbf{p}[t_{s}])\tau,\ \forall k,(i,j)$ (23b)
$\displaystyle\quad\sum_{k\in\mathcal{K}}p_{k}^{i,j}[t_{s}]\leq
P_{\max}^{i,j},\ \forall(i,j)$ (23c)

where
$r_{k}(\mathbf{p}[t_{s}])=\sum_{(i,j)\in\mathcal{P}_{k}^{*}}r_{k}^{i,j}(\mathbf{p}[t_{s}])$
with $r_{k}^{i,j}(\mathbf{p}[t_{s}])\triangleq
W\log_{2}\bigl{(}1+\gamma_{k}^{i,j}(\mathbf{p}[t_{s}])\bigr{)}$, and
$\bar{R}_{k}^{i,j}[t_{s}]\triangleq\sum_{\ell=1}^{t}\beta_{k}^{i,j}[\ell]\bar{A}_{k}\tau-(1-\epsilon_{k})\bar{A}_{k}\bar{d}_{k}-\sum_{\ell=1}^{t_{s-1}}r_{k}^{i,j}(\mathbf{p}[\ell])\tau.$

Problem (23) is nonconvex due to the nonconcavity of
$r_{k}^{i,j}(\mathbf{p}[t_{s}])$. We will now apply the inner approximation
(IA) method to effectively solve (23) in an iterative manner. Following from
inequality (A.2) in Appendix A with
$v=p_{k}^{i,j}[t_{s}]\|\mathbf{h}_{k}^{i,j}[t_{s}]\|_{2}^{2}$ and
$z=\Phi_{k}^{i,j}(\mathbf{p}[t_{s}])$, the global concave lower bound of
$r_{k}^{i,j}(\mathbf{p}[t_{s}])$ at the updated feasible point
$\mathbf{p}^{(n)}[t_{s}]$ found at iteration $n$, denoted by
${r}_{k}^{i,j(n)}(\mathbf{p}[t_{s}];{\mathbf{p}}^{(n)}[t_{s}])$, is given as

$\displaystyle r_{k}^{i,j}(\mathbf{p}[t_{s}])$ $\displaystyle\geq
r_{k}^{i,j}({\mathbf{p}}^{(n)}[t_{s}])-W\log_{2}e\biggl{[}\gamma_{k}^{i,j}({\mathbf{p}}^{(n)}[t_{s}])$
$\displaystyle\
-2\frac{\nu_{k}^{i,j}[t_{s}]\sqrt{{p}_{k}^{i,j(n)}[t_{s}]}\sqrt{p_{k}^{i,j}[t_{s}]}}{\Phi_{k}^{i,j}({\mathbf{p}}^{(n)}[t_{s}])}+\gamma_{k}^{i,j}({\mathbf{p}}^{(n)}[t_{s}])$
$\displaystyle\
\times\frac{p_{k}^{i,j}[t_{s}]\nu_{k}^{i,j}[t_{s}]+\Phi_{k}^{i,j}(\mathbf{p}[t_{s}])}{{p}_{k}^{i,j(n)}[t_{s}]\nu_{k}^{i,j}[t_{s}]+\Phi_{k}^{i,j}({\mathbf{p}}^{(n)}[t_{s}])}\biggl{]}$
$\displaystyle:={r}_{k}^{i,j(n)}(\mathbf{p}[t_{s}];{\mathbf{p}}^{(n)}[t_{s}])$
(24)

with
${r}_{k}^{i,j(n)}({\mathbf{p}}^{(n)}[t_{s}];{\mathbf{p}}^{(n)}[t_{s}])=W\log_{2}\bigl{(}1+\gamma_{k}^{i,j}(\mathbf{p}^{(n)}[t_{s}])\bigr{)}$.
As a result, we successively solve the following inner convex approximate
program of (23) at iteration $n$:

$\displaystyle\underset{\mathbf{p}[t_{s}]}{\mathrm{max}}$ $\displaystyle\
\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}{r}_{k}^{(n)}(\mathbf{p}[t_{s}])$
(25a) $\displaystyle{\mathrm{s.t.}}$ $\displaystyle\
\bar{R}_{k}^{i,j}[t_{s}]\leq{r}_{k}^{i,j(n)}(\mathbf{p}[t_{s}];{\mathbf{p}}^{(n)}[t_{s}])\tau,\
\forall k\in\mathcal{K},(i,j)\qquad$ (25b) $\displaystyle\
\sum_{k\in\mathcal{K}}p_{k}^{i,j}[t_{s}]\leq P_{\max}^{i,j},\ \forall(i,j)$
(25c)

and update the feasible point ${\mathbf{p}}^{(n)}[t_{s}]$ until convergence,
where
${r}_{k}^{(n)}(\mathbf{p}[t_{s}])=\sum_{(i,j)\in\mathcal{P}_{k}^{*}}r_{k}^{i,j(n)}(\mathbf{p}[t_{s}];$
${\mathbf{p}}^{(n)}[t_{s}])$. The proposed iterative procedure to solve (13)
is summarized in Algorithm 2. An initial feasible value for
$\mathbf{p}^{(0)}[t_{s}]$ to start Algorithm 2 is easily found by successively
solving the following simple convex program:

$\displaystyle\underset{\mathbf{p}[t_{s}]}{\mathrm{max}}$ $\displaystyle\
\varrho\triangleq\underset{\forall
k,(i,j)}{\min}\bigl{\\{}{r}_{k}^{i,j(n)}(\mathbf{p}[t_{s}];{\mathbf{p}}^{(n)}[t_{s}])\tau-\bar{R}_{k}^{i,j}[t_{s}]\bigr{\\}}\qquad$
(26a) $\displaystyle{\mathrm{s.t.}}$
$\displaystyle\quad\sum_{k\in\mathcal{K}}p_{k}^{i,j}[t_{s}]\leq
P_{\max}^{i,j},\ \forall(i,j)$ (26b)

until reaching $\varrho>0$.

0: Set $n:=1$ and generate an initial feasible value for
$\mathbf{p}^{(0)}[t_{s}]$ to constraints in (25)

1: repeat

2: Solve (25) to obtain the optimal transmission power $\mathbf{p}^{*}[t_{s}]$

3: Update ${\mathbf{p}}^{(n)}[t_{s}]:=\mathbf{p}^{*}[t_{s}]$

4: Set $n:=n+1$

5: until Convergence

6: Output: $\mathbf{p}^{*}[t_{s}]={\mathbf{p}}^{(n)}[t_{s}]$ and
$\mathbf{w}_{k}^{i,j,*}[t_{s}]=\frac{\sqrt{p_{k}^{i,j*}[t_{s}]}}{\sqrt{\nu_{k}^{i,j}[t_{s}]}}\mathbf{h}_{k}^{i,j}[t_{s}],\
\forall k,(i,j)$.

Algorithm 2 Proposed Iterative Algorithm for Solving (13) with MRT-Based
Transmission Design

Convergence and complexity analysis: The convergence of an IA-based algorithm
is already provided in [40]. In particular, Algorithm 2 generates an improved
solution after each iteration, which converges to at least a local optimal
solution of (13) when $n\rightarrow\infty$. The worst-case of per-iteration
complexity of Algorithm 2 is $\mathcal{O}\bigl{(}\sqrt{c}(v)^{3}\bigr{)}$ by
the interior-point method [41, Chapter 6], where $c=KJ+J$ and $v=KJ$ are the
numbers of linear constraints and scalar variables, respectively.

####  V-B2 ZFBF-Based Transmission Design

To make ZFBF efficient and feasible, the number of antennas of each RU $(i,j)$
is required to be larger than the number of UEs, i.e. $M_{i,j}>K,\
\forall(i,j)\in\mathcal{J}$, to cancel the inter-user interference transmitted
by this RU. In addition, the system bandwidth is equally allocated to each RU
$(i,j)$, i.e. $W^{i,j}=W/J$, to completely remove the intra-user interference
and interference caused by other RUs. Under the proposed ZFBF technique,
beamformer $\mathbf{w}_{k}^{i,j}[t_{s}]$ at RU $(i,j)$ is designed to satisfy
$(\mathbf{h}_{k^{\prime}}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{w}_{k}^{i,j}[t_{s}]=0,\forall
k^{\prime}\in\mathcal{K}\setminus\\{k\\}$. We denote by
$\mathbf{H}_{-k}^{i,j}[t_{s}]\triangleq\bigr{[}\mathbf{h}_{1}^{i,j}[t_{s}]\cdots\mathbf{h}_{k-1}^{i,j}[t_{s}]\
\mathbf{h}_{k+1}^{i,j}[t_{s}]\cdots\mathbf{h}_{K}^{i,j}[t_{s}]\bigr{]}\in\mathbb{C}^{M\times(K-1)}$
the channel matrix from RU ($i,j$) to UEs, except UE $k$. Let
$\mathbf{V}_{k}^{i,j}[t_{s}]\in\mathbb{C}^{M_{i,j}\times(M_{i,j}-K+1)}$ be the
null space of $(\mathbf{H}_{-k}^{i,j}[t_{s}])^{\mathsf{H}}$. We can then write
$\mathbf{w}_{k}^{i,j}[t_{s}]=\mathbf{V}_{k}^{i,j}[t_{s}]\tilde{\mathbf{w}}_{k}^{i,j}[t_{s}]$,
where $\tilde{\mathbf{w}}_{k}^{i,j}[t_{s}]\in\mathbb{C}^{(M_{i,j}-K+1)\times
1},\forall k,(i,j)$ are the solutions to the ZFBF-based problem. By defining
$\tilde{\nu}_{k}^{i,j}[t_{s}]\triangleq\|(\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\|_{2}^{2}$
with
$\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}]\triangleq(\mathbf{h}_{k}^{i,j}[t_{s}])^{\mathsf{H}}\mathbf{V}_{k}^{i,j}[t_{s}]\in\mathbb{C}^{1\times(M_{i,j}-K+1)}$,
we can equivalently express $\tilde{\mathbf{w}}_{k}^{i,j}[t_{s}]$ as
$\tilde{\mathbf{w}}_{k}^{i,j}[t_{s}]=\sqrt{\tilde{p}_{k}^{i,j}[t_{s}]}\frac{(\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}])^{\mathsf{H}}}{\sqrt{\tilde{\nu}_{k}^{i,j}[t_{s}]}}$,
where
$\tilde{\mathbf{p}}[t_{s}]\triangleq\bigl{[}\tilde{p}_{k}^{i,j}[t_{s}]\bigr{]}_{k,(i,j)\in\mathcal{P}_{k}}$
are the solutions to the following problem:

$\displaystyle\underset{\tilde{\mathbf{p}}[t_{s}]}{\mathrm{max}}$
$\displaystyle\quad\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}{r}_{k}(\tilde{p}_{k}^{i,j}[t_{s}])$
(27a) $\displaystyle{\mathrm{s.t.}}$
$\displaystyle\quad\bar{R}_{k}^{i,j}[t_{s}]\leq
r_{k}^{i,j}(\tilde{p}_{k}^{i,j}[t_{s}])\tau,\ \forall k,(i,j)$ (27b)
$\displaystyle\quad\sum_{k\in\mathcal{K}}\tilde{p}_{k}^{i,j}[t_{s}]\leq
P_{\max}^{i,j},\ \forall(i,j)$ (27c)

where $r_{k}^{i,j}(\tilde{p}_{k}^{i,j}[t_{s}])\triangleq
W^{i,j}\log_{2}\bigl{(}1+\frac{\tilde{p}_{k}^{i,j}[t_{s}]\tilde{\nu}_{k}^{i,j}[t_{s}]}{N_{0}}\bigr{)}$.
The function $r_{k}^{i,j}(\tilde{p}_{k}^{i,j}[t_{s}])$ is concave in
$\tilde{p}_{k}^{i,j}[t_{s}]$, leading to the convexity of problem (27). From
(27b), one can show that
$\tilde{p}_{k}^{i,j}[t_{s}]\geq\tilde{p}_{k,\min}^{i,j}[t_{s}]:=\frac{N_{0}}{\tilde{\nu}_{k}^{i,j}[t_{s}]}2^{\frac{\bar{R}_{k}^{i,j}[t_{s}]}{W^{i,j}\tau}-1}$.
We now develop an efficient method to solve (27) by formulating the partial
Lagrangian as

$\displaystyle L(\tilde{\mathbf{p}}[t_{s}],\boldsymbol{\mu})=$ $\displaystyle\
\sum_{k\in\mathcal{K}}\frac{\hat{q}_{k}[t_{s}]}{\tau}{r}_{k}(\tilde{p}_{k}^{i,j}[t_{s}])$
$\displaystyle+\sum_{(i,j)\in\mathcal{J}}\mu_{i,j}\bigl{(}P_{\max}^{i,j}-\sum_{k\in\mathcal{K}}\tilde{p}_{k}^{i,j}[t_{s}]\bigr{)}$
(28)

where $\boldsymbol{\mu}\triangleq\\{\mu_{i,j}\geq 0\\}_{(i,j)\in\mathcal{J}}$
are the Lagrange multipliers of constraint (27c). The dual function can be
written as $g(\boldsymbol{\mu})=\underset{\tilde{\mathbf{p}}[t_{s}]\geq
0}{\max}\\{L(\tilde{\mathbf{p}}[t_{s}],\boldsymbol{\mu})|\tilde{p}_{k}^{i,j}[t_{s}]\geq\tilde{p}_{k,\min}^{i,j}[t_{s}],\forall
k,$ $(i,j)\\}$. We note that $L(\tilde{\mathbf{p}}[t_{s}],\boldsymbol{\mu})$
is separable with respect to $\tilde{p}_{k}^{i,j}[t_{s}]$. Thus, by solving

$\displaystyle\tilde{p}_{k}^{i,j*}[t_{s}]=$
$\displaystyle\underset{\tilde{p}_{k}^{i,j}[t_{s}]\geq\tilde{p}_{k,\min}^{i,j}[t_{s}]}{\operatornamewithlimits{argmax}}\Bigr{\\{}\frac{\hat{q}_{k}[t_{s}]}{\tau}W\log_{2}\Bigl{(}1+\frac{\tilde{p}_{k}^{i,j}[t_{s}]\tilde{\nu}_{k}^{i,j}[t_{s}]}{N_{0}}\Bigr{)}$
$\displaystyle-\mu_{i,j}\tilde{p}_{k}^{i,j}[t_{s}]\Bigl{\\}}$ (29)

for a given $\mu_{i,j}$, the optimal solution to $\tilde{p}_{k}^{i,j}[t_{s}]$
is given as

$\displaystyle\tilde{p}_{k}^{i,j*}[t_{s}]=\max\Bigr{\\{}\tilde{p}_{k,\min}^{i,j}[t_{s}],\frac{\hat{q}_{k}[t_{s}]W^{i,j}}{\tau\mu_{i,j}\ln
2}-\frac{N_{0}}{\tilde{\nu}_{k}^{i,j}[t_{s}]}\Bigl{\\}}.$ (30)

0: Set $n:=1$ and generate initial values $\underline{\mu}_{i,j}=0$ and
$\overline{\mu}_{i,j}=+\infty,\forall(i,j)\in\mathcal{J}$

1: for each RU $(i,j)\in\mathcal{J}$ in parallel do

2: repeat

3: Compute $\mu_{i,j}^{(n)}=(\underline{\mu}_{i,j}+\overline{\mu}_{i,j})/2$
and $\tilde{p}_{k}^{i,j(n)}[t_{s}]$ as in (30)

4: if $\sum_{k\in\mathcal{K}}\tilde{p}_{k}^{i,j(n)}[t_{s}]-P_{\max}^{i,j}\leq
0$ then

5: Compute $\mu^{\prime}_{i,j}=(\underline{\mu}_{i,j}+\overline{\mu}_{i,j})/2$
and update $\overline{\mu}_{i,j}:=\mu^{\prime}_{i,j}$

6: else

7: Update $\mu^{\prime}_{i,j}=(\underline{\mu}_{i,j}+\overline{\mu}_{i,j})/2$
and update $\underline{\mu}_{i,j}:=\mu^{\prime}_{i,j}$

8: end if

9: Set $n:=n+1$

10: until $\overline{\mu}_{i,j}-\underline{\mu}_{i,j}\leq\delta$ {/*Satisfying
a given accuracy level*/}

11: end for

12: Output: $\mu_{i,j}^{*}=\mu_{i,j}^{(n)}$,
$\tilde{p}_{k}^{i,j*}[t_{s}]=\max\Bigr{\\{}\tilde{p}_{k,\min}^{i,j}[t_{s}],\frac{\hat{q}_{k}[t_{s}]W^{i,j}}{\tau\mu_{i,j}^{*}\ln
2}-\frac{N_{0}}{\tilde{\nu}_{k}^{i,j}[t_{s}]}\Bigl{\\}}$ and
$\mathbf{w}_{k}^{i,j,*}[t_{s}]=\frac{\sqrt{\tilde{p}_{k}^{i,j*}[t_{s}]}}{\sqrt{\tilde{\nu}_{k}^{i,j}[t_{s}]}}\mathbf{V}_{k}^{i,j}[t_{s}](\tilde{\mathbf{h}}_{k}^{i,j}[t_{s}])^{\mathsf{H}},\
\forall k,(i,j)$.

Algorithm 3 Proposed Low-Complexity Algorithm for Solving (13) with ZFBF-Based
Transmission Design

The optimal Lagrange multiplier $\mu_{i,j}$ is efficiently found by applying a
bisection search method between $\underline{\mu}_{i,j}=0$ and a sufficiently
large $\overline{\mu}_{i,j}$. An efficient algorithm for solving (13) with
ZFBF is summarized in Algorithm 3, which does not rely on existing convex
optimization solvers.

###  V-C O-RAN-based Implementation of Algorithm 1

Figure 4: O-RAN Alliance reference architecture for implementing the proposed
JFCS management scheme at time-frame $t$.

Fig. 4 illustrates the key steps for implementing the proposed JFCS management
scheme at time-frame $t$ in the O-RAN architecture.

  1. 1

At the beginning of time-frame $t>1$, the three-step RL procedure for solving
L-SP is carried out at Non-RT RIC based on the collected RAN data in SMO. The
collected data include performance/observation and resource updates from CU,
DU, RU and Near-RT RIC to SMO. For $t=1$, the flow-split decisions are
initialized as
$\boldsymbol{\beta}_{k}[1]=\frac{1}{|\mathcal{P}_{k}^{(1)}|}[1,\cdots,1],\forall
k$ where $\mathcal{P}_{k}^{(1)}$ is the set of RUs in the feasible
communication range of UE $k$.

  2. 3

The optimal flow-split decisions $\boldsymbol{\beta}^{*}[t]$ are sent to Near-
RT RIC via A1 (the standardized open interface) for real deployment.

  3. 5

Near-RT RIC hosts xApps (i.e. third party applications) which communicate with
CU/DU through the E2 interface. Given $\boldsymbol{\beta}^{*}[t]$, xAPPs
deployed in Near-RT RIC control congestion and optimize RAN resources and
functions in each time-slot $t_{s}$ of time-frame $t$ by solving S-SP1 and
S-SP2 to obtain the optimal solutions of congestion control
$\boldsymbol{a}^{*}[t_{s}]$ and beamformer $\mathbf{w}^{*}[t_{s}]$.

  4. 7

Subsequently, the RAN Data Analytic component in Near-RT RIC updates queue-
lengths as in Step 6 of Algorithm 1. The updated queue-lengths are sent back
to SMO through the O1 interface for periodic reporting.

  5. 9

Given $\boldsymbol{\beta}^{*}[t]$ and $\mathbf{w}^{*}[t_{s}]$, the optimal
service rate $\mathbf{r}(\mathbf{w}^{*}[t_{s}])$ is scheduled and applied to
CU and DUs through the E2 interface.

  6. 11

After $T_{f}$ time-slots in the short-term scale $t_{s}$, performance and
observations (e.g. $\mathbf{q}[t-1],\mathbf{A}[t-1]$) are updated to SMO
through the O1 interface to re-estimate the flow-split decision
$\boldsymbol{\beta}^{*}[t+1]$.

##  VI Performance Analysis of The JFCS Framework

In this section, we analyze the main theoretical performance results of
Algorithm 1 and discuss their key insights, followed by concrete proofs of the
theorems.

######  Assumption 2.

To facilitate the analysis, we make the following additional assumptions.

  * •

Under the limited transmit power budget at RUs, the achievable rate of UE $k$
is upper bounded by $r^{\max}>0$, i.e., $r_{k}(\mathbf{w}[t_{s}])\leq
r^{\max},\ \forall k,t_{s}$.

  * •

The congestion control variable $a_{k}[t_{s}]$ satisfies the condition
$\mathbb{E}\\{a_{k}^{2}[t_{s}]\\}\leq A_{1}^{\max}$, where $A_{1}^{\max}$ is a
sufficiently large positive constant [34].

######  Theorem 1 (Bounding the mean divergence of the auxiliary queue-
length).

For a given scaling factor $\varphi$, let
$\hat{\mathbf{q}}^{\infty}_{(\varphi)}$ and $\hat{\mathbf{q}}^{*}_{(\varphi)}$
be the steady-state and optimal queue-lengths, respectively. From Assumptions
1 and 2, the expected upper bound of the divergence of
$\hat{\mathbf{q}}^{\infty}_{(\varphi)}$ from
$\hat{\mathbf{q}}^{*}_{(\varphi)}$ is given as

$\mathbb{E}\\{\|\hat{\mathbf{q}}^{\infty}_{(\varphi)}-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}\\}\leq\mathsf{C}_{1}\sqrt{\varphi}=\mathcal{O}(\sqrt{\varphi})$
(31)

where
$\mathsf{C}_{1}\triangleq\sqrt{\frac{K\tau^{2}\Psi}{2}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}}$
is a positive constant.

The proof is detailed in Appendix B. Theorem 1 implies that the divergence of
the steady-state queue-length is bounded by $\mathcal{O}(\sqrt{\varphi})$. In
particular, the smaller the value of $\varphi$, the less the divergence of
$\hat{\mathbf{q}}^{\infty}_{(\varphi)}$. However, a small $\varphi$ will also
result in a small congestion control rate and a faster convergence. When
$\varphi$ is large, a better congestion control rate is achieved but with the
cost of larger steady-state queue-length divergence (i.e. larger delay and
slower convergence). Hence, there exists an appropriate value of $\varphi$ to
make this tradeoff more efficient. Theorem 1 will immediately lead to the
following result.

######  Corollary 1 (Queue-stability).

Given a scaling factor $\varphi$ and $\mathsf{C}_{1}$ in (31), the steady-
state total queue-length remains finite and scales as
$\mathcal{O}(\varphi)+\mathcal{O}(\sqrt{\varphi})$, i.e.

$\displaystyle\underset{t_{s}\to\infty}{\limsup}\
\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]\|_{1}\\}$
$\displaystyle\leq\tau\Psi
KA^{\max}\varphi+\sqrt{K}\mathsf{C}_{1}\sqrt{\varphi}$
$\displaystyle=\mathcal{O}(\varphi)+\mathcal{O}(\sqrt{\varphi}).$ (32)

###### Proof.

The proof of (1) is straightforward by noticing the fact that
$\underset{t_{s}\to\infty}{\limsup}\
\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]\|_{1}\\}=\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}\|_{1}\\}=\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}+\|\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}\\}$.
Applying the inequality $\|\mathbf{x}\|_{1}\leq\sqrt{K}\|\mathbf{x}\|_{2}$ for
any $\mathbf{x}\in\mathbb{R}_{+}^{K}$ yields:
$\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}+\|\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}\leq\sqrt{K}\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{2}+\|\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}$.
From (31) and Step 4 of Algorithm 1, it follows that

$\displaystyle\underset{t_{s}\to\infty}{\limsup}\
\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]\|_{1}\\}$
$\displaystyle\leq\sqrt{K}\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{2}\\}+\|\hat{\mathbf{q}}_{(\varphi)}^{*}\|_{1}$
$\displaystyle\leq\sqrt{K}\mathsf{C}_{1}\sqrt{\varphi}+\tau\sum_{k\in\mathcal{K}}U_{k}^{{}^{\prime}}\bigl{(}a_{k}^{*}\bigr{)}\varphi\leq\sqrt{K}\mathsf{C}_{1}\sqrt{\varphi}+\tau\Psi\sum_{k\in\mathcal{K}}a_{k}^{*}\varphi\
$ $\displaystyle\leq\sqrt{K}\mathsf{C}_{1}\sqrt{\varphi}+\tau\Psi
KA^{\max}\varphi\ (\text{due to}\ a_{k}^{*}\leq A^{\max},\forall k)$ (33)

showing (1).∎

Let
$\boldsymbol{a}^{\infty}_{(\varphi)}\triangleq[a^{\infty}_{(\varphi),k}]^{\mathsf{T}}_{k\in\mathcal{K}}$
with $a^{\infty}_{(\varphi),k}=\mathbb{E}\bigl{\\{}\min\\{A^{\max},$
$U_{k}^{{}^{\prime}-1}\bigl{(}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}\bigr{)}\\}\bigl{\\}}$
be the mean steady-state congestion control rate vector. We also denote by
$U(\boldsymbol{a})\triangleq\sum_{k\in\mathcal{K}}U_{k}(a_{k})$ the total
utility function of problem (7). The utility-optimality of Algorithm 1 is
stated by the following theorem, whose proof is given in Appendix C.

######  Theorem 2 (Optimality).

Given a scaling factor $\varphi$, Algorithm 1 produces the mean steady-state
congestion control rate vector $\boldsymbol{a}^{\infty}_{(\varphi)}$,
satisfying

$\displaystyle\|\boldsymbol{a}^{\infty}_{(\varphi)}-\boldsymbol{a}^{*}\|_{2}\leq\mathsf{C}_{2}\frac{1}{\sqrt{\varphi}}=\mathcal{O}(1/\sqrt{\varphi})$
(34)

where
$\mathsf{C}_{2}\triangleq\frac{\mathsf{C}_{1}}{\psi\tau}=\sqrt{\frac{K\Psi}{2\psi}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}}$.
Therefore, the optimal network utility maximization is bounded as

$\displaystyle
U(\boldsymbol{a}^{*})-\mathsf{C}_{3}\frac{1}{\varphi}=U(\boldsymbol{a}^{*})-\mathcal{O}(1/\varphi)\leq
U(\boldsymbol{a}^{\infty}_{(\varphi)})$ (35)

where
$\mathsf{C}_{3}\triangleq\frac{\Psi\mathsf{C}_{1}^{2}}{2\psi^{2}\tau^{2}}=\frac{K\Psi^{2}}{4\psi}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}$.

The analytical results in Theorem 2 show that the divergence of the steady-
state congestion control rate vector $\boldsymbol{a}^{\infty}_{(\varphi)}$
from $\boldsymbol{a}^{*}$ scales as $\mathcal{O}(1/\sqrt{\varphi})$, which is
the same as in [34, 42]. The utility-optimality gap can be reduced by
increasing $\varphi$, but this will also lead to a larger steady-state queue-
length divergence.

##  VII Numerical Results

In this section, we first present simulation setup and parameters in Section
VII-A and then provide numerical results of Algorithm 1 in Section VII-B. The
results and performance comparison over existing schemes will be provided in
Section VII-C.

###  VII-A Simulation Setups and Parameters

Figure 5: A system topology with $J=8$ RUs and $K=12$ UEs. TABLE I: Simulation
Parameters

Parameter | Value
---|---
System bandwidth, $W$ | 20 MHz
Number of RUs, $J$ | 8
Number of UEs, $K$ | 12
Number of antennas at RUs $(i,j)$, $M_{i,j}\equiv M$ | 16
RUs’ height | 10 m
UEs’ antenna altitude | 1.5 m
Power budget at RU $(i,j)$, $P_{\max}^{i,j}\equiv P_{\max}$ | 43 dBm
Noise figure, NF | 9 dB
Maximum average delay, $\bar{d}_{k}\equiv d$ | 10 ms
Require reliable communication, $\epsilon_{k}\equiv\epsilon$ | 0.95
Number of frames, $T$ | 10000
Number of time-slots per frame, $T_{f}$ | 10
Duration of one frame, $T_{c}$ | 10 ms
Duration of one time-slot, $\tau$ | 1 ms
Trade-off factor (Boltzmann temperature), $\lambda$ | 0.3

We consider a system topology given in Fig. 5, including 8 RUs and $12$ UEs
located within a circle of 1-km radius. There are two DUs, each connected to 4
RUs. RUs are uniformly distributed in the area, while those of UEs are
randomly located in each time-frame $t.$ The large-scale fading coefficient
$\xi[t]\in\\{\xi_{k}^{i,j}[t]\\}_{\forall(i,j),k}$ is modeled as the three-
slope path loss model [32], such as
$\xi[t]=\xi_{0}-35\log_{10}(d[t])+20c_{0}\log_{10}(d/d_{0})+15c_{1}\log_{10}(d/d_{1})$
where $\xi_{0}=-140.7+\mathsf{SF}$ dB, $d_{0}=10$ m, $d_{1}=50$ m, and $d$ is
the distance between an RU and a UE; here
$c_{i}=\max\\{0,\frac{d_{i}-d}{|d_{i}-d|}\\}$ with $i\in\\{0,1\\}$ and
$\mathsf{SF}\sim\mathcal{CN}(0,\sigma_{\mathsf{SF}})$ denotes the shadowing
factor with $\sigma_{\mathsf{SF}}=8$ dB. The Rician factor
$\kappa[t]\in\\{\kappa_{k}^{i,j}[t]\\}_{\forall(i,j),k}$ is given as
$\kappa=P_{\mathsf{LoS}}(d[t])/\big{(}1-P_{\mathsf{LoS}}(d[t])\big{)}$, where
the LoS probability follows the 3GPP–UMa model as
$P_{\mathsf{LoS}}(d[t])=\min\left(\frac{18}{d[t]},1\right)\big{(}1-\exp(-\frac{d[t]}{36})\big{)}+\exp(-\frac{d[t]}{36})$
[43]. We consider uniform linear arrays with half-wavelength distances between
array elements to model the LoS channels at RUs. The array response vector is
generated as $\bar{\mathbf{h}}^{i,j}_{k}[t]=\textbf{{a}}(\phi^{i,j}_{k}[t])$,
where each element $m$ is given as
$\big{[}\textbf{{a}}(\phi^{i,j}_{k}[t])\big{]}_{m}=\exp\bigr{(}j\pi(m-1)\sin\phi^{i,j}_{k}[t]\bigr{)}$
with $\phi^{i,j}_{k}[t]\in[-\pi/2,\pi/2)$ being the angle-of-departure (AoD)
at RU $(i,j)$. The noise power is modeled as
$N_{0}=-170+10\log_{10}(W)+\mathsf{NF}$ dBm, where $\mathsf{NF}=9$ dB denotes
the noise figure.

We run Algorithm 1 over $T=10000$ frames, each consists of $T_{f}=10$ time-
slots (subframes) and has duration of $T_{c}=10$ ms, followed by 5G NR Frame
structure [44]. In each time-frame $t$, UE $k$ is served by a subset of four
RUs. To illustrate the heterogeneity of UEs, we assume that the arrival rate
$A_{k}[t]$ is uniformly distributed in [1, 3] Gbps. The step sizes (learning
rates) are set to decrease after each frame as $\eta_{u}[t]=1/(t+1)^{0.51}$,
$\eta_{\theta}[t]=1/(t+1)^{0.55}$ and $\eta_{\beta}[t]=1/(t+1)^{0.6}$ [45]. We
adopt the proportional fairness metric to model the utility function as:
$U_{k}(r_{k})=\log(0.001+r_{k}),\forall k$ [46]. The key parameters are
summarized in Table I for ease of cross-referencing, followed by studies in
[43, 45, 44, 32, 38]. In the following figures, results are averaged over the
last 6000 frames.

Benchmark schemes: To demonstrate the benefits of the proposed JFCS algorithm,
we consider the following three benchmark schemes:

  * •

“NUM with fixed resource allocation (NUM-FRA)” [47]: Under Algorithm 1, RUs
allocate power equally to UEs.

  * •

“NUM with equal flow-split distribution (NUM-EFSD):” CU splits data-flows of
all UEs equally among the selected paths, i.e.,
$\beta_{k}^{i,j}[t]=1/|\mathcal{P}_{k}|,\forall(i,j)\in\mathcal{P}_{k}$.

  * •

“NUM with the nearest RU selection (NUM-NRU):” Under Algorithm 1, each UE $k$
selects only the nearest RU for the data transmission, i.e.
$\beta_{k}^{i,j}[t]=1$ if RU $(i,j)$ is the nearest RU to UE $k$.

###  VII-B Numerical Results of Algorithm 1

(a) Impact of $\varphi$ on congestion control rate

(b) Impact of $\lambda$ on estimated utility

Figure 6: Convergence behavior of Algorithm 1 with ZFBF.

We first study the impacts of $\varphi$ and $\lambda$ on the convergence
behavior of Algorithm 1 in Fig. 6. From Fig. 6(a), it can be observed that the
congestion control rates for different values of the scaling factor $\varphi$
converge to the same optimal solution, and $\|\boldsymbol{a}[t_{s}]\|$ is
almost independent of $\varphi$. In addition, increasing $\varphi$ results in
a smaller divergence of the steady-state congestion control rate (see Theorem
2), but also slows down the convergence rate of Algorithm 1. The reason is
attributed to the fact that for a large $\varphi$, the network utility
function $\sum_{k\in\mathcal{K}}U_{k}(a_{k}[t_{s}])$ in (8a) will prevail over
the Lyapunov drift function $\Delta L[t_{s}]$, which requires more iterations
to guarantee network stability. In Fig. 6(b), we increase the trade-off factor
$\lambda$ (i.e. Boltzmann temperature) from 0.05 to 0.7. The result shows that
the larger the value of $\lambda$, the better the estimated utility that can
be achieved with the cost of lower convergence speed of the RL process. From
(V-A), the paths associated with the highest estimated regret
$\hat{\theta}_{k}^{i,j}[t]$ will be selected to minimize the best response
function $f(\hat{\boldsymbol{\theta}}[t])$. Conversely, a low value of
$\lambda$ can speed up convergence by allocating traffic data uniformly to all
paths but leads to a very sub-optimal solution.

Figure 7: Performance of Algorithm 1 with different transmission strategies
versus the number of antennas at RUs, $M\equiv M_{i,j},\forall(i,j)$.

In Fig. 7, we evaluate the performance of Algorithm 1 with different
transmission strategies, namely MRT and ZFBF. For a fixed $\varphi=25$, we
vary the number of antennas at RUs $M\equiv M_{i,j},\forall(i,j)$ from 16 to
128. For each transmission design, we also plot the steady-state congestion
control rate $\mathbb{E}\\{\|\boldsymbol{a}_{(25)}^{\infty}\|\\}$ with the
equal flow-split distribution. As seen from Fig. 7 that the steady-state
congestion control rate of all schemes increases as $M$ increases.
Unsurprisingly, Algorithm 1 with ZFBF offers better performance in terms of
congestion control rate than that of MRT when the number of antennas at RUs is
sufficiently large to cancel the inter-user interference transmitted by the
same RU. It is obvious that the higher the effective data rate of a data-flow
in the downlink, the lower the total queue-length of that data-flow (or user),
resulting in a higher congestion control rate.

Since Algorithm 1 with MRT is based on the IA method that requires high
computation complexity and relies on existing convex optimization solvers, we
provide only the performance of Algorithm 1 with ZFBF in the following
section.

###  VII-C Performance Comparison

Figure 8: The steady-state congestion control rate with respect to the number
of antennas at RUs, $M\equiv M_{i,j},\forall(i,j)$.

Next, we show the performance comparison in terms of the steady-state
congestion control rate $\mathbb{E}\\{\|\boldsymbol{a}_{(25)}^{\infty}\|\\}$
among the considered schemes versus the number of antennas at RUs in Fig. 8.
We fix $\varphi=25$ and vary $M$ from 16 to 128 to investigate the impact of
the physical factor. As $M$ increases, the downlink instantaneous achievable
rates of all UEs also significantly increase since more degrees of freedom are
added to leverage multi-user diversity, resulting in lower queue-lengths. For
a fixed value of $\varphi$, the steady-state congestion control rate vector
increases monotonically with $M$. Clearly, Algorithm 1 outperforms the
benchmark schemes in all ranges of $M$, and the gap is deeper when $M$ is
small. In addition, the NUM-FRA and NUM-NRU, which fairly allocate the power
budget and fix the path selection to UEs, respectively, provide the worst
performance. These observations demonstrate the effectiveness of the proposed
Algorithm 1 by jointly optimizing the flow-split distribution, congestion
control, scheduling and radio resource allocation.

Lastly, the impacts of scaling factor $\varphi$ on the steady-state total
queue-length $\mathbb{E}\\{\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}\|_{1}\\}$
and average worst-case delay (i.e., the delay of slowest data-flow) are
plotted in Figs. 9 and 10, respectively. It can be seen from Fig. 9 that the
steady-state total queue-length of all schemes monotonically scales as
$\mathcal{O}(\varphi)+\mathcal{O}(\sqrt{\varphi})$, which confirms our
theoretical results in Corollary 1. We recall from Theorem 2 that the utility-
optimality gap can be narrowed by increasing $\varphi$, but with the cost of
higher delay, as shown in Fig. 10. When $\varphi$ is larger than 25, all the
considered schemes violate the maximum allowable average delay of $\bar{d}=10$
ms. It implies that the data traffic cannot be completely transmitted to UEs
in each time-frame. Nevertheless, Algorithm 1 still provides the best
performance out of the schemes considered.

Figure 9: The steady-state total queue-length with respect to $\varphi$.
Figure 10: Average worst-case delay with respect to $\varphi$.

##  VIII Conclusion

We have proposed a new holistic multi-layer optimization framework, called
JFCS, to enable intelligent traffic steering in a hierarchical O-RAN
architecture. In particular, we have developed an intelligent resource
management algorithm based on network utility maximization and stochastic
optimization to efficiently and adaptively direct traffic to appropriate RUs
by jointly optimizing the flow-split distribution, congestion control and
scheduling. JFCS is proved to achieve fast convergence, long-term utility-
optimality and significant delay reduction compared to state-of-the-art
approaches. To that end, the insights in this work will foster future studies
in this area, especially in the design of more advanced AI/ML solutions to
achieve enhanced control and flexibility in O-RAN.

##  Appendix A Derivation of Inequality

We will find the concave lower bound of $r_{k}^{i,j}[t_{s}]$. By [48, Appendix
A], it is true that the function $r(x,y)=-\ln(1-x^{2}/y)$ is convex in the
domain $y>x^{2}$ with $x,y\in\mathbb{R}_{+}$. The global concave lower bound
of $r(x,y)$ at the feasible point ($\bar{x},\bar{y}$) is given as

$\displaystyle r(x,y)$ $\displaystyle\geq
r(\bar{x},\bar{y})+\Bigl{<}\Bigr{(}\frac{\partial
r(\bar{x},\bar{y})}{\partial\bar{x}},\frac{\partial
r(\bar{x},\bar{y})}{\partial\bar{y}}\Bigl{)},(x-\bar{x},y-\bar{y})\Bigl{>}$
$\displaystyle=r(\bar{x},\bar{y})-\frac{\bar{x}^{2}}{\bar{y}-\bar{x}^{2}}+2\frac{\bar{x}x}{\bar{y}-\bar{x}^{2}}-\frac{\bar{x}^{2}}{\bar{y}-\bar{x}^{2}}\frac{y}{\bar{y}}$
(A.1)

by applying the first-order Taylor approximation. By the fact that
$\ln\bigl{(}1+\frac{x^{2}}{z}\bigr{)}=-\ln\bigl{(}1-\frac{x^{2}}{z+x^{2}}\bigr{)}$
and substituting $y=z+x^{2}$, $\bar{y}=\bar{z}+\bar{x}^{2}$, $x=\sqrt{v}$ and
$\bar{x}=\sqrt{\bar{v}}$ into (A.1), we obtain

$\displaystyle r(v,z)$
$\displaystyle\triangleq\ln\bigl{(}1+\frac{v}{z}\bigr{)}\geq
r(\bar{v},\bar{z})-\frac{\bar{v}}{\bar{z}}+2\frac{\sqrt{\bar{v}}\sqrt{v}}{\bar{z}}-\frac{\bar{v}(z+v)}{\bar{z}(\bar{z}+\bar{v})}$
$\displaystyle:=\bar{r}(v,z;\bar{v},\bar{z})$ (A.2)

where $\bar{r}(v,z;\bar{v},\bar{z})$ is concave and
$\bar{r}(\bar{v},\bar{z};\bar{v},\bar{z})=r(\bar{v},\bar{z})$ whenever
$(v,z)=(\bar{v},\bar{z})$.

##  Appendix B Proof of Theorem 1

For a given $\varphi$, the quadratic Lyapunov function defined in Section IV-A
is rewritten with respect to $\hat{\mathbf{q}}_{(\varphi)}[t_{s}]$ as:
$L(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])=\frac{1}{2\tau^{2}}\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}$.
Following [34, Theorem 3], the mean Lyapunov drift from time-slot $t_{s}$ to
$t_{s+1}$ is computed as

$\displaystyle\Delta\bar{L}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])$
$\displaystyle=\mathbb{E}\\{\Delta
L(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])\\}=\mathbb{E}\\{L(\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}])-L(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])\\}$
$\displaystyle=\frac{1}{2\tau^{2}}\mathbb{E}\Bigl{\\{}\bigl{(}\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}]+\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-2\hat{\mathbf{q}}^{*}_{(\varphi)}\bigr{)}^{\mathsf{T}}$
$\displaystyle\quad\times\bigl{(}\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}]-\hat{\mathbf{q}}_{(\varphi)}[t_{s}]\bigr{)}\Bigr{\\}}$
$\displaystyle\leq\frac{1}{2\tau}\mathbb{E}\Bigl{\\{}\bigl{(}2\hat{\mathbf{q}}_{(\varphi)}[t_{s}]+\bigr{(}\boldsymbol{a}[t_{s}]-\mathbf{r}(\mathbf{w}[t_{s}])\bigl{)}\tau-2\hat{\mathbf{q}}^{*}_{(\varphi)}\bigr{)}^{\mathsf{T}}$
$\displaystyle\quad\times\bigl{(}\boldsymbol{a}[t_{s}]-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\Bigr{\\}}$
$\displaystyle=\underbrace{\frac{1}{2}\mathbb{E}\\{\|\boldsymbol{a}[t_{s}]-\mathbf{r}(\mathbf{w}[t_{s}])\|_{2}^{2}\\}}_{\triangleq\mathsf{B}_{1}}$
$\displaystyle\quad+\underbrace{\frac{1}{\tau}\mathbb{E}\\{(\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\bigl{(}\boldsymbol{a}[t_{s}]-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\\}}_{\triangleq\mathsf{B}_{2}}$
(B.1)

by using the inequalities: $([x]^{+})^{2}\leq x^{2}$ and
$x^{2}-y^{2}=(x+y)(x-y)$, and the fact that
$\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}]-\hat{\mathbf{q}}^{*}_{(\varphi)}=\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)}+\bigr{(}\boldsymbol{a}[t_{s}]-\mathbf{r}(\mathbf{w}[t_{s}])\bigl{)}\tau.$

We first focus on providing the expected bound of $\mathsf{B}_{1}$ as

$\displaystyle\mathsf{B}_{1}$
$\displaystyle=\frac{1}{2}\mathbb{E}\\{\|\boldsymbol{a}[t_{s}]\|_{2}^{2}-2\boldsymbol{a}[t_{s}]^{\mathsf{T}}\mathbf{r}(\mathbf{w}[t_{s}])+\|\mathbf{r}(\mathbf{w}[t_{s}])\|_{2}^{2}\\}$
$\displaystyle\leq\frac{1}{2}\mathbb{E}\\{\|\boldsymbol{a}[t_{s}]\|_{2}^{2}+\|\mathbf{r}(\mathbf{w}[t_{s}])\|_{2}^{2}\\}$
$\displaystyle\leq\frac{K}{2}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}\triangleq\mathsf{B}_{1}^{\mathtt{UB}}$
(B.2)

where the last inequality follows from Assumption 2. To bound
$\mathsf{B}_{2}$, we first rewrite it equivalently as

$\displaystyle\mathsf{B}_{2}=$
$\displaystyle\frac{1}{\tau}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\bigl{(}\mathbb{E}\bigl{\\{}\boldsymbol{a}[t_{s}]\\}-\mathbf{r}^{*}\bigr{)}$
$\displaystyle+\frac{1}{\tau}\mathbb{E}\bigl{\\{}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\bigl{(}\mathbf{r}^{*}-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\bigr{\\}}.$
(B.3)

From (7), it follows that
$(\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\bigl{(}\mathbb{E}\bigl{\\{}\boldsymbol{a}[t_{s}]\\}-\mathbf{r}^{*}\bigr{)}\leq
0$. By applying the Cauchy–Schwarz inequality, i.e.
$|\mathbf{x}^{\mathsf{T}}\mathbf{y}|\leq\|\mathbf{x}\|_{2}\|\mathbf{y}\|_{2}$,
to the first term in (B), we have

$\displaystyle\frac{1}{\tau}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\bigl{(}\mathbb{E}\bigl{\\{}\boldsymbol{a}[t_{s}]\\}-\mathbf{r}^{*}\bigr{)}$
$\displaystyle\leq-\frac{1}{\tau}\sum_{k\in\mathcal{K}}|\hat{q}_{(\varphi),k}[t_{s}]-\hat{q}^{*}_{(\varphi),k}||a_{k}[t_{s}]-r^{*}_{k}|.$
(B.4)

By Assumption 1 on $\Psi$-smooth and Step 4 of Algorithm 1, it is true that
$a_{k}[t_{s}]-r^{*}_{k}=U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}[t_{s}]}{\varphi\tau}\bigl{)}$
$-U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}[t_{s}]}{\varphi\tau}\bigl{)}\leq
0$ and
$\bigr{|}U_{k}^{{}^{\prime}}\bigr{(}\frac{\hat{q}_{(\varphi),k}[t_{s}]}{\varphi\tau}\bigl{)}-U_{k}^{{}^{\prime}}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}[t_{s}]}{\varphi\tau}\bigl{)}\bigr{|}\leq\Psi\bigl{|}\frac{\hat{q}_{(\varphi),k}[t_{s}]}{\varphi\tau}-\frac{\hat{q}_{(\varphi),k}^{*}[t_{s}]}{\varphi\tau}\bigr{|}$.
In addition, we have
$\bigr{|}U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}[t_{s}]}{\varphi\tau}\bigl{)}-U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}[t_{s}]}{\varphi\tau}\bigl{)}\bigr{|}\geq\frac{1}{\Psi}\bigl{|}\frac{\hat{q}_{(\varphi),k}[t_{s}]}{\varphi\tau}-\frac{\hat{q}_{(\varphi),k}^{*}[t_{s}]}{\varphi\tau}\bigr{|}$
due to the inverse function lemma. From the fact that
$(\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\mathbf{r}^{*}-(\hat{\mathbf{q}}^{*}_{(\varphi)})^{\mathsf{T}}\mathbf{r}(\mathbf{w}[t_{s}])\geq
0$, we can further bound $\mathsf{B}_{2}$ as

$\displaystyle\mathsf{B}_{2}\leq-\frac{1}{\tau^{2}\Psi\varphi}\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}$
$\displaystyle\qquad+\frac{1}{\tau}\mathbb{E}\bigl{\\{}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])^{\mathsf{T}}\bigl{(}\mathbf{r}^{*}-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\bigr{\\}}$
(B.5)

where the term
$\mathbb{E}\bigl{\\{}(\mathbf{r}^{*}-\mathbf{r}(\mathbf{w}[t_{s}]))\bigl{\\}}$
is a constant with respect to $\hat{\mathbf{q}}_{(\varphi)}[t_{s}]$.
Substituting (B) and (B) into (B) yields

$\displaystyle\Delta\bar{L}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])\leq$
$\displaystyle-\frac{1}{\tau^{2}\Psi\varphi}\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}+\mathsf{B}_{1}^{\mathtt{UB}}$
$\displaystyle+\frac{1}{\tau}\mathbb{E}\bigl{\\{}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])^{\mathsf{T}}\bigl{(}\mathbf{r}^{*}-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\bigr{\\}}.$
(B.6)

We now compute the mean Lyapunov drift over $TT_{f}$ time-slots as

$\displaystyle\Delta\bar{L}$
$\displaystyle=\sum_{t=1}^{T}\sum_{s=1}^{T_{f}}\mathbb{E}\\{L(\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}])-L(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])|\hat{\mathbf{q}}_{(\varphi)}[1_{1}]\\}$
(B.7)
$\displaystyle=\sum_{t=1}^{T}\sum_{s=1}^{T_{f}}\sum_{\hat{\mathbf{q}}_{(\varphi)}\geq
0}\Bigl{(}\mathsf{Prob}\bigl{(}\hat{\mathbf{q}}_{(\varphi)}[t_{s}]=\hat{\mathbf{q}}_{(\varphi)}|\hat{\mathbf{q}}_{(\varphi)}[1_{1}]\bigr{)}$
$\displaystyle\times\mathbb{E}\\{L(\hat{\mathbf{q}}_{(\varphi)}[t_{s+1}])-L(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]=\hat{\mathbf{q}}_{(\varphi)}\\}\Bigr{)}.\qquad$

Let us denote by $\rho_{\hat{\mathbf{q}}_{(\varphi)}}^{\infty}$ the stationary
distribution of the Markov chain $\hat{\mathbf{q}}_{(\varphi)}[t_{s}]\geq 0$,
i.e.
$\rho_{\hat{\mathbf{q}}_{(\varphi)}}^{\infty}=\lim_{T\rightarrow\infty}\frac{1}{TT_{f}}\sum_{t=1}^{T}\sum_{s=1}^{T_{f}}\mathsf{Prob}\bigl{(}\hat{\mathbf{q}}_{(\varphi)}[t_{s}]=\hat{\mathbf{q}}_{(\varphi)}|\hat{\mathbf{q}}_{(\varphi)}[1_{1}]\bigr{)}$.
By substituting (B) into (B.7) and dividing both side with $TT_{f}$, we have

$\displaystyle\sum_{\hat{\mathbf{q}}_{(\varphi)}\geq
0}\rho_{\hat{\mathbf{q}}_{(\varphi)}}^{\infty}\Bigl{(}-\frac{1}{\tau^{2}\Psi\varphi}\|\hat{\mathbf{q}}_{(\varphi)}[t_{s}]-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}+\mathsf{B}_{1}^{\mathtt{UB}}$
$\displaystyle\quad+\,\frac{1}{\tau}(\hat{\mathbf{q}}_{(\varphi)}[t_{s}])^{\mathsf{T}}\mathbb{E}\bigl{\\{}\bigl{(}\mathbf{r}^{*}-\mathbf{r}(\mathbf{w}[t_{s}])\bigr{)}\bigr{\\}}\Bigr{)}$
$\displaystyle=-\frac{1}{\tau^{2}\Psi\varphi}\mathbb{E}\bigl{\\{}\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}\bigl{\\}}+\mathsf{B}_{1}^{\mathtt{UB}}$
$\displaystyle\qquad+\frac{1}{\tau}\mathbb{E}\bigl{\\{}(\hat{\mathbf{q}}_{(\varphi)}^{\infty})^{\mathsf{T}}\bigl{(}\mathbf{r}^{*}-\mathbf{r}^{\infty}\bigr{\\}}\geq
0$ (B.8)

where
$\mathbf{r}^{\infty}=\underset{r_{k}(\mathbf{w})\in{C}_{\mathbf{H}[\infty]},\forall
k\in\mathcal{K}}{\operatornamewithlimits{argmax}}\
\sum_{k\in\mathcal{K}}\hat{q}_{k}^{\infty}r_{k}(\mathbf{w})$. We note here
that
$(\hat{\mathbf{q}}_{(\varphi)}^{\infty})^{\mathsf{T}}\mathbf{r}^{\infty}=\underset{r_{k}(\mathbf{w})\in{C}_{\mathbf{H}[\infty]},\forall
k\in\mathcal{K}}{\max}$
$\sum_{k\in\mathcal{K}}\hat{q}_{k}^{\infty}r_{k}(\mathbf{w})\geq(\hat{\mathbf{q}}_{(\varphi)}^{\infty})^{\mathsf{T}}\mathbf{r}^{*}$,
yielding

$\displaystyle\frac{1}{\tau^{2}\Psi\varphi}\mathbb{E}\bigl{\\{}\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}^{2}\bigr{\\}}-\mathsf{B}_{1}^{\mathtt{UB}}\leq
0$ (B.9)

This implies that
$\mathbb{E}\bigl{\\{}\|\hat{\mathbf{q}}_{(\varphi)}^{\infty}-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}\bigr{\\}}\leq\sqrt{\frac{K\tau^{2}\Psi}{2}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}}\sqrt{\varphi}$
where
$\mathsf{B}_{1}^{\mathtt{UB}}=\frac{K}{2}\bigl{(}A_{1}^{\max}+(r^{\max})^{2}\bigr{)}$,
showing the inequality (31) in Theorem 1.

##  Appendix C Proof of Theorem 2

To prove (34), we first recall that
$a_{(\varphi),k}^{\infty}-a^{*}_{k}=U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}\bigl{)}$
$-U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}}{\varphi\tau}\bigl{)}$
and
$\bigr{|}U_{k}^{{}^{\prime}}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}\bigl{)}$
$-U_{k}^{{}^{\prime}}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}}{\varphi\tau}\bigl{)}\bigr{|}\geq\psi\bigl{|}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}-\frac{\hat{q}_{(\varphi),k}^{*}}{\varphi\tau}\bigr{|}$
using Assumption 1. By the inverse function lemma, we have
$\bigr{|}U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}\bigl{)}$
$-U_{k}^{{}^{\prime}-1}\bigr{(}\frac{\hat{q}_{(\varphi),k}^{*}}{\varphi\tau}\bigl{)}\bigr{|}\leq\frac{1}{\psi}\bigl{|}\frac{\hat{q}_{(\varphi),k}^{\infty}}{\varphi\tau}-\frac{\hat{q}_{(\varphi),k}^{*}}{\varphi\tau}\bigr{|}$,
which yields

$\displaystyle\|\boldsymbol{a}^{\infty}_{(\varphi)}-\boldsymbol{a}^{*}\|_{2}\leq\frac{1}{\psi\tau\varphi}\|\hat{\mathbf{q}}^{\infty}_{(\varphi)}-\hat{\mathbf{q}}^{*}_{(\varphi)}\|_{2}\overset{\eqref{eq_theo1eq}}{\leq}\frac{\mathsf{C}_{1}}{\psi\tau}\frac{1}{\sqrt{\varphi}}.$
(C.1)

Next, it is assumed that $U_{k}(\cdot)$ is twice continuously differentiable,
increasing, and strictly concave. If the utility function $U(\boldsymbol{a})$
has a maximizer $\boldsymbol{a}^{*}$, then

$\displaystyle
U(\boldsymbol{a}^{*})-U(\boldsymbol{a}^{\infty}_{(\varphi)})\leq\frac{\Psi}{2}\|\boldsymbol{a}^{*}-\boldsymbol{a}^{\infty}_{(\varphi)}\|_{2}^{2}\leq\frac{\Psi\mathsf{C}_{1}^{2}}{2\psi^{2}\tau^{2}}\frac{1}{\varphi}$
(C.2)

where the last inequality follows from (C.1). The proof is thus complete.

## References

  * [1] W. Saad, M. Bennis, and M. Chen, “A vision of 6G wireless systems: Applications, trends, technologies, and open research problems,” _IEEE Network_ , vol. 34, no. 3, pp. 134–142, 2020.
  * [2] K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y.-J. A. Zhang, “The roadmap to 6G: AI empowered wireless networks,” _IEEE Commun. Mag._ , vol. 57, no. 8, pp. 84–90, 2019.
  * [3] O-RAN Alliance, “ORAN-WG1 O-RAN architecture description v01.00.00.” _Technical Specification_ , Feb. 2020.
  * [4] SAMSUNG, “ORAN - The Open Road to 5G,” _White Paper_ , July 2019. [Online]. Available: https://www.samsung.com/ global/business/networks/insights/whitepapers /ORAN-the-open-road-to-5g/
  * [5] O-RAN Alliance, “ORAN Working Group 2: AI/ML workflow description and requirements,” _Tech. Rep._ , Mar. 2019.
  * [6] L. Bonati, S. D’Oro, M. Polese, S. Basagni, and T. Melodia, “Intelligence and learning in O-RAN for data-driven nextG cellular networks,” _IEEE Commun. Mag._ , vol. 59, no. 10, pp. 21–27, 2021.
  * [7] L. Gavrilovska, V. Rakovic, and D. Denkovski, “From cloud RAN to ORAN,” _Wireless Personal Communications_ , Mar. 2020.
  * [8] J. Wang, H. Roy, and C. Kelly, “OpenRAN: The next generation of radio access networks,” _Accesnture Startegy, Tech. Rep._ , Nov. 2019. [Online]. Available: https://telecominfraproject.com/openran/
  * [9] H. Kumar, V. Sapru, and S. K. Jaisawal, “O-RAN based proactive ANR optimization,” in _IEEE Global Commun. Conf. Workshops (IEEE GLOBECOM Wkshps.)_ , 2020, pp. 1–4.
  * [10] T. Pamuklu, M. Erol-Kantarci, and C. Ersoy, “Reinforcement learning based dynamic function splitting in disaggregated green open RANs,” in _IEEE Inter. Conf. Commun. (IEEE ICC 2021)_ , 2021, pp. 1–6.
  * [11] H. Lee, J. Cha, D. Kwon, M. Jeong, and I. Park, “Hosting AI/ML workflows on O-RAN RIC platform,” in _IEEE Global Commun. Conf. Workshops (IEEE GLOBECOM Wkshps.)_ , 2020, pp. 1–6.
  * [12] J. A. Ayala-Romero, A. Garcia-Saavedra, X. Costa-Perez, and G. Iosifidis, “Bayesian online learning for energy-aware resource orchestration in virtualized RANs,” in _IEEE Conf. Comput. Commun. (IEEE INFOCOM)_ , 2021, pp. 1–10.
  * [13] Y. Cao, S.-Y. Lien, Y.-C. Liang, K.-C. Chen, and X. Shen, “User access control in open radio access networks: A federated deep reinforcement learning approach,” _IEEE Trans. Wire. Commun._ , vol. 21, no. 6, pp. 3721–3736, 2022\.
  * [14] M. Karbalaee Motalleb, V. Shah-Mansouri, S. Parsaeefard, and O. L. Alcaraz López, “Resource allocation in an Open RAN system using network slicing,” _IEEE Trans. Netw. and Ser. Manag._ , vol. 20, no. 1, pp. 471–485, 2023.
  * [15] S.-Y. Lien and D.-J. Deng, “Intelligent session management for URLLC in 5G open radio access network: A deep reinforcement learning approach,” _IEEE Trans. Indus. Infor._ , vol. 19, no. 2, pp. 1844–1853, 2023.
  * [16] M. J. Neely, E. Modiano, and C.-P. Li, “Fairness and optimal stochastic control for heterogeneous networks,” _IEEE/ACM Trans. on Netw._ , vol. 16, no. 2, pp. 396–409, 2008.
  * [17] M. J. Neely, “Stochastic network optimization with application to communication and queueing systems,” _Synthesis Lectures Commun. Netw._ , vol. 3, no. 1, pp. 1–211, 2010.
  * [18] A. Eryilmaz and R. Srikant, “Joint congestion control, routing, and MAC for stability and fairness in wireless networks,” _IEEE J. Sel. Areas in Commun._ , vol. 24, no. 8, pp. 1514–1524, 2006.
  * [19] M. A. Habibi, M. Nasimi, B. Han, and H. D. Schotten, “A comprehensive survey of RAN architectures toward 5G mobile communication system,” _IEEE Access_ , vol. 7, pp. 70 371–70 421, 2019.
  * [20] J. Tang, W. P. Tay, and T. Q. S. Quek, “Cross-layer resource allocation with elastic service scaling in cloud radio access network,” _IEEE Trans. Wireless Commun._ , vol. 14, no. 9, pp. 5068–5081, 2015.
  * [21] P. Luong, F. Gagnon, C. Despins, and L.-N. Tran, “Joint virtual computing and radio resource allocation in limited fronthaul green C-RANs,” _IEEE Trans. Wireless Commun._ , vol. 17, no. 4, pp. 2602–2617, 2018.
  * [22] A. Douik, H. Dahrouj, T. Y. Al-Naffouri, and M.-S. Alouini, “Coordinated scheduling and power control in cloud-radio access networks,” _IEEE Trans. Wireless Commun._ , vol. 15, no. 4, pp. 2523–2536, 2016.
  * [23] A. Douik, H. Dahrouj, T. Y. Al-Naffouri, and M.-S. Alouini, “Low-complexity scheduling and power adaptation for coordinated cloud-radio access networks,” _IEEE Commun. Lett._ , vol. 21, no. 10, pp. 2298–2301, 2017.
  * [24] M. S. Al-Abiad, A. Douik, and S. Sorour, “Rate aware network codes for cloud radio access networks,” _IEEE Trans. Mobile Comput._ , vol. 18, no. 8, pp. 1898–1910, 2019.
  * [25] A. Douik, S. Sorour, T. Y. Al-Naffouri, and M.-S. Alouini, “Rate aware instantly decodable network codes,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 2, pp. 998–1011, 2017.
  * [26] M. S. Al-Abiad, A. Douik, S. Sorour, and M. J. Hossain, “Throughput maximization in cloud-radio access networks using cross-layer network coding,” _IEEE Trans. Mobile Comput._ , vol. 21, no. 2, pp. 696–711, 2022\.
  * [27] O-RAN.WG2.Use-Case-Requirements-v02.01, “Non-RT RIC & A1 interface: Use cases and requirements,” _Technical Specification_ , Nov. 2021. [Online]. Available: https://www.o-ran.org/specifications (accessed on 10 November 2021)
  * [28] M. R. Anwar, S. Wang, M. F. Akram, S. Raza, and S. Mahmood, “5G-enabled MEC: A distributed traffic steering for seamless service migration of internet of vehicles,” _IEEE Internet of Things J._ , vol. 9, no. 1, pp. 648–661, 2022.
  * [29] F. Kavehmadavani, V.-D. Nguyen, T. X. Vu, and S. Chatzinotas, “Intelligent traffic steering in beyond 5G Open RAN based on LSTM traffic prediction,” _IEEE Trans. Wireless Commun._ , pp. 1–1, 2023.
  * [30] T. K. Vu, M. Bennis, M. Debbah, and M. Latva-Aho, “Joint path selection and rate allocation framework for 5G self-backhauled mm-wave networks,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 4, pp. 2431–2445, 2019.
  * [31] S. Singh, M. Geraseminko, S.-p. Yeh, N. Himayat, and S. Talwar, “Proportional fair traffic splitting and aggregation in heterogeneous wireless networks,” _IEEE Commun. Lett._ , vol. 20, no. 5, pp. 1010–1013, 2016.
  * [32] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO versus small cells,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 3, pp. 1834–1850, Mar. 2017.
  * [33] P. Key, L. Massoulie, and D. Towsley, “Path selection and multipath congestion control,” in _IEEE Conf. Comput. Commun. (IEEE INFO- COM)_ , 2007, pp. 143–151.
  * [34] J. Liu, A. Eryilmaz, N. B. Shroff, and E. S. Bentley, “Understanding the impacts of limited channel state information on massive MIMO cellular network optimization,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 8, pp. 1715–1727, 2017.
  * [35] M. Razaviyayn, “Successive convex approximation: Analysis and applications,” Ph.D. dissertation, University of Minnesota, 2014.
  * [36] P. Billingsley, _Probability and Measure_ , 3rd ed. New York: Wiley, 1995.
  * [37] L. Tassiulas and A. Ephremides, “Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks,” _IEEE Trans. Automatic Control_ , vol. 37, no. 12, pp. 1936–1948, 1992.
  * [38] M. Bennis, S. M. Perlaza, P. Blasco, Z. Han, and H. V. Poor, “Self-organization in small cell networks: A reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 12, no. 7, pp. 3202–3212, 2013.
  * [39] D. S. Leslie and E. Collins, “Convergent multiple-timescales reinforcement learning algorithms in normal form games,” _Anna. Appl. Prob._ , vol. 13, no. 4, pp. 1231–1251, 2003.
  * [40] A. Beck, A. Ben-Tal, and L. Tetruashvili, “A sequential parametric convex approximation method with applications to nonconvex truss topology design problems,” _J. Global Optim._ , vol. 47, no. 1, pp. 29–51, May 2010.
  * [41] A. Ben-Tal and A. Nemirovski, _Lectures on Modern Convex Optimization._ Philadelphia: MPS-SIAM Series on Optimi., SIAM, 2001.
  * [42] A. Eryilmaz and R. Srikant, “Fair resource allocation in wireless networks using queue-length-based scheduling and congestion control,” _IEEE/ACM Trans. Net._ , vol. 15, no. 6, pp. 1333–1344, 2007.
  * [43] A. H. Jafari, D. López-Pérez, M. Ding, and J. Zhang, “Study on scheduling techniques for ultra dense small cell networks,” in _IEEE Veh. Technol. Conf. (VTC-Fall)_ , 2015, pp. 1–6.
  * [44] 3GPP, _NR, Physical channels and modulation (Release 15)_ , document 3GPP TS 38.211 version 15.2.0 Release 15, 2017.
  * [45] S. Samarakoon, M. Bennis, W. Saad, and M. Latva-aho, “Backhaul-aware interference management in the uplink of wireless small cell networks,” _IEEE Trans. Wireless Commun._ , vol. 12, no. 11, pp. 5813–5825, 2013.
  * [46] X. Lin, N. Shroff, and R. Srikant, “A tutorial on cross-layer optimization in wireless networks,” _IEEE J. Sel. Areas Commun._ , vol. 24, no. 8, pp. 1452–1463, 2006.
  * [47] E. Stai and S. Papavassiliou, “User optimal throughput-delay trade-off in multihop networks under NUM framework,” _IEEE Commun. Lett._ , vol. 18, no. 11, pp. 1999–2002, 2014.
  * [48] V.-D. Nguyen, T. Q. Duong, H. D. Tuan, O.-S. Shin, and H. V. Poor, “Spectral and energy efficiencies in full-duplex wireless information and power transfer,” _IEEE Trans. Commun._ , vol. 65, no. 5, pp. 2220–2233, May 2017\.