# Dynamic SDN-based Radio Access Network Slicing with Deep Reinforcement
Learning for URLLC and eMBB Services

Abderrahime Filali, Zoubeir Mlika, Soumaya Cherkaoui, and Abdellatif Kobbane
A. Filali, Z. Mlika, and S. Cherkaoui are with the INTERLAB Research
Laboratory, Faculty of Engineering, Department of Electrical and Computer
Science Engineering, Université de Sherbrooke, Sherbrooke (QC) J1K 2R1, Canada
(e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>soumaya.cherkaoui@usherbrooke.ca). A. Kobbane is with the UM5R ENSIAS, BP 713
Rabat, Morocco (e-mail: abdellatif.kobbane@um5.ac.ma).

###### Abstract

Radio access network (RAN) slicing is a key technology that enables 5G network
to support heterogeneous requirements of generic services, namely ultra-
reliable low-latency communication (URLLC) and enhanced mobile broadband
(eMBB). In this paper, we propose a two time-scales RAN slicing mechanism to
optimize the performance of URLLC and eMBB services. In a large time-scale, an
SDN controller allocates radio resources to gNodeBs according to the
requirements of the eMBB and URLLC services. In a short time-scale, each
gNodeB allocates its available resources to its end-users and requests, if
needed, additional resources from adjacent gNodeBs. We formulate this problem
as a non-linear binary program and prove its NP-hardness. Next, for each time-
scale, we model the problem as a Markov decision process (MDP), where the
large-time scale is modeled as a single agent MDP whereas the shorter time-
scale is modeled as a multi-agent MDP. We leverage the exponential-weight
algorithm for exploration and exploitation (EXP3) to solve the single-agent
MDP of the large time-scale MDP and the multi-agent deep Q-learning (DQL)
algorithm to solve the multi-agent MDP of the short time-scale resource
allocation. Extensive simulations show that our approach is efficient under
different network parameters configuration and it outperforms recent benchmark
solutions.

###### Index Terms:

Network slicing, software defined networking, URLLC, eMBB, deep reinforcement
learning.

##  I Introduction

The heterogeneous services supported by the fifth-generation (5G) new radio
(NR) can be classified mainly into enhanced mobile broadband (eMBB), ultra-
reliable low-latency communication (URLLC) and massive machine-type
communication (mMTC) services [1]. The eMBB services target the applications
that require a high data rate such as high definition (HD) video or large-
scale video streaming. The URLLC services accommodate low-latency and high
reliability applications such as autonomous driving or robotic surgery.
Finally, the mMTC services provide connectivity to a large number of devices,
e.g., massive access in Internet of things (IoT) networks [2, 3], that are
characterized by small data and sporadic traffic. To support these three 5G
services while respecting their heterogeneous and different requirements over
a common wireless network infrastructure, radio access network (RAN) slicing
[4] is introduced as a key enabling technology in the new generation of
cellular networks. Network slicing (NS) provides the ability to build several
independent logical networks, called network slices, each adapted to the
requirements of a specific service [5]. Therefore, each RAN slice can be
tailored and dedicated to support a specific service with distinctive
characteristics and requirements. The network operator can leverage the
network programmability provided by software defined networking (SDN) to
dynamically manage the provisioning of radio resources for the RAN slices [6].

Figure 1: Resource block allocation procedure

Unlike the cloud RAN (C-RAN) architecture, which presents major challenges in
its deployment with a multi-access edge computing (MEC) environment in terms
of maintaining service availability and leveraging MEC resource [7], SDN can
be recognized as an important 5G RAN enabler in a fog RAN (F-RAN)
architecture. Specifically, co-deploying SDN with F-RAN increases the ability
of exploiting radio resources using its global view of the network [8]. The
radio resource considered in this work is the resource block (RB), which is
the minimum resource element that can be allocated to an end-user and it is
identified by the frequency band and time slot pair. Each gNodeB should have
enough RBs from the shared radio RBs pool to meet the requirements of its end-
users. In a RAN slicing scenario, SDN can be used to allocate the appropriate
RBs for each RAN slice in each gNodeB, depending on the radio resource
availability and the services required by the end-users of the corresponding
gNodeB. A notable challenge here is how to achieve an optimal RBs allocation
for each gNodeB to satisfy the quality of service (QoS) requirements of its
end-users in different slices.

To solve this problem, a large variety of algorithms and schemes have been
proposed in recent years [9, 10, 11, 12, 13, 14, 15, 16, 17, 18] that mainly
present either centralized resource allocation solutions or multi-level
resource allocation solutions. However, we identify important gaps in these
related-works. In the centralized solution approaches, the radio resource
allocation decision relies only on a central entity (e.g., an SDN controller),
which increases the signaling overhead in the network caused by frequent
communications between the gNodeBs and the central entity, especially when the
latter has to perform the resource allocation of the different gNodeBs.
Therefore, the operation of allocating radio resources to end-users is
expected to be performed in a large time-scale. On the other hand, in the
multi-level resource allocation solution approaches, the SDN controller
allocates, in the upper level, radio resources to RAN slices or gNodeBs. The
latter are responsible, in the lower level and based on the pre-allocated
resources by the SDN controller, for allocating radio resources to end-users.
Although multi-level resource allocation solutions can efficiently allocate
RBs to end-users, when the pre-allocated resources by the SDN controller are
not sufficient to handle the required services, the low-level allocation
operation can fail since it must wait for the next resource reservation update
or immediately solicit the SDN controller for more resources, which can
significantly reduce the QoS that the network operator is expected to provide.

In this paper, we fill these two gaps by proposing a two-level RB allocation
mechanism. In the first level and in a large time-scale, the SDN controller
allocates to each gNodeB a number of RBs from the common radio RBs pool,
according to the gNodeB requirements. In the second level, each gNodeB
schedules the pre-allocated RBs to its associated end-users in a short time-
scale to satisfy their QoS requirements in terms of data rates and delay. This
mechanism avoids frequent communications between gNodeBs and the SDN
controller caused by abrupt and potentially unexpected fluctuations in
wireless network traffic. To further reduce communications between gNodeBs and
the SDN controller, a gNodeB can request additional RBs from other adjacent
gNodeBs when its pre-allocated RBs are not sufficient, which allows to
immediately respond to the requirements of the end-users.

In this work, two types of slices are considered, each one is dedicated to a
single 5G service, namely the eMBB service and the URLLC service. Then, we
formulate the two-level RB allocation problem as an optimization problem where
the objective is to maximize the total achievable data rate of eMBB and URLLC
end-users. This objective has to be achieved subject to the ultra-low latency
requirements of the URLLC services as well as to the minimum data rate
requirements of the eMBB services. The proposed mechanism to solve this
optimization problem will: (1) reduce the signaling overhead between the
gNodeBs and the SDN controller, and (2) quickly provide the required RBs for
different slices to meet the services requested by the corresponding end-
users.

The novelty of this work lies in two main parts. In the first part, we define
the two-level radio RB allocation problem using integer programming and we
analyze its NP-hardness. In the second part, we propose a single-agent multi-
agent reinforcement learning (SAMA-RL) framework to solve the two-level radio
RB allocation problem. Therefore, the proposed RL-based RAN slicing framework
is dynamic since RL algorithms can adapt their RB allocation policies
permanently according to different factors that mainly include the density of
the end-users in the network, the requirements of the eMBB and the URLLC
services, and the transmission conditions of the wireless channel. In
addition, the proposed RAN slicing approach is performed in two-time scales to
allocate resources for two levels including the SDN controller level and the
gNodeBs level. Precisely, the proposed two-time scales RAN slicing approach is
supervised by an SDN controller to manage the RB allocation policies
dynamically, which means that the programmability provided by the SDN
controller enables an automatic resource management. The main contributions of
this paper are summarized as follows:

  * •

We use mathematical programming techniques to model the global RBs allocation
for eMBB and URLLC end-users as a non-linear binary program and study its NP-
hardness.

  * •

Due to the NP-completeness result, obtaining an optimal solution to the RB
allocation problem is computationally expensive. Thus, we model each level of
the RB allocation problem as a Markov decision process (MDP).

  * •

To fairly partition RBs between gNodeBs according to their requirements, we
design a single agent RL-based algorithm to partition the RBs between gNodeBs
in the first allocation level.

  * •

In the gNodeBs level, based on the pre-allocated RBs by the SDN controller to
gNodeBs, we propose a multi-agent deep Q-learning (DQL) approach to allocate
RBs to eMBB and URLLC end-users and perform RBs sharing between gNodeBs.

  * •

We evaluate the performance of our proposed mechanism against benchmark
algorithms and perform extensive simulations to show the superiority of the
proposed framework.

The rest of the paper is organized as follows. Section II discusses analyzes
the related works. Section III presents the system model. Section IV
formulates the RB allocation problem as a mathematical program and studies its
NP-hardness. Section V presents the proposed learning solutions. Section VI
highlights the performances of the proposed mechanism and discusses the
obtained results. Finally, section VII concludes the paper.

##  II Related Work

In [9], the authors propose a framework for RAN slicing by using two machine-
learning approaches: (i) Long short-term memory (LSTM) is used to predict the
resources that should be allocated to a slice in a large time-scale (ii) a
multi-agent RL-algorithm, known as asynchronous actor-critic agent (A3C), is
exploited for on-line resource scheduling of RAN slices. However, the proposed
framework considers only eMBB end-users and the slicing for individual end-
users is not considered. In [10], the controller reserves resources for URLLC
and eMBB slices at each base station considering the minimum resource
requirement of each slice. Then, to meet the required QoS and increase the
resource utilization utility of slices, a deep RL (DRL) algorithm is executed
for each slice on each base station to dynamically update the allocated RBs
based on the reserved resources. Similarly, the authors of [11] improve the
user QoS satisfaction and resource utilization utility by adjusting the
resource provided to individual RAN slices. They leverage DQL approach with
the dueling DQN algorithm to solve the slice resource provisioning problem.
However, if the resources reserved for a given slice are not sufficient to
handle the required services, the slice must wait for the next resources
reservation update since the slice cannot occupy more resources than those
assigned by the controller. The authors of [12] present a dynamic radio
resource slicing scheme for a two-tier heterogeneous wireless network to
determine the optimal bandwidth slicing ratios for slices. An alternative
concave search algorithm is designed to solve the maximum network utility
optimization problem. Although the proposed scheme satisfies the QoS
requirement of machine-type and data slices, it cannot support URLLC slices
which need much lower latency. Also, resources for each slice are expected to
be updated only in a large time-scale. In [13], a two-level SDN-based radio
resource allocation framework is designed to improve RAN slicing over
different time scales. In a large time-scale, the SDN controller allocates RBs
to gNodeBs, while in a small time-scale the pre-allocated RBs are scheduled by
each gNodeB to eMBB and URLLC users. Moreover, each gNodeB can borrow RBs from
other gNodeBs when the pre-allocated RBs are insufficient. However, the
interactions between the base stations to share the RBs are not described. In
meeting the eMBB, URLLC and mMTC slice user requirements, the authors of [14]
formulate the user-association and resource allocation problem as a maximum
utility optimization problem. The optimization problem is decomposed in two
sub-problems using the hierarchical decomposition method. The base station-
slice user association sub-problem is solved by many-to-one matching game and
a genetic algorithm is adopted to solve the dynamic resource allocation sub-
problem. The authors of [15] present a dynamic framework to allocate radio
resources for two types of vehicular network services, i.e., delay-sensitive
service and delay-tolerant service. The radio resource allocation problem is
jointly formulated with that of computing resource allocation as an
optimization problem. To solve this problem, they use a two-layer constrained
RL algorithm. Based on the proposed RL algorithm, the SDN controller is
responsible for allocating resources to slices at all the base stations. In
order to provide their mobile users the access to the virtual computation and
communication slices, multiple service providers compete in [16] to
orchestrate channel access opportunities. The authors model such resource
allocation problem as a non-cooperative stochastic game and leverage a deep
learning approach based on double deep Q-network to approximate the optimal
allocation strategy. In [17], a two-step RAN slicing framework is proposed to
improve bandwidth utilization. In the first step, the framework selects a set
of users whose QoS can be satisfied simultaneously. In the second step, each
admissible user is associated with a slice via a specific base station and a
fraction of the base station bandwidth is allocated to it.

##  III System Model

We consider an SDN-enabled 5G RAN architecture composed of a finite set of
gNodeBs $\mathcal{B}=\\{1,2,\ldots,B\\}$. Two types of slices are considered,
namely the URLLC slice and the eMBB slice, which are denoted by $s_{u}$ and
$s_{e}$, respectively. Spectrum resources are represented as a shared RBs pool
denoted by $\mathcal{K}=\\{1,2,\ldots,K\\}$, where each RB represents the
minimum scheduling unit. The end-users $\mathcal{U}=\\{1,2,\ldots,U\\}$ are
randomly located across the network area and where they are URLLC end-users
and eMBB end-users. Each end-user $u\in\mathcal{U}$ is served by one gNodeB
and belong to one slice, i.e., $s_{u}$ or $s_{e}$. Each gNodeB
$b\in\mathcal{B}$ has a set of associated end-users denoted by
$\mathcal{U}_{b}$. We consider the orthogonal frequency division multiple
access (OFDMA) downlink (DL) scenario, where the RBs are organized as a
resource grid [18]. With OFDMA, the transmissions to end-users are scheduled
in an orthogonal manner to reduce interference.

The RB allocation procedure is performed in two levels. In the first level,
the SDN controller, since it has a global view of the network, allocates the
RBs to the gNodeBs using the available RBs pool, in a large time-scale. We
call this RB allocation level the SDN allocation level. The set of RBs
assigned by the SDN controller to a gNodeB $b\in\mathcal{B}$ is donated by
$\mathcal{K}_{b}\subseteq\mathcal{K}$. In the second level, the RB allocation
procedure is performed by the gNodeBs to allocate the necessary RBs to each
end-user. We call this RB allocation level the gNodeB allocation level. Each
gNodeB $b\in\mathcal{B}$ allocates to each of its associated end-users, during
a short time-scale, a number of RBs, from the pre-allocated RBs
$\mathcal{K}_{b}\subseteq\mathcal{K}$, to satisfy the various QoS requirements
in terms of data rate and latency of each end-user. To ensure the
orthogonality of DL transmissions among end-users which are served by the same
gNodeB $b\in\mathcal{B}$, each RB $k\in\mathcal{K}_{b}$ is exclusively
assigned to one end-user $u_{b}\in\mathcal{U}_{b}$. Also, if $b\in\mathcal{B}$
borrows $k^{\prime}\in\mathcal{K}\setminus\mathcal{K}_{b}$ to allocate it to
one of its end-users, then this RB $k^{\prime}$ should be unallocated. Thus,
we define the assignment of an RB $k\in\mathcal{K}$ to an associated end-user
$u_{b}\in\mathcal{U}_{b}$ with $b\in\mathcal{B}$ as a binary variable
$x_{u_{b}}^{k}$, where:

$x_{u_{b}}^{k}=\begin{cases}1,&\text{if }k\in\mathcal{K}\text{ is assigned to
}u_{b}\in\mathcal{U}_{b},b\in\mathcal{B},\\\ 0,&\text{otherwise.}\end{cases}$
(1) $\sum_{u_{b}\in\mathcal{U}_{b}}x_{u_{b}}^{k}\leq 1,\forall
k\in\mathcal{K},\forall b\in\mathcal{B}.$ (2)

Eq. (2) states that an RB must be allocated to only one end-user at a time,
which meets the OFDMA constraints.

In 5G NR, the scalable OFDM technology is a key innovation. The 3GPP 5G NR
Release 15 specifications [19] state that the waveform is scalable in the
sense that the subcarrier spacing of OFDM can be adapted to channel width. The
choice of the spacing parameter depends on several factors, including the
requirements of 5G services, e.g., low latency for URLLC services and high
data rate for eMBB services. Indeed, eMBB and URLLC services can be supported
simultaneously on the same carrier by multiplexing two different numerologies,
larger subcarrier spacing for URLLC services and lower subcarrier spacing for
eMBB services. In our approach, the RB allocation operation is performed
considering a given time-frequency resource grid model. In other words, for
each numerology strategy, i.e., a specific subcarrier spacing, such as 15 KHz,
30 KHz and 60 KHz, the appropriate trained model should be used to perform the
RB allocation operation. Since the gNodeB is responsible for deciding which
numerology strategy should be applied based on the channel state information,
it can select the appropriate trained model for the selected numerology to
allocate the RBs. Therefore, our approach can be applied with multiple 5G NR
OFDM numerologies to allocate the RBs, provided that the appropriate trained
model for each possible numerology strategy is available.

In this article, we assume that the gNodeBs have a (near) perfect knowledge of
the channel state information [20, 21]. The availability of (near) perfect
channel state information at the gNodeB in a real 5G network can be justified
when fading varies slowly over time and the mobility of the end-users is low
since the wireless channel does not change rapidly, which is similar to our
case. The channel state information can be accurately estimated in 5G networks
using, for example, deep learning algorithms [22].

The achievable data rate of the $u_{b}$-th end-user associated with the $b$-th
gNodeB over the $k$-th RB and belonging to a slice $s\in\\{s_{u},s_{e}\\}$ is
defined as follows:

$r_{u_{b},k}^{s}=W\log_{2}\left(1+\frac{P_{u_{b}}^{s}G_{u_{b},k}}{\sigma^{2}}\right),$
(3)

where $W$ denotes the bandwidth of an RB, $P_{u_{b}}^{s}$ is the transmission
power of $b\in\mathcal{B}$ to end-user $u_{b}\in\mathcal{U}_{b}$ in slice
$s\in\\{s_{u},s_{e}\\}$, $G_{u_{b},k}$ is the DL channel gain between
$b\in\mathcal{B}$ and its associated end-user $u_{b}\in\mathcal{U}_{b}$, and
$\sigma^{2}$ is the power of the additive white Gaussian noise (AWGN). We
assume that the bandwidth and the downlink transmission power are the same for
all RBs.

During the second level of resource allocation, if the SDN controller does not
allocate sufficient RBs to a gNodeB, the latter can request additional RBs
from other gNodeBs. In other words, we assume that a gNodeB can request
additional RBs from other gNodeBs only if all its pre-allocated RBs are
already assigned to its end-users. Mathematically, gNodeB $b$ can request
additional RBs from other gNodeBs and allocate them to its associated end-
users if and only if
$\sum_{u_{b}\in\mathcal{U}_{b}}\sum_{k\in\mathcal{K}_{b}}x_{u_{b}}^{k}\geq|\mathcal{K}_{b}|$.
This constraint can be defined as follows:

$x_{u_{b}}^{k^{\prime}}|\mathcal{K}_{b}|\leq\sum_{u_{b}\in\mathcal{U}_{b}}\sum_{k\in\mathcal{K}_{b}}x_{u_{b}}^{k},$
(4)

where $k^{\prime}\in\mathcal{K}_{b^{\prime}}$ is a borrowed RB by the gNodeB
$b\in\mathcal{B}$ from gNodeB $b^{\prime}\neq b\in\mathcal{B}$ and obviously
$x_{u_{b}}^{k^{\prime}}=1$.

The total achievable data rate of end-user $u_{b}\in\mathcal{U}_{b}$ belonging
to slice $s\in\\{s_{u},s_{e}\\}$ and associated to gNodeB $b\in\mathcal{B}$ is
defined as follows:

$r_{u_{b}}^{s}=\sum_{k\in\mathcal{K}_{b}}x_{u_{b}}^{k}r_{u_{b},k}^{s}+\sum_{\begin{subarray}{c}b^{\prime}\in\mathcal{B}\\\
b^{\prime}\neq
b\end{subarray}}\sum_{k^{\prime}\in\mathcal{K}_{b^{\prime}}}x_{u_{b}}^{k^{\prime}}r_{u_{b},k^{\prime}}^{s}$
(5)

To calculate the delay for URLLC and eMBB traffic, we consider the following
assumptions: (i) the arrival process of each gNodeB’s packets follows a
Poisson distribution, and (ii) the inter-arrival times of the packets are
independent and follow an exponential distribution [23]. Accordingly, the
queuing traffic model can be considered as an M/M/1 queuing system. In
addition, the packet lengths for different slices are different but they are
similar in a slice $s\in\\{s_{u},s_{e}\\}$. By applying Little’s law, we
calculate, in Eq. (6), the average delay $d^{s}_{u_{b}}$ experienced by an
end-user packet $u_{b}$ belonging to slice $s\in\ \\{s_{u},s_{e}\\}$ and
associated with $b\in\mathcal{B}$, where $\lambda_{u_{b}}^{s}$ is the packets
arriving rate of $u_{b}\in\mathcal{U}_{b}$ and belonging to slice
$s\in\\{s_{u},s_{e}\\}$.

$d^{s}_{u_{b}}=\frac{1}{r_{u_{b}}^{s}-\lambda_{u_{b}}^{s}},$ (6)

We choose the M/M/1 queuing system since it is widely used to characterize
wireless communication systems, particularly in RAN slicing approaches [24,
25, 15]. However, in some practical scenarios, the traffic becomes bursty and,
therefore, the M/M/1 assumption (i.e., the packet arrival process of each
gNodeB follows a Poisson distribution) may become too optimistic. In such
scenarios, the gNodeB cannot be considered as an M/M/1 system and the queue
cannot be modeled as a continuous-time Markov process. In this case, when the
traffic is bursty, the queuing system can be modeled as a discrete-time Markov
process [26, 27]. As shown in [28], many continuous-time Markov processes can
be transformed into discrete-time Markov processes by observing only the state
transitions. Therefore, since the M/M/1 queuing assumption can be seen as a
continuous-time Markov process, the analyses and results obtained in this work
are valid for most scenarios where the Poisson distribution cannot be applied.
Note that, under different queuing assumptions, the mathematical analysis may
be different and more complicated. Thus, for simplicity, we only assume the
M/M/1 case.

##  IV Problem Formulation and NP-Hardness

###  IV-A Problem Formulation

The main question of RB allocation problem in RAN is how to derive a two-level
optimal allocation of RBs to URLLC and eMBB end-users that meet their QoS
requirements in terms of data rate and delay. For this purpose, the global RB
allocation optimization problem is formulated as follows: [3]x∑_b
∈B∑_s∈{s_u,s_e}∑_u_b ∈U_b r_u_b^s ∑_k ∈Kx_u_b^k≤K_max, ∀u_b ∈U_b, ∀b ∈B ∑_u_b
∈U_b x_u_b^k≤1, ∀k ∈K, ∀b ∈B x_u_b^k’×—K_b—≤∑_u_b∈U_b∑_k ∈K_b x_u_b^k, ∀k’ ∈K
∖K_b, ∀u_b ∈U_b, ∀b∈B r_u_b^s_e≥R_min, ∀u_b ∈U_b, ∀b ∈B d_u_b^s_u≤D_max, ∀u_b
∈U_b, ∀b ∈B x_u_b^k ∈{0,1}, ∀k ∈K, ∀u_b ∈U_b, ∀b ∈B .

The objective function in (IV-A) maximizes the total sum data rates of the
URLLC and eMBB end-users. Constraints (IV-A) guarantee a fair RBs allocation
by forcing the number of RBs allocated to each end-user $u_{b}$ to not exceed
a maximum number $K_{\text{max}}$. Constraints (IV-A) respect the OFDMA
constraints by ensuring that each RB is allocated to only one end-user at a
time. Constraints (IV-A) guarantee that a gNodeB $b$ can request additonal RBs
from other gNodeBs if and only if all of its pre-allocated RBs are used by its
end-users. Constraints (IV-A) state that the data rate of the eMBB end-users
must be greater than a minimum required threshold $\mathcal{R}_{min}$.
Constraints (IV-A) ensure that the delay of URLLC end-users cannot exceed a
maximum required threshold $\mathcal{D}_{max}$. Finally, constraints (IV-A))
list the optimization variables.

Note that the dynamic resource allocation problem considers limited resources.
In fact, the formulated RB allocation problem consists of maximizing the total
achievable data rate of eMBB and URLLC end-users subject to QoS constraints
and limited resources constraints represented by the set of constraints given
in (IV-A). These constraints guarantee that every end-user cannot be allocated
more than a maximum number of resources. They limit the number of resources
that each end-user can have and thus can allocate the remaining resources more
efficiently and in a fair manner between end-users. The limitation of
resources in the considered problem is also stated by the borrowing concept.
Once a gNodeB is out of resources because all of its SDN-allocated resources
are used, it can borrow other gNodeBs-resources.

Due mainly to the binary nature of the optimization variables and the non-
linearity of the delay experienced by an end-user defined in Eq. (6), the RB
allocation problem is a non-linear binary programming problem. It is thus very
challenging to solve (IV-A) in general. In the sequel, we study its NP-
hardness.

###  IV-B NP-hardness

Here, we denote the problem (IV-A) by $P$. To prove that $P$ is NP-hard, we
reduce the 0-1 knapsack problem [29], which is NP-hard, to an instance of $P$.

######  Definition 1 (0-1 knapsack problem).

A 0-1 knapsack problem is defined as follows: given a set $\mathcal{N}$ of $n$
items, each one with its profit $p_{i}$ and weight $w_{i}$, and a knapsack of
capacity $C$. Each item can be put into the knapsack or not (1 or 0). The
objective of this problem is to find a subset
$\mathcal{N}^{\prime}\subseteq\mathcal{N}$ such that the total value of items
$\sum_{n_{i}\in\mathcal{N}^{\prime}}p_{i}$ is maximized and the total weight
of the selected items is less than or equal to the knapsack capacity, i.e.,
$\sum_{n_{i}\in\mathcal{N}^{\prime}}w_{i}\leq C$.

######  Theorem 1.

$P$ is NP-hard.

###### Proof:

We prove the theorem by considering a restricted version of $P$. This shows
that the problem is NP-hard in the general case as well. We consider the
following restricted version of $P$:

  * •

there is only one gNodeB denoted by $b$ with its associated end-users set
$\mathcal{U}_{b}$.

  * •

there is only the eMBB slice.

  * •

the RBs pre-allocated by the SDN controller $\mathcal{K}_{b}$ for $b$ are
known, i.e., the first level of the RB allocation procedure is already
performed by the SDN controller.

In this case, $P$ becomes equivalent to the following: [3]x∑_u_b ∈U_b
r_u_b^s_e ∑_k ∈K_bx_u_b^k≤—K_b—, ∀u_b ∈U_b ∑_u_b ∈U_b x_u_b^k≤1, ∀k ∈K_b
r_u_b^s_e≥R_min, ∀u_b ∈U_b, x_u_b^k ∈{0,1}, ∀k ∈K_b, ∀u_b ∈U_b, .

The mathematical statement of the problem in Eq. (IV-B) is equivalent to
finding an allocation of the RBs set $\mathcal{K}_{b}$ to the eMBB end-user
set $\mathcal{U}_{b}$ such that: (i) the sum data rates of the end-users is
maximized, (ii) the maximum number of pre-allocated RBs to gNodeB $b$ does not
exceed $K_{max}=|\mathcal{K}_{b}|$ (IV-B), (iii) each RB is allocated to only
one end-user at a time (IV-B), and (iv) the data rate of the eMBB end-users
must be greater than a minimum required threshold $\mathcal{R}_{min}$ (IV-B).

To reduce the 0-1 knapsack problem to (IV-B), we let (i) the number of items
$N$ be the number of eMBB end-users, (ii) the profit $p_{i}$ for item $i$ be
the achievable data rate $r_{u_{b}}^{s_{e}}$ for eMBB end-user $u_{b}$, (iii)
the weight $w_{i}$ for item $i$ be the allocated RBs for eMBB end-user
$u_{b}$, and (iv) the knapsack capacity $C$ be the number of the pre-allocated
RBs to gNodeB $b$. Problem $P$ is now clearly reduced to a knapsack problem.
The knapsack capacity is respected and, thus, the number of allocated RBs to
all eMBB end-users does not exceed $|\mathcal{K}_{b}|$.

Since the reduction is clearly done in a polynomial-time and the 0-1 knapsack
problem is NP-hard, we conclude that the restricted problem formulated in Eq.
(IV-B) is also NP-hard, which proves the theorem. ∎

##  V Single-Agent Multi-Agent Reinforcement Learning Based RAN Resource
Slicing

The resource allocation problem is very challenging to solve optimally in a
two-level procedure, i.e., in the controller level and in the gNodeBs level.
To overcome this challenge, we leverage machine learning techniques,
particularly RL for performing RB allocation tasks due to its excellent
capability to solve wireless network resource allocation problems in a
computationally efficient manner [30]. We propose a single-agent multi-agent
reinforcement learning (SAMA-RL), Fig. 2, framework to solve the two-level
radio RB allocation problem.

More precisely, in the SDN allocation level, we adapt the well-known
exponential-weight algorithm for exploration and exploitation (EXP3) [31]. The
SDN controller plays the role of an RL agent that performs the operation of
allocating the RBs to the gNodeBs. In the gNodeB allocation level, a
distributed multi-agent DRL approach is proposed. Each gNodeB, acting as a DRL
agent, schedules the pre-allocated RBs by the SDN controller to its associated
end-users and, if necessary, cooperate with the other gNodeBs to dynamically
share the unexploited RBs between them. To do so, we apply a DQL [32] model
where each gNodeB acts as an independent agent. To avoid any confusion between
the SDN controller agent and a gNodeB agent, the former will be called the
central agent.

We opt for a classical RL algorithm, i.e., EXP3, in the SDN allocation level
because this resource allocation level does not suffer from the curse of
dimensionality problem. In fact, the state and action spaces are not large,
which allows the central agent to learn in a reasonable time. EXP3 is a
promising algorithm because it does not depend on any assumption related to
the dynamicity of the SDN allocation system, which makes it relevant when the
channel gains offered by RBs vary randomly. Moreover, EXP3 provides a good
tradeoff between exploitation, the desire to make the best decision given
current information, and exploration, the desire to try a new decision which
may lead to better results. On the other hand, the gNodeB allocation level
suffers from the curse of dimensionality due to the multi-agent scenario and
presence of a large number of end-users, which leads the size of state space
to grow significantly. Therefore, using a simple RL framework becomes
computationally intractable. To overcome this challenge, we resort to a
distributed multi-agent DRL approach in the gNodeB allocation level. Indeed,
the agents are capable of discovering meaningful information through an
appropriate deep neural network architecture. Therefore, the agents can learn
close-optimal policies. In addition, DRL has been proven to be efficient in
solving many resource allocation problems in wireless networks, such as energy
scheduling [33] and orchestration of edge computing and caching resources
[34].

Before explaining in details the proposed SAMA-RL algorithms, we first model,
in what follows, each RB allocation problem of each level as a Markov decision
process (MDP) and we define the main components of each MDP in details.

Figure 2: Single-agent multi-agent interaction with the MDP environment.

###  V-A MDP formulation of the SDN allocation level

The MDP of the first resource allocation level is given by the triplet
$(\mathscr{S}_{c},\mathscr{A}_{c},\mathscr{R}_{c})$ where $\mathscr{S}_{c}$
represents the state space, $\mathscr{A}_{c}$ designates the action space and
$\mathscr{R}_{c}$ is the reward function.

####  V-A1 The state space

As discussed previously, the central agent is incorporated into the SDN
controller which has a global view of the network. In fact, the central
agent’s state contains information about its previously allocated RBs to the
gNodeBs and other global parameters of the network. More precisely, the state
space of the central agent is composed of the triple $\mathscr{S}_{c}$ given
as follows:

$\mathscr{S}_{c}=(\mathcal{B},\mathcal{K},\boldsymbol{\mathcal{K}_{h}}),$ (7)

where $\boldsymbol{\mathcal{K}_{h}}=\\{\mathcal{K}_{b},\forall
b\in\mathcal{B}\\}$ represents the allocated RBs previously to each gNodeB.
Also, the central agent’ state includes the sets $\mathcal{B}$ and
$\mathcal{K}$ that represent the sets of gNodeBs and RBs, respectively.

####  V-A2 The action space

Since the central agent performs the RB allocation in a large time-scale, it
has to decide which subset $\mathcal{K}_{b}\subseteq\mathcal{K}$ of RBs it
should allocate to each gNodeB $b\in\mathcal{B}$. Note that the central agent
has to make sure that an RB is assigned to only one gNodeB each time it
performs the RB allocation. Therefore, the action space of the central agent,
denoted by $\mathscr{A}_{c}$, is defined as follows:

$\mathscr{A}_{c}=\\{0,1\\}^{|\mathcal{K}|\times|\mathcal{B}|},$ (8)

where an action $\boldsymbol{a_{c}}\in A_{c}$ is given by the row vector
$[a_{1}^{1},\ldots,a_{1}^{|\mathcal{K}|},a_{2}^{1},\ldots,a_{2}^{|\mathcal{K}|},\ldots,a_{|\mathcal{B}|}^{1},\ldots,a_{|\mathcal{B}|}^{|\mathcal{K}|}]$.
If an element $a_{b}^{k}$ of vector $\boldsymbol{a_{c}}$ is equal to 1, it
means that the central agent has decided to allocate RB $k$ to gNodeB $b$.
Also, to avoid assigning the same RB $k$ to several gNodeBs, the central agent
has to make sure that the constraints $a_{b}^{k}\neq
a_{b^{\prime}}^{k},\forall k\in\mathcal{K}$ and $b\neq
b^{\prime}\in\mathcal{B}$ are satisfied.

####  V-A3 The reward function

Once the central agent observes the environment through its current state, it
chooses an action $\boldsymbol{a_{c}}$ from the set $\mathscr{A}_{c}$. After
choosing an action from the action space $\mathscr{A}_{c}$, the central agent
receives a reward $\mathscr{R}_{c}$. Since the objective is to maximize the
total sum-rate, the objective of the central agent has to be related to the
sum-rate of the network given in Eq. (IV-A). We define the reward as how the
assigned RBs to gNodeBs affect the achieved data rate of the entire system. In
other words, the more the total data rate of the system is higher, the better
is the action chosen by the central agent. Thus, the reward of the central
agent is given by Eq. (IV-A), where:

$\mathscr{R}_{c}=\sum_{b\in\mathcal{B}}\sum_{s\in\\{s_{u},s_{e}\\}}\sum_{u_{b}\in\mathcal{U}_{b}}r_{u_{b}}^{s}$
(9)

According to Eq. (9), the value of $\mathscr{R}_{c}$ depends on the RB
allocation decisions in the gNodeB level. Indeed, the central agent should
wait for the results of the second resource allocation level in order to
efficiently explore its environment. Therefore, the RB allocation is performed
in a joint manner between the two allocation levels.

###  V-B MDP formulation of the gNodeB allocation level

The MDP of the second resource allocation level is given by the triplet
$(\mathscr{S}_{b},\mathscr{A}_{b},\mathscr{R}_{b})$ where $\mathscr{S}_{b}$
represents the state space of agent $b$, $\mathscr{A}_{b}$ designates the
action space of agent $b$ and $\mathscr{R}_{b}$ is the reward function of
agent $b$.

####  V-B1 The state space

As illustrated in Fig. 2, we propose a multi-agent DQL algorithm, with each
gNodeB acts as an agent, to allocate the required RBs to eMBB and URLLC end-
users and performs RBs sharing between gNodeBs, if necessary. Indeed, based on
the chosen action by the central agent in the SDN allocation level, each agent
observes its local state $\mathscr{S}_{b}$. In our model, we consider that the
controller communicates to each agent the information about the RBs allocated
to all other agents. Accordingly, each agent $b$’s state
$\mathscr{S}_{b},\forall b\in\mathcal{B}$, is given by the following tuple:

$\mathscr{S}_{b}=(\mathcal{G}_{b},\mathcal{U}_{b},\mathcal{K}_{b},\mathcal{R}_{min},\mathcal{D}_{max}),$
(10)

where $\mathcal{U}_{b}$ and $\mathcal{K}_{b}$ represents the set of associated
end-users with agent $b$ and the set of pre-allocated RBs to agent $b$
respectively. The term $\mathcal{G}_{b}=(G_{{u_{b}},k}:u\in
U_{b},k\in\mathcal{K})$ represents the DL channel gain between agent $b$ and
its associated end-users in each RB $k\in\mathcal{K}$. The channel gains
$\mathcal{G}_{b}$ can be easily collected by agent $b$ as follows: (i) agent
$b$ broadcasts pilot signals to all of its associated end-users. Then, each
end-user estimates the channel state information and sends it back to the
corresponding agent through a feedback channel. The estimation of
$\mathcal{G}_{b}$ between each associated end-user over each RB in the system
helps agent $b$ to take good decisions, especially to borrow the needed RBs
from the other agents. The state $\mathscr{S}_{b}$ also includes the minimum
data rate threshold $\mathcal{R}_{min}$ and the maximum delay threshold
$\mathcal{D}_{max}$ that corresponds to the requirements of eMBB and URLLC
slices, respectively.

####  V-B2 The action space

In the gNodeB allocation level, each agent takes actions according to its own
allocation policy. In fact, the agent has to (i) assign the pre-allocated RBs
to its associated end-users and (ii) request additional RBs from other agents
when its pre-allocated RBs are not sufficient. Also, it is assumed that each
agent is aware of the number of RBs present in the entire system. This
information is acquired as follows: in each round of the SDN allocation level,
the controller communicates with each agent the information on the RBs
allocated to the other agents. Therefore, we define the space action of agent
$b$, denoted by $\mathscr{A}_{b}$, as follows:

$\mathscr{A}_{b}=\\{0,1\\}^{|\mathcal{K}|\times|\mathcal{U}_{b}|},$ (11)

where an action $\boldsymbol{a_{b}}\in\mathscr{A}_{b}$ is given by the row
vector
$[a_{1}^{1},\ldots,a_{1}^{|\mathcal{K}|},a_{2}^{1},\ldots,a_{2}^{|\mathcal{K}|},\ldots,a_{|\mathcal{U}_{b}|}^{1},\ldots,a_{|\mathcal{U}_{b}|}^{|\mathcal{K}|}]$.
Note that a vector $\boldsymbol{a_{b}}$ is equivalent to an association matrix
$[x_{u_{b}}^{k}]$ because each element $a_{u_{b}}^{k}$ of vector
$\boldsymbol{a_{b}}$ corresponds to the assignment of an RB $k\in\mathcal{K}$
to an associated end-user $u_{b}\in\mathcal{U}_{b}$. In other words, if an
element $a_{u_{b}}^{k}$ of vector $\boldsymbol{a_{b}}$ is equal to 1, it means
that agent $b$ has decided to allocate RB $k$ to end-user $u_{b}$. Also, if
all elements $a_{u_{b}}^{k^{\prime}},\forall
k^{\prime}\in\mathcal{K}\setminus\mathcal{K}_{b}$, of vector
$\boldsymbol{a_{b}}$ are equal to 0, it means that agent $b$ does not request
additional resources from the other gNodeBs. When constructing the action
space $\mathscr{A}_{b}$ of agent $b$, the constraints (IV-A), (IV-A) and
(IV-A) should be respected. By applying these constraints, we significantly
reduce the action space $\mathscr{A}_{b}$. As a result, the exploration phase
is significantly improved to discover better strategies, which considerably
accelerates the learning process of agent $b$. Further, considering that
agents should cooperate to dynamically share the unexploited RBs among
themselves, each agent communicates its chosen action to the others.
Therefore, each agent $b$ forms a joint action
$\boldsymbol{a}=(\boldsymbol{a}_{b},\boldsymbol{a}_{-b})$ where
$\boldsymbol{a}_{-b}$ denotes the actions chosen by the other agents.

####  V-B3 The reward function

Multi-agent reinforcement learning methods seek to learn a policy that
achieves the maximum expected total reward for all agents. Indeed, the
learning process of all agents is driven by the reward function. In our model,
the main objective is to maximize the total achievable data rate of the entire
system to meet the QoS requirement of eMBB and URLLC end-users. Therefore, the
reward function of agent $b$ relates to its total sum-rate subject to the
ultra-low latency requirements of the URLLC services as well as to the minimum
data rate requirements of the eMBB services.

The reward of agent $b$ depends on whether or not it has successfully
allocated the needed RBs to its associated end-users. An RB allocation
operation is considered to be feasible if the chosen action
$\boldsymbol{a_{b}}$ satisfies the constraints (IV-A), (IV-A), (IV-A), (IV-A)
and (IV-A), otherwise it is considered as an infeasible operation. The
constraints (7b), (7c) and (7d) are already verified in the action space
construction phase. However, as an allocation operation can require borrowing
some RBs from other agents, it is necessary to verify if a borrowed RB is: (i)
exploited by its owner and (ii) also chosen by at least one agent other than
its owner. Since each agent forms the joint action $\boldsymbol{a}$, (i) and
(ii) can be easily verified by agent $b$. If at least one of them is correct,
the constraint (IV-A) is not satisfied and thus the action chosen by agent $b$
is not feasible. As a result, the individual reward of agent $b$, denoted by
$\mathscr{R}_{b}$ is expressed as follows:

$\mathscr{R}_{b}=\begin{cases}\sum\limits_{s\in\\{s_{u},s_{e}\\}}\sum\limits_{u_{b}\in\mathcal{U}_{b}}r_{u_{b}}^{s},&\text{if
}\boldsymbol{a_{b}}\text{ is feasible},\\\ -1,&\text{if
}\boldsymbol{a_{b}}\text{ is not feasible}.\end{cases}$ (12)

When an action $\boldsymbol{a_{b}}\in\mathscr{A}_{b}$ is not feasible, it is
penalized with a negative reward, $\mathscr{R}_{b}=-1$, to prevent the agent
from choosing infeasible actions in the future.

###  V-C Single-agent EXP3 algorithm

In order to solve the SDN allocation level problem, we adopt an online
learning algorithm that is based on the multi-armed bandit (MAB) approach
[31]. In MAB, a player needs to choose, at each round of the game, one arm
from a finite set of arms, each characterized by an unknown reward, with the
objective of maximizing his expected cumulative reward. The single-agent MDP
is modeled as a MAB as follows. The central agent, i.e., the SDN controller,
represents the player and the set of arms is given by its action space
$\mathscr{A}_{c}$. We propose the EXP3 algorithm as a popular bandit strategy
to solve the SDN allocation level problem [2]. The SDN controller runs the
EXP3 algorithm where each action is assigned a weight to evaluate how good the
action is for the SDN controller, i.e., the higher the weight of an action,
the better the action is. At the beginning of the algorithm, the weights of
all actions are uniformly distributed. Then, the algorithm iterates several
rounds. For each round, the SDN controller:

  1. 1.

calculates the probability of choosing each action, which is proportional to
its weights;

  2. 2.

chooses an action according to the probability distribution calculated
previously and receives a reward; and

  3. 3.

uses the received reward to update the weights of each action by applying an
exponential weighting scheme. The advantage of such a scheme is that it
rapidly increases the probability of good actions, while rapidly reducing the
probability of bad actions.

The pseudo-code of the EXP3 algorithm is presented in Algorithm 1.

In detail, the EXP3 algorithm takes as input an exploration parameter
$\alpha\in[0,1]$ that controls the desire to choose an action uniformly at
random. It starts by assigning a weight $\psi_{i}$ to each action
$\boldsymbol{a_{c,i}}$, which is initialized to 1. Then, it iterates the
rounds. For each round, the central agent calculates the probability
$\pi_{i}$, given in Eq. (13), of choosing action $\boldsymbol{a_{c,i}}$. Based
on $\pi$, it selects an action $\boldsymbol{a_{c,i}}$ and receives a reward
$\mathscr{R}_{c,i}$. The obtained reward is scaled to the range $[0,1]$ and it
is denoted by $\mathscr{\bar{R}}_{c,i}$. Since $\mathscr{R}_{c,i}$ is the
total achievable data rate of the system, it can be scaled using Eq. (14).
After scaling the obtained reward, the central agent calculates an estimated
reward $\mathscr{\hat{R}}_{c,i}$ using Eq. (15). The idea behind estimating
the reward in such manner is to compensate for a potentially low probability
of obtaining the observed reward. Finally, the weights are updated as
$\psi_{i}=\psi_{i}\exp{(\alpha\mathscr{\hat{R}}_{c,j}/|\mathscr{A}_{c}|)}$.

$\pi_{i}=(1-\alpha)\frac{\psi_{i}}{\sum_{j=1}^{|\mathscr{A}_{c}|}\psi_{j}}+\frac{\alpha}{|\mathscr{A}_{c}|}$
(13) $\mathscr{\bar{R}}_{c,i}=1-\frac{1}{(1+\mathscr{R}_{c,i})}$ (14)
$\mathscr{\hat{R}}_{c,i}=\frac{\mathscr{\bar{R}}_{c,i}}{\pi_{i}}$ (15)

Algorithm 1 EXP3-based SDN allocation level

Parameters: $\alpha\in[0,1]$

Initialize $\psi_{i}=1$ for all $i\in\\{1,2,\ldots,|\mathscr{A}_{c}|\\}$

1: for each round do

2: for $i=1,2,\ldots,|\mathscr{A}_{c}|$ do

3: Calculate $\pi_{i}$ using Eq. (13)

4: end for

5: Select an action $\boldsymbol{a_{c,i}}$ according to $\pi_{i}$

6: Receive reward $\mathscr{R}_{c,i}$

7: Calculate $\mathscr{\bar{R}}_{c,i}$ using Eq. (14)

8: for $j=1,2,\ldots,|\mathscr{A}_{c}|$ do

9:
$\mathscr{\hat{R}}_{c,j}\leftarrow\mathscr{\bar{R}}_{c,j}/\pi_{i}\cdot\mathbbm{1}_{j=i}$

10:
$\psi_{i}\leftarrow\psi_{i}\exp{(\alpha\frac{\mathscr{\hat{R}}_{c,j}}{|\mathscr{A}_{c}|})}$

11: end for

12: end for

The EXP3 algorithm is simple to implement and does not require enormous
computational complexity since it simply updates the weights of choosing
actions by increasing or decreasing their probabilities according to their
performance. Note that the EXP3 algorithm is an online learning algorithm that
enables the SDN controller to increase or decrease the weight of the actions
according to the feedback received from the gNodeBs. Precisely, the SDN
controller selects an action $\boldsymbol{a_{c,i}}$ to allocate the RBs to the
gNodeBs. Then, it waits for the results of the second RB allocation level,
which is performed by the gNodeBs. Once the gNodeBs have assigned the RBs,
chosen by the SDN controller, to their end-users, each gNodeB: 1) calculates
its achieved data rate, and 2) communicates this information to the SDN
controller. The total sum-data rate of all gNodeBs will be the reward received
by the SDN controller after choosing action the $\boldsymbol{a_{c,i}}$, that
will be used by the EXP3 algorithm to update the actions’ weights.

###  V-D Multi-agent deep Q-Learning algorithm

The DQL algorithm [32] extends the classical Q-learning RL [35] algorithm by
approximating the Q-function using a deep neural network known as deep
Q-network (DQN). In order to be used as an efficient non-linear approximator,
such a network must be trained to observe the state of the agent and learn
weights in order to play actions that yield the highest rewards. Once the DQN
is properly trained, it is exploited by the agent to take actions based on the
observed state [36]. To solve the multi-agent MDP model, we propose a multi-
agent DQL algorithm. This algorithm consists of two main phases: the training
phase and the implementation phase. In the training phase, each agent trains a
deep neural network (DNN) in an offline manner using a large amount of
experiences (collected dataset). In the implementation phase, each agent
chooses actions in an online manner using its trained model. We describe, in
the following, the training and implementation phases of the proposed multi-
agent DQL approach.

####  V-D1 The training phase of DQL

DQN approximates the Q-value function $\mathscr{Q}(s,a)$ through a neural
network that performs a mapping between states and actions. In other words,
this network returns, for any given state-action pair, the estimated Q-value
$\mathscr{Q}(s,a;w)$, where $w$ represents the parameters of the network
(i.e., the weights). In order to improve the learning performance, DQN
introduces the experience replay memory strategy that overcomes the learning
stability issues. Indeed, this strategy stores the agent’s experiences that
include state transitions, actions and rewards, and then randomly samples from
these experiences to perform Q-learning. As a result, the experience replay
memory strategy reduces the correlation between the training samples, which
prevents the optimal policy from being conducted to a local minimum. Although
DQN can be effective, it still suffers from the problem of overestimating
action Q-values. Double DQN (DDQN) [37] is proposed to mitigate this
limitation and improve learning performance. The idea behind DDQN is to
decouple action selection from evaluation. To achieve this, two neural
networks are used, a main Q-network that selects an action and a target
Q-network that calculates the Q-value of the selected action. In our DQL-based
MARL algorithm, each agent $b$ has a DDQN that takes the current state as
input and outputs the Q-value function of all actions. The training phase of
the proposed multi-agent DDQN is given in Algorithm 2.

Algorithm 2 DQL Algorithm Training Phase

Input: Agents and environment

Output: Trained DDQNs

Start simulator: generate end-users and network parameters;
Initialize for each agent $b$: the main DQN, the target DQN and the replay
memory $\mathscr{M}_{b}$;

1: for each episode do

2: Reset and build the agents’ environment;

3: for each step do

4: for each agent $b$ do

5: Get observation $\mathscr{S}_{b}$;

6: Choose an action $\boldsymbol{a}_{b}$ using $\epsilon$-greedy;

7: end for

8: for each agent $b$ do

9: Obtain the joint action $\boldsymbol{a}$ and receive reward
$\mathscr{R}_{b}$;

10: Obtain the next observation $\mathscr{S}_{b}^{\prime}$;

11: Store the experience $exp_{b}$ in replay buffer $\mathscr{M}_{b}$;

12: if batch size then

13: Randomly sample a mini-batch from $\mathscr{M}_{b}$;

14: Calculate target Q-value;

15: Calculate loss between the main network and the target network;

16: Update the parameters of the main network using gradient descent to
minimze loss;

17: end if

18: if target step then

19: Update the target network parameters;

20: end if

21: end for

22: end for

23: end for

In detail, the training phase requires as input the environment of each agent
which includes gNodeBs, end-users, pre-allocated RBs, service requirements and
channel state information. As output, it returns the trained DQN of each
agent. The training starts by : (1) generating the network parameters, the
end-users including their positions on the grid, the service required (i.e.,
eMBB or URLLC) and their packet sizes, and (2) initializing the DQN of each
agent. Next, DQL iterates the episodes. At the beginning of each episode, each
agent’s environment is built by updating the end-user locations and the
channel coefficients (i.e., large scale fading). In each step, each agent $b$
observes the current state $\mathscr{S}_{b}$ of its environment and takes an
action $\boldsymbol{a_{b}}$ from its action space $\mathscr{A}_{b}$ by using
the $\epsilon$-greedy policy. With the $\epsilon$-greedy policy, an agent
selects an action randomly or using the Q-network. Precisely, this policy
chooses the action with the highest Q-value with probability $1-\epsilon$,
where the exploration rate $\epsilon$ represents the probability that an agent
will explore its environment rather than exploit it. As we go forward (i.e.,
after each step), each agent learns more about its environment and $\epsilon$
decays by some rate, so that exploring the environment becomes less probable.
Once all agents choose their action following the $\epsilon$-greedy policy,
they communicate them to each other. Accordingly, each agent $b$ forms its
joint action, calculates its reward $\mathscr{R}_{b}$ using Eq. (12) and moves
to a new state $\mathscr{S}_{b}^{\prime}$. Next, the obtained tuple
$(\mathscr{S}_{b},\boldsymbol{a_{b}},\mathscr{R}_{b},\mathscr{S}_{b}^{\prime})$,
called agent’s $b$ experience and denoted by $exp_{b}$, is stored in its
replay memory $\mathscr{M}_{b}$. In practice, since the size of the replay
memory is limited to a defined threshold $\mathcal{M}$, only the last
$\mathcal{M}$ experiences can be stored. After storing enough experiences,
each agent samples a random mini-batch from its replay memory. Note that the
size of the replay memory should be large enough to reduce the correlation
between the data that will be sampled from it. The obtained dataset is used by
the agent to perform the training. With the objective of minimizing the loss
function, given by Eq. (16), the main Q-network is used to approximate the
Q-value function while the target Q-network is used to outputs the target
Q-value.

$L_{b}(w_{b})=\mathbbm{E}[(y_{b}-\mathscr{Q}(\mathscr{S}_{b},\boldsymbol{a_{b}};w_{b}))^{2}],$
(16)

where $\mathscr{Q}_{b}(\mathscr{S}_{b},\boldsymbol{a_{b}};w_{b})$ is the
approximated Q-value function given by the main Q-network of agent $b$ with
weight parameter $w_{b}$ and $y_{b}$ denotes the target Q-value and it is
given as follows:

$y_{b}=\mathscr{R}_{b}+\gamma\mathscr{Q}(\mathscr{S}_{b},\underset{\boldsymbol{a_{b}}}{max}\,\\{\mathscr{Q}(\mathscr{S}_{b},\boldsymbol{a_{b}};w_{b})\\};w^{-}_{b}),$
(17)

where $0\leq\gamma\leq 1$ is called the discount factor, $w^{-}_{b}$ is the
weight parameter of the target Q-network. Note that the value of $y_{b}$ is
not necessarily the largest Q-value in the target Q-network, which allows to
avoid chossing an overestimated action.

After calculating the loss function, each agent performs a gradient descent to
update the parameters of the main Q-network. Finally, the parameters of the
target Q-network are updated, at each fixed target step, by copying the
parameters of the main Q-network.

Since the learning of the DDQNs is computationally heavy, the training phase
of the DQL algorithm is performed in an offline manner. Accordingly, the
training can be conducted using a large amount of dataset resulting from
different network topologies and channel conditions.

Algorithm 3 DQL Algorithm Implementation Phase

Input: The trained DDQNs

Output: RB allocation for end-users

Load the DDQN of each agent;

1: for each episode do

2: Reset and build the agents’ environment;

3: for each step do

4: for each agent $b$ do

5: Obtain observation $\mathscr{S}_{b}$;

6: Choose $\boldsymbol{a}_{b}$ that maximize the Q-function;

7: end for

8: Obtain the joint action $\boldsymbol{a}$;

9: Find a solution to RB allocation for end-users;

10: end for

11: end for

####  V-D2 The implementation phase of DQL

After the training phase, the parameters of the main Q-networks are used to
find an RB allocation solution for the end-users in the online implementation
phase of the DQL algorithm. The implementation phase is presented in Algorithm
3. This phase uses the trained DDQNs of the agents. At the beginning of each
episode, it builds the environment of each agent. Then, for each step, when a
new state is observed, each agent $b$ selects the action that maximizes the
Q-value of current state. Once all agents have chosen their actions, each
agent can form a joint action. Accordingly, an RBs allocation solution is
obtained.

##  VI Simulation Results

This section investigates the performance of the proposed two-level RB
allocation mechanism through several simulated scenarios.

###  VI-A Experiment scenarios and setup

We consider an SDN-enabled RAN architecture where the gNodeBs are deployed in
a square of area $1\text{Km}^{2}$. The end-users are uniformly distributed
across the entire coverage area where each one of them is associated to only
one gNodeB. Each end-user is assumed to be either an eMBB end-user or a URLLC
end-user. For the sake of simplicity, the transmission power $P_{u_{b}}^{s}$
is the same for all $u_{b}\in\mathcal{U}_{b}$ and $b\in\mathcal{B}$. The key
parameters of the simulations are summarized in Table I. In order to select an
effective Q-network model, the training phase of the DQL algorithm is
performed on a laptop with an Intel Core i7-8750H processor, 16 GB of RAM and
NVIDIA GeForce GTX 1070 graphic card. We create and train the DDQNs of the
agents using the PyTorch framework. To select the best hyperparameters values
for training the DDQN models, extensive simulations are performed. Indeed, we
tested random combinations of hyperparameters in a fine-grained set of values
chosen based on common settings in the literature [16, 38]. In particular,
each DDQN consists of two fully connected hidden layers, each with 256
neurons. Rectified linear unit (ReLU) is used as the activation function to
avoid the vanishing gradient problem in backpropagation, which accelerates the
learning process. The Adam optimizer is used with a learning rate of 0.001
since it is computationally efficient. During the training process, the
weights of the main Q-network are copied to the weights of the target
Q-network every 1000 steps to avoid overestimating the Q-values. In addition,
we consider end-users with low mobility, so the channel gains between a gNodeB
and an end-user remain unchanged for a certain period of time. To do so, we
fix the location of the end-user for a few training episodes, which helps the
learning algorithm to better acquire the dynamics of the end-users and, at the
same time, stabilize the training. The other hyperparameters of DDQN are given
in Table II.

TABLE I: Simulation parameters. Parameter |  Value
---|---
Number of gNodeBs |  2
Total number of end-users |  8
Bandwidth of an RB |  180 KHz
Transmit power of gNodeB, $P_{u_{b}}^{s}$ |  30 dBm
Power of AWGN, $\sigma^{2}$ |  -114 dBm
Packet arriving rate per end-user, $\lambda_{u_{b}}^{s}$ |  100 pachets/s
Packet length for an eMBB & URLLC end-user |  400 & 120 bits
Minimum data rate for eMBB end-user, $\mathcal{R}_{min}$ |  100 kbps
Maxim delay for URLLC end-user, $\mathcal{D}_{max}$ |  10 ms
TABLE II: Retained hyper-parameters for DDQN. Hyper-parameter |  Value
---|---
Learning rate |  0.001
Epsilon/$\epsilon$-greedy |  1
Discount factor |  0.996
$\epsilon$-min |  0.01
Size of replay memory |  100000
Size of mini-batch |  64
Target network update interval |  1000 steps
Loss function |  Mean squared error
Optimizer |  Adam
Activation function |  ReLu

###  VI-B DDQN training results

To assess the training performance of the proposed multi-agent DRL approach,
we observe the cumulative rewards per training episode and the behavior of the
loss function during the training process.

Fig. 3 shows the cumulative average rewards of agents per episode. From this
figure, the cumulative rewards improve as the number of training episodes
increases, which demonstrates the effectiveness of the proposed training
algorithm. When the training episode approximatively reaches 1800, the agents
gain interesting experiences and start to exploit better actions. Accordingly,
the cumulative reward approaches to a maximum value, indicating that the
training process converges after an acceptable number of training episodes.
Note that the convergence of the DQL algorithm does not present large
fluctuations which are principally due to the low-mobility of the end-users in
the environment.

Figure 3: Training rewards.

Fig. 4 illustrates the convergence of the loss function, Eq. (16), during the
training process of the DDQNs. It plots the average of the agents’ loss
results versus the episodes. The loss decreases with the increase in training
episodes. In the first few episodes, the loss declines gradually because
various new actions were explored randomly and as learning progressed, good
actions were selectively performed based on the Q-value function that had
become more reliable. Consequently, after episode 2500, the loss converges to
a minimum value, which demonstrates the accurate Q-value approximation.

Figure 4: Training loss.

It is worth noting that the convergence results of the reward and loss
function are obtained by assuming that the gNodeBs have perfect knowledge of
the channel state information, which is used as input of the DDQN algorithm.
In the case of imperfect channel state information, the convergence time of
the DDQN algorithm may increase [39]. Therefore, the DDQN algorithm takes more
time to accurately learn the appropriate policies. But, once the offline
training of the DDQN algorithm is performed, the learned policy can be applied
rapidly to obtain the resource allocation solution.

###  VI-C SAMA-RL performance evaluation

To benchmark our SAMA-RL approach, we implemented two schemes from the
literature, which we called 3-SRA [10] and 1-SRA [18] for three-stage resource
allocation and one-stage resource allocation, respectively. 3-SRA allocates
radio resources to end-users by considering two types of slices, namely the
rate constrained slice (i.e., eMBB slice) and the delay constrained slice
(i.e., URLLC slice). In the first stage, a central controller uses a heuristic
algorithm to reserve radio resources (i.e., a fraction of the system
bandwidth) to the eMBB and URLLC slices in each gNodeB. In the second stage,
the reserved radio resources for each slice in each gNodeB are adjusted using
a DRL algorithm. In the final stage, a heuristic algorithm is deployed to map
the fraction of bandwidth assigned to a slice with the physical RBs.
Accordingly, we have chosen 3-SRA since it: (i) is a competitive approach that
performs the slicing operation of radio resources in a hierarchical manner,
(ii) is integrated into a network architecture similar to our proposed
architecture, for instance, the presence of a central controller that has a
global view of the RAN, (iii) considers two types of slices, namely the eMBB
slice and the URLLC slice, and (iv) is based on a DRL algorithm to solve the
radio resource allocation problem in the RAN.

1-SRA performs the allocation of RBs to end-users in a single-stage process
using a greedy heuristic algorithm. Each gNodeB uses this algorithm to assign
the required RBs to its associated eMBB and URLLC end-users. This algorithm
computes the received signal-to-noise ratio (SNR) of the end-users on all the
available RBs of a gNodeB. Then, for each end-user, it: (1) identifies the
highest SNR given by an RB, (2) calculates the spectral efficiency (SE) that
corresponds to the highest SNR, (3) calculates the required number of RBs,
which is defined as the ratio of the traffic queue length to the calculated
SE, and (4) assigns the required RBs to it. Note that URLLC end-users are
prioritized over eMBB end-users, thus the proposed algorithm schedules URLLC
end-users first. We have chosen 1-SRA as a benchmark scheme since: (i) it
considers two types of slices, namely the eMBB slice and the URLLC slice, (ii)
the objective function is similar to our formulated objective function, which
maximizes the total sum data rates of the URLLC and eMBB end-users, and (iii)
it considers a minimum data rate for the eMBB end-user and a maximum latency
requirement for the URLLC end-users in the problem formulation, which is
similar to our optimization problem.

We adapt 3-SRA and 1-SRA to meet the requirements of our system model.
Specifically, in 3-SRA, two gNodeBs cannot serve a single end-user under the
3-SRA algorithm. On the other hand, in 1-SRA, we calculate the achievable data
rate of each RB using the Shannon rate formula instead of the modulation and
coding scheme and the queue length is given by the packet size of each end-
user. To fairly benchmark the performance of SAM-RL against 3-SRA and 1-SRA,
most of the simulation parameters are chosen similarly to those used in [10]
and [18]. Note that the comparison results are averaged over 1000 trials.

Figure 5: Impact of the number of end-users on the objective function.

Fig. 5 shows the performance of SAMA-RL compared to the benchmark approaches
when varying the number of end-users. SAMA-RL always achieves the best
performance in terms of the objective function value when the number of end-
users increases. We notice, for the three approaches, that the value of the
objective function, which represents the total sum data rates of the URLLC and
eMBB end-users, decreases as the number of end-users increases. This is due to
the increase of the competitiveness among end-users to obtain a sufficient
number of RBs that guarantees their requirements. Indeed, in the case of a low
number of end-users, several RBs can be assigned to one end-user to guarantee
its requirements in terms of data rate and delay. On the other hand, when the
number of end-users is large, each approach seeks to allocate a minimum number
of RBs to each end-user in order to satisfy as many end-users as possible in
terms of data rate and delay requirements. SAMA-RL maximizes the total sum
data rates of the end-users better than 3-SRA since the latter assigns the RBs
to the end-users according to the required data rate of the entire slice,
which includes a set of end-users, instead of relying on the required data
rate of each end-user individually. Therefore, it cannot efficiently allocate
the required RBs to the end-users. In the 1-SRA approach, the allocation of
RBs is based only on the highest estimated SNR for each end-user, which may
poorly determine the RBs required to satisfy the QoS of end-users.

Fig. 6 illustrates the impact of the minimum data rate threshold
($\mathcal{R}_{min}$) on the objective function Eq. (IV-A). Based on these
results, we make the following observations: (1) SAMA-RL outperforms 3-SRA and
1-SRA for all minimum data rate thresholds; (2) the higher is the minimum data
rate threshold, the larger is the performance gap between SAMA-RL and both
benchmark approaches. This is due to the multi-agent approach proposed in the
gNodeB allocation level where a gNodeB can borrow RBs from other adjacent
gNodeBs when the RBs pre-allocated by the SDN controller are insufficient.
This indeed illustrates the effectiveness of the proposed multi-agent method
and demonstrates its robustness.

Figure 6: Impact of the minimum data rate threshold ($\mathcal{R}_{min}$) on
the objective function.

Fig. 7 presents the comparison of SAMA-RL and both benchmark approaches, i.e.,
3-SRA and 1-SRA, in terms of the average number of end-users whose achieved
data rate is greater than or equal to the minimum data rate threshold
$\mathcal{R}_{min}$. We can see that SAMA-RL always outperforms 3-SRA and
1-SRA for all minimum data rate thresholds. These results can be justified as
follows: in 3-SRA, if the reserved resources for a given slice are not
sufficient to provide the required service, i.e., minimum data rate threshold,
the slice must wait for the next resources reservation update, while in SAMA-
RL, a gNodeB can request, if necessary, some RBs from other gNodeB. In 1-SRA,
if the SNR is not appropriately estimated, the RBs allocated to end -users may
not be sufficient to provide the required service, thus the end-users will be
served in the next resource allocation operation.

Figure 7: Impact of the minimum data rate threshold ($\mathcal{R}_{min}$) on
the number of end-users. Figure 8: Impact of the the maximum delay threshold
($\mathcal{D}_{max}$) on the number of end-users.

In Fig. 8, we compare the performance of SAMA-RL against the performance both
benchmark approaches in terms of the average number of end-users whose delay
experienced by a packet is less than or equal to the maximum delay threshold
$\mathcal{D}_{max}$. Similar to the obtained results in Fig. 7, we can once
again confirm that SAMA-RL gives better performance compared to 3-SRA and
1-SRA for all maximum delay thresholds.

##  VII Conclusion

Toward an efficient radio resource slicing of an SDN-enabled RAN, we proposed
a two-level RAN slicing mechanism where the allocation of radio RBs is
performed in the SDN level and in the gNodeBs level. The SDN level allocates
RBs to gNodeBs in a large time-scale while the gNodeBs allocate their RBs to
its associated end-users in a short time-scale to efficiently meet their
dynamic requirements. Moreover, each gNodeB can borrow some RBs from other
gNodeBs when the pre-allocated RBs are insufficient, which decreases the
signaling overhead between the two allocation levels and rapidly provide the
needed resources to its end-users. We formulated a data rate maximization
problem subject to the ultra-low latency requirements of URLLC services as
well as to the minimum data rate requirements of eMBB services. Subsequently,
we proved its NP-hardness. Then, we modeled the SDN allocation level problem
and the gNodeB allocation problem as a single MDP and a multi-agent MDP,
respectively. We adapt the EXP3 algorithm and the DQL algorithm to solve the
SDN allocation problem and the gNodeB allocation problem, respectively. In
addition, the DQL algorithm applies robust state-of-the-art approaches such as
double DQN and replay memory to improve its performance. Simulation results
have shown that the proposed two-level mechanism yields significant
improvements in terms of RBs allocation. Furthermore, our mechanism
outperformed a benchmark algorithm by ensuring the requirements of the URLLC
and eMBB services in terms of data rate and delay.

## References

  * [1] A. Dogra, R. K. Jha, and S. Jain, “A Survey on Beyond 5G Network With the Advent of 6G: Architecture and Emerging Technologies,” _IEEE Access_ , vol. 9, pp. 67 512–67 547, Oct. 2020.
  * [2] Z. Mlika _et al._ , “Massive IoT Access with NOMA in 5G Networks and Beyond using Online Competitiveness and Learning,” _IEEE Internet Things J._ , vol. 8, no. 17, pp. 13 624–13 639, Mar. 2021.
  * [3] B. Nour _et al._ , “A Network-Based Compute Reuse Architecture for IoT Applications,” _arXiv preprint arXiv:2104.03818_ , Apr. 2021.
  * [4] S. E. Elayoubi, S. B. Jemaa, Z. Altman, and A. Galindo-Serrano, “5G RAN Slicing for Verticals: Enablers and challenges,” _IEEE Commun. Mag._ , vol. 57, no. 1, pp. 28–34, Jan. 2019.
  * [5] A. Filali _et al._ , “Multi-Access Edge Computing: A Survey,” _IEEE Access_ , vol. 8, pp. 197 017–197 046, Oct. 2020.
  * [6] Z. Shu and T. Taleb, “A Novel QoS Framework for Network Slicing in 5G and Beyond Networks Based on SDN and NFV,” _IEEE Netw._ , vol. 34, no. 3, pp. 256–263, Apr. 2020.
  * [7] A. Reznik, L. M. C. Murillo, Y. Fang, W. Featherstone, M. Filippou, F. Fontes, F. Giust, Q. Huang, A. Li, C. Turyagyenda _et al._ , “Cloud RAN and MEC: A Perfect Pairing,” _ETSI White paper_ , no. 23, pp. 1–24, Feb. 2018\.
  * [8] K. Liang, L. Zhao, X. Chu, and H.-H. Chen, “An Integrated Architecture for Software Defined and Virtualized Radio Access Networks with Fog Computing,” _IEEE Netw._ , vol. 31, no. 1, pp. 80–87, Jan. 2017.
  * [9] M. Yan, G. Feng, J. Zhou, Y. Sun, and Y.-C. Liang, “Intelligent Resource Scheduling for 5G Radio Access Network Slicing,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 8, pp. 7691–7703, Aug. 2019.
  * [10] G. Sun, Z. T. Gebrekidan, G. O. Boateng, D. Ayepah-Mensah, and W. Jiang, “Dynamic Reservation and Deep Reinforcement Learning Based Autonomous Resource Slicing for Virtualized Radio Access Networks,” _IEEE Access_ , vol. 7, pp. 45 758–45 772, Apr. 2019.
  * [11] G. Sun, K. Xiong, G. O. Boateng, G. Liu, and W. Jiang, “Resource Slicing and Customization in RAN with Dueling Deep Q-Network,” _Journal of Network and Computer Applications_ , vol. 157, p. 102573, May 2020.
  * [12] Q. Ye, W. Zhuang, S. Zhang, A. Jin, X. Shen, and X. Li, “Dynamic Radio Resource Slicing for a Two-Tier Heterogeneous Wireless Network,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 10, pp. 9896–9910, Oct. 2018.
  * [13] J. Li, W. Shi, P. Yang, Q. Ye, X. S. Shen, X. Li, and J. Rao, “A Hierarchical Soft RAN Slicing Framework for Differentiated Service Provisioning,” _IEEE Wireless Commun._ , vol. 27, no. 6, pp. 90–97, Apr. 2020.
  * [14] S. O. Oladejo and O. E. Falowo, “Latency-Aware Dynamic Resource Allocation Scheme for Multi-Tier 5G Network: A Network Slicing-Multitenancy Scenario,” _IEEE Access_ , vol. 8, pp. 74 834–74 852, Apr. 2020.
  * [15] W. Wu, N. Chen, C. Zhou, M. Li, X. Shen, W. Zhuang, and X. Li, “Dynamic RAN Slicing for Service-Oriented Vehicular Networks via Constrained Learning,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 7, pp. 2076–2089, Dec. 2020.
  * [16] X. Chen, Z. Zhao, C. Wu, M. Bennis, H. Liu, Y. Ji, and H. Zhang, “Multi-Tenant Cross-Slice Resource Orchestration: A Deep Reinforcement Learning Approach,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 10, pp. 2377–2392, Aug. 2019.
  * [17] Y. Sun, S. Qin, G. Feng, L. Zhang, and M. Imran, “Service Provisioning Framework for RAN Slicing: User Admissibility, Slice Association and Bandwidth Allocation,” _IEEE Trans. Mobile Comput._ , pp. 1–1, Jun. 2020.
  * [18] P. Korrai, E. Lagunas, S. K. Sharma, S. Chatzinotas, A. Bandi, and B. Ottersten, “A RAN Resource Slicing Mechanism for Multiplexing of eMBB and URLLC Services in OFDMA Based 5G Wireless Networks,” _IEEE Access_ , vol. 8, pp. 45 674–45 688, Mar. 2020.
  * [19] 3GPP, “5GM; NR; Physical channels and modulation,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 38.211, 2019.
  * [20] P. Popovski, K. F. Trillingsgaard, O. Simeone, and G. Durisi, “5G Wireless Network Slicing for eMBB, URLLC, and mMTC: A Communication-Theoretic View,” _IEEE Access_ , vol. 6, pp. 55 765–55 779, Sep. 2018.
  * [21] E. N. Tominaga, H. Alves, O. L. A. López, R. D. Souza, J. L. Rebelatto, and M. Latva-Aho, “Network Slicing for eMBB and mMTC with NOMA and Space Diversity Reception,” in _2021 IEEE 93rd Vehicular Technology Conference (VTC2021-Spring), Helsinki, Finland_ , Apr. 2021, pp. 1–6.
  * [22] C. Luo, J. Ji, Q. Wang, X. Chen, and P. Li, “Channel State Information Prediction for 5G Wireless Communications: A Deep Learning Approach,” _IEEE Trans. Netw. Sci. Eng._ , vol. 7, no. 1, pp. 227–236, Jun. 2018.
  * [23] A. Filali _et al._ , “Preemptive SDN Load Balancing with Machine Learning for Delay Sensitive Applications,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 12, pp. 15 947–15 963, Nov. 2020.
  * [24] F. Rezazadeh, H. Chergui, and C. Verikoukis, “Zero-Touch Continuous Network Slicing Control via Scalable Actor-Critic Learning,” _arXiv preprint arXiv:2101.06654_ , Jan. 2021.
  * [25] M. K. Motalleb, V. Shah-Mansouri, and S. N. Naghadeh, “Joint Power Allocation and Network Slicing in an Open RAN System,” _arXiv preprint arXiv:1911.01904_ , Nov. 2019.
  * [26] Q. Xu, S. Li, T. Van, K. Jia, and N. Yang, “Performance Analysis of Cognitive Radio Networks with Burst Dynamics,” _IEEE Access_ , vol. 9, pp. 110 627–110 638, Aug. 2021.
  * [27] N. Pappas, “Performance Analysis of a System with Bursty Traffic and Adjustable Transmission Times,” in _2018 15th International Symposium on Wireless Communication Systems (ISWCS), Lisbon, Portugal_ , Oct. 2018, pp. 1–6.
  * [28] J. F. Shortle, J. M. Thompson, D. Gross, and C. M. Harris, _Fundamentals of queueing theory_. John Wiley & Sons, 2018, vol. 399.
  * [29] D. Pisinger and P. Toth, _Knapsack Problems_. Springer US, 1998, vol. 1, pp. 299–428.
  * [30] J. Wang, C. Jiang, H. Zhang, Y. Ren, K.-C. Chen, and L. Hanzo, “Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 3, pp. 1472–1514, Jan. 2020\.
  * [31] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire, “The Nonstochastic Multiarmed Bandit Problem,” _SIAM J. Comput._ , vol. 32, no. 1, p. 48–77, Jan. 2003.
  * [32] V. Mnih _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, Feb. 2015.
  * [33] Q. Zhang, M. Lin, L. T. Yang, Z. Chen, and P. Li, “Energy-Efficient Scheduling for Real-Time Systems Based on Deep Q-learning Model,” _IEEE Sustain. Comput._ , vol. 4, no. 1, pp. 132–141, Aug. 2019.
  * [34] Y. Dai, D. Xu, S. Maharjan, G. Qiao, and Y. Zhang, “Artificial Intelligence Empowered Edge Computing and Caching for Internet of Vehicles,” _IEEE Wireless Commun._ , vol. 26, no. 3, pp. 12–18, Jul. 2019.
  * [35] C. J. C. H. Watkins and P. Dayan, “Q-learning,” _Machine Learning_ , vol. 8, no. 3, pp. 279–292, May 1992.
  * [36] Z. Mlika _et al._ , “Network Slicing with MEC and Deep Reinforcement Learning for the Internet of Vehicles,” _IEEE Netw._ , vol. 35, no. 3, pp. 132–138, Jan. 2021.
  * [37] H. Van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-Learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, Mar. 2016.
  * [38] F. Wei, G. Feng, Y. Sun, Y. Wang, S. Qin, and Y.-C. Liang, “Network Slice Reconfiguration by Exploiting Deep Reinforcement Learning With Large Action Space,” _IEEE Trans. Netw. Service Manag._ , vol. 17, no. 4, pp. 2197–2211, Aug. 2020.
  * [39] A. Kaur and K. Kumar, “Imperfect CSI Based Intelligent Dynamic Spectrum Management Using Cooperative Reinforcement Learning Framework in Cognitive Radio Networks,” _IEEE Trans. Mobile Comput._ , Sep. 2020.