# Learning based E2E Energy Efficient in Joint Radio and NFV Resource
Allocation for 5G and Beyond Networks

Narges Gholipoor, Ali Nouruzi, Shima Salarhosseini, Mohammad Reza Javan, ,
Nader Mokari, , and Eduard A. Jorswieck, N. Gholipoor, Sh. Salarhosseini, A.
Nouruzi and N. Mokari are with the Department of ECE, Tarbiat Modares
University, Tehran, Iran (email: {Gholipoor.narges, shima.salarhosseini and
nader.mokari}@modares.ac.ir). Mohammad R. Javan is with the Department of
Electrical and Robotics Engineering, Shahrood University of Technology,
Shahrood, Iran (javan@shahroodut.ac.ir). Eduard A. Jorswieck is with TU
Braunschweig, Department of Information Theory and Communication Systems,
Braunschweig, Germany (jorswieck@ifn.ing.tu-bs.de).

###### Abstract

In this paper, we propose a joint radio and core resource allocation framework
for NFV-enabled networks. In the proposed system model, the goal is to
maximize energy efficiency (EE), by guaranteeing end-to-end (E2E) quality of
service (QoS) for different service types. To this end, we formulate an
optimization problem in which power and spectrum resources are allocated in
the radio part. In the core part, the chaining, placement, and scheduling of
functions are performed to ensure the QoS of all users. This joint
optimization problem is modeled as a Markov decision process (MDP),
considering time-varying characteristics of the available resources and
wireless channels. A soft actor-critic deep reinforcement learning (SAC-DRL)
algorithm based on the maximum entropy framework is subsequently utilized to
solve the above MDP. Numerical results reveal that the proposed joint approach
based on SAC-DRL algorithm could significantly reduce energy consumption
compared to the case in which R-RA and NFV-RA problems are optimized
separately.

###### Index Terms:

Resource allocation, Network Function Virtualization (NFV), E2E QoS, Energy
Efficiency (EE), Soft Actor Critic (SAC).

##  I introduction

To support the exponential growth of traffic and various services,
communication service providers (CSPs) need to redesign their infrastructure
or move toward a flexible programmable infrastructure [1]. On the other hand,
CSPs seek to increase capacity demands and upgrade their network in the
shortest possible time. Network function virtualization (NFV) and
softwarization are the key technologies that can meet the requirements of
increasing the exponential traffic, various quality of services (QoS)
requirements intended in the next generations of cellular networks, i.e.,
fifth-generation (5G) wireless cellular network. [2]. NFV is emerged as a
critical technology to reduce the network CAPEX and OPEX, and time to market
by virtualizing all the appliances such as servers, routers, storage, and
switches [3, 4, 5, 6, 7]. NFV technology allows network functions to be run
virtually on network servers as virtual network functions (VNFs), e.g.,
firewall, deep packet inspection, transcoding, and load balancing [2]. In this
paper, we assume that VNF and network function (NF) are the same [8, 9, 10].

###  I-A Background to NFV and Radio Resource Allocation

One of the critical challenges in the NFV-based network is resource allocation
(RA) [5, 10, 7]. NFV resource allocation (NFV-RA) consists of three phases: 1)
The first phase is the VNF service function chaining (VNF-SFC), in which the
chain and the connection of VNFs are determined [11], 2) The second phase is
VNF Placement, in which the VNFs are mapped to the servers/virtual machines
(VMs), and 3) The third phase is VNF scheduling in which the running time for
each VNF is determined [10]. Each of the mentioned phases has a significant
impact on the network performance, e.g., total cost and also QoS of users.
Therefore, optimizing all these phases together can have a significant impact
on reducing total costs and improving QoS of users, e.g., end to end (E2E)
delay. Besides, in the radio part, we are faced to limited power resources and
spectrum scarcity; thus, it is necessary to efficiently allocate radio
resources, i.e., radio resource allocation (R-RA), to provide a variety of
services while ensuring the QoS of each user. Therefore, to provide an E2E
service, the NFV-RA and R-RA should be considered jointly [12, 13].

Network services have various requirements such as delay and data rate. Since
delay is one of the requirements that affect all parts of the network, to
provide services that require a specific delay, it is necessary to consider
all parts of the network together. Also, due to the fact that different parts
of the network interact with each other, it is necessary to consider all parts
of the network together to optimize the total energy consumption. Machine
learning (ML) are new approaches to solve optimization problems in all
domains, e.g., network domain, that have recently received a great deal of
attention [14, 15, 16]. ML, due to its data-driven nature, automatically
learns the network and communication environment and dynamically adapts
protocols without human intervention [14, 15, 16].

###  I-B Related Works

In this section, we review the related works. For this purpose, we classify
the related works into two main categories: NFV-based works and learning
methods for RA.

####  I-B1 NFV-RA

As mentioned above, the NFV-RA problems are divided into three categories: 1)
SFC problems, 2) VNFs Placement problems, and 3) VNFs scheduling problems.
Each of these categories is discussed in the following.

##### VNF Chaining

In the NFV-based network, the flow should be passed through a sequence of
middleboxes in a particular order called SFC to provide a service [17]. A
resource allocation architecture which enables energy-aware SFC for a
software-defined network (SDN) based is proposed in [18] by considering
constraints on delay, link utilization, and server utilization. To solve this
problem, a heuristic algorithm is proposed. In [19], a VNF chaining problem is
addressed to minimize bandwidth utilization while computing results within
reasonable runtime. To solve the problem, a heuristic algorithm is proposed.
In [20], a VNF chaining problem is formulated to minimize total CAPEX and OPEX
costs. In [21], a VNF chaining problem, while considering the trade-off
between resource consumption and operational overhead, is studied. An integer
linear programming (ILP) model is utilized to solve this problem. In [22], a
VNF Selection and chaining problem in SDN/NFV-enabled networks by considering
minimum E2E delay is investigated.

##### VNF Placement

In [23], the VNF placement problem is formulated in which the computational
resources such as processing and storage capacity of CPUs and the capacity of
virtual links are considered. This problem is solved by a heuristic algorithm.
In [24], a VNF placement problem is proposed to minimize the total cost by
considering the limited space for allocating functions in each server. A
linear relaxation approximation is used to solve this problem. An algorithm
for VNF placement and CPU assignment to VNFs is proposed in [25], in which the
proposed problem is solved by decoupled solution strategy leveraging on
sequential decision making. A problem of placement and chaining is studied in
[26]. In this problem, nodes and links latency and capacities are considered.
In [27], the VNF placement optimization problem is formulated for achieving
lower bandwidth consumption and lower maximum link utilization. A genetic
algorithm is proposed to solve this problem. In [28], a VNFs placement and
chaining problem is formulated to minimize the total cost, including the cost
of deploying VNF instances, the cost of using surrogate servers, and the cost
of communication. Moreover, in that work, the communication delays and
processing delays of VNFs are considered. To solve the proposed algorithm, a
heuristic algorithm is presented. A VNF placement cost minimization problem,
taking the network stability into account, is formulated in [29]. The proposed
problem is decomposed into an SFC and VNF mapping problem, which are solved by
a genetic algorithm-based heuristic. In [30], VNF placement in the cloud data
center to minimize the number of activated physical machines and consider
time-varying workloads of physical machines is studied. The proposed problem
is formulated as an integer linear programming (ILP) model, which is solved by
a greedy-based algorithm. A Fully decentralized approach for online placement
and optimization of VMs for NFV-based network is proposed in [31]. A VNF
placement to optimize operational and traffic cost is formulated in [32]. To
solve that problem, a sampling-based Markov approximation (MA) approach is
applied which is a combination of Markov and matching algorithm. The joint VNF
placement and admission control to maximize the network provider revenue in
terms of bandwidth and capacity are investigated in [33]. The relaxation,
reformulation, and successive convex approximation (SCA) methods are employed
to solve this problem. In [34], a cross-layer resource optimization model and
solution for wireless-enabled SFC is proposed to optimize VNF placement and
route path by considering E2E downlink latency, including both wireless and
wired delay.

##### VNF Scheduling

In [3], a VNF scheduling and placement algorithm is proposed in which the
available buffer capacity and processing time of the VNFs are considered. A
greedy-based method is used to solve the proposed algorithm. In [7], a
scheduling problem for SDN-based system is proposed, which is solved with a
two-stage approach. In [35], a joint VNF scheduling and traffic steering
problem is formulated in which VNF transmission and processing delays are
considered. To solve this problem, a genetic algorithm-based method is
utilized. A matching-based algorithm for solving the VNF scheduling problem is
presented in [36]. A deadline-aware VNF scheduling problem for ultra-low
latency services is presented in [37] to maximize the number of admitted
services. The proposed problem is solved by utilizing a tabu search-based
heuristic method. In [38], an RA algorithm for VNFs placement and scheduling
is studied.

####  I-B2 Learning Methods for RA

In [39], a machine learning-enabled architecture is investigated to provide
the demands of advanced vehicular Internet infrastructures. In [15], a VNF
placement problem in SDN/NFV-enabled networks is formulated. To solve the
proposed problem, a deep reinforcement learning (RL) Algorithm is proposed. A
multi-objective resource allocation problem for NFV orchestration (NFVO), as a
Markov decision process, is formulated in [40] in which A Q-learning based
algorithm is proposed to solve this multiple objectives problem. In [41], a
problem of scheduling VNFs is studied to minimize the overall completion time
of all services by considering E2E delay requirements. To solve the problem
with high efficiency and high accuracy, the authors utilized the RL algorithm
to find the optimal scheduling policy. In [42], a novel method based on RL for
performing dynamic SFC resource allocation in NFV-SDN enabled network is
proposed. To design intelligent and efficient VNF selection and chaining for
service function chaining requests, a deep learning-based two-phase algorithm
is introduced in [22].

In all of the above works, the authors consider only one server hosts several
VMs or several servers each of which hosts only one VM. The critical point is
that several servers that host several VMs are not considered simultaneously
in any of the above works. However, in real systems, each server can process
multiple VMs simultaneously. As a result, considering multiple VMS on each
server increases the complexity of the system model and requires new
solutions. Most importantly, to provide some of the 5G and beyond (5G+)
services, such as ultra-low latency services, the network should be considered
from an E2E perspective which is not considered in all previous works.
Furthermore, due to the resource limitation and 5G+ services requirements such
as E2E delay, it is necessary to consider and optimize the NFV and radio parts
jointly. Moreover, the main drawback of the above works is that the new
solution methods, such as ML methods, are not applied to solve the
optimization problem in the NFV-RA area.

###  I-C Our Contributions

The main contributions of this paper can be summarized as follows:

  * •

In this paper, a joint NFV-RA and R-RA optimization problem from the E2E
perspective is proposed, in which QoS of the requested services are satisfied.
In our proposed problem, our aim is to minimize the energy per service by
considering the limitation of radio resources, i.e., power and spectrum, as
well as the computational and storage capacities of core servers. Moreover, we
consider multiple VMs per server in this work, which is a more realistic model
and leads to increase in the complexity of formulation and, thus, the problem
solution.

  * •

We introduce a new approach for VNF chaining, placement, and scheduling by
considering the network service delay in the closed-form expression with
mathematical representation.

  * •

Since the VNF chaining, placement, scheduling, and E2E-QoS for different
services and radio resources satisfy Markov property, we model the
optimization problem as a Markov decision process (MDP). The proposed MDP
problem is subsequently solved by utilizing a state-of-the-art off-policy deep
reinforcement learning (DRL) algorithm, namely soft actor-critic [43], which
is based on the maximum entropy framework.

  * •

A Python-based simulator is developed in the simulation to implement the
proposed algorithm and other baseline algorithms.

  * •

We provide numerical results for the performance evaluation of joint R-RA and
NFV-RA algorithms for different network configurations. Our simulation results
reveal joint R-RA and NFV-RA outperforms conventional ones by approximately
100 % by increasing computational complexity.

###  I-D Paper Organization

The rest of the paper is outlined as follows. In Section II, the system model
and problem formulation are explained. The proposed solution is presented in
Section III. In Section V, the computational complexity and convergence of our
solution are discussed. The simulation results are presented in Section VI.
Finally, in Section VII, the conclusion remarks is inferred.

Symbol Notations: Vector and matrices are indicated by bold lower-case and
upper-case characters, respectively. $\mathcal{A}$ denotes set
$\\{1,\dots,A\\}$, $a(i)$ is the $i$-th element of set $\mathcal{A}$, and
$\mathbb{R}^{n}$ is the set of $n$ dimension real numbers. Moreover,
$U_{d}[a,\leavevmode\nobreak\ b]$ denotes the uniform distribution in interval
$a$ and $b$ and $|.|$ indicates absolute value.

##  II system Model and problem formulation

As shown in Fig. 1, we consider an E2E network consisting of a multi-cell
radio access part and an NFV-based core part with several servers/nodes in
this system model. Therefore, we describe the system model from both aspects,
i.e., radio part and core part, in the following.

######  Remark 1.

It is worth noting that the wireless channel state information (CSI), i.e.,
radio access network (RAN) parameters, changes faster than the parameters of
the NFV, i.e., SFC. In this paper, we assume that the parameters of the radio
and NFV parts are fixed in each optimization problem similar to the existing
works [34, 44].

Figure 1: Considered system model with an example E2E flow for a user is
denoted by red dashed line. In this system model, the resource orchestrator
derives all of the optimization variables.

###  II-A Radio Access Network Description

We consider a multi-cell network with a set of $\mathcal{J}=\\{1,\dots,J\\}$
BSs which are connected to the core part. Moreover, we consider a set of
$\mathcal{U}=\\{1,\dots,U\\}$ users and a set $\mathcal{K}=\\{1,\dots,K\\}$
subcarriers with bandwidth $B$. We define the subcarrier assignment variable
$\rho_{u,j}^{k}$ with $\rho_{u,j}^{k}=1$ if subcarrier $k$ is allocated to
user $u$ at BS $j$ and otherwise $\rho_{u,j}^{k}=0$. We assume orthogonal
frequency division multiple access (OFDMA) as the transmission technology in
which each subcarrier is assigned at most to one user in each BS. Thus, the
following constraint is introduced:

$\text{C1:\leavevmode\nobreak\ }\sum_{u\in\mathcal{U}}\rho_{u,j}^{k}\leq
1,\forall k\in\mathcal{K},j\in\mathcal{J}.$ (1)

Let $h_{u,j}^{k}$ be the channel coefficient between user $u$ and BS $j$ on
subcarrier $k$, $\leavevmode\nobreak\ p_{u,j}^{k}$ be the transmit power from
BS $j$ to user $u$ on subcarrier $k$, $\sigma_{u,j}^{k}$ be the power of
additive white Gaussian noise (AWGN) for user $u$ at BS $j$ on subcarrier $k$,
and $I_{u,j}^{k}$ the is the inter-cell interference on user $u$ at BS $j$
over subcarrier $k$. Thus, the received signal to interference and noise ratio
(SINR) of user $u$ at BS $j$ on subcarrier $k$ is
$\gamma_{u,j}^{k}=\frac{p_{u,j}^{k}h_{u,j}^{k}}{\sigma_{u,j}^{k}+I_{u,j}^{k}}$,
and the achievable data rate (in bits per second/Hz) of user $u$ at BS $j$ on
subcarrier $k$ is given by

$\displaystyle r_{u,j}^{k}=\rho_{u,j}^{k}\log(1+\gamma_{u,j}^{k}),\forall
u\in\mathcal{U},k\in\mathcal{K},j\in\mathcal{J}.$ (2)

Hence, the total achievable rate of user $u$ at BS $j$ is given by
$R_{u,j}=\sum_{k\in\mathcal{K}}r_{u,j}^{k},\,\forall
u\in\mathcal{U},j\in\mathcal{J}$. The transmit power limitation of BS $j$ is

$\text{C2:\leavevmode\nobreak\
}\sum_{k\in\mathcal{K}}\sum_{u\in\mathcal{U}}\rho_{u,j}^{k}p_{u,j}^{k}\leq
P_{j}^{\max},\forall j\in\mathcal{J},$ (3)

where $P_{j}^{\max}$ is the maximum transmit power of BS $j$.

###  II-B NFV Environment Description

Here, we illustrate how the generated traffic of each user is handled in the
network by performing different NFs in the requested user’s NS111Defined by
European Telecommunications Standards Institute (ETSI) as the composition of
Network Function(s) and/or Network Service(s), defined by its functional and
behavioral specification [45]. on the different servers/physical nodes by
leveraging NFV222Standardized by ETSI organization for 5G and beyond [46].. In
this regard, we consider NFV-RA that consists of a new approach for the
placement and scheduling phases. In the placement phase, we map each NF on the
server that is capable to run that NF. Moreover, in the scheduling phase, all
NFs are scheduled on each server. Note that we do not consider mapping virtual
links on the physical links and leave it as an interesting future work as [3,
47]. Besides, similar to [48, 49, 50], we assume that the core network is
full-mesh and there is a direct path between any two nodes in the network.

Table I: Network parameters and notations

Notation Definition $\mathcal{J}/J/j$ Set/number/index of BSs
$\mathcal{U}/U/u$ Set/number/index of users $\mathcal{K}/K/k$ Set/number/index
of subcarriers with bandwidth $B$ $\mathcal{N}/N/n$ Set/number/index of
servers/physical nodes $\mathcal{V}/V/v$ Set/number/index of VMs
$\mathcal{F}/F/f$ Set/number/index of NFs $\mathcal{S}/S/s$ Set/number/index
of services $\Omega_{s}$ Set of NFs which constructs service $s$
$D^{\text{max}}_{s}$ Delay constraint for each packet of service $s$
$R^{\min}_{s}$ Minimum required data rate of service $s$ $P_{j}^{\max}$
Maximum transmit power of the BS $j$ ${\rho_{u,j}^{k}}$ Assignment of
subcarrier $k$ to user $u$ at BS $j$ ${p_{u,j}^{k}}$ Transmit power of user
$u$ from BS $j$ on subcarrier $k$ ${h_{u,j}^{k}}$ Channel coefficient between
user $u$ and the BS $j$ on subcarrier $k$ $\gamma_{u,j}^{k}$ SINR of user $u$
at BS $j$ on subcarrier $k$ $r_{u,j}^{k}$ Achieved rate of user $u$ at BS $j$
on subcarrier $k$ $y_{u,j}$ Packet size of the requested service of user $u$
at BS $j$ $q_{v,n}^{f_{m}^{s}}$, $\psi_{v,n}^{f_{m}^{s}}$ Processing and
storage demand of NF $f_{m}$ in service $s$ on VM $v$ on server $n$,
respectively $\tau_{u,j,n}^{f_{m}^{s},v}$ Processing delay of NF $f_{m}$ on VM
$v$ on server $n$ for service $s$ $L_{n},\Upsilon_{n}$ Processing and storage
capacity of server $n$, respectively
$\tilde{L}_{v}^{n},\tilde{\Upsilon}_{v}^{n}$ Required Processing and storage
capacity for running VM $v$ on server $n$, respectively $e_{v}^{f_{m}^{s}}$ VM
$v$ hosts NF $f_{m}^{s}$ $\chi_{u,j}^{s}$ User $u$ at BS $j$ requests service
$s$ $\beta_{u,j,n}^{f_{m}^{s},v}$ Server mapping between NF $f_{m}^{s}$ of
service $s$ for user $u$, and VM $v$ hosts on node $n$ $q_{v,n}^{f_{m}^{s}}$
Required processing capacity to run NF ${f_{m}^{s}}$ on VM $v$ over the
assigned server $n$ $\psi_{v,n}^{f_{m}^{s}}$ Required storage capacity to run
NF ${f_{m}^{s}}$ on VM $v$ over the assigned server $n$
$t_{u,j,n}^{f_{m}^{s},v}$ Start time of running NF $f_{m}^{s}$ of the
requested service $s$ for user $u$ at BS $j$ on server $n$ in VM $v$
$x_{u,j,n,u^{\prime},j^{\prime},n^{\prime}}^{f_{m}^{s},v,f_{m^{\prime}}^{s^{\prime}},v^{\prime}}$
Ordering indicator between NF $f_{m}^{s}$ of service $s$ for user $u$ and NF
$f_{m^{\prime}}^{s^{\prime}}$ of service $s^{\prime}$ for user $u^{\prime}$

We consider $S$ NS types whose set is $\mathcal{S}=\\{1,2,...,S\\}$ and $M$
NFs whose set is $\mathcal{F}=\\{f_{m}\leavevmode\nobreak\
\big{|}\leavevmode\nobreak\ m=1,\dots,M\\}$. Hereinafter, we use the term
service instead of NS. The considered parameters of the paper are stated in
Table I. Each service $s$ is defined by the tuple ${\Big{(}}\textit{source
node},\Omega_{s},D^{\text{max}}_{s},R^{\min}_{s}{\Big{)}}$333We assume that
the input node is specified for each service, and besides, the output node of
the path is the user requesting the service. where $\Omega_{s}$ is the set of
NFs which constructs service $s$ defined by
$\Omega_{s}=\left\\{f_{m}^{s}\right\\},\,m\in\\{1,\dots,M\\}$,
$D^{\text{max}}_{s}$ is the delay constraint for each packet of service $s$,
and $R^{\min}_{s}$ is the minimum required data rate of service $s$. Here, we
define a known binary variable $\chi_{u,j}^{s}\in\\{0,1\\}$ that indicates
user $u$ at BS $j$ requests service $s$. Based on the data rate requirement of
each service, we have:

$\displaystyle\text{C3:\leavevmode\nobreak\
}R_{u,j}\geq\sum_{s\in\mathcal{S}}\chi_{u,j}^{s}R^{\min}_{s},\,\forall
u\in\mathcal{U}.$

We consider a set of physical node/servers denoted by
$\mathcal{N}=\\{1,...,N\\}$ in the network each of which has a limited amount
of computing and storage resources. The CPU and storage resources of server
$n$ is defined by the tuple ${\Big{(}}L_{n},\Upsilon_{n}{\Big{)}}$ where
$L_{n}$ is processing capacity (CPU cycle per unit time) and $\Upsilon_{n}$ is
storage capacity of server $n$ (number of packets per unit time). Moreover, we
consider a set of VMs indicated by $\mathcal{V}=\\{1,...,V\\}$ running on top
of physical nodes and can host one or more NFs. Besides, VM $v$ needs a
processing resource ($\tilde{L}_{v}^{n}$) and storage resource
($\tilde{\Upsilon}_{v}^{n}$) to run on server $n$. Furthermore, we define a
known binary variable $e_{v}^{f_{m}^{s}}$ with $e_{v}^{f_{m}^{s}}=1$ if VM $v$
can host NF $f_{m}^{s}$ and otherwise $e_{v}^{f_{m}^{s}}=0$. We assume that
each VM can process at most one function at each time [3], but it can process
any NF [3], if capable to run it based on variable $e_{v}^{f_{m}}$.

To embed and schedule NFs, we first need to determine which server hosts which
VM. Then, it must be determined how the NFs are to be scheduled in the VMs and
servers. We introduce a binary variable $\beta_{u,j,n}^{f_{m}^{s},v}$ (i.e.,
VNF-placement variable) which denotes that NF $f_{m}^{s}$ of service $s$ for
user $u$ at BS $j$ is executed at node $n$ in VM $v$, and is defined as

$\begin{split}&\beta_{u,j,n}^{f_{m}^{s},v}=\begin{cases}1,&\text{NF\leavevmode\nobreak\
$f_{m}^{s}$\leavevmode\nobreak\ of service\leavevmode\nobreak\
$s$\leavevmode\nobreak\ for\leavevmode\nobreak\ user\leavevmode\nobreak\
$u$\leavevmode\nobreak\ at BS\leavevmode\nobreak\ $j$\leavevmode\nobreak\
is}\\\ &\text{executed\leavevmode\nobreak\ at server\leavevmode\nobreak\ $n$
in VM $v$}.\\\ 0,&\text{Otherwise}.\end{cases}\end{split}$

Each NF of each service is performed completely at only one VM and server at a
time [35]. Therefore, we have

$\displaystyle\text{C4:\leavevmode\nobreak\
}\sum_{n\in\mathcal{N}}\sum_{v\in\mathcal{V}}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\leq
1,\forall
u\in\mathcal{U},j\in\mathcal{J},f_{m}^{s}\in\Omega_{s},s\in\mathcal{S}.$

Moreover, we assume that NF ${f_{m}^{s}}$ needs a specific number of CPU
cycles per bit to run on VM $v$ over the assigned server $n$, i.e.,
$q_{v,n}^{f_{m}^{s}}$. From the physical resource perspective, we assume that
each server $n$ can provide at most $L_{n}$ CPU cycles per unit time.
Therefore, by considering total required CPU cycles, we have the following
constraint:

$\displaystyle\text{C5:\leavevmode\nobreak\
}\sum_{j\in\mathcal{J}}\sum_{u\in\mathcal{U}}\sum_{s\in\mathcal{S}}\sum_{v\in\mathcal{V}}\sum_{f_{m}^{s}\in\Omega_{s}}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\big{(}{y}_{u,j}q_{v,n}^{f_{m}^{s}}+\tilde{L}_{v}^{n}\big{)}\leq
L_{n},$ $\displaystyle\forall n\in\mathcal{N},$

where $y_{u,j}$ is the packet size of the service of user $u$ at BS $j$. Here,
we assume that the packet size is equal to the number of bits generated in a
unit time [3, 7, 35]. Hence, the elapsed time of each NF $f_{m}^{s}$ for each
bit on VM $v$ over server $n\in\mathcal{N}$ is equal to
$\frac{1}{q_{v,n}^{f_{m}^{s}}},\forall
n\in\mathcal{N},f_{m}^{s}\in\Omega_{s}$. Therefore, the total processing delay
of running NF $f_{m}^{s}$ on VM $v$ at server $n$ for each packet with packet
size ${y}_{u,j}$ of user $u$ at BS $j$ is obtained as

$\displaystyle\tau_{u,j,n}^{f_{m}^{s},v}=\frac{{y}_{u,j}}{q_{v,n}^{f_{m}^{s}}},\forall
n\in\mathcal{N},f_{m}^{s}\in\Omega_{s}.$ (4)

Additionally, we assume that each NF needs specific storage size, i.e.,
$\psi_{v,n}^{f_{m}^{s}}$, when it is running on VM $v$ on the server $n$.
Moreover, each packet consumes $y_{u,j}$ buffer capacity, when it is waiting
for running a NF on an assigned server. Hence, from the storage and buffer
resource perspective, we consider that each server has a limited buffer and
storage size, i.e., $\Upsilon_{n}$, which leads to the following constraint:

$\displaystyle\text{C6:\leavevmode\nobreak\
}\sum_{u\in\mathcal{U}}\sum_{\begin{subarray}{c}{s\in\mathcal{S}}\\\
f_{m}^{s}\in\Omega_{s}\end{subarray}}\sum_{j\in\mathcal{J}}\sum_{v\in\mathcal{V}}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\Big{(}(\psi_{v,n}^{f_{m}^{s}}+{y}_{u,j})+\tilde{\Upsilon}_{v}^{n}\Big{)}\leq\Upsilon_{n},$
$\displaystyle\forall n\in\mathcal{N}.$

###  II-C Latency Model

In NFV-RA, our main aim is to guarantee the service requirement, which
includes maximum tolerable delay for each packet with size ${y}_{u,j}$ of the
requested services while minimizing the energy consumption of physical nodes
[3, 7, 35]. The total delay that we consider in our system model results from
executing NFs, waiting time, and transmission time between nodes. In the
following, we calculate the total delay resulting from scheduling.

Each NF should wait until its preceding function is processed before its
processing can commence. The processing of each packet of service $s$ ends
when its last function is processed. Therefore, the total processing time is
the summation of the processing times of the NFs at the various VMs on
servers. For scheduling of each NF on a server, we need to determine the start
time of it. Therefore, we define $t_{u,j,n}^{f_{m}^{s},v}$ which is the start
time of running NF $f_{m}^{s}$ of the requested service $s$ for user $u$ at BS
$j$ on server $n$ in VM $v$. Furthermore, we introduce a new variable
$x_{u,j,n,u^{\prime},j^{\prime},n^{\prime}}^{f_{m}^{s},v,f_{m^{\prime}}^{s^{\prime}},v^{\prime}},$
in which, if NF $f_{m}^{s}$ of user $u$ at BS $j$ is running on VM $v$ at
server $n$ after NF $f_{m^{\prime}}^{s^{\prime}}$ of user $u^{\prime}$ at BS
$j^{\prime}$ is running on VM $v^{\prime}$ at server $n^{\prime}$, its value
is $1$, otherwise is $0$. By these definitions, the starting time of each NF
can be obtained in (5). The first part of this equation determines the
priority of performing the NFs of the requested service by a specific user.
The second part determines the priority of running each NF of user $u$ in
relation to each NF of user $u’$. Thus, the execution time and location of
each NF are determined. In the following, we clarify this equation with an
example.

$\displaystyle\text{C7:\leavevmode\nobreak\
}t_{u,j,n}^{f_{m}^{s},v}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\geq\max\Bigg{\\{}\mathop{\max}\limits_{\scriptstyle\forall
f_{m^{\prime}}^{s^{\prime}}\in\Omega_{s^{\prime}},\scriptstyle
u^{\prime}\in\mathcal{U}}\left\\{{x_{u,j,n,u^{\prime},j^{\prime},n}^{f_{m}^{s},v,f_{m^{\prime}}^{s^{\prime}},v}\beta_{u^{\prime},j^{\prime},n}^{f_{m^{\prime}}^{s^{\prime}},v}e_{v}^{f_{m^{\prime}}^{s^{\prime}}}(t_{u^{\prime},j^{\prime},n}^{f_{m^{\prime}}^{s^{\prime}},v}+\tau_{u^{\prime},j^{\prime},n}^{f_{m^{\prime}}^{s^{\prime}},v})}\right\\},$
$\displaystyle\mathop{\max}\limits_{\scriptstyle\forall
f_{m^{\prime\prime}}^{s}\in\\{{\Omega_{s}}-f_{m}^{s}\\},\scriptstyle
n^{\prime}\in\\{\mathcal{N}-n\\}}\left\\{{x_{u,j,n,u,j,n^{\prime}}^{f_{m}^{s},v,f_{m^{\prime\prime}}^{s},v^{\prime}}\beta_{u,j,n^{\prime}}^{f_{m^{\prime\prime}}^{s},v^{\prime}}e_{v^{\prime}}^{f_{m^{\prime\prime}}^{s}}(t_{u,j,n^{\prime}}^{f_{m^{\prime\prime}}^{s},v^{\prime}}+\tau_{u,j,n^{\prime}}^{f_{m^{\prime\prime}}^{s},v^{\prime}})}\right\\}\Bigg{\\}},$
$\displaystyle\forall
f_{m}^{s}\in{\Omega_{s}},f_{m^{\prime}}^{s^{\prime}}\in{\Omega_{s^{\prime}}},{\mkern
1.0mu}\forall s,s^{\prime}\in\mathcal{S},\forall n\in\mathcal{N},\,\forall
u,u^{\prime}\in\mathcal{U},\forall v,v^{\prime}\mathcal{V},\forall
j,j^{\prime}\in\mathcal{J}.$ (5)

To demonstrate how we formulate the scheduling of NFs, the proposed scheduling
policy is illustrated in Fig. 2. This figure is the state of the network
assuming two services with several NFs, four servers, and three VMs. The
processing time of each NF, $\tau_{u,j,n}^{f_{m}^{s},v}$ is obtained by (4),
i.e., $\tau_{1,1,4}^{f_{2}^{1},v3}=\frac{y_{1,1}}{q_{v3,4}^{f_{2}^{1}}}$. Each
NF has a start time (denoted by $t_{u,j,n}^{f_{m}^{s},v}$) to run on an
assigned VM and elapsed processing time (denoted by
$\tau_{u,j,n}^{f_{m}^{s},v}$) and is completed by the time given by
$t_{u,j,n}^{f_{m}^{s},v}+\tau_{u,j,n}^{f_{m}^{s},v}$ on VM $v$ at server $n$.
To make (5) clearer, suppose user $1$ at BS $1$ requests service $1$, and user
$2$ at BS $2$ requests service $2$. As can be seen, for user 1 $f_{1}^{1}$ is
executed after $f_{2}^{1}$ at VM $3$ on server $4$. Thus, we have
$x_{1,1,4,1,1,4}^{f_{1}^{1},v3,f_{2}^{1},v3}=1$ and
$x^{1,1,4,1,1,4}_{f_{1}^{1},v3,f_{2}^{1},v3}=0$. Moreover, as shown in this
figure, $f_{3}^{1}$ of user $1$ at BS $1$ is executed at VM $1$ on server $2$
after $f_{1}^{2}$ of user $2$ at BS $2$ is executed at VM $2$ on server $4$.
Thus, we have $x_{1,1,2,2,2,4}^{f_{3}^{1},v1,f_{1}^{2},v2}=1$ and
$x^{1,1,2,2,2,4}_{f_{3}^{1},v1,f_{1}^{2},v2}=0$. As can be seen, servers $3$
and $1$ are off, since based on our aim and solution algorithm, two servers
from four servers are sufficient to ensure the requested requirements.

Figure 2: Schematic illustration of the proposed scheduling and formulation of
(5).

It is worth noting that our problem is performed for a snapshot assuming all
the packets of the services which are generated in unit time are fetched into
the network at the beginning of each unit time. Hence, the arrival time of all
packets is the same and can be set to zero. Therefore, the total service chain
delay for each user $u$ at BS $j$ on the requested service is inferred in (6)
[36] in which $\hat{B}_{n,n^{\prime}}$ is bandwidth of link between node
$n^{\prime}$ to node $n$.

$\displaystyle D^{\text{Total}}_{u,j}=\max\limits_{\scriptstyle\forall
n\in{\cal N},v\in{\cal V},\hfill\atop\scriptstyle
f_{m}^{s}\in{\Omega_{s}},s\in{\cal
S}\hfill}\Big{\\{}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\Big{(}{t_{u,j,n}^{f_{m}^{s},v}}+\tau_{u,j,n}^{f_{m}^{s},v}+\sum_{n^{\prime}\in\mathcal{N}}\dfrac{\sum_{v^{\prime}\in\mathcal{V}}\beta_{u,j,n^{\prime}}^{f_{m-1}^{s},v^{\prime}}e_{v}^{f_{m}^{s}}y_{u,j}}{\hat{B}_{n,n^{\prime}}}\Big{)}\Big{\\}},\forall
u\in\mathcal{U}.$ (6)

Based on the delay requirement of each service, we have the following
constrain:

$\displaystyle\text{C8:\leavevmode\nobreak\ }D^{\text{Total}}_{u}\leq
D^{\text{max}}_{s},\leavevmode\nobreak\ \forall u\in\mathcal{U}.$

###  II-D Objective Function and Optimization Problem

Our aim is to minimize the total energy per service in the network. To this
end, we divide the total energy into three categories: 1) Radio part energy
consumption, 2) CPU energy consumption, and 3) memory energy consumption.
Therefore, the energy of the radio part is calculated as follows: where
$\tilde{\tau}$ is time unit, in which the optimization problem is solved.
Moreover, the energy consumption of the CPU of servers consists of two parts:
the energy consumption of the connected mode (when the server is working and
host one or several VMs), the energy consumption of the idle mode (when the
server is in sleep mode and no VMs run on it). Therefore, we have:

$\displaystyle
E(\mathbb{B},\boldsymbol{\tau})^{\text{CPU}}=\sum\limits_{\scriptstyle\forall
n\in{\cal N},v\in{\cal V},u\in\mathcal{U}\hfill\atop\scriptstyle
f_{m}^{s}\in{\Omega_{s}},s\in{\cal
S},j\in\mathcal{J}\hfill}P_{v,n}^{\text{CPU}}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\tau_{u,j,n}^{f_{m}^{s},v}+$
$\displaystyle\sum_{n\in\mathcal{N}}\hat{P}_{n}^{\text{CPU}}(\tilde{\tau}-\sum\limits_{\scriptstyle\forall
v\in{\cal V},u\in\mathcal{U}\hfill\atop\scriptstyle
f_{m}^{s}\in{\Omega_{s}},s\in{\cal
S},j\in\mathcal{J}\hfill}\beta_{u,j,n}^{f_{m}^{s},v}e_{v}^{f_{m}^{s}}\tau_{u,j,n}^{f_{m}^{s},v}),$
(7)

Therefore, the total energy consumption function is given as follows [51]:

$\displaystyle\Psi(\mathbb{P},\boldsymbol{\rho},\mathbb{B},\boldsymbol{\tau})=\mu_{1}E(\mathbb{P},\boldsymbol{\rho})^{\text{Radio}}+\mu_{2}E(\mathbb{B},\boldsymbol{\tau})^{\text{CPU}}$
(8)

where $\mu_{1}$ and $\mu_{2}$ are constants and are used for scaling and
balancing the energy of different resource types.

The gain of this approach is not only saving the consumption of the power in
the active mode (i.e., under load) but also saving the power of the servers in
the idle mode.

###  II-E Problem Formulation

Based on these definitions, our aim is to solve the following JRN-RA
optimization problem:

$\displaystyle\max_{\mathbb{P},\boldsymbol{\rho},\boldsymbol{\mathcal{\beta}}}\frac{\sum_{u\in\mathcal{U}}\sum_{j\in\mathcal{J}}\sum_{k\in\mathcal{K}}r_{u,j}^{k}}{\Psi(\mathbb{P},\boldsymbol{\rho},\mathbb{B},\boldsymbol{\tau})},$
(9a) s.t: C1-C8 (9b)
$\displaystyle\rho_{u,j}^{k}\in\\{0,1\\},\leavevmode\nobreak\ \forall
u\in\mathcal{U},j\in\mathcal{J},\leavevmode\nobreak\ k\in\mathcal{K},$ (9c)
$\displaystyle\beta_{u,j,n}^{f_{m}^{s},v}\in\\{0,1\\},\forall
u\in\mathcal{U},\,\forall f_{m}^{s}\in\Omega_{s},\forall s\in\mathcal{S},$
(9d) $\displaystyle
x_{u,j,n,u^{\prime},j^{\prime},n^{\prime}}^{f_{m}^{s},v,f_{m^{\prime}}^{s^{\prime}},v^{\prime}}\in\\{0,1\\},\forall
u,u^{\prime}\in\mathcal{U},u\neq u^{\prime},$ (9e) $\displaystyle\forall
f_{m}^{s},f_{m^{\prime}}^{s^{\prime}}\in\Omega_{s^{\prime}},\forall
v,v^{\prime}\in\mathcal{V},n,n^{\prime}\in\mathcal{N},$

where $\boldsymbol{\rho}=[\rho_{u,j}^{k}]$,
$\boldsymbol{\mathcal{\beta}}=[\beta_{u,j,n}^{f_{m}^{s},v}]$,
$\mathbb{{P}}=[p_{u,j}^{k}]$. In problem (9a), constraints C1-C8 are defined
in the previous section. Constraints (9c)-(9e) are for binary variables.

##  III MARKOV DECISION PROCESS BASED PROBLEM FORMULATION

Due to the arrival of unpredictable users’ requests, the state of the network,
traffic, and capacity of the servers will be changed and will not be static.
Thus, we are looking for a dynamic and appropriate solution to solve our
proposed system. In this paper, we want to perform different NFs on the
different VMs and assign VMs to the server/physical nodes. Since these choices
are not specific and may have high dimensional state spaces, we adopt deep RL
approaches for the joint radio and core resource allocation in the NFV-based
network from the EE perspective. Hence, we investigate deep learning methods.
Since we have compounded actions, such as continuous and discrete variables,
we propose a soft actor-critic (SAC) algorithm modeled as an MDP. The elements
of the MDP, i.e., state, action, and reward, are defined as follows:

For each time step $t_{p}$, the state is forwarded for the actor and critic.
The actual policy takes the state $s_{t_{p}}$ and outputs an action
$a_{t_{p}}$, which results in a new stat $\tilde{s}_{t_{p}}$ and a reward
$r_{t_{p}}$.

###  III-A System States:

The system state is an abstraction of the environment, and the learner makes
action decisions based on the states. The most influential parameter on the
state of the network environment is channel gain. Therefore, the system state
$\mathcal{S}$ is defined as is the state space, and the state
$s_{t_{p}}\in\mathcal{S}$ at $t_{p}$-th time slot that is equivalent to
channel gain of all users at $t_{p}$-th time slot is represented as:

$\displaystyle s_{t_{p}}=\\{h_{u,j}^{k}(t_{p})\\}.$ (10)

###  III-B Action space:

The learner takes an action by considering the network’s state because the
network operates in a new state and transients from the current state of the
network. The set of action space $\mathcal{A}$, which
$a_{t_{p}}\in\mathcal{A}$ executed in the ${t_{p}}$-th time slot is expressed
as:

$\displaystyle a_{t_{p}}=\\{P(t_{p}),\rho(t_{p}),\beta(t_{p})\\},$ (11)

where, $P(t_{p})$, $\rho(t_{p})$, and $\beta(t_{p})$ are the corresponding
actions of state that mentioned before in time slot $t_{p}$.

###  III-C Reward Function:

After an action is taken, the environment will return an immediate reward
$r_{t_{p}}$ to the agent. In this system, the agent tries to improve the
optimization problem (9), with the aim of maximization of EE. The benefit of
the action is defined as the reward $r_{t_{p}}$. Thus, the reward is denoted
as:

$\displaystyle
r_{t_{p}}=c\frac{\sum_{u\in\mathcal{U}}\sum_{j\in\mathcal{J}}\sum_{k\in\mathcal{K}}r_{u,j}^{k}}{\Psi(\mathbb{P},\boldsymbol{\rho},\mathbb{B},\boldsymbol{\tau})},$
(12)

where $c$ is a coefficient factor. Based on the actions that users take
according to the state of the system, the system’s state will be changed. For
instance, if the learning solution increases the objective function, the
reward will be positive; otherwise, the reward will be negative.

##  IV PROBLEM SOLUTION WITH SOFT ACTOR-CRITIC DRL ALGORITHM

To solve our proposed MDP problem, we employs SAC DRL. The SAC algorithm
consists of the following four parts:

  * •

Actor: Separate policy DNN,

  * •

Critic: value function DNNs,

  * •

An off-policy method which enables reuse of the past experience stored in the
memory $\mathcal{M}$,

  * •

Entropy maximization to ensure stability.

###  IV-A Soft Value Functions

The main objective of the RL agent is to find the policy $\pi(a_{t_{p}}\mid
s_{t_{p}})$ to maximize the expected reward. $\pi$ is the parameter to show
the stochastic policy that gives the probability of taking a specific action
when the agent is in a certain state. Despite existing RL algorithms, to
ensure that the agent can explore continually, an entropy term
$-\log\pi(a_{t_{p}}\mid s_{t_{p}})$ is added to the reward in [52]. The
objective with the expected entropy of the policy $\pi(a_{t_{p}}\mid
s_{t_{p}})$, which is the parameter to show the stochastic policy, gives the
probability of taking a certain action when the agent is in a certain state is
termed as the entropy objective and written as:

$\displaystyle
V^{\pi}=\mathbb{E}\\{{\sum_{i=1}^{T^{\prime}}}\nu^{t_{p}}[r_{t_{p}}-\lambda\log\pi(a_{t_{p}}\mid
s_{t_{p}})]\mid\pi\\}$ (13)

where $\nu^{t_{p}}$ shows the discount factor. Moreover, $\nu\in[0,1]$ and
$\lambda$ are the parameters that responsible for controlling the
stochasticity of the optimal policy. Furthermore, $\tilde{Q}^{\pi}$ is the
estimate of expected reward if the action $a_{t_{p}}$ in state $s_{t_{p}}$
according to the policy $\pi$ was taken. Q-value function:

$\displaystyle\tilde{Q}^{\pi}(s_{t_{p}},a_{t_{p}})=\mathbb{E}\\{{\sum_{i=1}^{T^{\prime}}}\nu^{\pi}[r_{t_{p}}-\lambda\log\pi(a_{t_{p}}\mid
s_{t_{p}})]\mid a_{t_{p}},s_{t_{p}},\pi\\}$ (14)

The relevance of state-value function and Q-function is:

$\displaystyle
V^{\pi}(s_{t_{p}})=\lambda\log\int_{A}\exp(\frac{1}{\lambda}\tilde{Q}^{\pi}(s_{t_{p}},a_{t_{p}}))da$
(15)

and the optimal policy for (13), according to the state and action that was
chosen is defined as below:

$\displaystyle\pi^{\ast}(0\mid
s_{t_{p}})=\text{exp}(\frac{1}{\lambda}(\tilde{Q}^{\pi}(s_{t_{p}},a_{t_{p}})-V^{\pi}(s_{t_{p}})))$
(16)

Compared to the conventional actor-critic DRL algorithm [53], and that the
(15) is a $LogSumExp$ function which is a soft maximum, the Q-value function
and the state-value function are also named as soft Q-value function and soft
state-value function, respectively.

###  IV-B Critic Networks:

In this section, among many effective methods [54], we apply DNN to
approximate value function due to its convergence and stability [54].
According to the goal of policy evaluation, soft Q-value
$Q_{w_{i}}(s_{t_{p}},a_{t_{p}})$ is approximated by a DNN which is symbolized
by weight $w_{i}$. In this regard, we use memory $\mathcal{M}$, in which the
new generated experience at each time step is stored, to increase system
stability and elude the divergence. In this process, $w_{i}$ is updated from
the memory $\mathcal{M}$ to fracture the correlations between samples. Hence,
the loss function can be expressed as follows:

$\displaystyle
L(w_{i})=\mathbb{E}[\frac{1}{2}(\tilde{Q}_{w_{i}}(s_{t_{p}},a_{t_{p}}\mid
w_{i})-H_{w_{i}}(s_{t_{p}},a_{t_{p}}))^{2}],$ (17)

where term $H_{w_{i}}(s_{t_{p}},a_{t_{p}})$ is:

$\displaystyle H_{w_{i}}(s_{t_{p}},a_{t_{p}})=r_{t_{p}}+\nu V^{\pi}.$ (18)

As previously explained, to update the weight of critic network, i.e, $w_{i}$,
with gradient decent method to minimize the loss function $L(w_{i})$, we have:

$\displaystyle
w_{i}(new)=w_{i}(old)-\alpha_{w_{i}}\nabla_{w_{i}}L(w_{i}(old)),$ (19)

where $\alpha_{w_{i}}$ is the learning rate of the critic which is a positive
value and $\nabla_{w_{i}}L(w_{i})$ is the gradient of $L(w_{i})$ and achieved
by:

$\displaystyle\nabla_{w_{i}}L(w_{i})=$ (20)
$\displaystyle\nabla_{w_{i}}\tilde{Q}_{w_{i}}(s_{t_{p}},a_{t_{p}})[\tilde{Q}_{w_{i}}(s_{t_{p}},a_{t_{p}})-H_{w_{i}}(s_{t_{p}},a_{t_{p}})].$

###  IV-C Actor Networks:

In this part based on a policy improvement approach, the objective function
can be written as:

$\displaystyle
L(\delta_{i})=\mathbb{E}[-\frac{1}{2}(\lambda\log\pi_{\delta_{i}}(a_{t_{p}}\mid
s_{t_{p}})+H_{\delta_{i}}(s_{t_{p}},a_{t_{p}}))^{2}],$ (21)

where term $H(s_{t_{p}},a_{t_{p}})$ is:

$\displaystyle
H_{\delta_{i}}(s_{t_{p}},a_{t_{p}})=Q_{\pi}(s_{t_{p}},a_{t_{p}})-V^{\pi}(s_{t_{p}}).$
(22)

In a similar way of critic network, to update the weight of actor network
$\delta_{i}$ with gradient decent method to minimize the loss function
$L(\delta_{i})$, we have:

$\displaystyle\delta_{i}(new)=\delta_{i}(old)+\alpha_{\delta_{i}}\nabla_{\delta_{i}}L(\delta_{i}(old))$
(23)

where $\alpha_{\delta_{i}}$ is the learning rate of the actor network and
$\nabla_{\delta_{i}}L(\delta_{i})$ is the gradient of $L(\delta_{i})$ and
achieved by:

$\displaystyle\nabla_{\delta_{i}}L(\delta_{i})=$ (24)
$\displaystyle\nabla_{\delta_{i}}V^{\pi}(s_{t_{p}})[-H(s_{t_{p}},a_{t_{p}})+\lambda\log\pi_{\delta_{i}}(a_{t_{p}})].$

###  IV-D The Soft Actor-Critic DRL Algorithm

The duty of agent is divided as follows:

  * •

The actor network: mapping states to actions,

  * •

The critic network: estimating states and state-action couple,

  * •

The memory $\mathcal{M}$: storing experience.

The actor network with the suitable approach of finding the right policy
$\pi_{\delta_{i}}(0\mid s_{t_{p}})$, which will be trained based on (16),
according to the current state$s_{t_{p}}$ selects and executes an action
$a_{t_{p}}$. In this step the neural network, depending on whether the action
is well chosen or not, the agent receive rewards or punishments, respectively.
Then, the next state ${\tilde{s}}_{t_{p}}$ is characterized and memory
$\mathcal{M}$ store this tuple
$(s_{t_{p}},a_{t_{p}},r_{t_{p}},{\tilde{s}}_{t_{p}})$. The critic network
samples randomly from the tuples that are stored in the memory $\mathcal{M}$,
with the approach of elimination of temporal correlations between samples and
train the state-value and Q-value. This learning process is intermittent
sampling of the environment and updating the network until the final state or
the network time-out. To understanding the better concept of this cycle, we
propose Algorithm 1.

1 Initialize parameters $w_{i},\delta_{i}$ and empty memory $\mathcal{M}$

2 Set target parameters equal to main parameters

3 $\mathbf{for}$ each iteration $\mathbf{for}$ each environment step: Observe
state $s_{t_{p}}$ and select action $a_{t_{p}}\sim\pi(0\mid s_{t_{p}})$
Execute $a_{t_{p}}$ in the environment Observe next state
${\tilde{s}}_{t_{p}}$, reward $r_{t}$ Store
$(s_{t_{p}},a_{t_{p}},r_{t_{p}},{\tilde{s}}_{t_{p}})$ in memory and update
$\mathcal{M}\leftarrow\mathcal{M}\cup\\{(s_{t_{p}},a_{t_{p}},r_{t},{\tilde{s}}_{t_{p}})\\}$
$\mathbf{end}$ $\mathbf{for}$ $\mathbf{for}$ each gradient step:
$w_{i}(new)\leftarrow w_{i}(old)-\alpha_{w_{i}}\nabla_{w_{i}}L(w_{i}(old))$
$\delta_{i}(new)\leftarrow\delta_{i}(old)+\alpha_{\delta_{i}}\nabla_{\delta_{i}}L(\delta_{i}(old))$
$\mathbf{end}$ $\mathbf{for}$

4 $\mathbf{end}$ $\mathbf{for}$

output parameters: $w_{i},\delta_{i}$

Algorithm 1 Soft Actor-critic Algorithm [55]

##  V COMPUTATIONAL COMPLEXITY AND CONVERGENCE ANALYSIS

###  V-A Computational Complexity

The proposed system’s computational complexity is one of the key and practical
criteria of the proposed system. Thus, we analyze the computational complexity
of our proposed system model from two perspectives: training process and
running process.

  * •

Training Process: In most of the papers such as [15],[22], the authors ignored
the computation complexity of the training process and assumed that the
training process happens in off-line mode, and its computation complexity is
proportional to the size of the training data and duration of the training. We
consider the number of neurons on $l$th and $k$th layer of the actor and
critic network as $I^{l}$ and $V^{k}$, respectively. Hence, the total
computational complexity of the training process is
$\mathcal{O}(\sum_{l=2}^{L-1}(I^{l-1}I^{l}+I^{l}I^{l+1})+\sum_{k=2}^{K-1}(V^{k-1}V^{k}+V^{k}V^{k+1}))$[56].

  * •

Running Process: In this part, we propose the computational complexity of the
actor critic network. The internal computations of neural network algorithms
depends on the architecture, the number of neurons, and the number of layers.
Besides, in an algorithm consistent with deep RL, the number of states and
actions is related to the number of neurons and neural network layers.
Therefore, the computational complexity is in order of
$\mathcal{O}(B\times(2\times HZ)\times(J\times U\times S\times V\times M))$ in
which $B$ is the training batch, $H$ is the number of neurons in hidden layer
$Z$, $J$, $U$, $S$, $V$, and $M$ are the numbers of BSs, users, NS, VMs, and
NFVs, respectively.

###  V-B Convergence Analysis

In this section, we evaluate the convergence of the SAC algorithm. According
to Q-function in (14), the solution can converge to the optimal Q-function as
$t\to\infty$ with probability 1, if the learning rates of the algorithm
$\alpha_{w_{i}}$ and $\alpha_{\delta_{i}}$ are deterministic, non-increasing,
and satisfy [57]:

$\displaystyle\sum_{0}^{\infty}\alpha_{w_{i}}=\infty,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\sum_{0}^{\infty}(\alpha_{w_{i}})^{2}\textless\infty,$ (25)
$\displaystyle\sum_{0}^{\infty}\alpha_{\delta_{i}}=\infty,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\sum_{0}^{\infty}(\alpha_{\delta_{i}})^{2}\textless\infty,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\lim_{t_{p}\to\infty}\frac{\alpha_{\delta_{i}}}{\alpha_{w_{i}}}=0,$ (26)

and $\mid r_{t_{p}}\mid$ be bounded[58]. With the aim of fast convergence and
training our neural network effectively, we utilize the inverse time decaying
learning rate that uses the large learning rate in the first episodes to
prevent the network from getting stuck in a bad local optimum trap near the
initial point and uses the small learning rate in the last training epochs in
order to converge to a good local optimum [59]. The convergence of our
proposed algorithm is also analyzed through simulations results in Section VI.

1 Initialize randomly $s,\theta,w$

2 For $t=1,\cdots,T$: 1.Sample reward $r_{t_{p}}\leavevmode\nobreak\
R_{t_{p}}(s,a)$ and next state $s^{\prime}\leavevmode\nobreak\
P(s^{\prime}\mid s,a)$; 2.Then sample the next action
$a^{\prime}\leavevmode\nobreak\ \Pi_{\theta}(a^{\prime}\mid s^{\prime})$;
3.Update the policy parameters:
$\theta_{i}(new)\leftarrow\theta_{i}(old)+a_{\theta}\nabla
J_{i}(\theta_{i}(old))$; 4.Compute the correction (TD error) for action-value
at time $t_{p}$: $\delta_{t_{p}}=\alpha[R_{s\rightarrow
s^{\prime}}+\nu\max\tilde{Q}_{t_{p}}(s^{\prime},a^{\prime})-\tilde{Q}(s,a)]$
5.and use it to update the parameters of action-value function:
$w_{i}(new)=w_{i}(old)+a_{w}\nabla L(w_{i}(old))$

Update $a\leftarrow a^{\prime}$ and $s\leftarrow s^{\prime}$.

Algorithm 2 Actor-critic methods consist of action-value $\tilde{Q}$ or state-
value $V_{\Pi}$

1 Initialize total reward $r_{t}=0$

2 for  _ $t=1,\cdots,T$_ do

3 for  _$u=1,\cdots,U$ _ do

4 The agent take an actoion $a$ with considering state $s$.

5 if  _constarints C1-C8_ then

6 Calcute the obtained rate for user $u$

7 Calcute the EE by 12

8

9

Algorithm 3 Join Radio and Core Resource Allocation

1 Initialize randomly $s,\theta,w$

2 For $t=1,\cdots,T$: 1.Sample reward $r_{t_{p}}\leavevmode\nobreak\
R_{t_{p}}(s,a)$ and next state $s^{\prime}\leavevmode\nobreak\
P(s^{\prime}\mid s,a)$; 2.Then sample the next action
$a^{\prime}\leavevmode\nobreak\ \Pi_{\theta}(a^{\prime}\mid s^{\prime})$;
3.Update the policy parameters:
$\theta_{i}(new)\leftarrow\theta_{i}(old)+a_{\theta}\nabla
J_{i}(\theta_{i}(old))$; 4.Compute the correction (TD error) for action-value
at time $t_{p}$: $\delta_{t_{p}}=\alpha[R_{s\rightarrow
s^{\prime}}+\nu\max\tilde{Q}_{t_{p}}(s^{\prime},a^{\prime})-\tilde{Q}(s,a)]$
5.and use it to update the parameters of action-value function:
$w_{i}(new)=w_{i}(old)+a_{w}\nabla L(w_{i}(old))$

Update $a\leftarrow a^{\prime}$ and $s\leftarrow s^{\prime}$.

Algorithm 4 Actor-critic methods consist of action-value $\tilde{Q}$ or state-
value $V_{\Pi}$

1 Initialization: Set the soft update parameter as $\tau=0.05$, the discount
factor as $\gamma=0.9$, the maximum memory size as $|\mathcal{M}|=100000$,
epsilon for exploration as $eps=1$, and decay epsilon factor
$eps_{decay}=0.9994$.

2 for  _ $t=1,\dots,E$_ do

3 Select action $a_{t}=\pi(S_{t}|\theta^{\pi})+n_{t}$

4 Take the action $a_{t}$ and observe the reward $r_{t}$ and observe the new
state $s_{t+1}$

5 Store transaction $<s_{t},a_{t},r_{t},s_{t+1}>$ in momory $\mathcal{M}$

6 Let $y_{t}=r_{t}+\gamma$

7 Update the critic network and minimize the loss function:

8
$\nabla_{\theta}J=\frac{1}{N}\sum_{i}\nabla_{a}Q(s,a|\theta^{Q})|_{s=s_{t},a=\pi(s_{t})}\nabla_{\theta}\pi(s|\theta^{\pi})|_{s_{t}}$

9 Update the target network parameters:

10 $\theta^{Q^{\prime}}\leftarrow\rho\theta^{Q}+(1-\rho)\theta^{Q^{\prime}}$

11
$\theta^{\pi^{\prime}}\leftarrow\rho^{\theta^{\pi}}+(1-\rho)\theta^{\pi^{\prime}}$

Algorithm 5 DDPG Algorithm

##  VI SIMULATION RESULTS

In this part, we consider a square area $1000m\times 1000m$ with 4 BSs, and
the users are randomly distributed there. Moreover, we assume 10 subchannels
with the frequency bandwidth of 200 kHz, the maximum power of each BS is set
to 40 watts, and the minimum data rate of the downlink is set to 1 bps/Hz. In
addition, our proposed system has 5 VNFs. Since each VNF require different
processing parameters as $\pi^{user}$, we define some VNFs in Table II [60].
Moreover, we represent three types of SFCs by changing the order of six VNFs
in Table III. Also, we show the performance requirements in terms of bandwidth
$\delta$ and maximum tolerated delay $\varphi$ for all SFCs. The upper bound
for the total service chain delay is set between 75 to 100 ms [15].
Furthermore, we assume that each node can host 6 VMs, and each VMs node can
host 6 VNF at most. The capacities of bandwidth and memory of each physical
link and node are set to $1$ Gbps and $1$ Gbyte, respectively. Moreover, the
CPU capacity of each node is set to 1200 CPU cycle/bit [61]. In the training
process, the value of loss function is Mean Square Error (MSE). From the
commuting services point of view, we should explain that at the start of each
experiment, we assume that each SFC request consists of six VNFs.

Table II: Processing Requirement (per User) for the VNFs

Virtual Network Function $\pi^{user}$ Network Address Translator (NAT) 0.00092
Firewall (FW) 0.0009 Traffic Monitor (TM) 0.0133 WAN Optimization Controller
(WOC) 0.0054 Intrusion Detection Prevention System (IDPS) 0.0107 Video
Optimization Controller (VOC) 0.0054

Table III: Performance Requirement for the SFCs

SFC Chained VNFs Latency$(\varphi)$ Bandwidth$(\delta^{user})$ Web Service
NAT-FW-TM-WOC-IDPS 500 ms 100 kb/s VoIP NAT-FW-TM-FW-NAT 100 ms 64 kb/s Video
Streaming NAT-FW-TM-VOC-IDPS 100 ms 4 Mb/s

It is worth noting that the source code of the proposed method is written with
Python language with TensorFlow library [62] with Adam optimizer. Also, more
details for configuration of the neural network are summarized in Table IV.
Besides, the sources codes of the proposed method and baselines are available
in [bibid].

Table IV: Simulation Parameters Parameter | Value
---|---
Number of sub-carriers | 10
Total available bandwidth | 200 KHz
Number of BSs | 4
Capacity of transaction memory | 100000
Batch size | 32
Actor network learning rate | 0.00001
Critic network learning rate | 0.0001
Moving average hyper parameter | 0.05
AWGN power | -170 dbm
Number of hidden layer of DNN | 2, 3, 4, 5, 6 [17]
Number for neuron in each layer | 1024, 500, 250 [62]

###  VI-A Comparison between Joint Approach and Disjoint Approach

In this system model, we consider a joint radio and core resource allocation
framework. In this part, we investigate the comparison between our proposed
joint framework and disjoint framework in which the resource allocation
optimization problem for RAN and core part are performed separately. To this
end, we rewrite the problem formulation (8) as two separate problems in the
following. Hence, the R-RA problem formulation can be written as:

$\displaystyle\max_{\mathbb{P},\boldsymbol{\rho},\boldsymbol{\mathcal{\beta}}}\frac{\sum_{u\in\mathcal{U}}\sum_{j\in\mathcal{J}}\sum_{k\in\mathcal{K}}r_{u,j}^{k}}{\mu_{1}E(\mathbb{P},\boldsymbol{\rho})^{\text{Radio}}},$
(27) $\displaystyle\textbf{s.t:\leavevmode\nobreak\ \leavevmode\nobreak\
}\text{C1-C2}.$

This new optimization problem can be solved by SAC algorithm. On the other
hand, NFV-RA optimization problem (8) can be summarized as follow:

$\displaystyle\max_{\mathbb{P},\boldsymbol{\rho},\boldsymbol{\mathcal{\beta}}}\dfrac{\sum_{u\in\mathcal{U}}\sum_{j\in\mathcal{J}}\sum_{k\in\mathcal{K}}r_{u,j}^{k}}{\mu_{2}E(\mathbb{B},\boldsymbol{\tau})^{\text{CPU}}},$
(28) $\displaystyle\textbf{ s.t:\leavevmode\nobreak\ \leavevmode\nobreak\
}\text{C3-C9}.$

We also solved the above problem by SAC algorithm. To solve above problems
independently, first we solve R-RA problem. After that, we consider the
obtained rate from radio part. Then we set the radio part delay as $10$ms. For
the sake of evaluating the performance of the proposed SAC method, we consider
DDPG and multi-agent DDPG (MA DDPG) as baselines.Table V shows the superiority
of the SAC method, in terms of complexity, and the comparison with other
method that considered as a baselines.

Table V: Performance Complexity of SAC Compare to Baselines

Methods Complexity SAC $\mathcal{O}(B(2\times HZ))$ DDPG .. MADDPG ..

As we depict in Fig.3, it is assumed that the core network and radio part have
independent agents in which the agents report the obtained reward to a central
neural network called global critic network. The global critic network
calculates the total reward and critics between the agents. By updating the
parameters of the radio and the core agents, these agent try to give better
reward and minimize their loss function, gradually. It’s worth noting that the
global critic just have the global reward that is obtained from radio and core
agents.

Figure 3: Comparing between the methods that are evaluated in this paper. In
disjoint method, the core agent and radio agent selected their action based in
the network state, independently. Similarly, in MA DDPG, each of the agents
take the actions independently, but, they just report the obtained reward to
the global critic. In addition the details of SAC and DDPG methods are written
in Al .4 and 5, respectively

As shown in Fig. 4, EE in the disjoint approach is one-half of the joint
approaches, i.e., SAC and DDPG. The reason for this is that in the disjoint
approach, two problems are solved entirely independently of each other, and
thus, we are faced with two problems that optimize the desired parts, e.g.,
RAN and core, regardless of the E2E view. As stated above, the global critic
network calculates the total reward and critics between the agents. Therefore,
it is expected that MA DDPG method has better performance than the disjoint
approach which is revealed in Fig. 4.

Figure 4: EE for number of the requested services

###  VI-B Effects of Number of the Service Requests

As can be seen in Fig. 4, by increasing the number of users to 10 users, the
EE increases with increasing number of users. This is due to the fact that by
increasing number of users, the network consumes more energy to ensure new
users’ rate and delay requirements. Thus, the total data rate in the network
is increased while the total energy is constant. Furthermore, as can be seen
by increasing the number of users up to 20, the EE remains constant. It stems
from the fact that by increasing the number of users, we reach a point that
all the network energy is consumed to guarantee the requirements of all users.
Thus, with the request of a new user, the network has no resources to
guarantee the requirements of that user. As a result, the new user or one user
who has a negative impact on EE is rejected. Therefore, the total data rate
remains constant from one point in the figure onwards. The total network
energy consumption is also constant and at its maximum value; hence, the EE
remains constant.

###  VI-C Effects of Minimum Data Rate

As we show in Fig.5, the average EE is decreased by increasing the minimum
data rate of the requested services. This is due to the fact that more energy
is required to provide higher data rate. On the other hand, because of the
resource limitations, i.e., energy limitation, the total accepted user is
decrees by increasing the minimum data rate of the services. Therefore, the EE
is decreased by increasing the minimum rate requirement.

Figure 5: Average EE versus the minimum data rate of the requested services.

###  VI-D Effects of Maximum Tolerable Time of the Services

Based on constraint (C8) and Eq. (6), it is obvious that more energy is
required to guarantee slighter delays, and as a result, fewer users can be
accepted due to the limitation of energy resources. As we depicted in Fig.6,
by increasing the maximum tolerable time of the services, average EE of the
network increases. Moreover, as stated before the disjoint approach performs
worse than the other approaches, e.g., MA DDPG, DDPG, and joint SAC.

Figure 6: Average EE versus the maximum tolerable time of the requested
services

###  VI-E Effects of the Coefficients

As can be seen in Fig. 7, by incensing the coefficient of core in the
different methods, the agents try to maximize the EE in the core network. As a
result, the agents select the VMs that have lower energy consumption. By
selecting these VMs, there is the possibility that the physical path between
the source node and destination can not guarantee the maximum tolerable delay
of the services. On the other hand, some users can not be admitted due to the
limited computation resources in these VMs. Moreover, the total energy
consumed in the core network is significantly less than the radio part; thus,
the average EE in the network is very similar to the average efficiency in the
radio part of the network, as is shown in Fig.7(c) and Fig.7(b). Furthermore,
due to the lower energy consumption in the core network, the average EE in the
core network is remarkably higher than EE in the radio part, as can be seen in
Fig.7(a).

###  VI-F Comparing Signaling Overhead in the proposed method and the
Baselines

In each method, the agents need to have information such as the states and
obtained rewards to take the actions. Thus, this information must be
intercommunicated between the resource orchestrator and the radio part and
core network [63]. To measure and model this information, we assume that each
element of the channel gain matrix can be decoded as a fixed-length 16-bit
binary string. Consequently, we use the type ’float16’ in Numpy library in
Python. Based on this, by considering a system with $B$ BSs and $J$
subchannels and $U$ users, the size of the channel gain matrix in a binary
model is equal to $16\times B\times J\times U$. Similarly, the local reward
that each agent obtains can be modeled as a fixed-length 16-bit binary string.
Finally, we summarize the total signal overhead in each episode for the
proposed method and the baselines in Table VI.

Table VI: Total Signal overhead for the methods

Method |  | Size of the data that intercommunicates
---
between the radio part and core network
| Size of the data that intercommunicates
---
between the resource orchestrator and core network
| Size of the data that intercommunicates
---
between the resource orchestrator and radio part
SAC Disjoint | Requested data rate in radio part=16 bits | Ignore able | Ignore able
SAC, DDPG | Requested data rate in radio part=16 bits | 16$\times B\times J\times U$ bits | 16$\times B\times J\times U$ bits
MA DDPG | Requested data rate in radio part=16 bits | Obtained local reward=16 bits | Obtained local reward=16 bits

(a) Average EE of core versus coefficient core

(b) Average EE of radio versus coefficient core

(c) Average EE versus coefficient core

Figure 7: Average EE in core, radio, and both of them versus coefficient core.

##  VII Conclusion

In this paper, we proposed a joint radio and NFV resource allocation to
maximize EE (EE) by satisfying E2E QoS for different service types. In this
regard, we formulated an optimization problem in which power and spectrum
resources are allocated in the radio part. In the core part, the chaining,
placement, and scheduling of VNFS are performed to guarantee the QoS of all
users. We modeled this joint optimization problem as a Markov decision process
(MDP) by considering time-varying characteristics of the available resources
and wireless channels. Thus, we applied a soft actor-critic deep reinforcement
learning (SAC-DRL) algorithm based on the maximum entropy framework to solve
the proposed MDP problem. Simulation results reveled that the proposed joint
approach based on SAC-DRL algorithm could significantly reduce energy
consumption compared to the case in which R-RA and NFV-RA problems are
optimized separately.

## References

  * [1] I. Afolabi, T. Taleb, K. Samdanis, A. Ksentini, and H. Flinck, “Network slicing and softwarization: A survey on principles, enabling technologies, and solutions,” _IEEE Communications Surveys Tutorials_ , vol. 20, no. 3, pp. 2429–2453, Mar. 2018.
  * [2] Z. Chang, Z. Zhou, S. Zhou, T. Chen, and T. Ristaniemi, “Towards service-oriented 5G: Virtualizing the networks for everything-as-a-service,” _IEEE Access_ , vol. 6, pp. 1480–1489, Dec. 2018\.
  * [3] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, and S. Davy, “Design and evaluation of algorithms for mapping and scheduling of virtual network functions,” in _Proc. IEEE Conference on Network Softwarization (NetSoft)_ , Apr. 2015, pp. 1–9.
  * [4] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, and R. Boutaba, “Network function virtualization: State-of-the-art and research challenges,” _IEEE Communications Surveys Tutorials_ , vol. 18, no. 1, pp. 236–262, Sep. 2016.
  * [5] J. Gil Herrera and J. F. Botero, “Resource allocation in NFV: A comprehensive survey,” _IEEE Transactions on Network and Service Management_ , vol. 13, no. 3, pp. 518–532, Sep. 2016.
  * [6] A. N. Al-Quzweeni, A. Q. Lawey, T. E. H. Elgorashi, and J. M. H. Elmirghani, “Optimized energy aware 5G network function virtualization,” _IEEE Access_ , vol. 7, pp. 44 939–44 958, Mar. 2019\.
  * [7] J. F. Riera, E. Escalona, J. Batallé, E. Grasa, and J. A. García-Espín, “Virtual network function scheduling: Concept and challenges,” in _Proc. 2014 International Conference on Smart Communications in Network Technologies (SaCoNeT)_ , June 2014, pp. 1–5.
  * [8] N. Alliance, “Description of network slicing concept,” _NGMN 5G P_ , vol. 1, p. 1, Jan. 2016.
  * [9] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz, J. Lorca, and J. Folgueira, “Network slicing for 5G with SDN/NFV: Concepts, architectures, and challenges,” _IEEE Communications Magazine_ , vol. 55, no. 5, pp. 80–87, May. 2017.
  * [10] R. Mijumbi, J. Serrat, J. Gorricho, N. Bouten, F. De Turck, and R. Boutaba, “Network function virtualization: State-of-the-art and research challenges,” _IEEE Communications Surveys Tutorials_ , vol. 18, no. 1, pp. 236–262, Sep. 2016.
  * [11] M. Mechtri, C. Ghribi, O. Soualah, and D. Zeghlache, “NFV orchestration framework addressing SFC challenges,” _IEEE Communications Magazine_ , vol. 55, no. 6, pp. 16–23, Jun. 2017.
  * [12] E. Hossain and M. Hasan, “5G cellular: key enabling technologies and research challenges,” _IEEE Instrumentation Measurement Magazine_ , vol. 18, no. 3, pp. 11–21, June 2015.
  * [13] H. Dahrouj, A. Douik, O. Dhifallah, T. Y. Al-Naffouri, and M. Alouini, “Resource allocation in heterogeneous cloud radio access networks: advances and challenges,” _IEEE Wireless Communications_ , vol. 22, no. 3, pp. 66–73, June 2015.
  * [14] P. Chemouil, P. Hui, W. Kellerer, Y. Li, R. Stadler, D. Tao, Y. Wen, and Y. Zhang, “Special issue on artificial intelligence and machine learning for networking and communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 6, pp. 1185–1191, 2019.
  * [15] J. Pei, P. Hong, M. Pan, J. Liu, and J. Zhou, “Optimal VNF placement via deep reinforcement learning in SDN/NFV-enabled networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 2, pp. 263–278, 2020.
  * [16] A. Martin, J. Egaña, J. Flórez, J. Montalbán, I. G. Olaizola, M. Quartulli, R. Viola, and M. Zorrilla, “Network resource allocation system for QoE-aware delivery of media services in 5G networks,” _IEEE Transactions on Broadcasting_ , vol. 64, no. 2, pp. 561–574, 2018.
  * [17] Y. Xie, Z. Liu, S. Wang, and Y. Wang, “Service function chaining resource allocation: A survey,” _arXiv preprint arXiv:1608.00095_ , 2016.
  * [18] M. M. Tajiki, S. Salsano, L. Chiaraviglio, M. Shojafar, and B. Akbari, “Joint energy efficient and QoS-aware path allocation and VNF placement for service function chaining,” _IEEE Transactions on Network and Service Management_ , vol. 16, no. 1, pp. 374–388, 2019.
  * [19] M. T. Beck and J. F. Botero, “Coordinated allocation of service function chains,” in _Proc IEEE Global Communications Conference (GLOBECOM)_ , San Diego, CA, USA, Dec. 2015, pp. 1–6.
  * [20] H. Li, L. Wang, X. Wen, Z. Lu, and L. Ma, “Constructing service function chain test database: An optimal modeling approach for coordinated resource allocation,” _IEEE Access_ , vol. 6, pp. 17 595–17 605, 2018\.
  * [21] J. Liu, W. Lu, F. Zhou, P. Lu, and Z. Zhu, “On dynamic service function chain deployment and readjustment,” _IEEE Transactions on Network and Service Management_ , vol. 14, no. 3, pp. 543–553, Sep. 2017.
  * [22] J. Pei, P. Hong, K. Xue, D. Li, D. S. L. Wei, and F. Wu, “Two-phase virtual network function selection and chaining algorithm based on deep learning in SDN/NFV-enabled networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 6, pp. 1102–1117, 2020.
  * [23] R. Riggio, A. Bradai, D. Harutyunyan, T. Rasheed, and T. Ahmed, “Scheduling wireless virtual networks functions,” _IEEE Transactions on Network and Service Management_ , vol. 13, no. 2, pp. 240–252, June 2016.
  * [24] R. Cohen, L. Lewin-Eytan, J. S. Naor, and D. Raz, “Near optimal placement of virtual network functions,” in _Proc. IEEE Conference on Computer Communications (INFOCOM)_ , 2015, pp. 1346–1354. Kowloon, Hong Kong. Apr.
  * [25] S. Agarwal, F. Malandrino, C. F. Chiasserini, and S. De, “VNF placement and resource allocation for the support of vertical services in 5G networks,” _IEEE/ACM Transactions on Networking_ , vol. 27, no. 1, pp. 433–446, 2019.
  * [26] A. Alleg, T. Ahmed, M. Mosbah, R. Riggio, and R. Boutaba, “Delay-aware VNF placement and chaining based on a flexible resource allocation approach,” in _2017 13th International Conference on Network and Service Management (CNSM)_ , 2017, pp. 1–7.
  * [27] J. Cao, Y. Zhang, W. An, X. Chen, J. Sun, and Y. Han, “VNF-FG design and VNF placement for 5G mobile networks,” _Science China Information Sciences_ , vol. 60, no. 4, p. 040302, 2017.
  * [28] M. Dieye, S. Ahvar, J. Sahoo, E. Ahvar, R. Glitho, H. Elbiaze, and N. Crespi, “CPVNF: Cost-efficient proactive VNF placement and chaining for value-added services in content delivery networks,” _IEEE Transactions on Network and Service Management_ , vol. 15, no. 2, pp. 774–786, 2018.
  * [29] L. Tang, H. Yang, R. Ma, L. Hu, W. Wang, and Q. Chen, “Queue-aware dynamic placement of virtual network functions in 5G access network,” _IEEE Access_ , vol. 6, pp. 44 291–44 305, 2018.
  * [30] D. Li, P. Hong, K. Xue, and j. Pei, “Virtual network function placement considering resource optimization and SFC requests in cloud datacenter,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 29, no. 7, pp. 1664–1677, July 2018.
  * [31] X. Chen, W. Ni, I. B. Collings, X. Wang, and S. Xu, “Automated function placement and online optimization of network functions virtualization,” _IEEE Transactions on Communications_ , vol. 67, no. 2, pp. 1225–1237, Feb. 2019.
  * [32] C. Pham, N. H. Tran, S. Ren, W. Saad, and C. S. Hong, “Traffic-aware and energy-efficient VNF placement for service chaining: Joint sampling and matching approach,” _IEEE Transactions on Services Computing_ , pp. 1–1, 2017.
  * [33] M. A. T. Nejad, S. Parsaeefard, M. A. Maddah-Ali, T. Mahmoodi, and B. H. Khalaj, “vSPACE: VNF simultaneous placement, admission control and embedding,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 3, pp. 542–557, Mar. 2018.
  * [34] J. Chen, H. Liu, and H. Jia, “Cross-layer resource allocation in wireless-enabled NFV,” _IEEE Wireless Communications Letters_ , pp. 1–1, IEEE Early Access, Feb. 2020.
  * [35] L. Qu, C. Assi, and K. Shaban, “Delay-aware scheduling and resource optimization with network function virtualization,” _IEEE Transactions on Communications_ , vol. 64, no. 9, pp. 3746–3758, Sep. 2016.
  * [36] C. Pham, N. H. Tran, and C. S. Hong, “Virtual network function scheduling: A matching game approach,” _IEEE Communications Letters_ , vol. 22, no. 1, Jan. 2018.
  * [37] H. A. Alameddine, L. Qu, and C. Assi, “Scheduling service function chains for ultra-low latency network services,” in _2017 13th International Conference on Network and Service Management (CNSM)_ , 2017, pp. 1–9.
  * [38] H. Kim, “Performance evaluation of revised virtual resources allocation scheme in network function virtualization (NFV) networks,” _Cluster Computing_ , vol. 22, no. 1, pp. 2331–2339, 2019.
  * [39] S. Khan Tayyaba, H. A. Khattak, A. Almogren, M. A. Shah, I. Ud Din, I. Alkhalifa, and M. Guizani, “5G vehicular network resource management for improving radio access through machine learning,” _IEEE Access_ , vol. 8, pp. 6792–6800, 2020.
  * [40] J. Chen, J. Chen, R. Hu, and H. Zhang, “QMORA: A Q-Learning based multi-objective resource allocation scheme for NFV orchestration,” in _2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring)_ , 2020, pp. 1–6.
  * [41] J. Li, W. Shi, N. Zhang, and X. S. Shen, “Reinforcement learning based VNF scheduling with End-to-End delay guarantee,” in _2019 IEEE/CIC International Conference on Communications in China (ICCC)_ , 2019, pp. 572–577.
  * [42] S. Troia, R. Alvizu, and G. Maier, “Reinforcement learning for service function chain reconfiguration in NFV-SDN metro-core optical networks,” _IEEE Access_ , vol. 7, pp. 167 944–167 957, 2019.
  * [43] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1861–1870.
  * [44] N. Yu, Z. Song, H. Du, H. Huang, and X. Jia, “Dynamic resource provisioning for energy efficient cloud radio access networks,” _IEEE Transactions on Cloud Computing_ , vol. 7, no. 4, pp. 964–974, Oct. 2019.
  * [45] ETSI, GSNFV, “Network functions virtualisation (NFV); terminology for main concepts in NFV,” _ETSI GS NFV_ , vol. 2, no. 2, p. V1, Aug. 2018.
  * [46] ——, “Network functions virtualisation (NFV): Architectural framework,” _ETSI GS NFV_ , vol. 2, no. 2, p. V1, 2018.
  * [47] M. S. Yoon and A. E. Kamal, “NFV resource allocation using mixed queuing network model,” in _in Proc. IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2016, pp. Washington, DC, USA, 1–6. Dec. 2016.
  * [48] Z. Shu and T. Taleb, “A novel QoS framework for network slicing in 5G and beyond networks based on SDN and NFV,” _IEEE Network_ , vol. 34, no. 3, pp. 256–263, 2020.
  * [49] G. Sun, Y. Li, D. Liao, and V. Chang, “Service function chain orchestration across multiple domains: A full mesh aggregation approach,” _IEEE Transactions on Network and Service Management_ , vol. 15, no. 3, pp. 1175–1191, 2018.
  * [50] S. Van Rossem, W. Tavernier, B. Sonkoly, D. Colle, J. Czentye, M. Pickavet, and P. Demeester, “Deploying elastic routing capability in an SDN/NFV-enabled environment,” in _2015 IEEE Conference on Network Function Virtualization and Software Defined Network (NFV-SDN)_ , 2015, pp. 22–24.
  * [51] M. Dayarathna, Y. Wen, and R. Fan, “Data center energy consumption modeling: A survey,” _IEEE Communications Surveys Tutorials_ , vol. 18, no. 1, pp. 732–794, 2016.
  * [52] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1352–1361.
  * [53] C. Zhong, Z. Lu, M. C. Gursoy, and S. Velipasalar, “A deep actor-critic reinforcement learning framework for dynamic multichannel access,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 4, pp. 1125–1139, 2019.
  * [54] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [55] F. Fu, Y. Kang, Z. Zhang, F. R. Yu, and T. Wu, “Soft actor–critic DRL for live transcoding and streaming in vehicular Fog-computing-enabled IoV,” _IEEE Internet of Things Journal_ , vol. 8, no. 3, pp. 1308–1321, 2021.
  * [56] Z. Li and C. Guo, “Multi-agent deep reinforcement learning based spectrum allocation for D2D underlay communications,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 2, pp. 1828–1840, 2020.
  * [57] I. Grondman, L. Busoniu, G. A. Lopes, and R. Babuska, “A survey of actor-critic reinforcement learning: Standard and natural policy gradients,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 42, no. 6, pp. 1291–1307, 2012.
  * [58] C. J. Watkins and P. Dayan, “Q-learning,” _Machine Learning_ , vol. 8, no. 4, p. 279–292, 1992.
  * [59] J. Hu, H. Zhang, L. Song, R. Schober, and H. V. Poor, _IEEE Transactions on Communications_.
  * [60] M. Savi, M. Tornatore, and G. Verticale, “Impact of processing-resource sharing on the placement of chained virtual network functions,” _IEEE Transactions on Cloud Computing_ , 2019.
  * [61] S. Ebrahimi, A. Zakeri, B. Akbari, and N. Mokari, “Joint resource and admission management for slice-enabled networks,” in _NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium_. IEEE, 2020, pp. 1–7.
  * [62] Y. Xiao, Q. Zhang, F. Liu, J. Wang, M. Zhao, Z. Zhang, and J. Zhang, “Nfvdeep: Adaptive online service function chain deployment with deep reinforcement learning,” in _Proceedings of the International Symposium on Quality of Service_ , 2019, pp. 1–10.
  * [63] D. W. K. Ng and R. Schober, “Resource allocation and scheduling in multi-cell ofdma systems with decode-and-forward relaying,” _IEEE Transactions on Wireless Communications_ , vol. 10, no. 7, pp. 2246–2258, 2011.