# Collaborative Multi-BS Power Management for Dense Radio Access Network using
Deep Reinforcement Learning

Yuchao Chang, Wen Chen, , Jun Li, , Jianpo Liu, Haoran Wei, , Zhendong Wang,
and Naofal Al-Dhahir Y. Chang and W. Chen are with the Department of
Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
(e-mail<EMAIL_ADDRESS>wenchen@sjtu.edu.cn).J. Li is with the
School of Electronic and Optical Engineering, Nanjing University of Science
Technology, Nanjing 210094, China (email: jun.li@njust.edu.cn).J. Liu is with
Science and Technology on Microsystem Laboratory, Shanghai Institute of
Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai
201800, China (email: liujp@mail.sim.ac.cn).H. Wei , Z. Wang, and N. Al-Dhahir
are with Department of Electrical and Computer Engineering, The University of
Texas at Dallas, Richardson, TX 75080, USA (email<EMAIL_ADDRESS><EMAIL_ADDRESS>aldhahir@utdallas.edu).Manuscript received
November 02, 2022; revised February 28, 2023 and March 29,
2023.(_Corresponding author: Wen Chen._)

###### Abstract

Network energy efficiency is a main pillar in the design and operation of
wireless communication systems. In this paper, we investigate a dense radio
access network (dense-RAN) capable of radiated power management at the base
station (BS). Aiming to improve the long-term network energy efficiency, an
optimization problem is formulated by collaboratively managing multi-BSs’
radiated power levels with constraints on the users’ traffic volume and
achievable rate. Considering stochastic traffic arrivals at the users and
time-varying network interference, we first formulate the problem as a Markov
decision process (MDP) and then develop a novel deep reinforcement learning
(DRL) framework based on the cloud-RAN operation scheme. To tackle the trade-
off between complexity and performance, the overall optimization of multi-BSs’
energy efficiency with the multiplicative complexity constraint is modeled to
achieve near-optimal performance by using a deep Q-network (DQN). In DQN, each
BS first maximizes its individual energy efficiency, and then cooperates with
other BSs to maximize the overall multi-BSs’ energy efficiency. Simulation
results demonstrate that the proposed algorithm can converge faster and enjoy
a network energy efficiency improvement by 5% and 10% compared with the
benchmarks of the Q-learning and sleep schemes, respectively.

###### Index Terms:

Network energy efficiency, multi-BS power management, deep reinforcement
learning, deep Q-Network.

##  I Introduction

Wireless communications technology has progressed from the first generation
(1G) to the latest fifth generation (5G) and beyond to meet the increasing and
diversified traffic requirements [1, 2]. The anticipated more than 1000-fold
increasing requirements of wireless traffic and global recognition of green
communications pose significant challenges for 5G and beyond wireless
communication design [3]. There are three basic means to improve wireless
communication performance: (1) network densification, (2) more spectrum, and
(3) increasing the spectral efficiency [4]. Therefore, future wireless
communication standards are likely to deploy access points with higher
density, to use new spectral bands, and to maximize the spectral efficiency
[5, 6]. Since the power consumption of a single 5G base station (BS) is more
than 1.5 times that of a 4G BS, and the number of required 5G BSs is more than
4 times that of 4G networks per coverage area [7], the power consumption of 5G
networks per coverage area is at least 6 times that of 4G networks. Therefore,
reducing the power consumption of 5G networks is a critical task, which
implies that green communication with high energy-efficiency is a key design
goal [8, 9, 10]. Meanwhile, different from the conventional cell-centric
wireless communications, 5G and beyond networks are more likely to be user-
centric systems due to diversified user service demands, implying that future
wireless communications need to be more intelligent for effective management
[11]. Hence, 5G and beyond wireless communications embraces two key themes:
_green_ and _intelligence_.

Currently, the global carbon emissions have continued to rise for years,
resulting in more harsh global weather and severe air pollution. Meanwhile,
unlike the third generation (3G) and the fourth generation (4G) wireless
communications, 5G and beyond wireless communications are going to be faced
with huge amounts of mobile traffic that is generated by tens of billions
mobile terminals [2, 12]. Mao _et al_ in [7] pointed out that the information
and communications technology (ICT) industry accounts for nearly 20% of the
total electricity consumption and keeps an annual growth rate of more than 6%.
That paper also presented main considerations for green communications and
research on artificial intelligence-based (AI-based) green communications. In
2021, ICT’s share of global greenhouse gas (GHG) emissions was estimated to be
1.8-2.8% of the global GHG emissions [13]. Many researchers have shown the
possibility of improving energy efficiency from different perspectives. The
authors in [14] conducted an extensive survey of energy efficiency and
discussed many tradeoff techniques in green communications. The research in
[15] analyzed four basic network tradeoffs: spectrum efficiency versus energy
efficiency, deployment efficiency versus energy efficiency, delay versus
power, and bandwidth versus power. The survey on green mobile networking in
[16] discussed detailed modeling methods for power consumption and energy
efficiency. In [17], Li _et al_ clearly pointed out that over 80% of the power
consumption took place in the radio access networks (RAN), implying that the
sleeping scheme is a very useful strategy for improving network energy
efficiency [18, 19, 20, 21]. To minimize the network power consumption, Dai
_et al_ developed an energy-efficient design of downlink C-RAN based on the
low-power sleeping mode while considering user rate constraints [18]. A novel
sleeping mechanism for heterogeneous network (HetNets) was proposed to
decrease energy consumption of multiple BSs by considering traffic dynamics
[19]. Furthermore, some resource allocation schemes have been proposed to
maximize network energy efficiency by exploring the BS switching operations.
Qin _et al._ in [20] proposed a resource-on-demand energy scheduling strategy
to make effective utilization of harvested energy. A multi-objective auction-
based switching-off scheme in heterogeneous networks was developed to improve
energy and cost savings [22]. In [23], the work enhanced the energy efficiency
of HetNets by switching off a part of small cells in HetNets. The works in
[24, 25] utilize a neural network (NN) to optimize the transmit power aiming
at minimizing the network energy consumption. Multiple access technologies,
for example rate splitting multiple access (RSMA) [26] and non-orthogonal
multiple access (NOMA) [27], also play an important role in enhancing power
management and has achieved positive results in RAN. Although the above
contributions have improved energy efficiency of wireless communication, they
fail to consider accurate downlink power management.

In 5G and beyond communication systems, the network scale and complexity have
increased substantially, which makes traditional network optimization
strategies inefficient and ineffective. Thanks to the abilities of AI
technology in tackling large-scale and complex tasks, it can significantly
improve the efficiency of network optimization [28, 29, 30, 31]. Hence,
similar to _green_ , _intelligence_ is also a critical characteristic of 5G
and beyond wireless communications. Being a key approach in machine learning,
reinforcement learning (RL) is an environment-based strategy and is realized
through continuous interaction in discrete time steps between the agent and
environment. To maximize the accumulative reward, RL continuously learns how
to take actions automatically by interacting with the environment [32]. The
basic elements of RL are a finite set of states, a finite set of actions, and
the reward function. In each episode, the agent identifies the environment
state and then takes action based on the optimal policy. At the end of the
episode, the agent evaluates the action based on the reward function. RL has
been applied to a wide range of engineering applications such as healthcare
and autonomous control [33, 34, 35, 36]. For example, the actor-critic
technology has been adopted to conserve network energy in [17, 37]. Wang _et
al._ adopted deep Q-network (DQN) to identify the optimal policy in channel
access, where the dynamic channel access problem was modeled as a partially
observable Markov decision process (POMDP) [38]. To explore a representative
user’s computation offloading to multi-BSs, Chen _et al._ adopted DQN to
determine the optimal offloading strategy [39]. In a MIMO-based UAV network,
the authors in [40] investigated DQN to maximize its coverage efficiency
calculated based on the received signal strength. As a well-established
artificial neural network (ANN) architecture, the back propagation neural
network (BPNN) is ideal as a DQN core and commonly used learning algorithm for
complex function approximation [41]. For example, the authors in [42] proposed
a novel adaptive learning hybrid model for precise solar intensity forecasting
by using BPNN. A general BPNN is designed to consist of one input layer,
multiple hidden layers, and one output layer. Considering the complexity of
future wireless communication networks, the reinforcement learning techniques
have great potential to improve the communication performance by using both
the continuous environmental interaction and the NN’s function approximation.

In [43], energy efficiency (EE) was proposed to evaluate the network energy
efficiency of a wireless communication link. The energy efficiency is defined
as the ratio of data rate to consumed power. Hence, to improve the network
energy efficiency, this paper proposes a novel precise downlink power model by
using deep reinforcement learning (DRL) based on BPNN to control power
intelligently. The main contributions of this paper are summarized as follows:

  * •

In dense-RAN, we propose a novel adaptive multiple base stations (multi-BSs)
downlink power management model to improve network energy efficiency. In the
studied model, based on the unpredictable differentiated traffic demands of
users, the central processor simultaneously optimizes the downlink power of
multi-BSs while guaranteeing the throughput requirement.

  * •

To tackle the power management model, we design a DRL framework, where the
central processor serves as an agent. The downlink power levels and network
energy efficiency are the action space and reward function, respectively.

  * •

Considering the trade-off between complexity and performance, the network
energy efficiency optimization of the DRL framework is modeled to achieve
near-optimal precise power management, where the cumulative energy efficiency
is maximized by using a DQN with multiplicative complexity. Moreover, BPNN is
used to tune the considered DQN parameter.

  * •

Simulation results show that our proposed algorithm enjoys a network energy
efficiency improvement of 5% and 10% compared with that of the Q-learning and
sleep schemes, respectively.

This paper is structured as follows: Section II describes the system model,
where the radio communication model and problem formulation are described,
respectively. Section III presents the deep reinforcement learning framework,
in which we model the precise downlink power management problem as MDP, and
adopt DQN based on BPNN to identify the optimal policy for precisely managing
downlink power. Section IV demonstrates the superiority of the proposed
algorithm in terms of both communication performance and computational
complexity. Section V concludes our work. Moreover, for ease of reference,
Table I lists the main acronyms.

TABLE I: Summary of main acronyms Acronym | Meaning
---|---
1/2/3/4/5G | First/second/third/fourth/fifth generation
ANN | Artificial neural network
BPNN | Back propagation neural network
BS | Base station
EE | Energy efficiency
DRL | Deep reinforcement learning
DQN | Deep Q-network
dense-RAN | Dense radio access network
GHG | Global greenhouse gas
MDP | Markov decision process
POMDP | Partially observable markov decision process
RSRP | Reference Signal Receiving Power
SINR | Signal-to-interference-plus-noise-ratio
UMa | Urban Macro
TABLE II: Summary of main notation Notation | Description
---|---
$\varphi_{[b^{{}^{\prime}},t]}$ | Active or sleep status of BS $b^{{}^{\prime}}$
${\gamma_{\left[{bu,t}\right]}}$ | SINR of user $u$
$\lambda$ | Discount factor
$\pi\left({s,a}\right)$ | Transition probability
$h_{[b,t]}$ | Radiating signal from BS $b$ at time $t$
$l_{b}$, $l_{[u,t]}$ | Coordinate of BS $b$ and user $u$
$o_{b}$, $o_{[u,t]}$ | Height of BS $b$ and user $u$
$y_{[u,t]}$ | Y-coordinate for user u at time $t$
${\mathcal{A}}$ | Action set in RL
${\mathcal{B}}$ | Set of BSs
$C_{[bu,t]}$ | Data rate from BS $b$ to user $u$
$\overline{C}_{t}^{\Sigma}$ | Cumulative average throughput
${\mathcal{D}}$ | Replay memory
${\mathcal{D}}_{m}$ | Mini-batch memory
$\operatorname{D}\\!\left({{l_{b}},{l_{[u,t]}}}\right)$ | Distance between BS $b$ and user $u$
${G_{\left[{bu,t}\right]}}$ | Total discounted rewards in RL
$H_{[bu,t]}$ | Channel gain from BS $b$ to user $u$
$\operatorname{L}\left({{\mathbf{W}}_{\text{p}}},{{\mathbf{W}}_{\text{o}}}\right)$ | Loss function
${\overline{N}_{T}^{*,\Sigma}}$ | Overall average number of iterations
${\overline{N}_{t}^{*,\Sigma}}$ | Cumulative average number of iterations
${\mathcal{P}}$ | Power levels’ set
$P_{[b,t]}$ | Power level of BS $b$
$\overline{P}_{t}^{\Sigma}$ | Cumulative average power
$\Delta\overline{P}_{t}^{S,\Sigma}$ | Cumulative average RSRP decline
$\Delta\overline{P}_{t}^{I,\Sigma}$ | Cumulative average interference decline
$Q_{\left[{bu,t}\right]}^{\pi}\left({s,a}\right)$ | State-action-value function
$R_{t}$ | Network energy efficiency
$\overline{R}_{T}^{\Sigma}$ | Overall average EE
$\overline{R}_{t}^{\Sigma}$ | Cumulative average EE
${\mathcal{S}}$ | State set in RL
${\mathcal{T}}$ | Set of time-steps
${\mathcal{U}}$ | Set of users
${V_{[bu,t]}}$ | Traffic volume from BS $b$ to $u$
${\mathbf{W}}_{\text{o}}$ | DQN parameters of target
${\mathbf{W}}_{\text{P}}$ | DQN parameters of predicted
$Y_{[u,t]}$ | Received signal of user $u$ in time-step $t$
${\overline{Z}_{T}^{\Sigma}}$ | Overall average success ratio
${\overline{Z}_{t}^{\Sigma}}$ | Cumulative average success ratio

Figure 1: The DRL-RAN architecture deploying BSs and users.

##  II SYSTEM MODEL

The studied system model is analyzed in a dense-RAN urban scenario, where
downlink power management for all BSs is explored. Meanwhile, to reduce the
back-lobe interference of the BS sector antenna, each BS is designed to
include three sectors. In particular, each sector has its own antenna and
serves different users and is independently laid out in its related BS [44].
The studied dense-RAN architecture considers a geographical area, where the
set of BSs, denoted by ${\mathcal{B}}=\left\\{{1,\ldots,b,\ldots,B}\right\\}$,
are pre-deployed by the operator and a group of randomly deployed terrestrial
users, denoted by ${\mathcal{U}}=\left\\{{1,\ldots,u,\ldots,U}\right\\}$,
generate unpredictable differentiated service demands over time. The dense-RAN
scenario is illustrated in Fig. 1. Specifically, the system model includes a
cloud server for optimization control and the network scenario for deploying
BSs and users. The cloud server communicates with BSs through the optical
fiber, while the BS communicates with the user through the radio channel. For
example, the coordinates of the BS $b$ and the user $u$ in time-step $t$ are
given by

$\left\\{\begin{gathered}{l_{b}}=\left({{x_{b}},{y_{b}},{o_{b}}}\right),\qquad\qquad\qquad\
\ \forall b\in{\mathcal{B}};\hfill\\\
{l_{[u,t]}}=\left({{x_{[u,t]}},{y_{[u,t]}},{o_{[u,t]}}}\right),\qquad\forall
u\in{\mathcal{U}};\hfill\\\ \end{gathered}\right.$ (1)

where $x_{b}$, $y_{b}$, $x_{[u,t]}$, and $y_{[u,t]}$ are plane coordinates,
while $o_{b}$ and $o_{\left[{u,t}\right]}$ are the heights. Moreover, the
distance between BS $b$ and user $u$ in time-step $t$ is calculated using the
relation

$\begin{gathered}\operatorname{D}\\!\left({{l_{b}},{l_{[u,t]}}}\right)=\hfill\\\
\sqrt{{{\left({{x_{b}}-{x_{[u,t]}}}\right)}^{2}}+{{\left({{y_{b}}-{y_{[u,t]}}}\right)}^{2}}\\!+\\!{{\left({{o_{b}}-{o_{[u,t]}}}\right)}^{2}}},\hfill\\\
\end{gathered}\\!\\!$ (2)

where $\operatorname{D}\left({{l_{b}},{l_{[u,t]}}}\right)$ is a smooth
function related to location change of user $u$ over time.

In long-term energy-efficient wireless networks, the user traffic arrives
randomly and network interference fluctuates over time. Hence, we adopt the
time-step structure in the dense-RAN system, where the operating period is
discretized into the time-step space
${\mathcal{T}}=\left\\{{0,1,\ldots,T-1}\right\\}$, where every time-step is
regulated by 1 msec. In orthogonal frequency division multiple access (OFDMA),
the different frequency sub-carriers are orthogonal to each other. Hence, we
optimize transmit power allocations between BSs at the same frequency sub-
carriers. Power management is more conducive to improving network energy
efficiency by interference coordination of the same frequency sub-carriers.
Hence, we configure an OFDMA-based network scenario for the dense-RAN, where
transmit power allocations at the same frequency sub-carriers are optimized.
Considering that the network scheduling optimization is out of our research
scope, the associations between BSs and users are known in advance based on
the maximal Reference Signal Receiving Power (RSRP) access. This indicates
that one user can choose the BS with the maximal RSRP as its association BS
when it has many candidate BSs. For ease of reference, Table II lists the main
notations of the system model.

###  II-A Radio Communication Model

Considering our research on power management, we simplify the channel
propagation model and reduce the estimation overhead by using widebeams [45].
Thus, we adopt the link budget calculation to explore the downlink power
management optimization problem in the studied dense-RAN [46]. The link budget
is a summary of the transmit power along with all the link’s gains and losses,
which enables the strength of the received signal at the user to be
calculated. The strength of the received signal is measured by RSRP [47]. It
is possible to estimate whether RSRP at the user is sufficient, too high, or
too low, implying that the corrective transmit power at the BS is managed for
ensuring satisfactory system operation. When it is larger than the required
RSRP for the user, the high transmit power at the BS can be reduced to
minimize the power cost while still meeting the communication demand of its
associated user.

Assume that BS $b$ is serving user $u$ and any other BS
$b^{{}^{\prime}}(b^{{}^{\prime}}\in{\mathcal{B}},b^{{}^{\prime}}\neq b)$ is
the interference BS. Then, the received signal of user $u$ in time-step $t$,
including the serving RSRP and interference RSRP, can be expressed as follows

$\displaystyle{Y_{[u,t]}}$
$\displaystyle=\underbrace{{h_{[b,t]}}{s_{[u,t]}}}_{{\text{desired
signal}}}+\underbrace{\sum\limits_{{b^{{}^{\prime}}}\in{\mathcal{B}}\backslash
b}{{h_{{[b^{{}^{\prime}},t]}}}{s_{[u^{{}^{\prime}},t]}}}}_{{\text{inter-user
interference signal}}}+\underbrace{{n_{0}}}_{{\text{noise signal}}}\hfill$ (3)
$\displaystyle=\left({P_{[b,t]}}{H_{[bu,t]}}\right){s_{[u,t]}}\hfill$
$\displaystyle\qquad+\sum\limits_{{b^{{}^{\prime}}}\in{\mathcal{B}}\backslash
b}{\left({P_{{[b^{{}^{\prime}},t]}}}{H_{[{b^{{}^{\prime}}}u,t]}}\right){s_{[{u^{{}^{\prime}}},t]}}}+{n_{0}},\hfill$

where $h_{[b,t]}={P_{[b,t]}}{H_{[bu,t]}}$ and
$h_{[b^{{}^{\prime}},t]}={P_{[b^{{}^{\prime}},t]}}{H_{[b^{{}^{\prime}}u,t]}}$.
${P_{[b,t]}}$ and ${P_{[b^{{}^{\prime}},t]}}$
$\left(P_{[b,t]},P_{[b^{{}^{\prime}},t]}\in{\mathcal{P}}\right)$ are,
respectively, the downlink power levels of BS $b$ and $b^{{}^{\prime}}$, and
${\mathcal{P}}=\left\\{{{P_{0}},\ldots,{P_{\kappa-1}}}\right\\}$ is the set of
available downlink power levels for all BSs. In addition, $s_{[u,t]}$ and
$s_{[{u^{{}^{\prime}}},t]}$ are the received signals for user $u$ and
$u^{{}^{\prime}}$, respectively, and $n_{0}$ is the Gaussian white noise
signal at the user.

Moreover, ${H_{[bu,t]}}$ is the gain that accounts for the total transmitter
gain, path loss, and total receiver gain from BS $b$ to user $u$ in time-step
$t$, while ${H_{[b^{{}^{\prime}}u,t]}}$ is defined similar to ${H_{[bu,t]}}$
and is the channel gain from BS $b^{{}^{\prime}}$ to user $u$. In particular,
based on the UMa (Urban Macro) propagation model in the 3GPP TR 38.901
standard [48], the channel gain from BS $b$ to user $u$ in time-step $t$ is
given by [49]

$\displaystyle{H_{\left[{bu,t}\right]}}$
$\displaystyle={H_{TX}}\times{PL_{[bu,t]}}\times{H_{RX}}\hfill$ (4)
$\displaystyle={H_{TX}}\times\left({\frac{c}{{4\pi\times{f_{c}}\times\operatorname{D}\left({{l_{b}},{l_{\left[{u,t}\right]}}}\right)}}}\right)\times{H_{RX}},\hfill$

where $H_{TX}$ and $H_{RX}$ are, respectively, the signal gain constants at
the transmitter and receiver. $PL_{[bu,t]}$ is the path loss from BS $b$ to
user $u$, c is the speed of light, and $f_{c}$ is the center frequency.

Accordingly, based on the definition in (3), the signal-to-interference-plus-
noise-ratio (SINR) for user $u$ in time-step $t$ is given by

$\displaystyle{\gamma_{\left[{bu,t}\right]}}$
$\displaystyle=\frac{{{h_{[bu,t]}}}}{{\sum\limits_{{b^{{}^{\prime}}}\in{\mathcal{B}}\backslash
b}{\left({\varphi_{{[b^{{}^{\prime}},t]}}}{h_{[b^{{}^{\prime}}u,t]}}\right)}+{\sigma^{2}}}}$
(5)
$\displaystyle=\frac{{{P_{\left[{b,t}\right]}}{H_{\left[{bu,t}\right]}}}}{{\sum\limits_{{b^{{}^{\prime}}}\in{\mathcal{B}}\backslash
b}{\left({\varphi_{{[b^{{}^{\prime}},t]}}}{P_{\left[{{b^{{}^{\prime}}},t}\right]}}{H_{\left[{{b^{{}^{\prime}}}u,t}\right]}}\right)}+{\sigma^{2}}}},$

where $\sigma^{2}$ is the noise power. In time-step $t$,
$\varphi_{[b^{{}^{\prime}},t]}=1$ when BS $b^{{}^{\prime}}$ is active for
serving a user, and $\varphi_{[b^{{}^{\prime}},t]}=0$ when BS
$b^{{}^{\prime}}$ is sleep for serving no user. Then, the downlink data rate
from BS $b$ to user $u$ in time-step $t$ is given by

${C_{[bu,t]}}=W{\log_{2}}\left({1+{\gamma_{[bu,t]}}}\right),$ (6)

where $W$ is the channel bandwidth. We assume that the downlink data rate of
user $u$ is $C_{bu}^{\max}$ when BS $b$ transmits data with the standard
downlink power level $P_{\text{max}}$. The power decreases of BS $b$ and the
downlink data rate variation in time-step $t$ are, respectively, expressed as
follows

$\displaystyle\Delta{P_{[b,t]}}$
$\displaystyle={\varphi_{[b,t]}}\left({{P_{\max}}-{P_{[b,t]}}}\right),$ (7)
$\displaystyle\Delta{C_{[bu,t]}}$
$\displaystyle={\varphi_{[b,t]}}\left({{C_{[b,t]}^{\max}}-{C_{[b,t]}}}\right).$
(8)

where $\varphi_{[b,t]}$ is defined similar to $\varphi_{[b^{{}^{\prime}},t]}$
with BS $b$.

By dividing the downlink data rate in (6) by the downlink power, the network
energy efficiency is given in (9) in units of bits per seconds per dBW
(Mbps/dBW) [43].

${R_{t}}=\frac{1}{{\sum\limits_{b=1}^{B}{{\varphi_{\left[{b,t}\right]}}}}}\sum\limits_{b=1}^{B}{{R_{\left[{bu,t}\right]}}},$
(9)

where

${R_{\left[{bu,t}\right]}}={\frac{{{C_{\left[{bu,t}\right]}}}}{{{P_{\left[{b,t}\right]}}}}},$
(10)

and the power unit translation between ”dBW” and ”W” is denoted as

$P\left[{{\rm{in\
dBW}}}\right]=10\cdot{\log_{10}}\left({\frac{{P\left[{{\rm{in\
W}}}\right]}}{{1\left[{{\rm{in\ W}}}\right]}}}\right).$ (11)

Besides, when user $u$ requests a traffic volume ${V_{[bu,t]}}$ from its
associated BS $b$ in time-step $t$, the number of required time-steps starting
from time-step $t$, denoted by ${T_{[bu,t]}}$, satisfies the following
constraint

${V_{[bu,t]}}=\sum\limits_{{k=0}}^{{T_{[bu,t]}}-1}{C_{[bu,t+k]}},$ (12)

where $T_{[bu,t]}\leq T$, and it is similar to the relation between any other
BS $b^{{}^{\prime}}$ and its corresponding associated user.

###  II-B Problem Formulation

In dense-RAN, based on constraints of the traffic volume and downlink data
rate variation, the long-term energy efficiency maximization problem is
formulated as follows

$\begin{gathered}\mathop{\operatorname{maximize}}\sum\limits_{t}{{R_{t}}};\hfill\\\
s.t.\quad\sum\limits_{{k=0}}^{{T_{[bu,t]}}-1}{C_{[bu,t+k]}}={V_{[bu,t]}};\hfill\\\
\qquad\ \ \sum\limits_{b=1}^{B}{{\Delta C_{[{bu,t}]}}}\geq 0;\hfill\\\ \qquad\
\ {P_{[b,t]}}\in{\mathcal{P}};\hfill\\\ \qquad\ \ b\in{\mathcal{B}},\
u\in{\mathcal{U}},\ t\in{\mathcal{T}}.\hfill\\\ \end{gathered}$ (13)

where the power level ${P_{[b,t]}}$ for any BS $b(b\in{\mathcal{B}})$ in time-
step $t$ is the optimization variable, the network energy efficiency
$\sum\limits_{t}{{R_{t}}}$ is the optimization objective, and other
parameters, for example the association between BS and user
$\varphi_{[b^{{}^{\prime}},t]}$ and the power set ${\mathcal{P}}$, are the
scenario setting. The first constraint is the traffic volume determined by the
traffic volume demand for any active user $u(\forall u\in{\mathcal{U}})$ and
its serving BS $b$. The second constraint shows the network throughput limit,
which is calculated by accumulating downlink data rate variations for all BSs
in any time-step $t(\forall t\in{\mathcal{T}})$. The third constraint
illustrates that the available downlink power levels are identical for all
BSs. By combining formulas (5) and (6), the downlink data rate in the first
constraint of the optimization problem (13) is computed as follows

$\begin{gathered}C_{[bu,t+k]}=\hfill\\\
W{\log_{2}}\\!\\!\left(\\!\\!{1\\!+\\!{\frac{{{P_{\left[{b,t+k}\right]}}{H_{\left[{bu,t+k}\right]}}}}{{\sum\limits_{{b^{{}^{\prime}}}\in{\mathcal{B}}\backslash
b}\\!\\!\\!{\left({\varphi_{{[b^{{}^{\prime}},t+k]}}}{P_{\left[{{b^{{}^{\prime}}},t+k}\right]}}{H_{\left[{{b^{{}^{\prime}}}u,t+k}\right]}}\right)}\\!+\\!{\sigma^{2}}}}}}\right)\\!\\!.\hfill\\\
\end{gathered}$ (14)

##  III Deep Reinforcement Learning Framework

Wireless random traffic has made large-scale and complex network optimization
techniques move from the rule-based to AI-based. The RL’s learning process is
realized through discrete time steps, implying that RL is a better alternative
for the dense-RAN scenario. As an example RL model, DQN has been introduced to
predict the Q-value for tackling the continuous and large state space and
action space by using deep neural networks. In [50], DQN was firstly proposed
to train an agent to learn a policy from its observations by Google DeepMind
based on an ANN, which overcomes the limitation of the traditional look-up
table approach in Q-learning. The actual solution of this optimization problem
is to obtain the near-optimal collaborative multi-BS power management while
meeting the network performance requirements [51]. RL is a continuous learning
algorithm that optimizes control through continuous observations, which can
deal with the changing state of complex systems in real time and realize an
optimized control scheme. Moreover, we learn that the traffic volume possesses
Markov process characteristics from (12). Hence, it is feasible to formulate
the optimization problem in (13) as an MDP problem. Our goal is to maximize
the long-term network energy efficiency with user traffic volume demand while
guaranteeing no network throughput loss.

###  III-A Reinforcement Learning in a Nutshell

The system state is designed with the residue transmitted traffic volume and
the users’ RSRP. In particular, the state space is defined as follows

${\mathcal{S}}=\left\\{{{S_{t}}\left|\begin{gathered}{S_{t}}=\left\\{{{V_{\left[{bu,t}\right]}},{Y_{\left[{u,t}\right]}}}\right\\},\hfill\\\
b\in{\mathcal{B}},u\in{\mathcal{U}},t\in{\mathcal{T}}.\hfill\\\
\end{gathered}\right.}\right\\}.$ (15)

Each BS chooses one available downlink power level from the downlink power
space $\mathcal{P}$ to serve its associated user. Overall, the action space
based on $\mathcal{P}$ is denoted by

${\mathcal{A}}=\left\\{{{A_{t}}\left|\begin{gathered}{A_{t}}=\left\\{{{A_{\left[{bu,t}\right]}}}\right\\},{A_{\left[{bu,t}\right]}}\in{\mathcal{P}};\hfill\\\
b\in{\mathcal{B}},u\in{\mathcal{U}},t\in{\mathcal{T}}.\hfill\\\
\end{gathered}\right.}\right\\}.$ (16)

Power management not only results in a downlink power decline, but also causes
an uncertain downlink data rate variation. The energy efficiency serves as the
immediate reward in the reinforcement learning optimization process. In
addition to a maximum immediate reward at the current time-step, we also
optimize the long-term reward in terms of a cumulative energy efficiency. To
this end, the long-term reward function [32] is defined as the total
discounted rewards from time-step $t$, and is computed as follows

$\displaystyle{G_{\left[{bu,t}\right]}}$
$\displaystyle=\sum\limits_{k=0}^{\infty}{{\lambda^{k}}{R_{\left[{bu,t+k+1}\right]}}}\hfill$
(17)
$\displaystyle={R_{\left[{bu,t+1}\right]}}+\lambda{R_{\left[{bu,t+2}\right]}}+{\lambda^{2}}{R_{\left[{bu,t+3}\right]}}+\cdots\hfill$
$\displaystyle={R_{\left[{bu,t+1}\right]}}+\lambda\left({{R_{\left[{bu,t+2}\right]}}+{\lambda}{R_{\left[{bu,t+3}\right]}}+\cdots}\right)\hfill$
$\displaystyle={R_{\left[{bu,t+1}\right]}}+\lambda{G_{\left[{bu,t+1}\right]}}.\hfill$

where $\lambda\left({0<\lambda\leq 1}\right)$ is the discount factor.

Furthermore, the long-term reward is used to build the state-action-value
function, which estimates how good it is to be at state $s$ with action $a$
according to policy $\pi$. Hence, the state-action-value function is given by

$\\!\\!\\!\begin{gathered}Q_{\left[{bu,t}\right]}^{\pi}\left({s,a}\right)\hfill\\\
\doteq{{\mathbb{E}}_{\pi}}\left[{{G_{\left[{bu,t}\right]}}\left|\begin{gathered}{S_{\left[{bu,t}\right]}}=s,{A_{\left[{bu,t}\right]}}=a\hfill\\\
\end{gathered}\right.}\right]\hfill\\\
={{\mathbb{E}}_{\pi}}\left[{{R_{\left[{bu,t+1}\right]}}+\lambda{G_{\left[{bu,t+1}\right]}}\left|\begin{gathered}{S_{\left[{bu,t}\right]}}=s,{A_{\left[{bu,t}\right]}}=a\hfill\\\
\end{gathered}\right.}\right]\hfill\\\
=\\!\\!\\!\sum\limits_{s^{{}^{\prime}}}\\!{{P}\left(s^{{}^{\prime}}|{s,a}\right)\\!\\!\left[{{{R_{\left[{bu,t+1}\right]}}}+\lambda\mathop{\max}\limits_{{a^{{}^{\prime}}}}Q_{\left[{bu,t+1}\right]}^{\pi}\left({{s^{{}^{\prime}}},{a^{{}^{\prime}}}}\right)}\\!\right]}.\hfill\\\
\end{gathered}$ (18)

The expression in (18) is the so-called Bellman equation for state $s$, and
shows a relationship between the value of state $s$ and the values of its
successive states.

Figure 2: Workflow diagram of precise downlink power management Based on DQN.

The transition probability ${P}\left(s^{{}^{\prime}}|{s,a}\right)$ depends on
the optimization policy $\pi$ and the historical action selection in replay
memory $\mathcal{D}$ defined in DQN. Therefore, ${\pi}\left({s,a}\right)$,
representing the transition probability at state $s$ with action $a$ according
to the policy $\pi$, is calculated as follows

$\pi\left({s,a}\right)=\left\\{\begin{gathered}\left({1-\varepsilon}\right)\cdot\frac{{\left|{{{\mathcal{D}}^{a}}}\right|}}{{\left|{\mathcal{D}}\right|}},\qquad{1-\varepsilon};\hfill\\\
\varepsilon\cdot\frac{1}{{\left|{\mathcal{P}}\right|}},\qquad\qquad\quad\varepsilon;\hfill\\\
\end{gathered}\right.$ (19)

where ${\left|{\mathcal{D}}\right|}$ is the size of replay memory
$\mathcal{D}$. ${{\mathcal{D}}^{a}}$ is the sample space related to action $a$
that is the subset of $\mathcal{D}$, and its size is
${\left|{\mathcal{D}}^{a}\right|}$. ${\left|{\mathcal{P}}\right|}$ is the size
of the downlink power space $\mathcal{P}$.

In MDP, policy $\pi$ specifies the action for each state. To achieve a good
trade-off between exploration and exploitation, we adopts the following
$\varepsilon$-greedy policy

${A_{\left[{bu,t}\right]}}=\left\\{\begin{gathered}\mathop{\operatorname{argmax}}\limits_{{a^{{}^{\prime}}}}Q\left({{S_{\left[{bu,t}\right]}},{a^{{}^{\prime}}}}\right),\quad
1-\varepsilon;\hfill\\\
\operatorname{random}(A),\qquad\qquad\qquad\varepsilon.\hfill\\\
\end{gathered}\right.$ (20)

Hence, the long-term energy efficiency maximization in (13) is equivalent to
finding the optimal policy $\pi^{*}$ such that the energy efficiency over time
is maximized based on multiple constraints. Mathematically, the optimization
problem in (13) can be transformed into the following deep reinforcement
learning problem

$\begin{gathered}{\pi^{*}}:=\mathop{\arg\max}\limits_{\pi}\sum\limits_{b}{Q_{\left[{bu,t}\right]}^{\pi}\left({s,a}\right)};\hfill\\\
s.t.\quad\sum\limits_{{k=0}}^{{T_{[bu,t]}}-1}{C_{[mn,t+k]}}={V_{[bu,t]}};\hfill\\\
\qquad\ \ \sum\limits_{b=1}^{B}{{\Delta C_{\left[{bu,t}\right]}}}>=0;\hfill\\\
\qquad\ \ {P_{[b,t]}}\in{\mathcal{P}};\hfill\\\ \qquad\ \ b\in{\mathcal{B}},\
u\in{\mathcal{U}},\ t\in{\mathcal{T}}.\hfill\\\ \end{gathered}$ (21)

where the optimization objective function, that is expressed by the cumulative
energy efficiency in (13), is replaced by finding the optimal policy based on
the long-term reward in (18).

###  III-B Power Management Based on Deep-Q-Network

DQN improves Q-Learning with deep neural networks to make reinforcement
learning solve complex and nonlinear optimization problems. The typical DQN
framework is composed of the replay memory, mini-batch sample, target DQN,
predicted DQN, weight updating, and loss function. By refining Fig. 1 with the
RL framework, the workflow diagram of precise downlink power management based
on DQN is illustrated in Fig. 2. In DQN, the experience in each time-step is
stored into the replay memory $\mathcal{D}$ for being accessed to update the
DQN parameter. For example, after BS $b$ executes action
${A_{\left[{bu,t}\right]}}$ at state ${S_{\left[{bu,t}\right]}}$, it returns
reward ${R_{\left[{bu,t+1}\right]}}$ and transfers to the next state
${S_{\left[{bu,t+1}\right]}}$. Hence, the tuple
$\left({{S_{\left[{bu,t}\right]}},{A_{\left[{bu,t}\right]}},{R_{\left[{bu,t+1}\right]}},{S_{\left[{bu,t+1}\right]}}}\right)$
is stored into the replay memory $\mathcal{D}$ as follows

${\mathcal{D}}\leftarrow{\mathcal{D}}\cup\left\\{{\left({{S_{\left[{bu,t}\right]}},{A_{\left[{bu,t}\right]}},{R_{\left[{bu,t+1}\right]}},{S_{\left[{bu,t+1}\right]}}}\right)}\right\\}.$
(22)

At each update process, the DQN parameter is updated based on the mini-batch
samples memory ${\mathcal{D}}_{m}$ that comes from $\mathcal{D}$. The
predicted network inputs the current state-action pair
$\left({{S_{\left[{bu,t}\right]}},{A_{\left[{bu,t}\right]}}}\right)$ and
outputs the predicted value, i.e.,
${Q}\left({{S_{{}_{[bu,t]}}},{A_{{}_{[bu,t]}}};{{\mathbf{W}}_{\text{p}}}}\right)$.
The target network inputs the next state $s^{{}^{\prime}}$ and outputs the
maximum Q-value of the next state-action pair. Therefore, the target value of
$\left({{S_{\left[{bu,t}\right]}},{A_{\left[{bu,t}\right]}}}\right)$ is given
by

$\\!\\!\\!\\!\\!\begin{gathered}Q\left({{S_{\left[{bu,t}\right]}},{A_{\left[{bu,t}\right]}};{{\mathbf{W}}_{\text{o}}}}\right)={R_{\left[{bu,t+1}\right]}}+\lambda\mathop{\max}\limits_{{a^{{}^{\prime}}}}Q\left({{s^{{}^{\prime}}},{a^{{}^{\prime}}};{{\mathbf{W}}_{\text{o}}}}\right).\hfill\\\
\end{gathered}$ (23)

where $a^{{}^{\prime}}(a^{{}^{\prime}}\in{\mathcal{P}})$ is the candidate
action for the next state.

The loss function is used to measure whether the DQN parameter is optimized or
not. In particular, the DQN parameter is optimized when the loss function is
stable and tends to 0; and it is not optimized otherwise. Hence, based on the
mini-batch sample set ${\mathcal{D}}_{m}$, the loss function measures how good
is the predicted DQN as follows

$\begin{gathered}\operatorname{L}\left({{\mathbf{W}}_{\text{p}}},{{\mathbf{W}}_{\text{o}}}\right)\hfill\\\
=\frac{1}{{2\left|{{\mathcal{D}}_{m}}\right|}}\sum\limits_{k=1}^{\left|{{\mathcal{D}}_{m}}\right|}{{{\left[{Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{p}}}}\right)-Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{o}}}}\right)}\right]}^{2}}},\hfill\\\
\end{gathered}$ (24)

where $\left|{{\mathcal{D}}_{m}}\right|$ is the size of mini-batch sample
memory.

Algorithm 1 Downlink Power Management using DQN

Initialization:  $\mathcal{B}$, $\mathcal{U}$, $\mathcal{P}$, $\mathcal{T}$,
$\mathcal{D}$, ${\mathcal{D}}_{m}$;

for $t=1$ to $T$ ($T$ is the number of episodes) do

Obtain state
${S_{t}}=\left\\{{{V_{\left[{bu,t}\right]}},{X_{\left[{bu,t}\right]}}}\right\\},b\in{\mathcal{B}}$
and $\zeta_{t}=0$;

for $n=1$ to $N$ ($N$ is the maximum iteration) do

Initialization vectors: ${\mathcal{A}}_{n}=\emptyset$ and
${\mathcal{Q}}_{n}^{\pi}=\emptyset$;

for $b=1$ to $B$ do

if $1=={\varphi_{[b,t]}}$ then

Select action:
${A_{\left[{bu,t}\right]}}={P_{\left[{b,t}\right]}}\sim\pi\left({{S_{\left[{bu,t}\right]}}}\right)$;

Update action vector:
${\mathcal{A}}_{n}\leftarrow{\mathcal{A}}_{n}\cup{A_{\left[{bu,t}\right]}}$;

end if

end for

for $b=1$ to $B$ do

Calculate ${X_{[bu,t]}}$, $\Delta{C_{[bu,t]}}$, $R_{[bu,t]}$;

Update traffic volume: ${V_{[bu,t+1]}}\leftarrow{V_{[bu,t]}}-{C_{[bu,t]}}$;

Obtain the reward ${{R_{[bu,t+1]}}}$;

Transfer to state:
${S_{\left[{bu,t+1}\right]}}=\left\\{{{V_{\left[{bu,t+1}\right]}},{X_{\left[{bu,t+1}\right]}}}\right\\}$;

Calculate $Q_{\left[{bu,t}\right]}^{\pi}$ based on (18);

Update the value vector:
${\mathcal{Q}}_{n}^{\pi}\leftarrow{\mathcal{Q}}_{n}^{\pi}\cup{Q_{\left[{bu,t}\right]}^{\pi}}$;

end for

if $\sum\limits_{b=1}^{B}{\Delta{C_{[{bu,t}]}}}>=0$ then

$\zeta_{t}$=1;

end if

end for

Find the optimal tuple based on
${\pi^{*}}:=\mathop{\operatorname{argmax}}\limits_{\pi}{\mathcal{Q}}_{n}^{\pi}$;

if $\zeta_{t}==1$ then

Update $\mathcal{D}$ and record $n_{t}^{*}$ based on the optimal tuple;

end if

if $\left({\left|{\mathcal{D}}\right|>{\left|{{\mathcal{D}}_{m}}\right|}}\ \&\
0==\\!\\!\\!\\!\\!\mod(t,T_{m})\right)$ then

Randomly extract mini-batch samples ${{\mathcal{D}}_{m}}$ from
${\mathcal{D}}$;

for $k=1$ to ${\left|{{\mathcal{D}}_{m}}\right|}$ do

Obtain $Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{P}}}}\right)$ and
$Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{o}}}}\right)$;

end for

Calculate $\operatorname{L}\left({\mathbf{W}}_{\text{P}}\right)$and update
${\mathbf{W}}_{\text{P}}$;

After a fixed interval, update ${\mathbf{W}}_{\text{o}}$ as
${\mathbf{W}}_{\text{o}}={\mathbf{W}}_{\text{P}}$.

end if

end for

The structure of precise downlink power management based on DQN is described
in Algorithm 1, which includes two modules, named downlink power management
optimization and the DQN training process. In the downlink power management
optimization module, the optimal policy for each BS is learned to achieve a
satisfactory reward considering downlink data rate variations described from
line 3 to 22. During each optimization iteration, the vectors for storing the
selected actions and their respective state-action-values of BSs are
initialized in line 4. Each BS selects an action based on the
$\varepsilon-$greedy policy in line 7, and stores it into the action vector
${\mathcal{A}}_{n}$ in line 8. From lines 11 and 18, a series of computations
associated with the reward and state transition for each BS are performed. The
satisfactory conditions for the training iteration in terms of the cumulative
downlink data rate variation are defined from lines 19 to 21. Based on the
state-action-value, the optimal policy is identified to obtain its
corresponding tuple in line 23. The second module is described from line 27 to
34, wherein the DQN is trained by the transition pairs stored in replay memory
$\mathcal{D}$. In line 28, the replay memory $\mathcal{D}$ and records
$n_{t}^{*}$ are updated based on the optimal tuple. From lines 28 to 31, the
mini-batch samples ${{\mathcal{D}}_{m}}$ are randomly extracted from
$\mathcal{D}$ to calculate the predicted value and target value. In line 28,
the loss function is calculated to update ${\mathbf{W}}_{\text{p}}$ based on
(24), where $Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{P}}}}\right)$ and
$Q\left({{s_{k}},{a_{k}};{{\mathbf{W}}_{\text{o}}}}\right)$ are, respectively,
the predicted and target values of the $k^{th}$ mini-batch sample from
${{\mathcal{D}}_{m}}$. The gradient descent method is adopted to update
${{\mathbf{W}}_{\text{P}}}$ of the predicted network, and
${{\mathbf{W}}_{\text{o}}}$ is updated after a fixed interval in line 33.

##  IV NUMERICAL RESULTS

In this section, simulation results are presented to evaluate the performance
of the proposed algorithm. We adopt the sleep scheme in [19] as the benchmark,
and evaluate performance improvement of the precise power management
strategies that include the proposed algorithm and Q-learning. In particular,
we first show the superiority of the proposed algorithm in terms of network
energy efficiency over the Q-learning and sleep scheme. Next, the downlink
average throughput and power of all BSs are analyzed based on the network
energy efficiency definition in (9). Furthermore, we present the effect of
precise power management on the RSRP and interference reduction, which
determines the downlink network throughput variation of all the BSs. Finally,
the computation complexities of the proposed algorithm and Q-learning are
verified by the optimization success ratio and iterations.

Figure 3: Cumulative average energy efficiency statistics and overall average
energy efficiency statistics.

The basic simulation parameters are classified into the network topology
parameters, BS equipment parameters, and algorithm setting parameters. The
network topology parameters for the dense-RAN are the number of BSs $B=19$ and
the maximum number of users scheduled at a time-step $U=57$. Meanwhile,
hundreds and thousands of users are served by BSs in the multiple time-steps.
The BS equipment parameters are $P_{\text{max}}=$15.2dBW, $w_{0}=$-125dBW, and
$W=$10MHz; and the algorithm setting parameters are $T=20000$, $T_{m}=500$,
the reduced downlink power $\Delta P_{\max}=$2dBW or 5dBW, $\lambda=0.9$,
$\varepsilon=0.1$, $N=$100 or 200, $\left|{\mathcal{D}}\right|=5000$, and
$\left|{\mathcal{D}}_{m}\right|=1000$.

###  IV-A Energy Efficiency Analysis

In this subsection, we analyze the reward statistics of all algorithms based
on overall and cumulative average network energy efficiency, which are given
by

$\displaystyle\overline{R}_{T}^{\Sigma}=\frac{1}{T}\sum\limits_{t=0}^{T-1}{\left({\frac{1}{B}\sum\limits_{b=1}^{B}{{R_{\left[{bu,t}\right]}}}}\right)},$
(25)
$\displaystyle{\overline{R}_{t}^{\Sigma}}=\frac{1}{t+1}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{\left({\frac{1}{B}\sum\limits_{b=1}^{B}{{R_{\left[{bu,{t^{{}^{\prime}}}}\right]}}}}\right)}.$
(26)

The overall average network energy efficiency for the proposed algorithm,
Q-learning, and sleep scheme are depicted in Fig. 3-(1). First, the proposed
algorithm achieves the largest overall average energy efficiency for different
values of the tuple $(N,\Delta P_{\max})$ among all the algorithms. Second,
when $\Delta P_{max}$ increases, the overall average energy efficiency of the
proposed algorithm achieves better performance, while that of Q-learning
improves slightly. Third, the iteration has little effect on the overall
average energy efficiency of both the proposed algorithm and Q-learning. Based
on the overall average energy efficiency analysis, we set $(N,\Delta
P_{\text{max}})=$(100, 2dBW) and $(N,\Delta P_{\text{max}})=$(100, 5dBW) to
show the cumulative average energy efficiency of all the algorithms in Fig.
3-(2). It can be learned that the proposed algorithm outperforms Q-learning
based on the cumulative average energy efficiency, while Q-learning remains
superior to the sleep scheme. Moreover, we enlarge the curves as shown in the
small sub-image of Fig. 3-(2), which shows that the energy efficiency
fluctuates slightly over time due to a variable arrival rate. In particular,
the fluctuating curve of the sleep scheme demonstrates that the variable
request arrival rate leads to diversified BSs’ sleep over the time-step. Next,
we further analyze how the downlink power management affects the energy
efficiency by controlling the downlink power and data rate of BSs in Section
IV-B.

###  IV-B Effect of Downlink Power Management

Because the downlink power and data rate of BSs are, respectively, the
numerator and denominator of the energy efficiency, it is important to explore
how the downlink power and data rate of BSs vary with the increasing number of
episodes during the adaptive downlink power management process. Similar to
${\overline{R}}_{t}^{\Sigma}$, the cumulative average throughput of BSs is
expressed as follows

$\overline{C}_{t}^{\Sigma}=\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{\left({\frac{1}{B}\sum\limits_{b=1}^{B}{{C_{\left[{bu,{t^{{}^{\prime}}}}\right]}}}}\right)}.$
(27)

The average downlink power of BSs in time-step $t$ is given by

${\overline{P}_{t}}=10\cdot{\log_{10}}\left({\frac{1}{{B}}\sum\limits_{b=1}^{B}{{{10}^{\left({{{{P_{\left[{b,t}\right]}}}\mathord{\left/{\vphantom{{{P_{\left[{b,t}\right]}}}{10}}}\right.\kern-1.2pt}{10}}}\right)}}}}\right).$
(28)

Hence, the cumulative average power of BSs is given by

$\overline{P}_{t}^{\Sigma}=\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{{{\overline{P}}_{{t^{{}^{\prime}}}}}}.\hfill\\\
$ (29)

$\left\\{\begin{gathered}\Delta{\overline{P}_{t}^{S}}=10\cdot{\log_{10}}\left({\frac{1}{B}\sum\limits_{b=1}^{B}{{\varphi_{\left[{b,t}\right]}}\left({{{10}^{{}^{\left({{{{P_{\max}}}\mathord{\left/{\vphantom{{{P_{\max}}}{10}}}\right.\kern-1.2pt}{10}}}\right)}}}-{{10}^{\left({{{{P_{\left[{b,t}\right]}}}\mathord{\left/{\vphantom{{{P_{\left[{b,t}\right]}}}{10}}}\right.\kern-1.2pt}{10}}}\right)}}}\right)}}\right),\hfill\\\
\Delta\overline{P}_{t}^{I}=10\cdot{\log_{10}}\left({\frac{1}{B}\sum\limits_{b=1}^{B}{\sum\limits_{{b^{{}^{\prime}}}=1,{b^{{}^{\prime}}}\neq
b}^{B}{{\varphi_{\left[{{b^{{}^{\prime}}},t}\right]}}\left({{{10}^{{}^{\left({{{{P_{\max}}}\mathord{\left/{\vphantom{{{P_{\max}}}{10}}}\right.\kern-1.2pt}{10}}}\right)}}}-{{10}^{\left({{{{P_{\left[{{b^{{}^{\prime}}},t}\right]}}}\mathord{\left/{\vphantom{{{P_{\left[{{b^{{}^{\prime}}},t}\right]}}}{10}}}\right.\kern-1.2pt}{10}}}\right)}}}\right)}}}\right).\hfill\\\
\end{gathered}\right.$ (30)
Figure 4: Statistics for the network throughput and downlink power with
diverse $\Delta P_{max}$.

Fig. 4 depicts the variations of the cumulative average downlink transmit
power and data rate of BSs with increasing number of episodes for the settings
$(N,\Delta P_{\text{max}})=$(100, 2dBW) and $(N,\Delta P_{\text{max}})=$(100,
5dBW). Fig. 4-(1) and Fig. 4-(3) imply that the average optimized downlink
power of BSs has a slight effect on the average network throughput for the
proposed algorithm and Q-learning in comparison to that of the sleep scheme,
suggesting that it is feasible to optimize the downlink power without network
throughput loss. Fig. 4-(2) and Fig. 4-(4) show the average downlink power of
BSs for the proposed algorithm, Q-learning, and sleep scheme with increasing
number of episodes. It can be learned that our proposed algorithm achieves
better downlink power efficiency in comparison with Q-learning and sleep
scheme, indicating that the downlink power optimization is almost the sole
main contributor to the energy efficiency improvement. Meanwhile, we also
enlarge the fluctuating sleep scheme curve as shown in the sub-images of Fig.
4-(2) and -(4), indicating that the variable request arrival rate leads to
diversified BSs’ sleep over the time-step.

Figure 5: Statistics for the average RSRP and interference decline.

The average downlink powers of BSs suffer from decline while the average
throughputs of BSs hardly change, which inspires us to explore how downlink
power management affects the network interference of BSs. In time step $t$,
the average RSRP and interference decline of BSs in dBW are, respectively,
expressed in (30). Moreover, the cumulative average RSRP and interference
decline of BSs are, respectively, given by

$\left\\{\begin{gathered}\Delta\overline{P}_{t}^{S,\Sigma}=\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{\Delta\overline{P}_{{t^{{}^{\prime}}}}^{S}},\hfill\\\
\Delta\overline{P}_{t}^{I,\Sigma}=\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{\Delta\overline{P}_{{t^{{}^{\prime}}}}^{I}}.\hfill\\\
\end{gathered}\right.$ (31)

Fig. 5 depicts declining variations of RSRP and interference for all BSs with
increasing number of episodes for the settings $(N,\Delta
P_{\text{max}})=$(100, 2dBW) and $(N,\Delta P_{\text{max}})=$(100, 5dBW). Fig.
5-(1) and Fig. 5-(4) show that the average RSRP decline of all the BSs of the
proposed algorithm compared to the sleep scheme is much larger than that of
Q-learning, whose trends match that of the average downlink power in Fig.
4-(2) and Fig. 4-(4), respectively. From Fig. 5-(2) and -(5), it can be
learned that the average interference decline of all BSs of the proposed
algorithm compared to the sleep scheme is also much larger than that of
Q-learning. The enlarged curves of the sub-images in Fig. 5-(1), -(2), -(4)
and -(5) show that the optimization process can converge in less 200
iterations, implying that the optimization is fast enough to cope with the
necessary power adjustments. Moreover, the gaps between the average
interference decline and the average RSRP decline for the settings $(N,\Delta
P_{\text{max}})=$(100, 2dBW) and $(N,\Delta P_{\text{max}})=$(100, 5dBW) are,
respectively, shown in Fig. 5-(3) and Fig. 5-(6). These figures indicate that
better downlink power management of BSs can make the average interference
decline larger than the average RSRP decline, which helps improve the average
downlink networks throughput.

Figure 6: Decline of RSRP, interference, and data rate for active BSs.

However, the improvement of average downlink network throughput is less
prominent due to the much smaller gap.

For example, we randomly select one episode to analyze the RSRP decline,
interference decline, and downlink data rate decline of active BSs for the
setting $(N,\Delta P_{\text{max}})=$(100, 2dBW), which is illustrated in Fig.
6. For any active BS, it can be learned that the downlink data rate increases
when RSRP decline is smaller than interference decline, and that the downlink
data rate declines when RSRP decline is larger than interference decline.
However, the difference between RSRP decline and interference decline is a
nonlinear function with the downlink data rate decline due to (6). Based on
(30), the average RSRP decline and interference decline of all the BSs are
calculated as 1.43dBW and 1.04dBW, respectively. The average downlink data
rate decline of all BSs, denoted by
$\frac{1}{B}\sum\limits_{b=1}^{B}{\Delta{C_{\left[{bu,t}\right]}}}$, decreases
by 11.1Kb/s.

Fig. 5 and 6 show that power management of multi-BSs has different effect on
declines of RSRP and interference for various BSs. The RSRP decline for one BS
can simultaneously lead to the interference declines for its interference-
related BSs, while the BS also enjoys the cumulative interference reduction
from the RSRP decline of its interference-related BSs based on common sense of
interaction. Moreover, Fig. 3 and 4 demonstrate that the optimization
algorithm can converge quickly no matter how much BS reduces its downlink
power, and that the proposed algorithm has good advantages. We can learn that
collaborative multi-BSs power management for dense-RAN is to search the
optimal power reduction combination of multi-BSs while considering the
downlink data rate, and that DRL has advantages in handling the massive
states’ problem caused by multi-BSs through continuous learning. Therefore, it
represents the joint optimization of performance and power for artificial
intelligence (COPPAI) in dense-RAN.

The above analysis implies that the adaptive downlink power management
achieves the optimal downlink power for each BS. The optimal downlink power
generates satisfactory RSRP decline and the corresponding interference decline
at users, which weakens the loss of the average downlink network throughput.
Hence, the optimal downlink power management improves the energy efficiency
while guaranteeing the average downlink network throughput. Moreover, the
proposed algorithm achieves better communication performance in comparison
with Q-learning and the sleep scheme. This is why the proposed algorithm can
learn the dense-RAN state to more precisely manage the downlink power of BSs.

###  IV-C Discussion on Optimization Complexity

In the proposed algorithm, the main time consuming step is to search for the
optimal power value combination of all BSs, which requires $O(2^{B})$ floating
point operations (flops). To quantify the complexity of near-optimal solution,
Fig. 3-(1) shows that the two numbers of iterations, $N=100$ and $N=200$, have
comparable effect on the energy efficiency improvement, which inspires us to
further explore the computational complexity of both the proposed algorithm
and Q-learning. In this subsection, the computational complexity of the
proposed algorithm and Q-learning are analyzed in terms of the optimization
success ratio and the number of iterations. First, the overall and cumulative
average optimization success ratio are, respectively, defined as follows

$\left[{\begin{array}[]{*{20}{c}}{\overline{Z}_{T}^{\Sigma}}&{\overline{Z}_{t}^{\Sigma}}\end{array}}\right]=\left[{\begin{array}[]{*{20}{c}}{\frac{1}{T}\sum\limits_{t=0}^{T-1}{\zeta_{t}}}&{\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{\zeta_{t}}}\end{array}}\right],$
(32)

where $\zeta_{t}=1$ represents optimization success and $\zeta_{t}=0$ is
optimization failure. Then the overall and cumulative average number of
iterations are, respectively, defined as follows

$\\!\\!\left[{\begin{array}[]{*{20}{c}}{\overline{N}_{T}^{*,\Sigma}}&{\overline{N}_{t}^{*,\Sigma}}\end{array}}\right]=\left[{\begin{array}[]{*{20}{c}}{\frac{1}{T}\sum\limits_{t=0}^{T-1}{n_{t}^{*}}}&{\frac{1}{{t+1}}\sum\limits_{{t^{{}^{\prime}}}=0}^{t}{n_{t}^{*}}}\end{array}}\right],$
(33)

where ${n_{t}^{*}}$ is the optimal number of iterations in time-step $t$.

Figure 7: Statistics for the average optimization success ratio and
iterations.

In Fig. 7-(1) and Fig. 7-(2), the overall and cumulative average optimization
success ratio are, respectively, depicted with increasing number of episodes.
In particular, Fig. 7-(1) demonstrates that the overall average optimization
success ratio of the proposed algorithm is approaching 1 and is much better
than that of Q-learning. Fig. 7-(2) shows that the cumulative average
optimization success ratio of the proposed algorithm achieves rapid
convergence in comparison with that of Q-learning. Meanwhile, the cumulative
average optimization success ratio of the proposed algorithm remains stable
with increasing number of episodes, while it drops slightly for Q-learning.
Further, the curves in Fig. 7-(2) are enlarged in the sub-image to demonstrate
fast convergence for the necessary power adjustments. This is the reason that
the Q-table of Q-learning fails to fully learn the dense-RAN state.

In Fig. 7-(3) and Fig. 7-(4), the overall and cumulative average number of
iterations are, respectively, depicted with increasing number of episodes. It
can be learned that the overall and cumulative average number of iterations of
the proposed algorithm are much smaller than those of Q-learning. Hence, like
the communication performance, the computational complexity of the proposed
algorithm is lower than Q-learning. It is insufficient for the Q-table of
Q-learning to learn the dense-RAN state, which results in more explorations
for Q-learning in comparison with the proposed algorithm.

##  V CONCLUSION

In this paper, we propose a novel multi-BSs downlink power optimization
algorithm to improve the long-term energy efficiency in dense-RAN, which is
defined as the ratio of the downlink data rate by the downlink power. In the
studied model, the transmitting traffic volume, generated from many users with
unpredictable service demands, is integrated with the network throughput to
formulate the constraints. Based on the cloud-RAN operation scheme, we design
a deep reinforcement learning framework to tackle the optimization problem.
Considering the Markov characteristics of traffic volume, we transform the
maximization problem of long-term energy efficiency into the Bellman equation,
and adopt a DQN to optimize multi-BSs downlink power management. In the
studied DQN algorithm, BPNN is used to tune the DQN parameter to find the
mapping approximation from the dense-RAN state to the cumulative reward.
Simulation results show that the proposed algorithm demonstrates superiority
of communication performance and computational complexity over Q-learning and
the sleep scheme. Moreover, our numerical results demonstrate that precise
downlink power management lowers the downlink power and guarantees the network
throughput, which is key to improve the energy efficiency.

## Acknowledgments

The work of N. Al-Dhahir was supported by Erik Jonsson Distinguished
Professorship at UT-Dallas

## References

  * [1] D. He, B. Ai, K. Guan, L. Wang, Z. Zhong, and T. Kürner, “The Design and Applications of High-Performance Ray-Tracing Simulation Platform for 5G and Beyond Wireless Communications: A Tutorial,” _IEEE Commun. Surveys Tuts._ , vol. 21, no. 1, pp. 10–27, 2019.
  * [2] Y. Xu, F. Yin, W. Xu, J. Lin, and S. Cui, “Wireless Traffic Prediction With Scalable Gaussian Process: Framework, Algorithms, and Verification,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 6, pp. 1291–1306, 2019.
  * [3] N. Saxena, A. Roy, and H. Kim, “Traffic-Aware Cloud RAN: A Key for Green 5G Networks,” _IEEE J. Sel. Areas Commun._ , vol. 34, no. 4, pp. 1010–1021, 2016.
  * [4] T. L. Marzetta, E. G. Larsson, H. Yang, and H. Q. Ngo, _Fundamentals of Massive MIMO_. Cambridge University Press, 2016.
  * [5] W.-S. Liao, M. G. Kibria, G. P. Villardi, O. Zhao, K. Ishizu, and F. Kojima, “Coordinated Multi-Point Downlink Transmission for Dense Small Cell Networks,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 1, pp. 431–441, 2019\.
  * [6] H.-J. Song and T. Nagatsuma, “Present and Future of Terahertz Communications,” _IEEE Trans. Terahertz Sci. Technol._ , vol. 1, no. 1, pp. 256–263, 2011.
  * [7] B. Mao, F. Tang, Y. Kawamoto, and N. Kato, “AI models for green communications towards 6G,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 1, pp. 210–247, 2021.
  * [8] Z. Hasan, H. Boostanimehr, and V. K. Bhargava, “Green Cellular Networks: A Survey, Some Research Issues and Challenges,” _IEEE Commun. Surveys Tuts._ , vol. 13, no. 4, pp. 524–540, 2011.
  * [9] Y. Chang, H. Tang, Y. Cheng, Q. Zhao, and B. L. a. Yuan, “Dynamic Hierarchical Energy-Efficient Method Based on Combinatorial Optimization for Wireless Sensor Networks,” _Sensors_ , vol. 17, no. 7, 2017.
  * [10] K. Wang, Q. Wu, W. Chen, Y. Yang, and D. W. K. Ng, “Energy-Efficient Buffer-Aided Relaying Systems With Opportunistic Spectrum Access,” _IEEE Trans. Green Commun. Netw._ , vol. 4, no. 3, pp. 731–744, 2020.
  * [11] D. C. Nguyen, P. Cheng, M. Ding, D. Lopez-Perez, P. N. Pathirana, J. Li, A. Seneviratne, Y. Li, and H. V. Poor, “Enabling AI in Future Wireless Networks: A Data Life Cycle Perspective,” _IEEE Commun. Surveys Tuts._ , vol. 23, no. 1, pp. 553–595, 2021.
  * [12] Y. Chang, X. Yuan, B. Li, D. Niyato, and N. Al-Dhahir, “Machine-learning-based parallel genetic algorithms for multi-objective optimization in ultra-reliable low-latency wsns,” _IEEE Access_ , vol. 7, pp. 4913–4926, 2019.
  * [13] C. Freitag, M. Berners-Lee, K. Widdicks, B. Knowles, G. Blair, and A. Friday, “The climate impact of ICT: A review of estimates, trends and regulations,” 2021.
  * [14] R. Mahapatra, Y. Nijsure, G. Kaddoum, N. U. Hassan, and C. Yuen, “Energy efficiency tradeoff mechanism towards wireless green communication: A survey,” _IEEE Commun. Surveys Tuts._ , vol. 18, no. 1, pp. 686–705, 2015\.
  * [15] S. Zhang, Q. Wu, S. Xu, and G. Y. Li, “Fundamental Green Tradeoffs: Progresses, Challenges, and Impacts on 5G Networks,” _IEEE Commun. Surveys Tuts._ , vol. 19, no. 1, pp. 33–56, 2017.
  * [16] M. Ismail, W. Zhuang, E. Serpedin, and K. Qaraqe, “A survey on green mobile networking: From the perspectives of network operators and mobile users,” _IEEE Commun. Surveys Tuts._ , vol. 17, no. 3, pp. 1535–1556, 2014.
  * [17] R. Li, Z. Zhao, X. Chen, J. Palicot, and H. Zhang, “TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks,” _IEEE Trans. Wireless Commun._ , vol. 13, no. 4, pp. 2000–2011, 2014.
  * [18] B. Dai and W. Yu, “Energy Efficiency of Downlink Transmission Strategies for Cloud Radio Access Networks,” _IEEE J. Sel. Areas Commun._ , vol. 34, no. 4, pp. 1037–1050, 2016.
  * [19] R. Tao, W. Liu, X. Chu, and J. Zhang, “An Energy Saving Small Cell Sleeping Mechanism With Cell Range Expansion in Heterogeneous Networks,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 5, pp. 2451–2463, 2019.
  * [20] M. Qin, W. Wu, Q. Yang, R. Zhang, N. Cheng, H. Zhou, R. R. Rao, and X. Shen, “Green-Oriented Dynamic Resource-on-Demand Strategy for Multi-RAT Wireless Networks Powered by Heterogeneous Energy Sources,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp. 5547–5560, 2020.
  * [21] R. Shinkuma, N. Kishi, K. Ota, M. Dong, T. Sato, and E. Oki, “Smarter Base Station Sleeping for Greener Cellular Networks,” _IEEE Network_ , vol. 35, no. 6, pp. 98–103, 2021.
  * [22] A. Bousia, E. Kartsakli, A. Antonopoulos, L. Alonso, and C. Verikoukis, “Multiobjective Auction-Based Switching-Off Scheme in Heterogeneous Networks: To Bid or Not to Bid?” _IEEE Trans. Veh. Technol._ , vol. 65, no. 11, pp. 9168–9180, 2016.
  * [23] M. Oikonomakou, A. Antonopoulos, L. Alonso, and C. Verikoukis, “Evaluating Cost Allocation Imposed by Cooperative Switching Off in Multioperator Shared HetNets,” _IEEE Trans. Veh. Technol._ , vol. 66, no. 12, pp. 11 352–11 365, 2017.
  * [24] B. Matthiesen, A. Zappone, K.-L. Besser, E. A. Jorswieck, and M. Debbah, “A globally optimal energy-efficient power control framework and its efficient implementation in wireless interference networks,” _IEEE Trans. Signal Process._ , vol. 68, pp. 3887–3902, 2020.
  * [25] R. Dong, C. She, W. Hardjawana, Y. Li, and B. Vucetic, “Deep learning for radio resource allocation with diverse quality-of-service requirements in 5G,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 4, pp. 2309–2324, 2020\.
  * [26] Z. Yang, M. Chen, W. Saad, and M. Shikh-Bahaei, “Optimization of Rate Allocation and Power Control for Rate Splitting Multiple Access (RSMA),” _IEEE Trans. Commun._ , vol. 69, no. 9, pp. 5988–6002, 2021.
  * [27] Z. Ding, X. Lei, G. K. Karagiannidis, R. Schober, J. Yuan, and V. K. Bhargava, “A survey on non-orthogonal multiple access for 5G networks: Research challenges and future trends,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 10, pp. 2181–2195, 2017.
  * [28] Y. Shi, K. Yang, T. Jiang, J. Zhang, and K. B. Letaief, “Communication-efficient edge AI: Algorithms and systems,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 4, pp. 2167–2191, 2020.
  * [29] M. Bennis, S. M. Perlaza, P. Blasco, Z. Han, and H. V. Poor, “Self-Organization in Small Cell Networks: A Reinforcement Learning Approach,” _IEEE Trans. Wireless Commun._ , vol. 12, no. 7, pp. 3202–3212, 2013.
  * [30] Y. Chang, X. Yuan, B. Li, D. Niyato, and N. Al-Dhahir, “A Joint Unsupervised Learning and Genetic Algorithm Approach for Topology Control in Energy-Efficient Ultra-Dense Wireless Sensor Networks,” _IEEE Commun. Lett._ , vol. 22, no. 11, pp. 2370–2373, 2018.
  * [31] A. Feriani and E. Hossain, “Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial,” _IEEE Commun. Surveys Tuts._ , vol. 23, no. 2, pp. 1226–1252, 2021.
  * [32] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. Massachusetts London, England: The MIT Press Cambridge, 2018.
  * [33] M. Mahmud, M. S. Kaiser, A. Hussain, and S. Vassanelli, “Applications of Deep Learning and Reinforcement Learning to Biological Data,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 29, no. 6, pp. 2063–2079, 2018.
  * [34] P. Liu, Y. Chang, Z. Deng, Y. Zhang, and L. Wang, “Research on Tourist City Traffic Signal Intelligent Control Method,” in _Engineering Solutions for Manufacturing Processes_ , ser. Advanced Materials Research, vol. 655\. Trans Tech Publications Ltd, 3 2013, pp. 1383–1389.
  * [35] Z. Wang, H. Wei, J. Wang, X. Zeng, and Y. Chang, “Security Issues and Solutions for Connected and Autonomous Vehicles in a Sustainable City: A Survey,” _Sustainability_ , vol. 14, no. 19, p. 12409, 2022.
  * [36] B. Kiumarsi, K. G. Vamvoudakis, H. Modares, and F. L. Lewis, “Optimal and Autonomous Control Using Reinforcement Learning: A Survey,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 29, no. 6, pp. 2042–2062, 2018.
  * [37] Y. Wei, F. R. Yu, M. Song, and Z. Han, “User scheduling and resource allocation in HetNets with hybrid energy supply: An actor-critic reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 17, no. 1, pp. 680–692, 2017.
  * [38] S. Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 4, no. 2, pp. 257–265, 2018.
  * [39] X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Optimized Computation Offloading Performance in Virtual Edge Computing Systems Via Deep Reinforcement Learning,” _IEEE Internet Things J._ , vol. 6, no. 3, pp. 4005–4018, 2019.
  * [40] H. Huang, Y. Yang, H. Wang, Z. Ding, H. Sari, and F. Adachi, “Deep Reinforcement Learning for UAV Navigation Through Massive MIMO Technique,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 1, pp. 1117–1121, 2020.
  * [41] N. A. Al-Sammarraie, Y. M. H. Al-Mayali, and Y. A. Baker El-Ebiary, “Classification and diagnosis using back propagation Artificial Neural Networks (ANN),” in _2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE)_ , 2018, pp. 1–5.
  * [42] Y. Wang, Y. Shen, S. Mao, G. Cao, and R. M. Nelms, “Adaptive Learning Hybrid Model for Solar Intensity Forecasting,” _IEEE Trans. Ind. Informat._ , vol. 14, no. 4, pp. 1635–1645, 2018.
  * [43] J. N. Murdock and T. S. Rappaport, “Consumption factor: A figure of merit for power consumption and energy efficiency in broadband wireless communications,” in _Proc. IEEE Globecom Workshops_ , 2011, pp. 1393–1398.
  * [44] R. Yamaguchi, Y. Ohta, H. Omote, Y. Sugita, and T. Fujii, “Throughput improvement by combined tilting of base station antenna,” in _Proc. EUCAP_ , Apr. 2013, pp. 871–873.
  * [45] H. Lee, S. Kim, and J. Choi, “Efficient Channel AoD/AoA Estimation Using Widebeams for Millimeter Wave MIMO Systems,” in _2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_ , 2019, pp. 1–5.
  * [46] A. V. Lopez, A. Chervyakov, G. Chance, S. Verma, and Y. Tang, “Opportunities and Challenges of mmWave NR,” _IEEE Wireless Commun._ , vol. 26, no. 2, pp. 4–6, 2019.
  * [47] R. Q. Hu and Y. Qian, _Connected-Mode Mobility in LTE Heterogeneous Networks_ , 2013, pp. 199–214.
  * [48] G. R. A. N. W. Group _et al._ , “Study on channel model for frequencies from 0.5 to 100 GHz (Release 15),” 3GPP TR 38.901, Tech. Rep., 2018.
  * [49] R. Nur Esa, A. Hikmaturokhman, and A. Rizal Danisya, “5G NR Planning at Frequency 3.5 GHz : Study Case in Indonesia Industrial Area,” in _2020 2nd International Conference on Industrial Electrical and Electronics (ICIEE)_ , 2020, pp. 187–193.
  * [50] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” _CoRR_ , vol. abs/1312.5602, 2013. [Online]. Available: http://arxiv.org/abs/1312.5602
  * [51] M. Abdelhakim, Y. Liang, and T. Li, “Mobile Coordinated Wireless Sensor Network: An Energy Efficient Scheme for Real-Time Transmissions,” _IEEE J. Sel. Areas Commun._ , vol. 34, no. 5, pp. 1663–1675, 2016.