# Cooperative Hierarchical Deep Reinforcement Learning based Joint Sleep,
Power, and RIS Control for Energy-Efficient HetNet

Hao Zhou, Medhat Elsayed, Majid Bavand, Raimundas Gaigalas, Steve Furr,
Melike Erol-Kantarci  This work has been supported by MITACS and Ericsson
Canada, and NSERC Collaborative Research and Training Experience Program
(CREATE) under Grant 497981. H. Zhou and M. Erol-Kantarci are with the School
of Electrical Engineering and Computer Science, University of Ottawa, Ottawa,
ON K1N 6N5, Canada. (email:{hzhou098<EMAIL_ADDRESS>M.
Elsayed and M. Bavand, are with Ericsson Canada, ON K2K 2V6, Ontario, Canada.
<EMAIL_ADDRESS>S. Furr is with Amazon,
USA. The work was done while he was at Ericsson Canada. (email:
<EMAIL_ADDRESS>R. Gaigalas is with Ericsson Sweden, 164 80, Stockholm
County, Sweden<EMAIL_ADDRESS>
###### Abstract

Energy efficiency (EE) is one of the most important metrics for 5G and future
6G networks to reduce energy costs and control carbon footprint. Sleep
control, as a cost-efficient approach, can significantly lower power
consumption by switching off network devices selectively. Meanwhile,
reconfigurable intelligent surface (RIS) has emerged as a promising technique
to enhance the EE of 5G beyond and 6G networks. In this work, we jointly
consider sleep and transmission power control for reconfigurable intelligent
surface (RIS)-aided energy-efficient heterogeneous networks (Hetnets). In
particular, we first propose a fractional programming (FP) method for RIS
phase-shift control, which aims to maximize the sum-rate under given
transmission power levels. Then, considering the timescale difference between
sleep control and power control, we introduce a cooperative hierarchical deep
reinforcement learning (Co-HDRL) algorithm, including a cross-entropy enabled
meta-controller for sleep control, and correlated equilibrium-based sub-
controllers for power control. Moreover, we proposed a surrogate optimization
method as one baseline for RIS control, and conventional HDRL as another
baseline for sleep and power control. Finally, simulations show that the RIS-
assisted sleep control can achieve more than 16% lower energy consumption and
30% higher energy efficiency than baseline algorithms.

###### Index Terms:

Reconfigurable Intelligent Surfaces, Hierarchical Deep Reinforcement Learning,
energy efficiency.

##  I Introduction

Energy efficiency (EE) is a critical metric for sustainable 5G and 6G
networks. Sleep control has been one of the widely considered approaches to
enhance the EE of radio access network (RAN)[1]. Sleep control turns network
devices such as base stations (BS) to sleep mode selectively [2]. On the other
hand, reconfigurable intelligent surface (RIS) has emerged as a promising
technique for amplifying signals in 5G beyond and 6G networks. In particular,
a huge number of small and low power consumption elements are integrated into
RISs where each is equipped with reflect-arrays to amplify and forward the
incoming signal. Since no power amplifier is needed, RISs consume much less
energy than conventional relay transceivers. Therefore, RISs can become an
ideal solution to improve the EE of RAN [3]. However, these potential
solutions for energy-efficient RAN may greatly increase the network management
complexity as multiple network functions should be jointly considered to adapt
to network dynamics. Fortunately, machine learning techniques offer promising
opportunities for intelligent network management and control [4]. The
advantage of machine learning algorithms has been extensively studied in many
works [5].

In this work, we jointly consider sleep control, transmission power control
and RIS control to improve EE performance. Note that the BS transmission power
control is involved because: i) if one BS enters the sleep mode, the rest
active BSs may need to increase the transmission power to serve the user
equipment (UE) that is previously associated with the BS that has switched to
sleep mode; ii) BS transmit power and RIS phase shifts are usually jointly
optimized, which is known as joint active and passive beamforming[6]; iii)
power control can reduce the overall interference level, increase the average
Signal-to-interference-plus-noise ratio (SINR), and further improve the EE[7].
However, compared with  transmit power and RIS control, sleep control is a
long-term decision that affects the network performance for several
consecutive time slots. By contrast, power control applies faster decision-
making to adapt to the changing traffic demand of UEs. Therefore, such
timescale obstacles in decision-making prevent the application of conventional
reinforcement learning algorithms [8].

To this end, we deploy a hierarchical architecture for joint sleep and power
control, including a meta-controller for long-term sleep control and several
sub-controllers for short-term power control. The sleep control is decided
every few time slots as a high-level policy for sub-controllers, and sub-
controllers can adjust the transmission power in each time slot accordingly.
HRL has been used for joint relay selection and power optimization in [9] and
multichannel sensing in [10]. But in the existing schemes, the meta-controller
uses the average reward of sub-controllers as feedback to assess the high-
level policy. Nevertheless, the low-level action selection is constantly
changing, and the average reward metric may have difficulty representing the
sub-controller status. Learning multiple levels of policies simultaneously can
be difficult due to the inherent instability. Specifically, changes in a
policy at one level of the hierarchy may lead to changes in the policies of
higher levels, making it difficult to jointly learn multiple levels of
policies. For instance, a high average reward may be produced by very unstable
sub-controllers that cannot guarantee a satisfying performance. Such unstable
low-level policies can mislead the decision-making of higher-level
controllers, which has become a critical issue for HRL applications [11].
Consequently, we propose a novel cooperative hierarchical deep reinforcement
learning (Co-HDRL) algorithm. For high-level meta-controller, we apply a
cross-entropy enabled policy to monitor the stability of sub-controllers,
which aims at evaluating the reliability of low-level reward feedback. For
low-level sub-controllers, we introduce a correlated equilibrium-based
cooperation strategy to stabilize action selections. Therefore, Co-HDRL is
expected to produce a stable performance under a hierarchical scheme while
allowing a long-term RAN policy to interact with short-term optimizations of
network functions.

On the other hand, compared with sleep and power control, RIS phase control
has more stringent requirements for fast and real-time optimization [1]. Each
RIS element should respond to the incoming signal immediately, and the channel
condition such as phase difference may be randomly changed in the next time
slot, which requires an optimization algorithm with a fast response. For
example, [6] applies an alternating optimization algorithm for active and
passive beamforming in RIS-assisted networks, and [12] utilizes fractional
programming (FP) techniques to maximize the weighted sum-rate. To better
combine sleep and power control with RIS control, we propose an FP-based
algorithm to maximize the sum-rate of all UEs[13]. We first apply the
quadratic transformation to tackle the multiple-ratio FP terms, and then
transform the optimization into a semi-definite programming problem by using
the well-known Schur complement.

The main contributions of this work are summarized as follows:
1) We provide a novel hierarchical, intelligent scheme by combining convex
optimization-based FP with machine learning-enabled Co-HDRL to improve the EE
of heterogeneous networks (HetNets). The Co-HDRL decides the sleep control and
BS transmission power level, and then the transmission power is sent to FP for
RIS phase-shift optimization.

2) An FP-based RIS phase-shift control method is introduced for multi-BS and
multi-RIS HetNet scenarios. It applies an iterative optimization method to
maximize the total transmission rate under given power levels. Moreover, we
design a surrogate optimization-based baseline algorithm for RIS control,
which yields an approximation for the objective function by using surrogate
models.

3) We propose a Co-HDRL algorithm for the joint sleep and power control of
HetNets, which is capable of handling problems with different timescale
actions. More specifically, a cross-entropy enabled policy is applied for
high-level meta-controller to evaluate the low-level stability, and correlated
equilibrium is deployed for low-level sub-controllers to stabilize the joint
action selections. In addition, such joint control capabilities of handling
timescale differences are critical to many existing platforms such as O-RAN
and associated control loops; i.e. non-RT (with decision horizon in hours) and
near-RT RIC (with decision horizon in seconds or milliseconds)[14].

Finally, our simulations show that FP and Co-HDRL can achieve 20% higher
throughput, 16% lower power consumption, and 35% higher energy efficiency than
other baseline algorithms. Moreover, Co-HDRL presents more stable action
selections and higher rewards than conventional HDRL algorithms.

The rest of this work is organized as follows. Section II presents the related
work, and Section III gives the network and system model. The FP-based RIS
control algorithm is shown in section IV, section V introduces the proposed
Co-HDRL, and baseline algorithms are given in section VI. Finally, section VII
shows the simulation results, and section VIII concludes this work.

##  II Related Work

Machine learning techniques, including Q-learning[15], deep Q-learning[16,
17], transfer learning[18], actor-critic learning[19] have been widely used
for wireless network management. For example, Q-learning and deep Q-learning
are used in [17] for energy sharing of renewable energy powered base stations
to reduce energy footprint. [19] combines actor-critic reinforcement learning
with spatial-temporal networks for traffic-aware BS sleep control. [20]
defined a deep Q-learning-based scheme with action-wise experience replay and
adaptive reward scaling for sleep control. However, it is worth noting that
most existing works require actions with the same timescales, which means all
the actions have to be decided simultaneously. But in practice, the actions
may belong to different agents that apply various decision-making timescales.

Consequently, HRL is proposed to overcome this issue [21]. [9] proposed an
HRL-based method for joint relay selection and power optimization in two-hop
cooperative relay networks to reduce outage probability. In [10], HRL is used
for dynamic spectrum access control, which aims to reduce the complexity of
the whole optimization process. In these works, the meta-controller uses the
average reward of sub-controllers to evaluate the high-level policy. But the
low-level stability issue is not considered since the low-level action
selection is constantly changing[11, 22]. Then the unreliable reward feedback
from sub-controllers may mislead the policy selection of meta-controllers and
degrade the overall performance. In our former work [23], we provide a
hierarchical Q-learning-based method for joint sleep and power control, but
RIS control is not included in our prior work. In addition, the sub-controller
stability is not considered in [23] by using conventional $\epsilon$-greedy
policy for the action and goal selection for the sub-controller and meta-
controller, which may lead to unstable policy selection. In this paper, we
propose a Co-HDRL algorithm to overcome this issue, including a cross-entropy
enabled meta-controller policy to monitor the low-level stability, and a
correlated equilibrium to stabilize the action selection of sub-controllers.

Compared with existing sleep control and power control studies, another
difference is that we investigate how RIS technique can contribute to sleep
control and improve the EE of HetNets. RISs, as a cost-efficient approach,
have become a promising technique to improve the EE of 5G beyond and 6G
networks. [24] proposed two algorithms for the transmit power allocation and
the phase shifts of RIS elements to maximize the EE metric. [25] studies the
tradeoff between energy efficiency and spectral efficiency in RIS-aided
networks. Different than these existing works, here, we consider a novel
multi-BS and multi-RIS HetNet and study how RISs can assist sleep and power
control. We focus on the phase shift optimization of RIS elements and propose
an FP-based control method. The proposed RIS control method receives the
transmission power information from Co-HDRL as input and optimizes the phase
shift to maximize the sum-rate.

##  III Network and System Model

Figure 1: RIS-aided heterogeneous network.

As shown in Fig.1, we consider a multi-BS and multi-RIS HetNets environment.
Firstly, the small base stations (SBSs) can enter sleep mode when the
transmission demand drops, which will reduce the energy consumption and
increase overall energy efficiency. Then the MBS will take over the active UEs
that are previously covered by sleeping SBSs. Meanwhile, SBS can adjust its
transmission power dynamically to achieve the desired SINR for attached UEs
and reduce the interference on other SBSs. In this work, we propose a Co-HDRL
method for joint sleep and power control of SBSs.

On the other hand, the direct transmission between BSs and UEs may suffer high
penetration loss due to dense buildings. Then, RISs are deployed to reshape
the signal transmission path from BS to UEs and increase the SINR. Note that
one RIS may be shared by several BSs, which means the RIS control will
simultaneously affect the performance of several BSs, and we propose an FP-
based algorithm for the RIS phase-shift control.

###  III-A RIS-Assisted Channel Model

The direct transmissions between BS and UEs are usually considered as non-
line-of-sight (NLOS) with a high penetration loss caused by dense buildings
and complicated propagation environments. Consequently, we assume UEs mainly
receive signals by the indirect RIS-assisted transmission that consists of BS-
RIS and RIS-UE links[24].

RIS elements are usually deployed on the surface of high buildings, and then
the BS-RIS link is considered as line-of-sight (LOS) link:

$\bm{H}_{b,m}=g_{b,m}[\bm{h}_{b,m,1},...,\bm{h}_{b,m,n},...,\bm{h}_{b,m,|\mathcal{N}_{m}|}],$
(1)

where $g_{b,m}$ is the path loss from $b^{th}$ BS to $m^{th}$ RIS,
$\mathcal{N}_{m}$ is the total number of elements of $m^{th}$ RIS. The phase
difference $\bm{h}_{b,m,n}$ is given by
$\bm{h}_{b,m,n}=\exp\left(\frac{{-2j\pi d_{b,m,n}}}{\lambda}\right)$, where
$j$ is the imaginary unit, $d_{b,m,n}$ is the distance from $b^{th}$ BS to
$m^{th}$ RIS, $\lambda$ is the signal wavelength.

Then, the signal will be reflected by RIS to UEs by a phase shift matrix:

$\bm{\Theta}_{m}=\omega_{m}{\color[rgb]{0,0,0}{\text{diag}}}(\theta_{m,1},...,\theta_{m,n},...,\theta_{m,\mathcal{N}_{m}}),$
(2)

where diag indicates the diagonal operator, $\omega_{m}$ is the amplitude
reflection coefficient of $m^{th}$ RIS, and $\omega_{m}\in[0,1]$.
$\theta_{m,n}=e^{j\theta^{\prime}_{m,n}}$ is the phase shift of $n^{th}$ RIS
elements with
$\theta^{\prime}_{m,n}\in\left\\{0,\frac{2\pi}{2^{\mu}},\cdots,\frac{2\pi(2^{\mu}-1)}{2^{\mu}}\right\\}$,
and $\mu$ is the resolution of the RIS element’s phase shifter.

Considering the complex environment on the UE side, the RIS-UE link follows
the Rician fading, which is a combination of LOS and NLOS transmissions[26]:

$\bm{G}_{m,k}=\sqrt{\frac{\tau_{m,k}}{\tau_{m,k}+1}}\overline{\bm{G}}_{m,k}+\sqrt{\frac{1}{\tau_{m,k}+1}}\widetilde{\bm{G}}_{m,k},$
(3)

where $\tau_{m,k}$ is the Rician factor between $m^{th}$ RIS and $k^{th}$ UE.
$\overline{\bm{G}}_{m,k}$ is the deterministic LOS component, which is
calculated in a similar way as (1). $\widetilde{\bm{G}}_{m,k}$ is the NLOS
component, which is given by independent and identically distributed complex
Gaussian distribution.

For a downlink transmission between BS $j$ and UE $k$, the channel capacity
is:

$\begin{aligned}
C_{b,k}=&b_{k}\log{\color[rgb]{0,0,0}{\bigg{(}}}1+&\frac{P_{b}|\sum\limits_{m\in\mathcal{M}}\bm{H}_{b,m}\bm{\Theta}_{m}\bm{G}_{m,k}^{{\dagger}}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}_{-b}}P_{b^{\prime}}|\sum\limits_{m^{\prime}\in\mathcal{M}}\bm{H}_{b^{\prime},m^{\prime}}\bm{\Theta}_{m^{\prime}}\bm{G}_{m^{\prime},k}^{{\dagger}}|^{2}+N_{0}^{2}}{\color[rgb]{0,0,0}{\bigg{)}}},\end{aligned}$

(4)

where $b_{k}$ is the bandwidth allocated to UE $k$, $\bm{G}_{m,k}^{{\dagger}}$
represents the conjugate transpose of vector $\bm{G}_{m,k}$, $N_{0}^{2}$ is
the noise power, and $P_{b}$ is the transmission power of $b^{th}$ BS.
$\mathcal{B}_{-b}$ denotes the set of BSs except $b^{th}$ BS, and
$\mathcal{M}$ is the set of RISs in the environment.

###  III-B Energy Consumption Model

We assume only SBS can enter the sleep mode, and the energy consumption of the
$b^{th}$ SBS is[27]:

$E_{b}=\left\\{\begin{array}[]{lcl}P_{active}+\delta_{BS}P_{b},&&\text{if\,\,}q_{b}=1;\\\
P_{sleep},&&\text{if\,\,}q_{b}=0,\end{array}\right.$ (5)

where $q_{b}=1$ indicates active mode, and $q_{b}=0$ means sleep mode.
$P_{active}$ is the constant energy consumption of SBS in active mode,
$\delta_{BS}$ is the slope of load-dependent energy consumption factor,
$P_{b}$ is the transmission power, and $P_{sleep}$ is the SBS energy
consumption in sleep mode.

###  III-C Problem Formulation

The overall objective is to maximize the overall EE, and the problem
formulation is given by:

$\displaystyle\max\limits_{{\color[rgb]{0,0,0}{q_{b},P_{b},\bm{\Theta}_{m}}}}\qquad$
$\displaystyle\frac{\sum_{b\in\mathcal{B}}\sum_{k\in\mathcal{K}_{b}}D_{b,k}}{\sum_{b\in\mathcal{B}}E_{b}}-\phi\sum_{b\in\mathcal{B}}n_{b}$
(6) s.t.
$\displaystyle(\ref{eq2})\,(\ref{eq4})\,(\ref{eq5})\,(\ref{eq7})\,(\ref{eq8}),$
(6a)

where $D_{b,k}$ is the throughput of $k^{th}$ UE associated to $b^{th}$
BS111Note that the throughput level $D_{b,k}$ depends on the user traffic
demand and available channel capacity, which means that BS can adjust the
transmission power dynamically to save energy consumption when user’s traffic
demand is off-peak. Therefore, such definitions can better adapt to wireless
network dynamics.. $n_{b}$ is a binary variable: $n_{b}=1$ means the $b^{th}$
BS is overloaded; otherwise $n_{b}=0$. Here, overloading indicates the
transmission demand has exceeded the BS channel capacity, and consequently
attached UEs may experience a long queuing delay. We define $\phi$ as a
penalty factor to avoid overloading and guarantee network performance. The
power consumption of RIS elements is not included here because it is much
lower than BS power in practice[24].

The defined problem (6) includes three control variables $q_{b}$, $P_{b}$, and
$\bm{\Theta}_{m}$. First, the binary variable $q_{b}$ in equation (5)
indicates the sleep control of SBSs, which will affect the energy consumption
$\sum_{b\in\mathcal{B}}E_{b}$ in the objective. Second, the BS transmission
power level $P_{b}$ in equation (4) will affect both energy consumption and
channel capacity. Finally, the channel capacity also depends on the RIS phase
shift $\bm{\Theta}_{m}$, which is given by equation (4). In summary, we aim to
jointly consider the sleep control, transmission power control, and RIS phase-
shift control to optimize the objective defined in equation (6).

##  IV Fractional Programming based RIS Phase-shift Control

In this section, we introduce the proposed FP-based RIS phase control. Based
on equation (4), the SINR of UE $k$ is:

$\psi_{k}=\frac{P_{b}|\sum\limits_{m\in\mathcal{M}}\bm{H}_{b,m}\bm{\Theta}_{m}\bm{G}_{m,k}^{{\dagger}}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}_{-b}}P_{b^{\prime}}|\sum\limits_{m^{\prime}\in\mathcal{M}}\bm{H}_{b^{\prime},m^{\prime}}\bm{\Theta}_{m^{\prime}}\bm{G}_{m^{\prime},k}^{{\dagger}}|^{2}+N_{0}^{2}}$
(7)

The objective of RIS phase-shift control is to maximize the total data rate
for all UEs, which is given by:

$\displaystyle\max\limits_{\bm{\Theta}_{m}}\qquad$ $\displaystyle
f_{1}(\bm{\Theta}_{m})=\sum_{k\in\mathcal{K}}b_{k}\log(1+\psi_{k})$ (8) s.t.
$\displaystyle|\theta_{m,n}|^{2}=1,m\in\mathcal{M},n\in\mathcal{N}_{m},$ (8a)

where $\mathcal{K}$ is the set of UEs, and $b_{k}$ is the bandwidth allocated
to UE $k$. In equation (8), note that the control variable is
$\bm{\Theta}_{m}$, and the BS transmission power $P_{b}$ depends on the Co-
HDRL algorithm, which will be introduced in the next section.

Proposition 1: The problem (8) is equivalent to:[13]

$\begin{aligned}
\max\limits_{\bm{\Theta}_{m},\bm{\beta}}&f_{2}(\bm{\Theta}_{m},\bm{\beta})=\sum_{k\in\mathcal{K}}{\color[rgb]{0,0,0}{\bigg{(}}}b_{k}\log(1+\beta_{k})\\\
&\qquad\qquad\qquad\qquad-
b_{k}\beta_{k}+\frac{b_{k}(1+\beta_{k})\psi_{k}}{1+\psi_{k}}{\color[rgb]{0,0,0}{\bigg{)}}}\\\
\text{s.t.}&(\ref{e9:d}),\end{aligned}$

(9)

where $\bm{\beta}=[\beta_{1},\beta_{2},...,\beta_{|\mathcal{K}|}]$ is the
auxiliary variables given by Lagrangian dual transform.

Proof: See Appendix A.

Given equation (9), we take the ratio term out of the logarithm by Lagrangian
transform. To solve the problem (9), we apply an iterative method to update
the $\bm{\Theta}_{m}$ and $\bm{\beta}$ alternatively. First, given
$\bm{\Theta}_{m}$, setting $\frac{\partial f_{2}}{\partial\beta_{k}}=0$ claims
$\beta^{*}_{k}=\psi_{k}$. Then, given $\beta^{*}_{k}$, optimizing
$\bm{\Theta}_{m}$ means:

$\begin{aligned}
\max\limits_{\bm{\Theta}_{m}}\quad&f_{3}(\bm{\Theta}_{m})=\sum_{k\in\mathcal{K}}\frac{b_{k}(1+\beta_{k})\psi_{k}}{1+\psi_{k}}\\\
&=\sum_{k\in\mathcal{K}}\frac{b_{k}(1+\beta_{k})P_{b}|\sum\limits_{m\in\mathcal{M}}\bm{H}_{b,m}\bm{\Theta}_{m}\bm{G}_{m,k}^{{\dagger}}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}}P_{b^{\prime}}|\sum\limits_{m^{\prime}\in\mathcal{M}}\bm{H}_{b^{\prime},m^{\prime}}\bm{\Theta}_{m^{\prime}}\bm{G}_{m^{\prime},k}^{{\dagger}}|^{2}+N_{0}^{2}}\\\
\text{s.t.}\quad&(\ref{e9:d}),\end{aligned}$

(10)

For ease of notation, we define $\bm{\hat{\theta}}_{m}$ by:

$\bm{\hat{\theta}}_{m}=[\theta_{m,1},\theta_{m,2},...,\theta_{m,\mathcal{N}_{m}}]\in\mathbb{C}^{1\times
N},$ (11)

and $\sqrt{P_{b}}\bm{H}_{b,m}\bm{\Theta}_{m}\bm{G}_{m,k}^{{\dagger}}$ can be
easily transformed to
$\omega_{m}\bm{\hat{\theta}}_{m}$diag$(\bm{H}_{b,m})\bm{G}_{m,k}^{{\dagger}}\sqrt{P_{b}}$.
For notation brevity, we further define
$\bm{v}_{b,m,k}=\omega_{m}{\color[rgb]{0,0,0}{\text{diag}}}(\bm{H}_{b,m})\bm{G}_{m,k}^{{\dagger}}\sqrt{P_{b}}$.
Then, equation (10) can be rewritten by:

$\begin{aligned} \max\limits_{\bm{\hat{\Theta}}}\
&f_{3}(\bm{\hat{\Theta}})=\sum_{k\in\mathcal{K}}\frac{b_{k}(1+\beta_{k})|\sum\limits_{m\in\mathcal{M}}\bm{\hat{\theta}}^{{\dagger}}_{m}\bm{v}_{b,m,k}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}}|\sum\limits_{m^{\prime}\in\mathcal{M}}\bm{\hat{\theta}^{{\dagger}}}_{m^{\prime}}\bm{v}_{b^{\prime},m^{\prime},k}|^{2}+N_{0}^{2}}\\\
&\qquad=\sum_{k\in\mathcal{K}}\frac{b_{k}(1+\beta_{k})|\bm{\hat{\Theta}}\bm{V}_{b,k}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{\hat{\Theta}}\bm{V}_{b^{\prime},k}|^{2}+N_{0}^{2}}\\\
\text{s.t.}\ &(\ref{e9:d}),\end{aligned}$

(12)

where
$\bm{\hat{\Theta}}=[\bm{\hat{\theta}}^{{\dagger}}_{1},\bm{\hat{\theta}}^{{\dagger}}_{2},...,\bm{\hat{\theta}}^{{\dagger}}_{|\mathcal{M}|}]$
represents all the RIS phase control variables, and
$\bm{V}_{b,k}=[\bm{v}_{b,1,k},\bm{v}_{b,2,k},...,\bm{v}_{b,|\mathcal{M}|,k}]$.

Proposition 2: Based on quadratic transformation, equation (12) is equivalent
to[13]:

$\begin{aligned} \max\limits_{\bm{\hat{\Theta}},\eta_{k}}\
&f_{4}(\bm{\hat{\Theta}},\eta_{k})=\sum_{k\in\mathcal{K}}(2\sqrt{b_{k}(1+\beta_{k})}\\\
&\qquad{\color[rgb]{0,0,0}{\text{Re}}}\\{\eta_{k}^{{\dagger}}(\bm{\hat{\Theta}}\bm{V}_{b,k})\\}-\eta_{k}^{{\dagger}}(\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{\hat{\Theta}}\bm{V}_{b^{\prime},k}|^{2}+N_{0}^{2})\eta_{k})\\\
\text{s.t.}\ &(\ref{e9:d}),\end{aligned}$

(13)

where $\bm{\eta}=[\eta_{1},\eta_{2},...\eta_{|\mathcal{K}|}]$ a collection of
auxiliary variables, and $\text{Re}\\{\\}$ refers to the real number part of a
complex number.

Proof: See Appendix B.

Given proposition 2, we rewrite equation (13) to optimize $\bm{\hat{\Theta}}$:

$\begin{aligned} \max\limits_{\bm{\hat{\Theta}}}\
&f_{5}(\bm{\hat{\Theta}})=2{\color[rgb]{0,0,0}{\text{Re}}}\\{\bm{\hat{\Theta}^{{\dagger}}}\sum_{k\in\mathcal{K}}2\sqrt{b_{k}(1+\beta_{k})}\eta_{k}^{{\dagger}}\bm{V}_{b,k}\\}\\\
&-\bm{\hat{\Theta}^{{\dagger}}}\sum_{k\in\mathcal{K}}|\eta_{k}|^{2}\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{V}_{b^{\prime},k}|^{2}\bm{\hat{\Theta}}+\sum_{k\in\mathcal{K}}|\eta_{k}|^{2}N_{0}^{2}\\\
&=-\bm{\hat{\Theta}^{{\dagger}}}\Lambda\bm{\hat{\Theta}}+2{\color[rgb]{0,0,0}{\text{Re}}}\\{\bm{\hat{\Theta}^{{\dagger}}}\Psi\\}+\sum_{k\in\mathcal{K}}|\eta_{k}|^{2}N_{0}^{2}\\\
\text{s.t.}\ &(\ref{e9:d}),\end{aligned}$

(14)

where
$\Lambda=\sum_{k\in\mathcal{K}}|\eta_{k}|^{2}\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{V}_{b^{\prime},k}|^{2}$
and
$\Psi=\sum_{k\in\mathcal{K}}2\sqrt{b_{k}(1+\beta_{k})}\eta_{k}^{{\dagger}}\bm{V}_{b,k}$
for ease of notations. Note that equation (14) is a quadratically constrained
quadratic programming problem, but the phase shift constraint
$|\theta_{m,n}|^{2}=1$ is non-convex. Therefore, we relax this constraint by
allowing $|\theta_{m,n}|^{2}\leq 1$ for convexity, which means:

$\displaystyle\max\limits_{\bm{\hat{\Theta}}}$ $\displaystyle
f_{6}(\bm{\hat{\Theta}})=-\bm{\hat{\Theta}^{{\dagger}}}\Lambda\bm{\hat{\Theta}}+2{\color[rgb]{0,0,0}{\text{Re}}}\\{\bm{\hat{\Theta}^{{\dagger}}}\Psi\\}$
(15) s.t. $\displaystyle|\theta_{m,n}|^{2}\leq
1,m\in\mathcal{M},n\in\mathcal{N}_{m},$

which is an optimization problem with strong convexity. The Lagrange dual of
this problem can be written as an unconstrained problem:

$\displaystyle\max\limits_{\bm{\hat{\Theta}},\bm{\hat{\sigma}}}$
$\displaystyle
f_{7}(\bm{\hat{\Theta}},\bm{\hat{\sigma}})=-\bm{\hat{\Theta}^{{\dagger}}}\Lambda\bm{\hat{\Theta}}+2{\color[rgb]{0,0,0}{\text{Re}}}\\{\bm{\hat{\Theta}^{{\dagger}}}\Psi\\}$
(16)
$\displaystyle\qquad\qquad-\sum\limits_{m\in\mathcal{M}}\sum\limits_{n\in\mathcal{N}_{m}}\sigma_{m,n}(|\theta_{m,n}|^{2}-1),$

where $\sigma_{m,n}$ is the Lagrange dual variable for each constraint with
$\bm{\hat{\sigma}}=[\bm{\sigma}_{1},...,\bm{\sigma}_{m},...,\bm{\sigma}_{\mathcal{M}}]$,
and each $\bm{\sigma}_{m}=[\sigma_{m,1},...,$
$\sigma_{m,n},...,\sigma_{m,|\mathcal{N}_{m}|}]$.

Proposition 3: the optimal value of equation (15) can be achieved by updating
optimal $\bm{\hat{\Theta}}$ and $\bm{\hat{\sigma}}$ iteratively.

Proof: Setting $\frac{\partial f_{7}}{\partial\bm{\hat{\Theta}}}=0$, then we
have the optimal $\bm{\hat{\Theta}^{*}}$ by:

$\bm{\hat{\Theta}^{*}}=\frac{\Psi}{\Lambda+{\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}})}.$
(17)

Then, given $\bm{\hat{\Theta}^{*}}$, we differentiate (16):

$\begin{aligned} \frac{\partial
f_{7}}{\partial\sigma_{m,n}}&=\frac{\partial(\sum\limits_{m\in\mathcal{M}}\sum\limits_{n\in\mathcal{N}_{m}}\sigma_{m,n}(1-|\theta_{m,n}|^{2}))}{\partial\sigma_{m,n}}\\\
&=1-\frac{\partial(\bm{\hat{\Theta}}\bm{\hat{\Theta}^{{\dagger}}}{\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}}))}{\partial\sigma_{m,n}}\\\
&=1-|\theta_{m,n}|^{2},\end{aligned}$

(18)

Setting $\frac{\partial f_{7}}{\partial\sigma_{m,n}}=0$ will bring
$|\theta_{m,n}|^{2}=1$, which means each element in $\bm{\hat{\Theta}^{*}}$
will satisfy the constraints (8a).

Finally, substituting (17) into (16) claims:

$f_{8}=\frac{\Psi\Psi^{{\dagger}}}{\Lambda+{\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}})}+tr({\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}})),$
(19)

which can be solved by the well-known Schur complement:

$\displaystyle\max\limits_{\bm{\hat{\sigma}},\kappa}\ $ $\displaystyle
f_{9}(\bm{\hat{\sigma}},\kappa)=\kappa-
tr({\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}}))$ (20) s.t.
$\displaystyle\left[\begin{matrix}\Lambda+{\color[rgb]{0,0,0}{\text{diag}}}(\bm{\hat{\sigma}})&\Psi\\\
\Psi^{{\dagger}}&-\kappa\\\ \end{matrix}\right]\succeq 0$ .

Finally, equation (20) can be efficiently solved as a semidefinite programming
(SDP) problem, and we summarize the FP-based RIS phase control in Algorithm 1,
which applies an alternative method for optimization.

Algorithm 1 Fractional programming based RIS phase control

1: Initialize: Wireless channel parameters and BS transmission powers. Setting
iteration number $y=1$, selecting a feasible initial value for
$\bm{\Theta}^{(y)}$ based on the constraints.

2: repeat

3: Given $\bm{\Theta}^{(y)}$, updating auxiliary variable
$\beta_{k}^{*(y+1)}=\psi_{k}^{(y)}$ by equation (7).

4: Calculating $\bm{\hat{\Theta}}^{(y+1)}$ and $\bm{V}_{b,k}^{(y+1)}$ which
has been defined in equation (12).

5: Calculating $\eta^{*(y+1)}$ as in Appendix B, and generating
$\Lambda^{(y+1)}$ and $\Psi^{(y+1)}$ accordingly.

6: Calculating $\bm{\hat{\sigma}}^{(y+1)}$ by solving problem (20).

7: Updating $\bm{\hat{\Theta}^{(y)}}$ by equation (17), transforming
$\bm{\hat{\Theta}^{(y)}}$ to $\bm{\Theta}^{(t)}$.

8: $y=y+1$

9: until Problem (9) converges, or the iteration number $y$ reaches the
predefined maximum value.

10: Output: Maximum data transmission rate and optimal RIS phase shifts under
given transmission power of BSs.

##  V Cooperative Hierarchical Deep Reinforcement Learning for Sleep Control
and Power Control

We introduce the proposed Co-HDRL algorithm in this section. Co-HDRL applies a
hierarchical architecture for decision-making, including a meta-controller for
sleep control, and multiple sub-controllers for transmission power control.

###  V-A Hierarchical SMDP, Meta-controller and Sub-controller Definitions

In this section, we introduce a hierarchical semi Markov decision process
(SMDP), which is used to transform our defined problem formulation into MDP
context.

(a) Conventional MDP architecture

(b) SMDP architecture.

Figure 2: MDP and SMDP comparison.

A conventional MDP scheme can be defined by a tuple $<S,A,T,R>$, where $S$ is
the state set, $A$ is the action set, $T$ is the transition probability, and
$R$ is the reward function, respectively. As shown in Fig.2(a), the agents
select an action $a^{t}$ based on current state $s^{t}$, then it receives the
reward $r^{t}$ and moves to the next state $s^{t+\Delta t}$. However,
conventional MDP has difficulty in defining more complicated and realistic
problems such as tasks with sparse rewards and different timescale actions.
For example, sleep control is obviously a long-term decision, since turning
on/off SBSs will greatly affect the network performance. By contrast, power
control is a delicate and short-term decision to change the SINR of UEs.
Therefore, we include a novel hierarchical SMDP framework to better handle the
timescale difference between sleep control and power control.

Compared with conventional MDP, the main difference of SMDP lies in the
hierarchical architecture. The SMDP here can be defined by a tuple
$<S,A,T,R,\mathcal{G}>$, where $\mathcal{G}$ is the set of goals. For example,
Fig. 2(b) includes three agents in different levels, and each agent has its
own MDPs. Given current state $s_{h}^{t}$, the highest level agent will first
select a goal $g_{h}^{t}$ at time slot $t$, and this goal serves as a high-
level policy instruction for middle-level agent from time slot $t$ to
$t+4\Delta t$. Similarly, $g_{m}^{t}$, which is the goal selected by middle-
level agent, can affect the low-level action selection from $t$ to $t+2\Delta
t$. On the other hand, low-level agents will provide reward feedback to
higher-level agents, which can be used to evaluate higher-level goals. For
instance, $r_{l}^{t}$ and $r_{l}^{t+\Delta t}$, which are low-level rewards,
may affect the $r_{m}^{t}$ calculation, i.e. the middle-level reward.

Moreover, these agents can apply different timescales for decision making:
e.g., the highest level selects $g_{h}^{t}$ every $4\Delta t$, the middle
level chooses $g_{m}^{t}$ every $2\Delta t$, and the low level makes decisions
every $\Delta t$. This hierarchical timescale setting enables higher
flexibility for SMDP in more complicated and realistic environments, which
makes it an ideal model to describe the joint sleep control and power control
problem. Meanwhile, Fig. 2(b) reveals that the high-level agents rely on the
reward feedback of lower levels for goal evaluation. Unstable feedback from
low-level agents can mislead the high-level controllers, and sub-optimal
policy may be selected, which will finally degrade the overall system reward.
Note that SMDP has many variants, and here we define our SMDP to solve the
hierarchical control problem[28]. We use three layers as an example in Fig.
2(b), but the proposed SMDP scheme is compatible with any number of layers.

Figure 3: Overall architecture of the proposed Co-HDRL.

Based on Fig. 2(b), we define an SMDP with two layers. MBS is defined as a
meta-controller for sleep control, which will decide the long-term on/off
status of attached SBSs. Then, each SBS is considered as a sub-controller to
adjust its short-term transmission power level. Sleep control is refereed to
high-level policy instruction for SBSs, and SBSs can provide reward feedback
to the meta-controller to evaluate the sleep control policies.

For the SBS sub-controller, we define the state, action and reward by:

  * •

State: For the $b^{th}$ SBS, the state $s_{sub}$ is defined by the total
transmission demand level of attached UEs:

$s_{sub}=\frac{\sum_{k\in\mathcal{K}_{b}}W_{b,k}}{W^{max}_{b}},$ (21)

where $W_{b,k}$ represents the transmission demand of UE $k$,
$\mathcal{K}_{b}$ represents the set of UEs that are associated with $b^{th}$
SBS. $W^{max}_{b}$ is the max transmission demand of $b^{th}$ SBS, which is
referred to as a constant value to normalize transmission demand in the
current time slot. It is assumed the UE daily transmission demand follows
specific patterns as in [29].

  * •

Action: The SBS sub-controller can adjust its transmission power level $P_{b}$
to satisfy the transmission demand in the current time slot. Therefore, the
action of $b^{th}$ SBS sub-controller is $a_{sub}=\\{P_{b}\\}$.

  * •

Low-level reward: The low-level reward of sub-controller is:

$r_{sub}=\left\\{\begin{array}[]{lcl}\frac{\sum_{k\in\mathcal{K}_{b}}W_{b,k}}{E_{b}}-\phi
n_{b},&&\text{if\,\,}q_{b}=1\\\ \qquad
0,&&\text{if\,\,}q_{b}=0\end{array}\right.$ (22)

where $\phi$ and $n_{b}$ are overloading penalty factor and overload
indicator, respectively. The overload definition has been given in equation
(6). $q_{b}$ has been defined in equation (5) as the sleep control indicator.
$r_{sub}$ aims to maximize the EE of $b^{th}$ SBS, and we assume the $r_{sub}$
is 0 in sleep mode.

The MBS is defined as a meta-controller, which will produce high-level sleep
control policies for the SBS sub-controllers:

  * •

State: The state of the MBS meta-controller includes the transmission demand
level of all BSs:

$s_{meta}={\color[rgb]{0,0,0}{\bigg{[}}}\frac{\sum\limits_{k\in\mathcal{K}_{1}}W_{1,k}}{W^{max}_{1}},\frac{\sum\limits_{k\in\mathcal{K}_{2}}W_{2,k}}{W^{max}_{2}},...,\frac{\sum\limits_{k\in\mathcal{K}_{|\mathcal{B}|}}W_{|\mathcal{B}|,k}}{W^{max}_{|\mathcal{B}|}}{\color[rgb]{0,0,0}{\bigg{]}}},$
(23)

where $\mathcal{B}$ is the set of all BSs.

  * •

High-level goals: Given the transmission demand level of all BSs, the meta-
controller can produce sleep control decisions for SBS sub-controllers:

$g_{meta}=[q_{1},q_{2},...,q_{b},...,q_{|\mathcal{B}_{SBS}|}],$ (24)

where $q_{b}$ has been defined in equation(5) as sleep mode indicator of
$b^{th}$ SBS, and $\mathcal{B}_{SBS}$ is the set of SBSs.

  * •

High-level reward: The meta-controller is responsible for the overall
performance of all BSs. Accordingly, the high-level reward is given by the
objective function we have defined in the problem formulation:

$r_{meta}=\frac{\sum_{b\in\mathcal{B}}\sum_{k\in\mathcal{K}_{b}}D_{b,k}}{\sum_{b\in\mathcal{B}}E_{b}}-\phi\sum_{b\in\mathcal{B}}n_{b},$
(25)

###  V-B Co-HDRL Framework and Network Training

The overall framework of our proposed Co-HDRL algorithm is given in Fig. 3,
which includes one meta-controller and multiple sub-controllers. The goal
selected by the meta-controller will decide the on/off status of sub-
controllers.

In each controller, we deploy a double deep Q-learning (DDQN) scheme for the
Q-value prediction. In conventional deep Q-learning, the loss function $Er$ of
network training is:

$L(w)=Er(r^{t}+\gamma\max\limits_{a}Q(s^{t+1},a,w^{\prime})-Q(s^{t},a^{t},w)),$
(26)

where $s^{t}$, $a^{t}$ and $r^{t}$ are the state, action and reward at time
slot $t$, respectively. $w$ and $w^{\prime}$ are the weight of the main and
target networks. $\gamma$ is the discount factor ($0<\gamma<1$).
$Q(s^{t},a^{t},w)$ is the current Q-value that is predicted by the main
network, and $Q(s^{t+1},a,w^{\prime})$ is the target Q-value produced by
target network. In equation (26), note that both action selection and
evaluation depend on the target network, which is indicated by
$\max\limits_{a}Q(s^{t+1},a,w^{\prime})$. Consequently, the max operator will
constantly generate overoptimistic values for the loss function, which will
further affect the Q-value prediction in main network training [30]. To this
end, the DDQN scheme is proposed to separate the action selection and
evaluation by:

$\displaystyle L$ $\displaystyle(w)=Er(r^{t}+$ (27) $\displaystyle\gamma
Q(s^{t+1},\arg\max\limits_{a}Q(s^{t+1},a,w),w^{\prime})-Q(s^{t},a^{t},w)),$

where $\arg\max\limits_{a}Q(s^{t+1},a,w)$ indicates the main network selects
the next action, and the target network evaluates the action by
$Q(s^{t+1},\arg\max\limits_{a}Q(s^{t+1},a,w),w^{\prime})$. Decoupling the
action selection and evaluation can provide a more accurate target for the
main network training, which will further reduce the Q-value prediction error.

In Co-HDRL, first, the meta-controller selects a high-level goal $g_{meta}$,
which indicates the sleep control decisions on sub-controllers. $g_{meta}$ is
temporarily fixed in the following several time slots, and sub-controllers
select actions, receive rewards, and train their networks accordingly. The
transmission power of sub-controllers is sent to the RIS control block for
phase shift optimization. Based on the long-term performance of sub-
controllers, the meta-controller receives an average reward $r_{meta}$ from
the wireless environment and moves to the next state $s_{meta}$. The new
experience tuple $<s_{meta}^{t},g_{meta}^{t},r_{meta}^{t},s_{meta}^{t+1}>$
will be sent to the experience pool (shown as the left bottom block in Fig.
3). Then the agent will sample a random mini batch from the experience pool,
and train the main network as equation (27). The target network will copy the
main network weight after several training, which guarantees a stable target
for the main network training.

In Fig. 3, it is worth noting that we define a high-level cross-entropy
calculation module (shown by the blue block on the top), and a low-level
action selection cooperation module (indicated by the red block on the right).
In HRL, the meta-controller produces goals to instruct sub-controllers, and
sub-controllers provide reward feedback to the meta-controller to evaluate the
high-level goal. However, although the high-level goal $g_{meta}$ is
temporarily fixed, the action of sub-controllers may be constantly changing
during this period. Indeed, the sub-controllers’ exploration increases the
uncertainty of the reward feedback to the meta-controller. Therefore, this
feedback uncertainty can mislead the goal evaluation of meta-controller[11,
22]. To this end, we proposed a cross-entropy enabled policy for high-level
meta-controller, and a correlated equilibrium-based cooperation strategy for
low-level sub-controllers. Following, we will introduce two techniques in
detail.

Figure 4: Co-HDRL system update between meta-controller and sub-controllers.

###  V-C High-level Cross-entropy enabled Meta-controller Policy

In this section, we propose a cross-entropy enabled policy for meta-
controller. More specifically, we use the cross-entropy as a metric to
evaluate the stationarity of sub-controllers’ actions, then the defined metric
is used for high-level goal exploration. For example, goals with high reward
feedback but low stationarity may be unreliable, since the low stationarity
metric means the sub-controllers are still exploring action combinations. On
the other hand, goals that bring high stationarity can be less visited in the
exploration phase, because the sub-controllers have entered a stable status
and it requires less training[31]. Following, we first present the cross-
entropy metric definition, then we introduce how to use the metric for goal
exploration and selection.

Given a random variable $x$, and $N^{X}$ is the total number of possible
outcomes of $x$, then the entropy of $x$ in set $X$ is defined by:

$I(X)=-\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(x_{i}),$ (28)

where $pr(x_{i})$ is the probability of $x_{i}$ in set $X$, and
$\sum_{i=1}^{N^{X}}pr(x_{i})=1$. $I(X)$ indicates the average level of
information and uncertainty of variable $x_{i}$ in set $X$.

Then we introduce the Kullback–Leibler divergence to define the relative
entropy from one distribution $X$ to another distribution $Y$ of variable $x$
[32] :

$D_{KL}(X||Y)=\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}\frac{pr(x_{i})}{pr^{{}^{\prime}}(x_{i})},$
(29)

where $pr^{{}^{\prime}}(x_{i})$ is the probability of $x_{i}$ in set $Y$.
$D_{KL}(X||Y)$ indicates the difference of set $X$ and $Y$ in terms of
probability distribution of $x$.

In this work, we consider $X$ as the set of action selection history of sub-
controllers, and $Y$ is the set of actions selected in the current time
interval $\Delta t$. Consequently, we can apply $D_{KL}(X||Y)$ to measure the
stationarity of the low-level action selection policy. The idea behind is that
a stationary policy will produce similar action selections in different time
intervals, which is indicated by a lower $D_{KL}(X||Y)$ value. By contrast, a
higher $D_{KL}(X||Y)$ means sub-controllers require more training to stabilize
the low-level policies.

Proposition 4: The Kullback–Leibler divergence has the minimum value when
$pr(x_{i})=pr^{{}^{\prime}}(x_{i})$.

Proof 4: Based on Gibbs’ inequality[33]:

$\begin{aligned}
D_{KL}(X||Y)&=\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}\frac{pr(x_{i})}{pr^{{}^{\prime}}(x_{i})}\\\
&=-\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(\frac{pr^{{}^{\prime}}(x_{i})}{pr(x_{i})})\\\
&\qquad(\text{Using ${\color[rgb]{0,0,0}{\log}}(x)<x-1$ for all $x>0$})\\\
&\geq-\sum_{i=1}^{N^{X}}pr(x_{i})(\frac{pr^{{}^{\prime}}(x_{i})}{pr(x_{i})}-1)\\\
&=\sum_{i=1}^{N^{X}}pr(x_{i})-\sum_{i=1}^{N^{X}}pr^{{}^{\prime}}(x_{i})\\\
&=0.\end{aligned}$

(30)

$D_{KL}(X||Y)=0$ if and only if $X$ and $Y$ has the same distribution
$pr(x_{i})=pr^{{}^{\prime}}(x_{i})$.

Moreover, we rewrite equation (29) by:

$\displaystyle D_{KL}(X||Y)$
$\displaystyle=\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}\frac{pr(x_{i})}{pr^{{}^{\prime}}(x_{i})}$
(31)
$\displaystyle=\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(pr(x_{i}))-\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(pr^{{}^{\prime}}(x_{i}))$
$\displaystyle=-I(X)-\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(pr^{{}^{\prime}}(x_{i})),$

where $I(X)$ is the entropy of the action selection distribution history, and
$Y$ is the action selection distribution in the current time interval. The
history distribution $X$ is generally more stable than the current
distribution $Y$. It means
$\sum_{i=1}^{N^{X}}pr(x_{i}){\color[rgb]{0,0,0}{\log}}(pr^{{}^{\prime}}(x_{i}))$
item contributes more to the uncertainty, which is known as the cross-
entropy[34]. In this work, we use cross entropy to represent the stationarity
of the sub-controllers.

We include multiple sub-controller in this work, and the cross-entropy of
$b^{th}$ sub-controller under high-level goal $g_{meta}$ is:

$I_{(}X_{b,t-1},Y_{b,t}|g_{meta})=-\sum_{i=1}^{A_{g_{meta}}}pr(a_{i}){\color[rgb]{0,0,0}{\log}}(pr^{{}^{\prime}}(a_{i})),$
(32)

where $X_{b,t-1}$ is the accumulated action selection distribution of sub-
controllers in former $t-1$ time slots under high-level goal $g_{meta}$,
$Y_{b,t-1}$ is the action selection distribution under $g_{meta}$ in current
time slot, and $A_{g_{meta}}$ is the action set of sub-controllers under
$g_{meta}$. The cross-entropy metric $I_{(}X_{b,t-1},Y_{b,t}|g_{meta})$
defines the stationary of sub-controller action selections under $g_{meta}$.

Finally, in the exploration phase, the meta-controller selects high-level
goals by:

$\begin{aligned}
pr(g_{meta}|&s_{meta})=\frac{{\color[rgb]{0,0,0}{\tanh}}(\sum_{b\in\mathcal{B}_{-M}}I(X_{b,t-1},Y_{b,t}|g_{meta}))}{\sum\limits_{g_{meta}\in\mathcal{G}}{\color[rgb]{0,0,0}{\tanh}}(\sum_{b\in\mathcal{B}_{-M}}I(X_{b,t-1},Y_{b,t}|g_{meta}))},\end{aligned}$

(33)

where $\mathcal{B}_{-M}$ indicates the set of controllers except the MBS meta-
controller. The ${\color[rgb]{0,0,0}{\tanh}}$ indicates the Tanh function,
which is applied to normalize all the cross-entropy values.

Algorithm 2 Co-HDRL based joint sleep and power control

1: Initialize: Wireless network and Co-HDRL parameters

2: for every $N_{T}\Delta t$, the MBS do

3: Updating the cross-entropy metric under $g_{meta}$ based on the action
selection history of last $N_{T}\Delta t$.

4: Calculating the average EE metric of former $N_{T}\Delta t$ as high-level
reward $r_{meta}$ by equation (6).

5: Updating new state $s_{meta}$, and saving $(s_{meta},g_{meta},$
$r_{meta},s_{meta}^{\prime})$ to the experience pool.

6: Sampling a mini-batch from the experience pool randomly. Generating target
Q-values $Q^{Tar}(s_{meta},g_{meta})$=
$\left\\{\begin{array}[]{ccl}r_{meta}&if\;done\\\ r_{meta}+\gamma
Q(s_{meta}^{\prime},\arg\max\limits_{g}Q(s_{meta}^{\prime},g,w),w^{\prime})&else\end{array}\right.$

7: Updating $w$ using gradient descent by minimizing the loss
$L(w)=Er(Q^{Tar}(s_{meta},g_{meta})-Q(s_{meta},g_{meta},w))$.

8: Copying $w$ to $w^{\prime}$ after several training.

9: Selecting a high-level goal $g_{meta}$ of the next $N_{T}\Delta t$ by
following method:

10: with probability $\epsilon$, selecting high-level goal $g_{meta}$ by
equation (33); otherwise, choosing goal $g_{meta}$ by greedy policy
$\arg\max\limits_{a}Q(s_{meta},g_{meta},w)$.

11: end for

12: for every $\Delta t$, active SBSs do

13: Selecting the transmission power level $a_{sub}$ by:

14: with probability $\epsilon$, sub-controllers select $a_{sub}$ randomly;
otherwise, choosing joint action $\vec{a}_{sub}$ by correlated equilibrium as
equation (34).

15: Sending the transmission power to RIS control Algorithm 1, then receives
the data transmission rate from Algorithm 1, and calculates the low-level
reward.

16: Updating new state $s_{sub}$, and saving $(s_{sub},a_{sub},$
$r_{sub},s_{sub}^{\prime})$ to the experience pool. Each sub-controller
samples a mini-batch from its experience pool, and trains the network similar
to the meta-controller.

17: end for

18: Output: Performance of the network and the learning algorithm.

In summary, equation (33) shows that goals that lead to lower stability will
be more frequently selected in the exploration phase, which is indicated by a
high cross-entropy value. It guarantees that sub-controllers can provide
reliable and stable feedback to the meta-controller for goal evaluation. The
meta-controllers and sub-controllers are updated by Fig.4. The meta-controller
will first select a goal $g^{t}_{meta}$ at time $t$, which is temporarily
fixed in the next $N_{T}\Delta t$ time slot. Then, sub-controllers select the
actions in each $\Delta t$ under current state $s_{sub}$ and high-level goal
$g^{t}_{meta}$. Finally, the actions and rewards of sub-controllers from $t$
to $t+N_{T}\Delta t$ are collected by meta-controllers for cross-entropy
calculation and goal selection as shown by equation (33).

###  V-D Low-level Sub-controllers Cooperation

We propose a correlated equilibrium-based cooperation strategy for low-level
sub-controllers. Correlated equilibrium is proposed as a multi-agent
collaboration strategy[35]. Compared with existing methods such as Nash
equilibrium, the existence and convergence of correlated equilibrium can be
better guaranteed by linear programming solutions. Indeed, cooperative action
selections can bring higher stability than independent action selections. With
correlated equilibrium, the SBS sub-controllers choose actions by:

$\displaystyle\max_{a_{sub}\in A_{sub}}\sum_{\vec{a}_{sub}\in A_{sub}}$
$\displaystyle pr(\vec{a}_{sub}|s_{sub})Q(s_{sub},\vec{a}_{sub},w)$ (34)
$\displaystyle sub.to$ $\displaystyle\quad 0\leq pr(\vec{a}_{sub}|s_{sub})\leq
1,$ (34a) $\displaystyle\sum_{\vec{a}\in A}pr(\vec{a}_{sub}|s_{sub})=1,$ (34b)
$\displaystyle\sum_{a_{-b}\in A_{-b}}$ $\displaystyle
pr(\vec{a}_{sub}|s_{sub})(Q(s_{sub},\vec{a}_{sub},w)$
$\displaystyle-Q(s_{sub},\vec{a}_{-b},a_{b},w))\geq 0,$ (34c)

where $pr(\vec{a}_{sub}|s_{sub})$ is the probability of selecting action
combination $\vec{a}_{sub}=(a_{1},a_{2},...,a_{|\mathcal{B}_{-M}|})$ under
state $s_{sub}$, $\vec{a}_{-b}$ indicates the joint action of all sub-
controllers except $b^{th}$ sub-controller, and $w$ is the main network
weight. $\mathcal{B}_{-M}$ has been defined in equation (33) as the set of
controllers except the MBS meta-controller, $A_{-b}$ is the set of
$\vec{a}_{-b}$.

Equation (34) uses Q-values predicted by the main network to represent the
expected reward. It aims to maximize the total potential reward of all sub-
controllers by finding an optimal action selection probability distribution
$pr(\vec{a}_{-b}|s_{sub})$. Constraints (34a) and (34b) are probability
distribution constraints. The constraint (34c) indicates that $\vec{a}_{-b}$
produces higher expected reward with probability $pr(\vec{a}_{-b}|s_{sub})$,
which is indicated by
$pr(\vec{a}_{sub}|s_{sub})(Q(s_{sub},\vec{a}_{sub},w)-Q(s_{sub},\vec{a}_{-b},a_{b},w))\geq
0$.

Finally, the proposed Co-HDRL algorithm is summarized in Algorithm 2.

###  V-E Computational Complexity Analyses

In this section, we analyze the computational complexity of the proposed
algorithms. For the RIS control part, the complexity lies in solving equation
(20), which is a semidefinite programming (SDP) problem. SDP can be
efficiently solved by many methods such as the interior point method, first-
order method and so on. Based on the model proposed in [36], the computational
complexity is given by
$O(\sqrt{\rho}(c\rho^{2}+c^{l}+\rho^{l}){\color[rgb]{0,0,0}{\log}}(\frac{1}{\zeta}))$,
where $\rho$ is the variable size, $c$ is the number of constraints, $l$ is
the cost of matrix multiplication, and $\zeta$ is the relative accuracy. For
the Co-HDRL part, the computational complexity depends on neural network
training. In this paper, we deploy Long short-term memory (LSTM) networks for
the Q-values prediction, which is a special recurrent neural network that can
better capture the long-term dependency of input data. The computational
complexity of updating LSTM networks is $O(\alpha^{2}\chi^{2})$, where
$\alpha$ indicates the memory blocks number, and $\chi$ represents the memory
cell number of one block [30].

##  VI Baseline Algorithms

Algorithm 3 HDRL based joint sleep and power control

1: Initialize: Wireless network and HDRL parameters

2: for every $N_{T}\Delta t$, the MBS do

3: Calculating the average EE metric of former $N_{T}\Delta t$ as high-level
reward $r_{meta}$ by equation (6).

4: Updating new state $s_{meta}$, and saving $(s_{meta},g_{meta},$
$r_{meta},s_{meta}^{\prime})$ to the experience pool.

5: Sampling a mini-batch from experience pool randomly. Generating target
Q-values $Q^{Tar}(s_{meta},g_{meta})$=
$\left\\{\begin{array}[]{ccl}r_{meta}&if\;done\\\ r_{meta}+\max\gamma
Q(s_{meta}^{\prime},g_{meta},w^{\prime})&else\end{array}\right.$

6: Updating $w$ using gradient descent by minimizing the loss
$L(w)=Er(Q^{Tar}(s_{meta},g_{meta})-Q(s_{meta},g_{meta},w))$.

7: Copying $w$ to $w^{\prime}$ after several training.

8: With probability $\epsilon$, selecting a high-level goal randomly for the
next $N_{T}\Delta t$; otherwise, choosing goal $g_{meta}$ by greedy policy
$\arg\max\limits_{a}Q(s_{meta},g_{meta},w)$.

9: end for

10: for every $\Delta t$, active SBSs do

11: Selecting the transmission power level $a_{sub}$ by:

12: with probability $\epsilon$, sub-controllers select $a_{sub}$ randomly;
otherwise, choosing action $a_{sub}$ by greedy policy.

13: Sending the transmission power to RIS control Algorithm 1, then receives
the data transmission rate from Algorithm 1, and calculating the low-level
reward.

14: Updating new state $s_{sub}$, and saving $(s_{sub},a_{sub},$
$r_{sub},s_{sub}^{\prime})$ to the experience pool. Each sub-controller
samples a mini-batch from its experience pool, and trains the network similar
to the meta-controller.

15: end for

16: Output: Performance of the network and the learning algorithm.

###  VI-A Hierarchical Deep Reinforcement Learning

To better demonstrate the capability of our proposed Co-HDRL algorithm, we
propose a conventional hierarchical deep reinforcement learning (HDRL) based
method as a benchmark. In HDRL, both meta-controller and sub-controllers apply
$\epsilon$-greedy policy for goal and action selections, respectively, and
there is no high-level cross-entropy and low-level cooperation between agents.
The HDRL is given in Algorithm 3.

###  VI-B Surrogate Optimization based RIS phase control

This section introduces the surrogate optimization-based RIS phase-shift
control as a baseline algorithm. In practice, the phase shift can only be
selected from a set of fixed values, which mainly depends on the predefined
resolution settings of phase shifters. Therefore, we formalize the RIS control
problem to maximize the total transmission rate of all UEs by:

$\displaystyle\max\limits_{\mathcal{Z}}$ $\displaystyle\quad
f_{10}(\mathcal{Z})=\sum_{k\in\mathcal{K}}b_{k}\log{\color[rgb]{0,0,0}{\bigg{(}}}1+$
$\displaystyle\frac{P_{b}|\sum\limits_{m\in\mathcal{M}}\bm{H}_{b,m}\bm{\Theta}_{m}\bm{G}_{m,k}^{{\dagger}}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}_{-b}}P_{b^{\prime}}|\sum\limits_{m^{\prime}\in\mathcal{M}}\bm{H}_{b^{\prime},m^{\prime}}\bm{\Theta}_{m^{\prime}}\bm{G}_{m^{\prime},k}^{{\dagger}}|^{2}+N_{0}^{2}}{\color[rgb]{0,0,0}{\bigg{)}}}$
(35) s.t.
$\displaystyle\theta^{{}^{\prime}}_{m,n}=z_{m,n}\frac{2\pi}{2^{\mu}}$ (35a)
$\displaystyle z_{m,n}\in\left\\{0,1,\cdots,(2^{\mu}-1)\right\\}$ (35b)

where $\theta^{{}^{\prime}}_{m,n}$ has been defined in equation (2) as the RIS
phase-shift angle. $z_{m,n}$ belongs to the matrix $\mathcal{Z}$, which will
decide the phase shift angle $\theta^{{}^{\prime}}_{m,n}$. Note that the
decision variable $z_{m,n}$ can only be selected from fixed integer values,
which means equation (35) is a Mixed Integer Nonlinear Programming (MINLP)
problem, then a surrogate optimization method is applied to solve this
problem.

Different than the FP-based method, the surrogate optimization treats the
optimization as a black-box problem[37]. It yields a reasonable and accurate
approximation for the original optimization problem by a surrogate model,
which avoids the complexity of designing a dedicated optimization solution.
The proposed surrogate optimization-based RIS phase control is summarized in
Algorithm 4. Note that there are many surrogate models and here we choose the
radial basis function model for its low computation complexity.

Algorithm 4 Surrogate optimization based RIS phase control

1: Initialize: Wireless channel and surrogate optimization parameters, and BS
transmission powers. Setting iteration number $y=1$, and initializing the
surrogate model.

2: repeat

3: Obtaining an estimated solution to equation (35) by optimizing the
surrogate.

4: Evaluating the surrogate model at the estimated solution produced in last
step.

5: Updating the radial basis function-based surrogate model using the new
high-fidelity model data.

6: $y=y+1$

7: until Problem (35) converges, or the iteration number $y$ reaches the
predefined maximum value.

8: Output: Maximum data transmission rate and optimal RIS phase shifts under
given transmission power of BSs.

TABLE I: Parameters settings Wireless network settings |  Traffic model
---|---
3GPP urban macro network |  20% Poisson distribution
Bandwidth: 20MHz |  80% constant bit rate
Number of RBs: 100 |  Learning and algorithm settings
Subcarriers in each RB: 12 |  Network layers: 4
Subcarrier bandwidth: 15kHz |  2 LSTM network as hidden layers
UE and BSs settings |  Hidden layer has 10 nodes
1 MBS, 3 SBSs |  Experience pool size: 600
MBS cell radius: 400m |  Initial network learning rate: 0.005
SBS cell radius: 100m |  Minibatch size: 180
MBS max Tx power: 20 W |  Discount factor: 0.3
SBS max Tx power: 6.3 W |  Epsilon value: 0.05
20 UE with random distribution |  Training frequency: 240 iterations
RIS settings |  Network copy: every 600 iterations
6 RIS deployed in the cell |  Time resolution difference $N_{T}$:10
Each RIS includes 10 elements |  Overloading penalty factor: 0.2

##  VII Performance Evaluation

###  VII-A Simulation Settings

(a) Power consumption of RIS and no-RIS cases

(b) Throughput comparison of RIS and no-RIS cases

(c) Energy efficiency comparison of RIS and no-RIS cases

(d) Daily SBS on/off status analyses

Figure 5: Performance comparison of sleep control and RIS.

(a) Convergence performance of surrogate optimization based RIS control

(b) Convergence performance of fractional programming (FP) based RIS control

(c) Power consumption comparison of different RIS control methods

(d) Throughput consumption comparison of different RIS control methods

(e) Energy efficiency comparison of different RIS control methods

(f) Daily SBS on/off status analyses of different RIS control methods

Figure 6: Performance comparison of difference RIS control strategies.

We consider a HetNet that includes 1 MBS and 3 SBSs. The radius of the MBS and
SBS are 400 m and 100 m, respectively. The carrier frequency for the MBS and
SBS are both 4 GHz[38]. The load-dependent power consumption factor is 4.7 and
2.6 for MBS and SBS, respectively[29]. The whole cell includes 20 UEs that are
randomly distributed. 6 RISs are uniformly distributed, and each contains 10
elements with 2 bits resolution. For learning settings, we apply 2 LSTM
networks as hidden layers for each controller. The hyperparameters such as
neural network learning rate and training frequency are selected by the grid
search method, which means we try different parameter combinations and find
the best results accordingly. The simulations are repeated 10 runs on MATLAB
with 95% confidence interval, and other settings are summarized in Table I.

###  VII-B Performance Comparison under Sleep Control and RIS

This section investigates the performance under sleep control and RIS
techniques. The proposed FP and Co-HDRL methods are deployed for RIS control
and sleep control, respectively. As shown in Fig.5, four cases are considered
by combining sleep control with RIS control.

Fig. 5(a) shows that sleep control can significantly save power consumption by
switching off SBSs intelligently. Combining sleep control with RIS can achieve
the lowest energy consumption. Moreover, Fig. 5(b) demonstrates that RIS can
greatly increase the average throughput, which can be explained by RIS’
capability to reshape the signal transmission path. Consequently, shown by
Fig. 5(c), sleep control + RIS produces the highest energy efficiency, no
sleep control + no RIS shows the lowest energy efficiency, and the other two
combinations present a comparable performance.

To better explain how RIS can help reduce energy consumption, we compare the
daily SBS on/off status of RIS and no-RIS condition in Fig.5(d). The vertical
axis is the probability of turning on SBSs, which indicates the power
consumption level during specific time slots. The blue shade represents the
preset traffic load pattern. Fig.5(d) shows that most SBSs are switched off
during the low-peak period (from 3:00 to 7:00) in both RIS and no-RIS cases.
However, when the traffic load increases to mid-peak and on-peak periods, no-
RIS case has to turn on most SBSs to satisfy the increasing traffic demand;
otherwise, the overloading may lead to a high penalty. By contrast, using RIS
can greatly increase the channel capacity, which means the MBS can still
switch off most SBSs to save energy without causing overloading.

In summary, the simulations in Fig.5 demonstrate that combining sleep control
and RIS can greatly save energy consumption, increase throughput and achieve
higher energy efficiency.

(a) Energy efficiency comparison of Co-HDRL and HDRL

(b) Reward comparison of Co-HDRL and HDRL

(c) Stationarity comparison of different action selection strategies

(d) Daily SBS on/off status of Co-HDRL and HDRL

(e) Daily SBS on/off status of Co-HDRL under various traffic load

(f) Average data rate under various RIS elements numbers and phase shift
resolutions

Figure 7: Performance analyses of Co-HDRL and HDRL.

###  VII-C RIS Control Strategy Analyses

Consequently, we compare the proposed FP-based RIS control method with the
surrogate optimization strategy.

Firstly, Fig. 6(a) shows the convergence performance of surrogate optimization
in one episode. One can observe that surrogate optimization relies on random
and adaptive samples to find better solutions. The surrogate model is
constantly updated in each iteration to provide a more accurate estimation of
the given problem. By contrast, in Fig. 6(b), our proposed FP method reaches a
much faster convergence. This faster convergence can be explained by the
dedicated designed FP optimization model for RIS control. In the proposed FP,
the alternative optimization method guarantees that the next iteration can
always perform better than the former iterations.

Similarly, we consider the combination of sleep control with different RIS
control methods, and the power consumption, throughput and EE results are
given in Fig. 6(c), (d) and (e), respectively. Combining FP-based RIS control
with sleep control can bring lower power consumption, higher throughput and EE
than other cases. The proposed FP method can achieve 35% lower power
consumption and 16% higher EE when the peak traffic load is 8 Mbps. Moreover,
the SBS on/off status is shown in Fig. 6(f). It demonstrates that FP-based RIS
control can maintain low power consumption even in peak-load periods ( from
17:00 to 23:00). On the contrary, surrogate-based RIS control has to turn on
SBSs to adapt to the increasing transmission demand to prevent overloading
penalty.

###  VII-D Machine Learning Algorithm Comparison

In this section, we compare the proposed Co-HDRL algorithm with conventional
HDRL, which aims to prove that the cooperation strategy can bring better
performance. The EE and reward performance of 200 episodes are given in
Fig.7(a) and (b), in which the proposed Co-HDRL algorithm achieves a higher EE
and reward than the HDRL baseline. In Co-HDRL, we apply the cross-entropy-
based policy for meta-controller to monitor the stability of sub-controllers,
and the correlated equilibrium is deployed for joint action selection of sub-
controllers. These cooperation methods enable higher stability for sub-
controller action selection and training, which brings better performance than
conventional HDRL. In Fig.7(c), we further present the stationarity of various
low-level action selection strategies. The proposed multi-agent correlated
equilibrium enables a more stable action selection than independent action
selection that is applied in HDRL, which is indicated by a lower stationarity
metric. Meanwhile, Fig.7(c) demonstrates that correlated equilibrium can reach
a stable state much faster than the independent action selection method.

Meanwhile, the satisfying performance of Co-HDRL can also be observed in
Fig.7(d), in which the Co-HDRL switches on/off SBSs dynamically during on-peak
and off-peak periods. By contrast, HDRL produces the same sleep control
decision regardless of different traffic load conditions ( traffic patterns
shown by the light blue shade). Moreover, the Co-HDRL performance from 4 to 10
Mbps is presented in Fig.7(e), in which the proposed Co-HDRL make decisions
intelligently under various traffic load level. For example, when the traffic
load level reaches 10 Mbps, the meta-controller turns on most SBSs to serve
the high transmission demand. On the other hand, when the traffic load is 4
Mbps, most SBSs are switched off to reduce power consumption.

We show the average data rate under various RIS element numbers and phase
shift resolutions in Fig.7(f). It demonstrates that increasing the number of
RIS elements can improve the system data rate. Meanwhile, higher phase shift
resolutions can achieve a higher data rate, because the phase can be more
accurately shifted.

###  VII-E Performance under various cell sizes and SBS numbers.

Finally, this subsection investigates the algorithm performance with different
cell sizes and numbers of SBSs. Specifically, Fig. 8(a) shows that increasing
cell size will lower EE and increase power consumption, which can be explained
by lower SINR due to a larger cell radius. Meanwhile, the EE in Fig. 8(b)
increases first and then decreases. In the increasing phase, the MBS has
enough channel capacity to support the UEs that are previously associated with
the SBSs in sleep mode. However, when the number of SBSs keeps increasing, the
probability of turning on SBSs increases as illustrated in Fig. 8(b). The main
reason is that MBS cannot serve all the UEs when SBSs enter sleep mode, and
some SBSs must keep active to fulfill the traffic demand. Consequently, the
energy efficiency becomes lower with the increasing number of SBSs.

(a) Energy efficiency and power consumption under various cell radius.

(b) Energy efficiency and SBS active probability under different SBS numbers.

Figure 8: Algorithm performance under various cell sizes and SBS numbers.

##  VIII Conclusion

Machine learning is a promising technique for network management in 5G and
future 6G networks. In this work, we propose a cooperative hierarchical deep
reinforcement learning method for RIS-assisted energy efficient RAN, including
a cross-entropy enabled meta-controller policy and cooperative action
selections for sub-controllers. Besides, a fractional programming-based RIS
control strategy is introduced for phase shift control. The proposed
architecture enables joint sleep, power, and RIS control with different
timescales, increasing the flexibility for RAN management. This is
particularly important for 6G and O-RAN where intelligence will be more
abundant and control will span several time scales. The proposed method is
compared with conventional hierarchical deep reinforcement learning and
surrogate optimization-based RIS control, and the simulations show that the
proposed algorithms achieve better energy efficiency. In the future, we will
focus more on the RIS control methods, and investigate how the RIS location
affects the network performance. In addition, we will investigate the
effectiveness of the proposed method through experiments and empirical
studies.

##  Appendix A Proof of Proposition 1

In equation (9), when $\psi_{k}$ is fixed, note that $f_{2}$ is a
differentiable concave function for $\bm{\beta}$ with
$\frac{\partial^{2}f_{2}}{\partial\bm{\beta}^{2}}<0$. Then we have

$\displaystyle\frac{\partial
f_{2}(\bm{\Theta}_{m},\bm{\beta})}{\partial\beta_{k}}$
$\displaystyle=\frac{\partial\bigg{(}b_{k}\log(1+\beta_{k})-b_{k}\beta_{k}+\frac{b_{k}(1+\beta_{k})\psi_{k}}{1+\psi_{k}}\bigg{)}}{\partial\beta_{k}}$
$\displaystyle=\frac{b_{k}}{1+\beta_{k}}-b_{k}+\frac{b_{k}\psi_{k}}{1+\psi_{k}}$
$\displaystyle=b_{k}(\frac{\psi_{k}}{1+\psi_{k}}-\frac{\beta_{k}}{1+\beta_{k}})$
$\displaystyle=b_{k}\frac{\psi_{k}-\beta_{k}}{(1+\psi_{k})(1+\beta_{k})}$

Setting $\frac{\partial f_{2}}{\partial\beta_{k}}$ and obviously
$\beta^{*}_{k}=\psi_{k}$, where $\beta^{*}_{k}$ is the optimal $\beta_{k}$.
Substituting ${\beta}^{*}_{k}$ back to equation (9) will recover (8). As such,
the optimal values of these two equations are equivalent.

##  Appendix B Proof of Proposition 2

In equation (12), for ease of notations, we use $A_{k}$ to represent
$\sqrt{b_{k}(1+\beta_{k})}(\bm{\hat{\Theta}}\bm{V}_{b,k})$, and $B_{k}$ to
represent
$\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{\hat{\Theta}}\bm{V}_{b^{\prime},k}|^{2}+N_{0}^{2}$.
Then, we can rewrite equation (13) as
$2\text{Re}\\{\eta_{k}^{{\dagger}}A_{k}\\}-\eta_{k}^{{\dagger}}B_{k}\eta_{k}$,
which can be reformulated by
$\eta_{k}^{{\dagger}}A_{k}+A_{k}^{{\dagger}}\eta_{k}-\eta_{k}^{{\dagger}}B_{k}\eta_{k}$.
Then we rewrite it as
$A_{k}^{{\dagger}}B^{-1}_{k}A_{k}-(\eta_{k}^{{\dagger}}-B^{-1}_{k}A_{k}^{{\dagger}})B_{k}(\eta_{k}-B^{-1}_{k}A_{k})$.
It is obvious that the optimal value is $\eta^{*}_{k}=B^{-1}_{k}A_{k}$, which
means

$\eta^{*}_{k}=\frac{\sqrt{b_{k}(1+\beta_{k})}\bm{\hat{\Theta}}\bm{V}_{b,k}}{\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{\hat{\Theta}}\bm{V}_{b^{\prime},k}|^{2}+N_{0}^{2}}.$

Then substituting $\eta^{*}_{k}$ back to equation (13) will get the optimal
value as
$\sum_{k\in\mathcal{K}}\frac{b_{k}(1+\beta_{k})|\bm{\hat{\Theta}}\bm{V}_{b,k}|^{2}}{\sum\limits_{b^{\prime}\in\mathcal{B}}|\bm{\hat{\Theta}}\bm{V}_{b^{\prime},k}|^{2}+N_{0}^{2}}$,
which is exactly the objective of equation (12). As such, the equivalence is
demonstrated.

## Acknowledgement

We would like to thank Dr. Long Kong for his useful discussions when starting
this work.

## References

  * [1] M. Usama and M. Erol-Kantarci, “A survey on recent trends and open issues in energy efficiency of 5G,” _Sensors_ , vol. 19, no. 14, pp. 1–23, Jul. 2019.
  * [2] J. Wu, Y. Zhang, M. Zukerman, and E. K.-N. Yung, “Energy-efficient base-stations sleep-mode techniques in green cellular networks: A Survey,” _IEEE Communications Surveys and Tutorials_ , vol. 17, no. 2, pp. 803–826, 2ndquater 2015.
  * [3] M. Di Renzo, A. Zappone, M. Debbah, M.-S. Alouini, C. Yuen, J. de Rosny, and S. Tretyakov, “Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 11, pp. 2450–2525, Nov. 2020.
  * [4] M. Elsayed and M. Erol-Kantarci, “Ai-enabled future wireless networks: Challenges, opportunities, and open issues,” _IEEE Vehicular Technology Magazine_ , vol. 14, no. 3, pp. 70–77, Sep. 2019.
  * [5] G. Vallero, D. Renga, M. Meo, and M. A. Marsan, “Greener RAN operation through machine learning,” _IEEE Transaction on Network and Service Management_ , vol. 16, no. 3, pp. 896–908, Sep. 2019.
  * [6] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 11, pp. 5394–5409, Nov. 2019.
  * [7] V. G. Douros and G. C. Polyzos, “Review of some fundamental approaches for power control in wireless networks,” _Computer Communications_ , vol. 34, no. 13, pp. 1580–1592, Aug. 2011.
  * [8] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT Press, 2018.
  * [9] Y. Geng, E. Liu, R. Wang, and Y. Liu, “Hierarchical reinforcement learning for relay selection and power optimization in two-hop cooperative relay network,” _IEEE Transactions on Communications_ , vol. 70, no. 1, pp. 171–184, Jan. 2022.
  * [10] S. Liu, J. Wu, and J. He, “Dynamic multichannel sensing in cognitive radio: Hierarchical reinforcement learning,” _IEEE Access_ , vol. 9, pp. 25 473–25 481, Feb. 2021.
  * [11] M. H. Buysse, K. Mets, and S. Latré, “Hierarchical reinforcement learning: A survey and open research challenges,” _Machine Learning and Knowledge Extraction_ , vol. 4, no. 1, pp. 172–221, Feb. 2022.
  * [12] H. Guo, Y. Liang, J. Chen, and E. G. Larsson, “Weighted sum-rate maximization for reconfigurable intelligent surface aided wireless networks,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 5, pp. 3064–3076, May 2020.
  * [13] K. Shen and W. Yu, “Fractional programming for communication systems—part i: Power control and beamforming,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 10, pp. 2616–2630, May 2018.
  * [14] P. Iturria-Rivera, H. Zhang, H. Zhou, S. Mollahasani, and M. Erol-Kantarci, “Multi-agent team learning in virtualized open radio access networks (o-ran),” _Sensors_ , vol. 22, no. 14, pp. 1–13, Sep. 2022\.
  * [15] H. Zhou and M. Erol-Kantarci, “Ran resource slicing in 5G using Multi-agent correlated Q-Learning,” in _Proc. IEEE PIMRC_ , Sep. 2021, pp. 1–6.
  * [16] M. Elsayed and M. Erol-Kantarci, “Deep reinforcement learning for reducing latency in mission critical services,” in _Proc. 2018 IEEE GLOBECOM_ , Dec. 2018, pp. 1–6.
  * [17] N. Piovesan, D. López-Pérez, M. Miozzo, and P. Dini, “Joint load control and energy sharing for renewable powered small base stations: A machine learning approach,” _IEEE Trans. Green Commun. Netw._ , vol. 5, no. 1, pp. 512–525, Mar. 2021.
  * [18] H. Zhou, M. Erol-Kantarci, and V. Poor, “Learning from peers: Transfer reinforcement learning for joint radio and cache resource allocation in 5g network slicing,” _arXiv:2109.07999_ , pp. 1–15, Sep. 2021.
  * [19] Q. Wu, X. Chen, Z. Zhou, L. Chen, and J. Zhang, “Deep reinforcement learning with spatio-temporal traffic forecasting for data-driven base station sleep control,” _IEEE/ACM Trans. Netw._ , vol. 29, no. 2, pp. 935–948, Apr. 2021.
  * [20] J. Liu, B. Krishnamachari, S. Zhou, and Z. Niu, “Surrogate-based analysis and optimization,” _IEEE Internet of Things Journal_ , no. 6, pp. 4273–4282, Jun. 2018.
  * [21] P. Iturria-Rivera, M. Elsayed, M. Bavand, R. Gaigalas, S. Furr, and M. Erol-Kantarci, “Hierarchical deep q-learning based handover in wireless networks with dual connectivity,” in _Proc. of IEEE GLOBECOM (Accepted)_ , Dec. 2022, p. 1–6.
  * [22] O. Nachum, S. Gu, H. Lee, and S. Levine, “Data-efficient hierarchical reinforcement learning,” in _Proc. of Advances in Neural Information Processing Systems 31_ , Dec. 2018, pp. 1–11.
  * [23] H. Zhou, L. Kong, M. Elsayed, M. Bavand, R. Gaigalas, S. Furr, and M. Erol-Kantarci, “Hierarchical reinforcement learning for ris-assisted energy-efficient ran,” in _Proc. of IEEE GLOBECOM_ , Dec. 2022, p. 1–6.
  * [24] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen, “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 8, pp. 4157–4170, Aug. 2019.
  * [25] L. You, J. Xiong, D. W. Ng, C. Yuen, W. Wang, and X. Gao, “Energy efficiency and spectral efficiency tradeoff in ris-aided multiuser mimo uplink transmission,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 1407–1421, Dec. 2020.
  * [26] H. Zhang, B. Di, L. Song, and Z. Han, “Reconfigurable intelligent surfaces assisted communications with limited phase shifts: How many phase shifts are enough?” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 4, pp. 4498–4502, Feb. 2020.
  * [27] P. Ren and M. Tao, “A decentralized sleep mechanism in heterogeneous cellular networks with qos constraints,” _IEEE Wireless Communications Letters_ , vol. 3, no. 5, pp. 509–512, Oct. 2014.
  * [28] S. Pateria, B. Subagdja, A. Tan, and C. Quek, “Hierarchical reinforcement learning: A comprehensive survey,” _ACM Computing Surveys_ , vol. 54, no. 5, pp. 1–35, Jun. 2022.
  * [29] G. Auer, V. Giannini, C. Desset, and et.al, “How much energy is needed to run a wireless network?” _IEEE Wireless Commun._ , vol. 18, no. 5, pp. 40–49, Oct. 2011.
  * [30] H. Zhou, A. Aral, I. Brandic, and M. Erol-Kantarci, “Multi-agent bayesian deep reinforcement learning for microgrid energy management under communication failures,” _IEEE Internet of Things Journal (Early Access)_ , pp. 1–14, Dec. 2021.
  * [31] R. Wang, R. Yu, B. An, and Z. Rabinovich, “I2hrl: Interactive influence-based hierarchical reinforcement learning,” in _Proc. of International Joint Conference on Artificial Intelligence_ , Jan. 2021, pp. 3131–3138.
  * [32] T. V. Erven and P. Harremos, “Rényi divergence and kullback-leibler divergence,” _IEEE Transactions on Information Theory_ , no. 7, pp. 3797–3820, Jul. 2014.
  * [33] D. MacKay, _Information Theory, Inference and Learning Algorithms_. Cambridge University Press, 2003.
  * [34] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” in _Proc. of Advances in Neural Information Processing Systems_ , Dec. 2018, p. 8792–8802.
  * [35] A. Greenwald and K. Hall, “Correlated q-learning,” in _Proc. of International Conference on Machine Learning_ , Aug. 2003, p. 84–89.
  * [36] H. Jiang, T. Kathuria, Y. T. Lee, S. Padmanabhan, and Z. Song, “A faster interior point method for semidefinite programming,” in _Proc. of IEEE Annual Symposium on Foundations of Computer Science_ , Nov. 2020, p. 1–9.
  * [37] N. V. Queipo, R. T. Haftka, R. V. W. Shyy, T. Goel, and P. K. Tucker, “Surrogate-based analysis and optimization,” _Progress in Aerospace Sciences_ , no. 1, pp. 1–28, Jan. 2005.
  * [38] M. Elsayed and M. Erol-Kantarci, “Reinforcement learning-based joint power and resource allocation for URLLC in 5G,” in _Proc. of IEEE Global Communications Conference (GLOBECOM)_. IEEE, Dec 2019, pp. 1–6.


* [61] Ivan Gonzalez, Lin Jiu, and Victor H. Moll. “An extension of the method of brackets. Part 2”. Open Mathematics, 18(1):983–995, 2020