# Hybrid Reinforcement Learning for STAR-RISs: A Coupled Phase-Shift Model
Based Beamformer

Ruikang Zhong, Yuanwei Liu,
Xidong Mu, Yue Chen,
Xianbin Wang, Lajos Hanzo Ruikang Zhong, Yuanwei Liu, and Yue Chen are with
the school of School of Electronic Engineering and Computer Science, Queen
Mary University of London, London E1 4NS, U.K. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>yue.chen@qmul.ac.uk). Xidong Mu is with the School of
Artificial Intelligence, Beijing University of Posts and Telecommunications,
Beijing, 100876, China. He is also with the school of School of Electronic
Engineering and Computer Science, Queen Mary University of London, London E1
4NS, U.K. (email:muxidong@bupt.edu.cn). Xianbin Wang is with Department of
Electrical and Computer Engineering, Western University, London, ON N6A5B9,
Canada (e-mail: xianbin.wang@uwo.ca). Lajos Hanzo is with the School of
Electronics and Computer Science, University of Southampton, Southampton SO17
1BJ, U.K. (e-mail:lh@ecs.soton.ac.uk). R. Zhong would like to acknowledge the
financial support of the China Scholarship Council (No.201908610187). L. Hanzo
would like to acknowledge the financial support of the Engineering and
Physical Sciences Research Council projects EP/P034284/1 and EP/P003990/1
(COALESCE) as well as of the European Research Council’s Advanced Fellow Grant
QuantCom (Grant No. 789028)

###### Abstract

A simultaneous transmitting and reflecting reconfigurable intelligent surface
(STAR-RIS) assisted multi-user downlink multiple-input single-output (MISO)
communication system is investigated. In contrast to the existing ideal STAR-
RIS model assuming an independent transmission and reflection phase-shift
control, a practical coupled phase-shift model is considered. Then, a joint
active and passive beamforming optimization problem is formulated for
minimizing the long-term transmission power consumption, subject to the
coupled phase-shift constraint and the minimum data rate constraint. Despite
the coupled nature of the phase-shift model, the formulated problem is solved
by invoking a hybrid continuous and discrete phase-shift control policy.
Inspired by this observation, a pair of hybrid reinforcement learning (RL)
algorithms, namely the hybrid deep deterministic policy gradient (hybrid DDPG)
algorithm and the joint DDPG & deep-Q network (DDPG-DQN) based algorithm are
proposed. The hybrid DDPG algorithm controls the associated high-dimensional
continuous and discrete actions by relying on the hybrid action mapping. By
contrast, the joint DDPG-DQN algorithm constructs two Markov decision
processes (MDPs) relying on an inner and an outer environment, thereby
amalgamating the two agents to accomplish a joint hybrid control. Simulation
results demonstrate that the STAR-RIS has superiority over other conventional
RISs in terms of its energy consumption. Furthermore, both the proposed
algorithms outperform the baseline DDPG algorithm, and the joint DDPG-DQN
algorithm achieves a superior performance, albeit at an increased
computational complexity.

###### Index Terms:

Beamforming, deep reinforcement learning (DRL), reconfigurable intelligent
surfaces (RISs), simultaneous transmitting and reflecting reconfigurable
intelligent surfaces (STAR-RISs)

##  I Introduction

Each generation of wireless networks has aimed for improving the quality of
service (QoS) [1]. While the fifth-generation (5G) network is being rolled
out, investigators turned to the next-generation (6G) research both in
academia and industry. In this context, reconfigurable intelligent surfaces
(RISs) have emerged as a competitive 6G component [2, 3]. They constitute a
low-cost solution for improving the propagation conditions for edge users [4]
and millimeter wave (mmWave) communications [5], as well as energy-efficient
communication [6]. Furthermore, RISs can also provide additional functions
beyond signal enhancement, such as integrated sensing and reflecting [7],
channel estimation [8], and user localization [9].

Due to their similar functions and roles, the performance of RISs and wireless
relays has been comprehensively compared in [10, 11]. Despite the appealingly
low complexity and noise figure [11], an undeniable fact is that the
reflecting-only RISs are likely to have a coverage disadvantage compared to
omnidirectional relays due to their $180^{\circ}$ half-plane reflection
limitation. However, in practical scenarios, users roam at both sides of the
RIS, and the reflecting-only RIS cannot provide signal enhancements for users
located at the back of the RIS. This deficiency of reflecting-only RISs
inspires the design of double-sided RISs to achieve $360^{\circ}$ coverage
boundary.

Hence, a simultaneous transmitting and reflecting reconfigurable intelligent
surface (STAR-RIS) model emerged as an ameliorated version of the reflecting-
only RIS [12]. As an advanced derivative, an additional transmission function
is facilitated by the simultaneous transmitting and reflecting (STAR)
elements. The STAR-RIS accommodate a number of STAR elements on a surface to
separate the incident electromagnetic waves into transmitted and reflected
signals. Consequently, the transmitted signals and reflected signals can form
a pair of sectors on both sides of the surface simultaneously, thereby solving
the limitations of RISs in terms of their coverage region. In contrast to the
reflecting-only RIS, which has to be in the vicinity of either the base
station (BS) or the users, STAR-RIS may be positioned flexibly. Furthermore,
if the BS or users are out of the optimal orientation of the RIS, the gain
provided by the reflecting-only RIS deteriorates distinctly [13]. Fortunately,
the emergence of STAR-RISs can relax the orientation requirement of (STAR)
RISs111The abbreviation ’(STAR) RISs’ refers to both reflecting-only RISs and
STAR-RISs., since the transmission and reflection sectors have the opposite
orientation.

###  I-A The State of the Art

Given its short history, the research of STAR-RIS is still in its infancy. Zhu
et al. [14, 15] presented the design of metasurfaces to control the amplitudes
and phase-shifts of the transmission (refraction) and reflection of
electromagnetic waves. The implementation of STAR elements was discussed in
[14], which laid a foundation of STAR-RISs. While Xu et al. [16] proposed a
physical model for the STAR-RIS and demonstrated that it is feasible to extend
reflecting-only RISs to STAR-RISs. Three protocols were conceived for STAR-
RISs in [12], namely the energy splitting, mode switching, and time switching
models. The phase-shift optimization problem of STAR-RISs was investigated in
[17], where a branch-and-bound based algorithm was proposed for maximizing the
downlink’s spectral efficiency. The coverage performance of STAR-RIS networks
was investigated by Wu et al. [18], and a one-dimensional search-based
algorithm was proposed for optimizing the coverage of STAR-RISs. The
simulation results in [18] confirmed that STAR-RISs exhibit significant
coverage advantage over reflecting-only RISs. By considering a coupled
transmitting and reflecting model, Liu et al. [19] proposed an efficient
element-wise alternating optimization algorithm for minimizing the transmit
power of a STAR-RIS assisted non-orthogonal multiple access (NOMA) network.
Unfortunately, although several STAR-RIS models have been proposed in the
aforementioned research, these models assume arbitrary transmission
coefficients (TCs) and reflection coefficients (RCs) for the STAR elements.
However, as pointed out in [20], a passive device such as the STAR-element is
not likely to provide arbitrary phase and amplitude responses.

Deep learning (DL) and deep reinforcement learning (DRL) are capable of
intelligently managing RISs [21]. DL is primarily used for channel estimation
in RIS-assisted networks, in order to extract the channel state information
(CSI) in support of RIS-based passive beamforming. Gao et al. [22] employed a
synthetic deep neural network (DNN) for sequentially estimating the BS-RIS
channel and RIS-user channel, while reducing the pilot overhead and
guaranteeing the estimation accuracy. DL was also invoked for the phase-shift
control of RISs. For example, an unsupervised DL algorithm was proposed for
joint active and passive beamforming optimization in [23]. In terms of DRL, by
adopting a deep deterministic policy gradient (DDPG) algorithm, Huang et al.
[24] conceived the joint optimization of the transmit beamforming matrix of
the BS and the phase-shift matrix of the RIS. Yang et al. [25] proposed a deep
Q network (DQN) for secure beamforming guarding against eavesdroppers in
dynamic environments. Their simulation results verified that the DQN approach
is capable of enhancing the secrecy rate and the satisfaction probability of
users. Furthermore, a DRL scheme was invoked in [26] for optimizing a RIS-
assisted NOMA network, where a long short-term memory (LSTM) based echo state
network (ESN) algorithm collaborated with a decaying double deep Q-network
(D3QN) for intelligently controlling the RIS according to the users’ data
demand. In [27], a proximal policy optimization (PPO) algorithm was developed
for minimizing the expected Age-of-Information (AoI) for an aerial RIS.
Specifically for the STAR-RIS scenario, a federated learning algorithm was
proposed in [28] for maximizing the achievable data rate of a STAR-RIS
assisted heterogeneous NOMA network.

###  I-B Motivations

Although STAR-RISs have the aforementioned advantages, the effective design of
the RC and TC of STAR-RISs has become a new challenge. Firstly, the STAR-RIS
requires joint transmission and reflection beamforming, which is exceedingly
more complex than reflection-only beamforming. What aggravates the situation
further is that the STAR-RISs cannot independently adjust the TCs and RCs in
practice, since the electric and magnetic impedances are unlikely to leave
arbitrary values, but they depend on the electromagnetic properties of the
STAR elements [14]. Furthermore, the coupling of the TCs and RCs requires a
hybrid continuous and discrete control scheme for the phase-shift design.
Given the above-mentioned adversities, it is a challenge to jointly solve the
transmission and reflection beamforming problem for STAR-RISs, especially
considering that the existing convex optimization and machine learning
solutions basically only support either continuous or discrete control.
Although several hybrid algorithms have been proposed in the field of computer
science [29, 30, 31], they are designed for minuscule action dimensions. For
example, only four discrete actions were assumed in [30] since gaming
controllers usually have four buttons. However, the possible number of actions
can be $a^{N}$ for the STAR-RIS scenario, where $a$ represents the possible
number of actions for a single STAR element and $N$ represents the total
number of elements employed. Based on the current assumptions, prototypes of
(STAR) RISs are likely to have a massive number of elements, which implies
that the action dimension of STAR-RISs substantially exceeds the design in
existing algorithms. Since there are lacking suitable hybrid algorithms for
the optimization of STAR-RISs, we propose two hybrid reinforcement learning
(RL) algorithms for joint active and passive beamforming design for the BS and
the STAR-RIS.

###  I-C Our contributions

In this paper, we are committed to establishing a practical STAR-RIS model
having coupled TCs and RCs, which can provide basis for potential research.
According to the proposed model, we embark upon the investigation and
optimization of the performance of STAR-RISs. Specifically, we provide the
following contributions.

  * •

We conceive a STAR-RIS model for broadening the coverage of reflecting-only
RISs. Specifically, the practical electromagnetic property of STAR elements
are considered, resulting in a coupled phase-shift of the transmission and
reflection. Based on the proposed model, a joint active and passive
beamforming problem that requires hybrid control for the phase-shift and
amplitude is formulated for minimizing the long-term power consumption.

  * •

We propose a hybrid DDPG algorithm for solving the hybrid control problem
caused by the energy splitting nature of STAR elements. The hybrid control is
carried out by mapping each output node of the actor network to the
transmission and reflection actions of each STAR element. The proposed hybrid
DDPG solution provides high-dimensional continuous and discrete phase-shift
optimization for STAR-RISs.

  * •

We develop a joint DDPG-DQN algorithm as a high performance-solution. The
joint DDPG-DQN scheme can handle hybrid control by employing two collaborated
agents, where a DDPG agent is in charge of the continuous control and a DQN
agent is invoked for the discrete control.

  * •

The performance of the STAR-RIS relying on the proposed algorithms is
evaluated by computer simulation revealing that it outperforms both the
reflecting-only and the double spliced RISs. Furthermore, the hybrid DDPG
algorithm outperforms its plain DDPG counterpart without increasing its
complexity, while the joint DDPG-DQN algorithm attains optimality at an
increased complexity.

###  I-D Organizations

Section II illustrates the model of our STAR-RIS assisted wireless network,
including the coupled phase-shift model, channel model, signal model, and the
problem formulation. Section III introduces the hybrid DDPG algorithm as a low
complexity solution, while Section IV presents the high-performance
amalgamated DDPG-DQN algorithm. Section V provides the performance analysis of
the proposed framework and algorithms. Section VI concludes the paper.

TABLE I: Key Notations Parameter | Description
---|---
$M,N$ | number of antenna and STAR elements
$K,U,I$ |  number of total users, $\mathcal{R}$ users and $\mathcal{T}$ users
$\beta_{\mathcal{R},n}$ | amplitude response for reflection
$\mathbf{\Theta}_{\mathcal{R}},\mathbf{\Theta}_{\mathcal{T}}$ | phase shift of for reflection and transmission
$\bm{H}_{b,\mathcal{R}},\bm{H}_{b,\mathcal{T}}$ |  channel from the BS to $\mathcal{R}\&\mathcal{T}$ users
$\bm{H}_{b,r}$ | channel from the BS to the STAR-RIS
$\bm{H}_{r,\mathcal{R}},\bm{H}_{r,\mathcal{T}}$ |  channel from the STAR-RIS to $\mathcal{R}\&\mathcal{T}$ users
$\mathbf{w}_{k,t}$ |  beamforming vector for user $k$ at time $t$
$\mathbf{x}_{b,t}$ | transmitted signal at the BS
$\mathbf{x}_{r,t}$ | received signal at the STAR-RIS
$y_{u,t},y_{i,t}$ |  received signal at user $\mathcal{R}_{u}$ and $\mathcal{T}_{i}$
$\gamma_{u,t},\gamma_{i,t}$ |  SINR at user $\mathcal{R}_{u}$ and $\mathcal{T}_{i}$
$R_{k,t}$ |  achievable data rate for user $k$
$\mathbf{s}_{t}$ | state for DRL agents
$\mathbf{a}^{c}_{t},\mathbf{a}^{d}_{t}$ | continuous and discrete actions
$\mathbf{a}^{h}_{t}$ | hybrid actions
$r_{t}$ | reward for DRL agent
$\bm{\omega}^{\mu}_{t},\bm{\omega}^{Q}_{t}$ | parameters for actor and critic networks
$\bm{\omega}^{\mathcal{Q}}_{t}$ | parameters for the deep Q network
$\bm{\omega}^{\mu^{\prime}}_{t},\bm{\omega}^{Q^{\prime}}_{t},\bm{\omega}^{\mathcal{Q}^{\prime}}_{t}$ | parameters for target networks

##  II System Model

###  II-A Model of STAR-RISs

We employ an energy splitting model for supporting simultaneous transmission
and reflection [12], where the STAR-RIS is capable of splitting the incident
signal into the transmitted and reflected signals, partitioning the space into
the transmission and reflection zones. Mobile users can be served by the
transmitted or reflected signal, respectively, depending on which region they
happen to be roaming in. In order to perform joint beamforming to covering
both the transmission and reflection sectors, the TC and RC of each STAR
element have to be appreciatively integrated, which are denoted as
$\beta_{\mathcal{R},n}e^{j\theta_{\mathcal{R},n}}$ and
$\beta_{\mathcal{T},n}e^{j\theta_{\mathcal{T},n}},n=1,2,...,N$.

It is worth noting that for any STAR element, the TCs an RCs are determined by
its resistance and reactance. Therefore, it is non-trivial to independently
adjust the coefficients. For a given RC of
$\beta_{\mathcal{R},n}e^{j\theta_{\mathcal{R},n}}$, according to the
conservation of energy222We assume that there is no energy loss in the process
of transmission and reflection. The energy loss has an impact on the phase
shift relationship between transmission and reflection [14], which would make
the beamforming design more complex. We would consider the non-conservation
model in our future work., we have
$\beta_{\mathcal{T},n}=\sqrt{1-\beta_{\mathcal{R},n}^{2}}$. Then, simplifying
RC $\beta_{\mathcal{R},n}$ as $\beta_{n}$, the TC can be calculated as
$\sqrt{1-\beta_{n}^{2}}e^{j\theta_{\mathcal{T},n}}$. As pointed out in [14],
for STAR elements, the coupling between the TC’s phase-shift
$\theta_{\mathcal{T},n}$, the RC’s phase-shift $\theta_{\mathcal{R},n}$ and
the amplitude $\beta_{n}$ follows a relationship as

$\displaystyle\beta_{n}\sqrt{1-\beta_{n}^{2}}\text{
cos}(\theta_{\mathcal{R},n}-\theta_{\mathcal{T},n})=0.$ (1)

Thus, for a STAR-RIS having $N$ elements, the transmission and reflection
matrices have a diagonal structure given by

$\displaystyle{\mathbf{\Theta}_{\mathcal{R}}}={\rm{diag}}\left({{\beta_{1}e^{j{\theta_{\mathcal{R},1}}}},{\beta_{2}e^{j{\theta_{\mathcal{R},2}}}},\cdots,{\beta_{N}e^{j{\theta_{\mathcal{R},N}}}}}\right){\text{}},$
(2)
$\displaystyle{\mathbf{\Theta}_{\mathcal{T}}}={\rm{diag}}\left({{\sqrt{1-\beta_{1}^{2}}e^{j{\theta_{\mathcal{T},1}}}},\cdots,{\sqrt{1-\beta_{N}^{2}}e^{j{\theta_{\mathcal{T},N}}}}}\right){\text{}}.$
(3)

###  II-B System Description

We consider a downlink scenario of Fig. 1, where the AP is equipped with $M$
antennas, and the STAR-RIS has $N$ STAR elements. There are $K$ randomly
roaming users and each having a single antenna. Each of the $N$ STAR elements
has the amplitude response $\beta_{n},n=1,2,...,N$. We denote the locations of
the BS, RIS, and users as ${\left({{x_{b}},{y_{b}},{z_{b}}}\right)^{T}}$,
${\left({{x_{r}},{y_{r}},{z_{r}}}\right)^{T}}$, and
${\left({{x_{k}},{y_{k}},{z_{k}}}\right)^{T}}$, respectively. The STAR-RIS
naturally partitions the users into two groups according to their locations.
The users between the BS and the STAR-RIS receive direct signals from the BS
and reflected signals from the STAR-RIS. This fraction of the users in the
reflective region of the STAR-RIS are denoted by $\mathcal{R}_{u}$. For
simplicity, we term this set of users as $\mathcal{R}$ users in the rest of
the text. Correspondingly, the users served by direct BS signals and
transmitted STAR-RIS signals is denoted by $\mathcal{T}_{i}$. The number of
users obeys $K=U+I$, where $U$ and $I$ are the numbers of $\mathcal{R}$ users
and $\mathcal{T}$ users333For simplicity of equations, the subscript
$k=1,2,...,K$ refers to any user, user $\mathcal{R}_{u},u=1,2,...,U$ or
$\mathcal{T}_{i},i=1,2,...,I$ refers to a $\mathcal{R}$ user or $\mathcal{T}$
user..

Figure 1: System model of STAR-RIS assisted wireless networks

###  II-C Channel Model

In the STAR-RIS scenario, multiple channels have to be considered, including
the BS to STAR-RIS channel $\bm{H}_{b,r}\in{\mathbb{C}^{M\times N}}$, the
direct channel spanning the BS to $\mathcal{R}$ and $\mathcal{T}$ users
$\bm{H}_{b,\mathcal{R}}\in{\mathbb{C}^{M\times
U}},\bm{H}_{b,\mathcal{T}}\in{\mathbb{C}^{M\times I}}$, and the channel
impinging from the STAR-RIS to $\mathcal{R}$ and $\mathcal{T}$ users
$\bm{H}_{r,\mathcal{R}}\in{\mathbb{C}^{N\times
U}},\bm{H}_{r,\mathcal{T}}\in{\mathbb{C}^{N\times I}}$. For each specific user
$\mathcal{R}_{u}$ and $\mathcal{T}_{i}$, the direct channels, and the
$\mathcal{T}\&\mathcal{R}$ channels can be denoted as
$\bm{h}_{b,u}\in{\mathbb{C}^{M\times 1}},\bm{h}_{b,i}\in{\mathbb{C}^{M\times
1}},\bm{h}_{r,u}\in{\mathbb{C}^{N\times 1}}$ and
$\bm{h}_{r,i}\in{\mathbb{C}^{N\times 1}}$, respectively.

All channels are assumed to follow the quasi-static block fading model, where
the fading coefficient remains constant in each time slot (TS) $t$. We assume
that the channel $\bm{H}_{b,r}$ has a line-of-sight (LoS) path and obeys to
the Rician distribution, since the BS and STAR-RIS have a LoS component owing
to their selected positions. Thus, upon considering the path loss and the
small scale fading, the Rician channel can be formulated as

$\displaystyle{\bm{H}_{b,r,t}}\\!=\\!\\!\sqrt{\mathcal{L}_{b,r,t}\left({{d_{b,r,t},f_{c}}}\right)}\\!\left(\\!\\!{\sqrt{\frac{{{K}}}{{{K}+1}}}{{\bm{L}_{b,r,t}}}\\!\\!+\\!\\!\sqrt{\frac{1}{{{K}+1}}}{{\bm{G}_{b,r,t}}}}\\!\\!\right),$
(4)

where $\mathcal{L}_{b,r,t}$ represents the pathloss in the power domain, $K$
is the Rician factor, $\mathbf{G}_{b,r,t}\sim\mathcal{CN}(0,1)$ denotes the
scattered paths, and ${{\bm{L}_{b,r,t}}}$ represents the LoS path between the
BS and the RIS. Furthermore, we have
${{\bm{L}_{b,r,t}}}=\bm{a}_{r}(\varphi^{A},\psi^{A})\bm{a}_{b}(\varphi^{D})^{H},\bm{a}_{b}\in\mathbb{C}^{M\times
1},\bm{a}_{r}\in\mathbb{C}^{N\times 1}$, where $\varphi^{A/D}\in[0,2\pi)$ and
$\psi^{A}\in[-2/\pi,2/\pi)$ represent the azimuth and elevation angle-of-
arrival (AoA)/angle-of-departure (AoD),
$\varphi^{A}=\arcsin(\frac{{y_{r}}-{y_{b}}}{\sqrt{{{\left({{x_{r}}-{x_{b}}}\right)}^{2}}+{{\left({{y_{r}}-{y_{b}}}\right)}^{2}}}})$,
$\varphi^{D}=\pi/2-\varphi^{A}$, and
$\psi^{A}=\arcsin(\frac{{{z_{r}}-{z_{b}}}}{{\sqrt{{{\left({{x_{r}}-{x_{b}}}\right)}^{2}}+{{\left({{y_{r}}-{y_{b}}}\right)}^{2}}}}})$.

We assume that the BS has linear array antennas and the STAR RIS has a uniform
planar array of reflective elements, where the antenna space and the
reflective element space are $d_{a}$ and $d_{e}$. According to [32], we have

$\displaystyle a_{b}[m]=$ $\displaystyle
e^{j2\pi(m-1)d_{a}\sin(\varphi^{D})/\lambda},$ (5) $\displaystyle a_{r}[n]=$
$\displaystyle e^{j2\pi(n-1)d_{e}(\lfloor
n/N_{x}\rfloor\sin(\varphi^{A})\sin(\psi^{A})}$ $\displaystyle e^{(n-\lfloor
n/N_{x}\rfloor N_{x})\sin(\varphi^{A})\cos(\psi^{A}))/\lambda},$ (6)

where $M_{x}$ denotes the number of reflective elements in each row and
$\lfloor\rfloor$ is the floor function.

The pathloss $\mathcal{L}$ follows the urban propagation model presented in
3GPP specification TR 36.873 [33]. For the path with LoS, the path loss can be
given by

$\displaystyle\mathcal{L}_{\text{LoS}}\left(d,f_{c}\right)=22.0\log_{10}{d}+28.0+20\log_{10}{f_{c}},$
(7)

where $d$ represents the 3D distance between the transmitter and the receiver,
while $f_{c}$ is the carrier frequency. For the NLoS propagation, the path
loss is given by

$\displaystyle\mathcal{L}_{\text{NLoS}}=\max[\mathcal{L}_{\text{LoS}}\left(d,f_{c}\right),\mathcal{L}_{\text{NLoS}}\left(d,f_{c}\right)],$
(8) $\displaystyle\mathcal{L}_{\text{NLoS}}\left(d,f_{c}\right)=$
$\displaystyle 36.7\log{10}{d}+22.7$
$\displaystyle+26\log{10}{f_{c}}-0.3(z_{r}-1.5).$ (9)

On the other hand, due to the random movements of users, LoS propagation may
not necessarily be guaranteed, regardless whether the transmitting side is the
BS or the STAR-RIS. Therefore,
$\bm{H}_{b,{\mathcal{R}},t},\bm{H}_{b,\mathcal{T},t},\bm{H}_{r,{\mathcal{R}},t}$
and $\bm{H}_{r,{\mathcal{T}},t}$ are assumed to be NLoS channels and follow
Rayleigh fading, which can be expressed as
$\bm{H}_{b,{\mathcal{R}},t}\\!=\\!\sqrt{\mathcal{L}_{b,\mathcal{R},t}}\bm{G}_{b,{\mathcal{R}},t},\bm{H}_{b,{\mathcal{T}},t}\\!=\\!\sqrt{\mathcal{L}_{b,\mathcal{T},t}}\bm{G}_{b,{\mathcal{T}},t},\bm{H}_{r,{\mathcal{R}},t}\\!=\\!\sqrt{\mathcal{L}_{r,\mathcal{R},t}}\bm{G}_{r,{\mathcal{R}},t},\bm{H}_{r,{\mathcal{T}},t}\\!=\\!\sqrt{\mathcal{L}_{r,\mathcal{T},t}}\bm{G}_{r,{\mathcal{T}},t}$.

###  II-D Signal Model

We denote the information sequence and the active beamforming vectors for user
$k$ at the BS by $s_{k,t}$ and $\mathbf{w}_{k,t},\in{\mathbb{C}^{M\times K}}$.
The signal transmitted at TS $t$ can be expressed as

$\displaystyle\mathbf{x}_{b,t}=\sum\limits_{k=1}^{K}\mathbf{w}_{k,t}s_{k,t}.$
(10)

Then, the incident signal at the STAR-RIS is given by

$\displaystyle\mathbf{x}_{r,t}=\bm{H}_{b,r,t}\sum\limits_{k=K}^{U}\mathbf{w}_{k,t}s_{k}+n_{0},$
(11)

where $n_{0}$ represents the Gaussian noise, and the received signal of user
$\mathcal{R}^{u}$ is given by

$\displaystyle
y_{u,t}=[\bm{h}_{b,u,t}+\bm{h}_{r,u,t}\mathbf{\Theta}_{\mathcal{R},t}\bm{H}_{b,r,t}]\sum\limits_{u=1}^{U}\mathbf{w}_{u,t}s_{u,t}+n_{0}.$
(12)

Correspondingly, the received signal of user $\mathcal{T}^{i}$ can be
represented in a similar form as

$\displaystyle
y_{i,t}=[\bm{h}_{b,i,t}+\bm{h}_{r,i,t}\mathbf{\Theta}_{\mathcal{T},t}\bm{H}_{b,r,t}]\sum\limits_{i=1}^{I}\mathbf{w}_{i,t}s_{i,t}+n_{0}.$
(13)

Given the received signal, the signal-to-interference-plus-noise ratio (SINR)
of user $\mathcal{R}^{u}$ and $\mathcal{T}^{i}$ is given by

$\displaystyle\gamma_{u,t}=\frac{\mid[\bm{h}_{b,u,t}+\bm{h}_{r,u,t}\mathbf{\Theta}_{\mathcal{R},t}\bm{H}_{b,r,t}]\mathbf{w}_{u,t}\mid^{2}}{\mid[\bm{h}_{b,u,t}+\bm{h}_{r,u,t}\mathbf{\Theta}_{\mathcal{R},t}\bm{H}_{b,r,t}]\sum\limits_{k\leq
K,k\neq u}\mathbf{w}_{k,t}\mid^{2}+\sigma^{2}},$ (14)
$\displaystyle\gamma_{i,t}=\frac{\mid[\bm{h}_{b,i,t}+\bm{h}_{r,i,t}\mathbf{\Theta}_{\mathcal{T},t}\bm{H}_{b,r,t}]\mathbf{w}_{i,t}\mid^{2}}{\mid[\bm{h}_{b,i,t}+\bm{h}_{r,i,t}\mathbf{\Theta}_{\mathcal{T},t}\bm{H}_{b,r,t}]\sum\limits_{k\leq
K,k\neq i}\mathbf{w}_{k,t}\mid^{2}+\sigma^{2}},$ (15)

where $\sigma^{2}$ represents the noise power. Therefore, given a bandwidth
$B$, the achievable data rate of each user is given by

$\displaystyle R_{k,t}=B\log_{2}\left({1+\gamma_{k,t}}\right).$ (16)

###  II-E Problem Formulation

We aim for minimizing the power consumption of the BS by jointly optimizing
the beamforming vector $\mathbf{w}_{k}$ for the BS and the TCs as well as RCs
of the STAR-RIS. Again, the TCs and RCs of the STAR-RIS can be represented by
the phase-shifts $\bm{\Theta}_{\mathcal{T}},\bm{\Theta}_{\mathcal{R}}$, and
amplitude coefficients $\bm{\beta}$. Therefore, the optimization problem can
be formulated as

$\displaystyle\min_{\mathbf{w},\bm{\Theta}_{\mathcal{T}},\bm{\Theta}_{\mathcal{R}},\bm{\beta}}\sum_{t=1}^{T}\sum_{k=1}^{K}\parallel\mathbf{w}^{2}_{k,t}\parallel,$
(17a) s.t. $\displaystyle-\pi\leq\theta_{\mathcal{T},n,t}\leq\pi,\forall
n,\forall t,$ (17b)
$\displaystyle-\pi\leq\theta_{\mathcal{R},n,t}\leq\pi,\forall n,\forall t,$
(17c) $\displaystyle R_{k,t}\geq R_{\text{QoS}},\forall k,\forall t,$ (17d)
$\displaystyle 0<\beta_{n,t}\leq 1,\forall n,\forall t,$ (17e)
$\displaystyle\beta_{n,t}\sqrt{1-\beta_{n,t}^{2}}\cos(\theta_{\mathcal{R},n,t}-\theta_{\mathcal{T},n,t})=0,$
(17f) $\displaystyle P_{b,t}\leq P_{\text{max}},$ (17g)

where constraint (17b) and (17c) represent the legitimate range of the TC and
RC phase-shifts. Constraint (17d) is a QoS constraint specifically the minimum
data rate. Since (STAR) RISs are passive devices, their amplitude response is
limited by the conservation of energy, as shown in (17e). Constraint (17f)
characterizes the phase and amplitude relationship of the TCs and RCs.
Finally, (17g) is the maximum power constraint for the BS.

The challenge of solving our problem is not only owing to the joint
consideration of TCs and RCs, but also due to the constraint (17e). Given the
coupling between $\theta_{\mathcal{T},n}$ and $\theta_{\mathcal{R},n}$, the
STAR elements $n$ cannot have independent arbitrary TCs and RCs. Furthermore,
although the STAR-RIS aspire continuous phase-shift control for both
transmission and reflection, one party of them can only have dualistic
options. For example, assuming $\beta_{n}\neq 0$444If $\beta_{n}=0$ or
$\beta_{n}=1$, the STAR-RIS operates in either the full transmission or full
reflection mode, which is not a preferred mode., once the TC is determined as
$\theta_{\mathcal{T},n}$, the RC can only select the phase-shift from
$\\{\theta_{\mathcal{T},n}+\frac{\pi}{2},\theta_{\mathcal{T},n}-\frac{\pi}{2}\\}$.
Therefore, the coupled phase-shift model of STAR-RISs requires hybrid
continuous and discrete control for the transmission and reflection, which
motivates us to develop hybrid DRL algorithms for solving this challenge.

##  III Solution I: The Hybrid DDPG Algorithm

The DDPG algorithm was shown to constitute an efficient solution for
continuous control problems [34]. For applying DRL approaches to solve our
(STAR) RIS optimization problem, the transmission period has to obey a Markov
decision process (MDP) [35]. In TS $t\in T$, by checking the CSI of the
current channels, the agent determines the current state
$\mathbf{s}_{t}\in\mathbf{S}$ and decides to carry out the action
$\mathbf{a}_{t}\in\mathbf{A}$, where $\mathbf{S}$ and $\mathbf{A}$ represent
the state space and action space. The action refers to a vector storing the
active and passive beamforming coefficients at the BS and STAR-RIS.

Since passive and active beamforming are jointly considered, the MDP state for
each TS $t$ includes the CSI of the BS to STAR-RIS, BS to users (both
$\mathcal{R}$ and $\mathcal{T}$ users), and the STAR-RIS to users channels, as
shown in Fig. 1. Thus, $\mathbf{s}_{t}$ is given by

$\displaystyle\mathbf{s}_{t}=\\{\bm{H}_{b,r,t},\bm{H}_{b,\mathcal{R},t},\bm{H}_{b,\mathcal{T},t},\bm{H}_{r,\mathcal{R},t},\bm{H}_{r,\mathcal{T},t}\\}.$
(18)

Once action $\mathbf{a}_{t}$ is executed, the agent has to determine the
reward $r_{t}$ according to the data rate and power consumption of the
transceiver, and then the state would be constrained to
$\mathbf{s}_{t+1}\in\mathbf{S}$. Once the above steps are completed,
$(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})$ would be saved as a
Markov transition in the replay buffer for the agent’s training.

###  III-A DDPG training

The objective of the DRL based agent training is to find the specific action
$\mathbf{a}_{t}$ for each state $\mathbf{s}_{t}$, which maximizes the expected
accumulated reward $\mathbb{E}[\sum\nolimits_{i=t}^{T}\gamma r_{t+1}]$, where
$\gamma$ represents the discount factor $\gamma\in[0,1]$. For a DDPG agent,
the Q value of action $\mathbf{a}_{t}$ can be quantified by the Bellman
equation of [36]

$\displaystyle
Q^{\mu}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbb{E}\big{[}r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma
Q^{\mu}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})\big{]},$ (19)

where $\mu$ represents the action policy function
$\mathbf{S}\leftarrow\mathbf{A}$. The training of DRL agents aims for
ascertaining the optimal action yielding the maximum Q value, as given by

$\displaystyle
Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})=\mathbb{E}\big{[}r(\mathbf{s}_{t},\mathbf{a}_{t})+\mathop{\max}_{\mathbf{a}\in\mathbf{A}}\gamma
Q^{*}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})\big{]}.$ (20)

We consider a parameterized actor function $\mu(\mathbf{s}|\bm{\omega}^{\mu})$
and a function approximator associated with a batch of parameters denoted by
$\bm{\omega}^{Q}$. By sampling the aforementioned transition experiences in
the memory of the replay buffer, the DRL agent can be trained by minimizing
the loss function

$\displaystyle
L(\bm{\omega}^{Q})=\frac{1}{e}\sum_{e}[y_{t}-Q(\mathbf{s}_{t},\mathbf{a}_{t}|\bm{\omega}^{Q}_{t})]^{2},$
(21)

where $e$ is the size of the sampled transitions. However, to avoid
oscillations or divergence during the training process, $y_{t}$ has to be
provided by the target network, which has the same structure as the training
network, but associated with a deferred parameter update. Upon denoting the
parameters of the target networks by $\bm{\omega}^{Q^{\prime}}$, $y_{t}$ can
be expressed as

$\displaystyle y_{t}=r_{t}(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma
Q^{\prime}[\mathbf{s}_{t},\mu^{\prime}(\mathbf{s}_{t}|\bm{\omega}^{\mu^{\prime}}_{t})|\bm{\omega}^{Q^{\prime}}_{t}].$
(22)

According to the principle of the original DDPG algorithm [34], the actor
network is trained by the policy gradient calculated by the critic network as

$\displaystyle\nabla_{\bm{\omega}^{\mu}}J=$
$\displaystyle\frac{1}{e}\sum_{e}\nabla_{\mathbf{a}}Q(\mathbf{s}_{e},\mathbf{a}_{e}|\bm{\omega}^{Q})|_{\mathbf{s}_{e}=\mathbf{s}_{t},\mathbf{a}_{e}=\mu(\mathbf{s}_{t})}\nabla_{\bm{\omega}^{\mu}}\mu(\mathbf{s}_{e}|\bm{\omega}^{\mu}|_{\mathbf{s}_{e}=\mathbf{s}_{t}}).$
(23)

(a) Flow diagram of the conventional DDPG algorithm with output
discretization.

(b) Flow diagram of the proposed hybrid DDPG algorithm.

Figure 2: Flow diagrams of the DDPG/hybrid DDPG algorithms. Algorithm 1
DDPG/Hybrid DDPG algorithm

1: Initialize the environment and the agent with the actor network
$\bm{\omega}^{\mu}$, critic network $\bm{\omega}^{Q}$, target actor network
$\bm{\omega}^{\mu^{\prime}}$, target critic network $\bm{\omega}^{Q^{\prime}}$

2: for each episode do

3: Reinitialize the environment to $\mathbf{s}_{t=0}$

4: for each step in $t_{0}\leq t\leq T$ do

5: Observe $\mathbf{s}_{t}$

6: Choose $\mathbf{a}_{t}$ according to (25)

7: if DDPG algorithm then

8: Discretize a part of $\mathbf{a}_{t}$ into $\mathbf{a}_{t,d}$

9: end if

10: if Hybrid DDPG algorithm then

11: Map $\mathbf{a}_{t}$ with $\bm{\Theta}_{\mathcal{R},t}$ and
$\bm{\Theta}_{\mathcal{T},t}$

12: end if

13: Execute $\mathbf{a}_{t}$ in the environment

14: Calculated the reward $r_{t}$ and observe the next state
$\mathbf{s}_{t+1}$

15: Record $e\\{\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1}\\}$ in
memory buffer

16: Random sample a batch of transection $e$ from memory buffer

17: Calculate target according to (22)

18: Train critic network $Q(\mathbf{s}_{t},\bm{\omega}^{Q}_{t})$ with a
gradient descent step (21)

19: Train actor network ${\mu}(s_{t},\bm{\omega}^{\mu}_{t})$ with (III-A)

20: Update the target networks
$\bm{\omega}^{\mathcal{\mu^{\prime}}}_{t}\leftarrow(1-\tau)\bm{\omega}^{\mathcal{\mu^{\prime}}}_{t}+\tau\bm{\omega}^{\mathcal{\mu}}_{t}$,
$\bm{\omega}^{Q^{\prime}}_{t}\leftarrow(1-\tau)\bm{\omega}^{Q^{\prime}}_{t}+\tau\bm{\omega}^{Q}_{t}$

21: $\mathbf{s}_{t}\leftarrow\mathbf{s}_{t+1}$

22: end for

23: end for

###  III-B Continuous-discrete actions and hybrid DDPG

The actions designed for the DRL agent have to contain all optimized
variables, resulting in the action space of
$\mathbf{a}_{t}=\\{\mathbf{w},\bm{\Theta}_{\mathcal{T}},\bm{\Theta}_{\mathcal{R}},\bm{\beta}\\}$.
Among these optimization variables, $\mathbf{w}$ and $\bm{\beta}$ can be
handled by a continuous control scheme to achieve precise control. Therefore,
in this subsection we focus our attention on the discussion of
$\bm{\Theta}_{\mathcal{T}}$ and $\bm{\Theta}_{\mathcal{R}}$. As described
above, due to the existence of the constraint (17f), the transmission and
reflection phase-shifts cannot be adjusted independently by the STAR-RIS. For
element $n$, assume $\beta_{n}\neq 0$ and $\theta_{\mathcal{R},n}$ is
determined, in order to satisfy the constraint (17f). In this context, it is
no hard to discover that
$\theta_{\mathcal{T},n}=\theta_{\mathcal{R},n}\pm\frac{\pi}{2}$. Therefore,
regardless of whether the phase-shift $\theta_{\mathcal{R},n}$ or
$\theta_{\mathcal{T},n}$ is selected to be continuously controlled, the phase
control of the other one is no longer a continuous control problem, but a
binary selection problem. Therefore, (17a) requires continuous control for
$\mathbf{w},\bm{\Theta}_{\mathcal{R}},\bm{\beta}$ and discrete control for
$\bm{\Theta}_{\mathcal{T}}$, which lead to a hybrid action space associated
with continuous and discrete actions. The continuous sub-space and the
discrete sub-space are denoted by
$\mathbf{a}^{c}=(a^{c}_{1},a^{c}_{2}...a^{c}_{n}...)$ and
$\mathbf{a}^{d}=(a^{d}_{1},a^{d}_{2}...a^{d}_{n}...)$555For the convenience of
presentation, we assume that $\bm{\Theta}_{\mathcal{R}}\subset a^{c}$ and
$\bm{\Theta}_{\mathcal{T}}\subset a^{d}$., respectively.

Figure 3: Amplitude response over normalized action output of hybrid DDPG
algorithm for STAR-RIS ($\beta=0.5$).

####  III-B1 Action Space for the DDPG Algorithm

If the DDPG algorithm is applied for solving the optimization problem
associated with the hybrid action space, a direct and intuitive approach is to
discretize a part of the continuous output of the DDPG algorithm as presented
in Fig. 2(a) and Algorithm 1. Including the action noise, a classic action
policy is given for the DDPG algorithm by

$\displaystyle\mathbf{a}_{t}=\mu(\mathbf{s}_{t}|\bm{\omega}^{\mu}_{t})+\mathcal{N}_{\text{OU}}(0,\xi),$
(24)

where $\mathcal{N}_{\text{OU}}(0,\xi)$ represents the zero mean Ornstein
Uhlenbeck (OU) noise [37] that follows $\mathcal{N}_{\text{OU}}\sim
OU(0,\xi)$, and $\xi$ is the volatility of the OU noise.

Thus, the continuous action space can be formulated as

$\displaystyle\mathbf{a}_{t}^{c}=\\{\mathbf{a}_{t}^{\mathbf{w}},\mathbf{a}_{t}^{\bm{\Theta}_{\mathcal{R}}},\mathbf{a}_{t}^{\bm{\beta}}\\}.$
(25)

Then, the normalized output of the actor network may be decoded into
executable actions for our communication environment by following the actions
of

$\displaystyle\mathbf{w}\leftarrow\mathbf{a}_{t}^{\mathbf{w}},$ (26)
$\displaystyle\bm{\Theta}_{\mathcal{R}}\leftarrow\mathbf{a}_{t}^{\bm{\Theta}_{\mathcal{R}}},$
(27) $\displaystyle\bm{\beta}\leftarrow\mathbf{a}_{t}^{\bm{\beta}}.$ (28)

Since the mapping of $\mathbf{a}_{t}^{c}$ and the continuous actions are
trivial, it is not necessary to discuss them in detail. On the other hand, for
a specific STAR element $n$, $\theta_{\mathcal{T},n}$ can be obtained by the
binary discretized $a_{n,t}^{d}$

$\displaystyle\theta_{\mathcal{T},n,t}=\begin{cases}\theta_{\mathcal{R},n,t}+\frac{\pi}{2},&a_{n,t}^{d}>0,\\\
\theta_{\mathcal{R},n,t}-\frac{\pi}{2},&a_{n,t}^{d}\leq 0.\end{cases}$ (29)

####  III-B2 Action Space for our Hybrid DDPG Algorithm

In order to deal with the high-dimensional continuous and discrete action
components, specifically for the STAR-RIS scenario, we propose a hybrid DDPG
algorithm. By exploiting $\bm{\Theta}_{\mathcal{R}}$ and
$\bm{\Theta}_{\mathcal{T}}$ have the same dimension for the STAR-RIS, we
design the codebook between the actor outputs and the phase-shifts to achieve
hybrid control. The hybrid action policy for the STAR-RIS at time $t$ can be
expressed by

$\displaystyle\mu(\mathbf{s}_{t}|\bm{\omega}^{\mu}_{t})$
$\displaystyle=\mu^{c}(\mathbf{s}_{t}|\bm{\omega}^{\mu}_{t})\mu^{d}(\mathbf{s}_{t}|\bm{\omega}^{\mu}_{t})$
$\displaystyle=\prod_{n=1}^{N}\mu^{c}(s_{n,t}|\bm{\omega}^{\mu}_{t})\prod_{n=1}^{N}\mu^{d}(s_{n,t}|\bm{\omega}^{\mu}_{t}).$
(30)

Specifically, for the STAR element $n$, the values of
$\theta_{\mathcal{R},n,t}$ and $\theta_{\mathcal{T},n,t}$ are compounded and
given in a normalized output as

$\displaystyle\mu(s_{n,t}|\bm{\omega}^{\mu}_{t})=\mu^{c}(s_{n,t}|\bm{\omega}^{\mu}_{t})\mu^{d}(s_{n,t}|\bm{\omega}^{\mu}_{t}).$
(31)

In the face of the action noise similar to (24), the hybrid action
$a^{h}_{n,t}$ can be obtained from (31). Then, the normalized action
$a^{h}_{n,t}$ has to be mapped to $\theta_{\mathcal{R},n,t}$ and
$\theta_{\mathcal{R},n,t}$ as

$\displaystyle\theta_{n,\mathcal{R},t}=2\pi a^{h}_{n,t},$ (32)
$\displaystyle\theta_{n,\mathcal{T},t}=\begin{cases}\theta_{n,\mathcal{R},t}+\frac{\pi}{2},&a^{h}_{n,t}>0,\\\
\theta_{n,\mathcal{R},t}-\frac{\pi}{2},&a^{h}_{n,t}\leq 0.\end{cases}$ (33)

Since the normalized action $a^{h}_{n,t}$ is in the interval $[-1,1]$,
$\theta_{n,\mathcal{R},t}$ and $\theta_{\mathcal{T}}$ have to be mapped with
$a^{h}_{n,t}$. Furthermore, $\theta_{n,\mathcal{R},t}$ vs $a^{h}_{n,t}$ is
modelled by a periodic linear function and $\theta_{\mathcal{T}}$ vs
$a^{h}_{n,t}$ by a piecewise linear function. The amplitude response
$a^{h}_{n,t}$ of a s single STAR element is plotted in Fig. 3 for $\beta=0.5$.
Since STAR-RIS is generally equipped with a large number of STAR elements, in
this case, the hybrid DDPG algorithm has a significantly smaller action
dimension than the conventional DDPG algorithm, since we have
$|\mathbf{a}^{h}_{t}|=\frac{|\mathbf{a}^{c}_{t}|+|\mathbf{a}^{d}_{t}|}{2}$.

######  Remark 1.

In some existing hybrid DRL schemes, the agent can only output a single action
for the discrete action space, which is a disadvantage for problems associated
with high-dimensional discrete action spaces of multi-antenna or multi-
element-RIS scenarios. By contrast, the proposed hybrid scheme is eminently
suitable for the high-dimensional hybrid action spaces, where the discrete
action dimension is no larger than the continuous action dimension.

###  III-C Reward Function

The plain DDPG algorithm and the hybrid DDPG algorithm have identical reward
functions. In order to ensure that the agent meets the users’ QoS constraint,
each satisfied $\mathcal{T}$ and $\mathcal{R}$ user contributes a
distinguishable positive reward. Meanwhile, to minimize the power consumption,
the sum power cost associated with $\mathbf{w}$ results in a negative reward.
Thus, the reward function at TS $t$ can be formulated as

$r_{t}=\sum_{k=1,R_{k,t}>R_{c}}^{K}\dot{r}-\sum_{t=1}^{T}\sum_{k=1}^{K}\hat{r}\parallel\mathbf{w}^{2}_{k,t}\parallel,$
(34)

where the constant coefficient $\dot{r}$ represents the reward gleaned for
satisfying the data rate requirement per user, and the coefficient $\hat{r}$
cost (negative reward) for power consumption. It is worth noting that, to
guarantee the primary goal of the agent is satisfying the QoS requirement of
each and every user rather than saving energy, the coefficients have to
satisfy $\dot{r}>\sum_{k=1}^{K}\hat{r}P_{\text{max}}$.

###  III-D Neural Network Structure

In order to ensure accurate fitting, the structure and scale of the (target)
actor network and the (target) critic network have to be selected
appropriately. The actor networks include the input layer, batch normalization
(BN) layer, and activation layer(s) with a ’relu’ function in turn. Moreover,
’tanh’ is assigned as the activation function of the output layer. To ensure a
valid range for the input values of the output layer, another BN layer has to
be employed above the output layer. The critic networks consist of an input
layer, BN layer, concatenate layer, and activation layers. The size of the
hidden layers has to be determined according to the state and action
dimension, which is dependent both on the number of antennas $M$ and the
number of STAR elements $N$. It is also worth noting that due to the
difference in output dimensions between the DDPG scheme and the hybrid DDPG
scheme, the appropriate size of the hidden layer may vary, especially when a
large number of STAR elements is considered.

##  IV Solution II: Joint DDPG-DQN Algorithm

The hybrid DDPG algorithm aims for covering the action space $\mathbf{a}^{c}$
and $\mathbf{a}^{d}$ with the aid of a compound action mapping. By contrast,
in this section, we explore another promising option, namely that of employing
two agents for covering the action spaces $\mathbf{a}^{c}$ and
$\mathbf{a}^{d}$, respectively. Thus, a DDPG algorithm is harnessed for
optimizing
$\mathbf{a}^{c}=\\{\mathbf{a}_{t}^{\mathbf{w}},\mathbf{a}_{t}^{\bm{\Theta}_{\mathcal{R}}},\mathbf{a}_{t}^{\bm{\beta}}\\}$,
and a DQN algorithm is responsible for giving
$\mathbf{a}^{d}=\mathbf{a}_{t}^{\bm{\Theta}_{\mathcal{T}}}$. DDPG and DQN
algorithms can be regarded either as a joint agent, or as a pair of
cooperative agents. For the convenience of the presentation, we regard the
DDPG agent and the DQN agent of the joint DDPG-DQN algorithm as a pair of
components, but they have to be installed on the same device in practice.

###  IV-A MDP for Joint DDPG-DQN

The DDPG and DQN algorithms are capable of optimizing independent continuous
and discrete problems [38]. Nonetheless, how to harness them in the interest
of joint optimization results is an open conundrum. If two agents adopt a
parallel relationship at $\mathbf{s}_{t}$ to output $\mathbf{a}^{c}_{t}$ and
$\mathbf{a}^{d}_{t}$, then according to the classic MDP model, the execution
of $\mathbf{a}_{t}$ will result in reward $r_{t}$. However, both agents need
corresponding rewards for estimating the action value of $\mathbf{a}^{c}_{t}$
and $\mathbf{a}^{d}_{t}$ independently. The dilemma is that in the scenario
considered, $r_{t}$ is the result of the combined action of
$\mathbf{a}^{c}_{t},\mathbf{a}^{d}_{t}$ and cannot be split into
$r^{c}_{t},r^{d}_{t}$. In other words, from the perspective of the wireless
network, the user-side SINR of the STAR-RIS network is jointly determined by
active beamforming, reflection beamforming, and transmission beamforming. As a
consequence, we cannot solely and unambiguously attribute any gain or loss. If
$r_{t}$ is regarded as a common rewards for both two agents, the agents will
authenticate the MDP transitions as
$\\{\mathbf{s}_{t},\mathbf{a}^{c}_{t},r_{t},\mathbf{s}_{t+1}\\}$ and
$\\{\mathbf{s}_{t},\mathbf{a}^{d}_{t},r_{t},\mathbf{s}_{t+1}\\}$.
Unfortunately, these transitions are not correct, as in fact the correct
transitions have to be formulated as
$\\{\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1}\\}$ or
$\\{\mathbf{s}_{t},\mathbf{a}^{c}_{t},r^{c}_{t},\mathbf{s}^{c}_{t+1}\\},\\{\mathbf{s}_{t},\mathbf{a}^{d}_{t},r^{d}_{t},\mathbf{s}^{d}_{t+1}\\}$.
In order to resolve this dilemma, inspired by [35], we artificially add an
inner environment to formulate the MDPs for a pair of agents.

Figure 4: Flow diagram of the proposed DDPG-DQN algorithm.

As shown in Fig. 4, an extra environment is derived from the original
environment, and the environments refer to the ’inner environment’ and ’outer
environment’ to distinguish them. Thus, the DDPG agent interacts with the
outer environment and the DQN agent interacts with the inner environment. The
outer environment represents the STAR-RIS assisted wireless network in the
reality, the inner environment is only a fictitious environment that contains
the knowledge of the outer environment for the agent’s training. From the
perspective of the DQN agent, the inner environment can be regarded as a
collection of the outer environment and the DDPG agent. Based on this
framework, for the DQN agent and the inner environment, the MDP can be
formulated as
$\\{\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t},r_{i,t},\mathbf{s}_{i,t+1}\\}$ and
for the outer environment as
$\\{\mathbf{s}_{o,t},\mathbf{a}^{c}_{o,t},r_{o,t},\mathbf{s}_{o,t+1}\\}$. In
the remainder of this section, we discuss the details of the DQN and DDPG
agents, respectively.

Algorithm 2 Joint DDPG-DQN algorithm

1: Initialize the outer environment and the DDPG agent with the actor network
$\bm{\omega}^{\mu}$, critic network $\bm{\omega}^{Q}$, target actor network
$\bm{\omega}^{\mu^{\prime}}$, target critic network $\bm{\omega}^{Q^{\prime}}$

2: Initialize the DQN agent with the deep Q network
$\bm{\omega}^{\mathcal{Q}}$ and target Q network
$\bm{\omega}^{\mathcal{Q^{\prime}}}$

3: for each episode do

4: Reinitialize the outer environment to $\mathbf{s}_{t=0}$

5: for each step in $t_{0}\leq t\leq T$ do

6: Observing $\mathbf{s}_{o,t}$ and DDPG agent choose $\mathbf{a}^{c}_{o,t}$
according to (25)

7: DQN agent choose inner action $\mathbf{a}^{d}_{i,t}$ with (39)

8: Execute action $\mathbf{a}^{d}_{i,t}$ in the inner environment

9: Calculated the reward $r_{i,t}$ and observe the next state
$\mathbf{s}_{i,t+1}$

10: Record
$e\\{\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t},r_{i,t},\mathbf{s}_{i,t+1}\\}$

11: Sample random transitions of $e$ from DQN memory

12: Train $\omega^{\mathcal{Q}}$ with (IV-B1)

13: DQN agent choose outer action $\mathbf{a}^{d}_{o,t}$ with (40)

14: Execute $\mathbf{a}^{c}_{o,t}$ and $\mathbf{a}^{d}_{o,t}$ in the outer
environment

15: Calculated the reward $r_{o,t}$ and observe the next state
$\mathbf{s}_{r,t+1}$

16: Record
$e\\{\mathbf{s}_{o,t},\mathbf{a}_{o,t},r_{o,t},\mathbf{s}_{o,t+1}\\}$ in DDPG
memory buffer

17: Random sample a batch of transection $e$ from memory buffer

18: Calculate target according to (22)

19: Train critic network $Q(\mathbf{s}_{t},\bm{\omega}^{Q}_{t})$ with a
gradient descent step (21)

20: Train actor network ${\mu}(\mathbf{s}_{t},\bm{\omega}^{\mu}_{t})$ with
(III-A)

21: Update the target networks:
$\bm{\omega}^{\mathcal{\mu^{\prime}}}_{t}\leftarrow(1-\tau)\bm{\omega}^{\mathcal{\mu^{\prime}}}_{t}+\tau\bm{\omega}^{\mathcal{\mu}}_{t}$,
$\bm{\omega}^{Q^{\prime}}_{t}\leftarrow(1-\tau)\bm{\omega}^{Q^{\prime}}_{t}+\tau\bm{\omega}^{Q}_{t}$,
$\bm{\omega}^{\mathcal{Q^{\prime}}}_{t}\leftarrow(1-\tau)\bm{\omega}^{\mathcal{Q^{\prime}}}_{t}+\tau\bm{\omega}^{\mathcal{Q}}_{t}$

22: $\mathbf{s}_{o,t}\leftarrow\mathbf{s}_{o,t+1}$

23: end for

24: end for

###  IV-B Inner Environment and the DQN agent

####  IV-B1 DQN Training

As a value-based RL algorithm, the DQN algorithm identifies the action values
as (19), since the DQN agent also aims for the maximum long-term reward. In
the inner environment, the Q function is given by

$\displaystyle
Q(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})=\mathbb{E}\big{[}r(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})+\gamma
Q(\mathbf{s}_{i,t+1},\mathbf{a}_{i,t+1})\big{]}.$ (35)

In the training process, the Q value for the actions has to be updated in each
step by following

$\displaystyle Q_{t+1}(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})$
$\displaystyle\leftarrow(1-\alpha)Q_{t}(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})$
$\displaystyle+\alpha[r_{i,t}+\gamma\max
Q_{t}(\mathbf{s}_{i,t+1},\mathbf{a}^{d}_{i,t+1})],$ (36)

where $\alpha$ represents the learning rate ( $0<\alpha\leq 1$). In the DQN
algorithm, the Q function is approximated by a DNN having the parameter vector
$\bm{\omega}^{\mathcal{Q}}$, and we have
$Q(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})\approx
Q(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t}|\omega^{\mathcal{Q}})$. In order to
accurately fit the Q function, the DNN has to be appreciatively trained.
Similar to the DDPG algorithm, the transitions of the DQN agent are stored in
the DQN memory buffer and we can adopt the memory replay technique for
training the DNN by the following loss function

$\displaystyle
L(\omega^{\mathcal{Q}})=\frac{1}{e}\sum_{e}\big{[}r_{i,t}(\mathbf{s}_{i,t},\mathbf{a}^{d}_{i,t})+$
$\displaystyle\gamma\\!\\!\mathop{\max}_{\mathbf{a}_{i,t+1}\in\mathbf{A}^{d}}Q^{\prime}(\mathbf{s}_{i,t+1},\mathbf{a}_{i,t+1}|\omega^{\mathcal{Q}^{\prime}}_{t})-Q(\mathbf{s}_{i,t},\mathbf{a}_{i,t}|\omega^{\mathcal{Q}}_{t})\big{]}^{2}.$
(37)

####  IV-B2 Inner State and Action Space

As designed above, the DQN agent regards the DDPG agent as a part of the inner
environment. Therefore, the state of the inner environment consists of two
parts, including the state inherited from the outer environment and the
actions $\mathbf{a}^{c}_{t}$ of the DDPG agent. Thus the inner state can be
formulated as

$\displaystyle\mathbf{s}_{i,t}=\\{\bm{H}_{b,r,t},\bm{H}_{b,\mathcal{R},t},\bm{H}_{b,\mathcal{T},t},\bm{H}_{r,\mathcal{R},t},\bm{H}_{r,\mathcal{T},t},\mathbf{a}^{c}_{o,t}\\}.$
(38)

The action space of the DQN agent is
$\mathbf{a}^{d}_{i,t}=\bm{\Theta}_{\mathcal{T}}$, since the DQN agent is only
responsible for the discrete actions. The state space and action indicate a
pair of main facts. Primarily, the decision of the DDPG agent has to be known
to the DQN agent, when it is interacting with the inner environment.
Additionally, it also suggests that the DQN agent has to ’interrupt’ the
interactions of the DDPG agent. Specifically, as shown in Algorithm 2, the
timing of activating the inner environment and the DQN agent is after the
action selection of the DDPG agent, but before the outer environment’s
execution of the action, since the transition of the outer environment relies
on $\mathbf{a}_{o,t}=\\{\mathbf{a}^{c}_{o,t},\mathbf{a}^{d}_{o,t}\\}$.

####  IV-B3 Action Policy

For conventional DQN agents, the $\epsilon-\text{greedy}$ action policy
constitutes an efficient technique of carefully balancing exploration and
exploitation. To elaborate, the $\epsilon-\text{greedy}$ policy authorizes the
DQN agent to choose a random action with the probability of $\epsilon$, and
the optimal action with a probability of $1-\epsilon$. However, it is worth
noting that in the joint DDPG-DQN algorithm, the DQN agent has to output a
pair of actions one for the inner and one for the outer environment. Action
$\mathbf{a}^{d}_{i,t}$ for the inner environment is adopted to form the inner
transitions for the training process of the DQN agent. On the other hand, the
action $\mathbf{a}^{d}_{o,t}$ produced by the DQN for the outer environment
has to assist the DDPG agent. Therefore, once the $\epsilon-\text{greedy}$
action policy is adopted in the outer environment, it may impose interference
on $r_{o,t}$ and lead to an estimation error on the Q value of
$\mathbf{a}^{c}_{o,t}$. Thus, the action policy of the DQN agent can be
formulated as

$\mathbf{a}^{d}_{i,t}=\begin{cases}\text{random action},&\epsilon,\\\
\mathop{\arg\max}\limits_{\mathbf{a}_{i,t}\in\mathbf{A}^{d}}Q(\mathbf{s}_{i,t+1},\mathbf{a}_{i,t}|\bm{\omega}^{Q}_{t}),&1-\epsilon,\end{cases}$
(39)

$\mathbf{a}^{d}_{o,t}=\mathop{\arg\max}\limits_{\mathbf{a}_{i,t}\in\mathbf{A}^{d}}Q(\mathbf{s}_{o,t},\mathbf{a}_{o,t}|\bm{\omega}^{Q}_{t}).$
(40)

######  Remark 2.

In the joint DDPG-DQN algorithm, the DQN is trained in the inner environment.
The DQN agent produces actions for the outer environment, which may be viewed
as a service instead of training for the DQN. Therefore, we choose the
$\epsilon-\text{greedy}$ action policy in the inner environment to train the
DQN agent and the optimal action policy in the outer environment to obtain
optimal actions.

####  IV-B4 Reward Function

Since the reward function depends on the optimization goal and on the
constraints, similarly to (34), the reward function of the DQN agent is also
given by

$r_{i,t}=\sum_{k=1,R_{k,t}>R_{c}}^{K}\dot{r}-\sum_{t=1}^{T}\sum_{k=1}^{K}\hat{r}\parallel\mathbf{w}^{2}_{k,t}\parallel,$
(41)

where the constant settings have to be consistent with the corresponding
discussion of Sub-section III-D.

###  IV-C Outer Environment and the DDPG agent

The principle and algorithm’s flow of the outer environment and the DDPG agent
are the same as in Section III, thus we do not repeat the principle and
training process here but highlight the difference. For the DDPG agent, state
$\mathbf{s}_{o,t}$ is the same as (18), and the reward is
$r_{o,t}=\sum_{k=1,R_{k,t}>R_{c}}^{K}\dot{r}-\sum_{t=1}^{T}\sum_{k=1}^{K}\hat{r}\parallel\mathbf{w}_{k}^{2}\parallel$,
but the difference is that the action space is reduced to
$\mathbf{a}^{c}_{o,t}=\\{\mathbf{a}_{t}^{\mathbf{w}},\mathbf{a}_{t}^{\bm{\Theta}_{\mathcal{R}}},\mathbf{a}_{t}^{\bm{\beta}}\\}$.
Hence, for any TS $t$, the optimized value
$\\{\mathbf{w}_{t},\bm{\Theta}_{\mathcal{R},t},\bm{\beta}_{t}\\}$ can be
completely covered by $\mathbf{a}^{c}_{o,t}$ using the mapping approach
described in (26) (27) and (28).

In the joint DDPG-DQN algorithm, since both agents are employed and each has
its own DNNs, their optimal DNN scales are likely to be smaller than these of
the hybrid DDPG algorithm. For the DDPG agent in the joint algorithm, we adopt
the same structure as for the DNNs of the hybrid DDPG algorithm, not only as
its optimal experimental structure, but also to ensure the fairness of the
performance comparison.

###  IV-D Discussions

Although facing an identical formulated problem, Solution I and Solution II
revealed two disparate measures. The hybrid DDPG approach aimed for achieving
hybrid control via carefully tailored mapping function without any additional
DNN or other processes. The advantage of this scheme is that its complexity is
not increased. By contrast, since it has a lower output dimension than the
DDPG agent, i.e.
$|\mathbf{a}^{h}_{n}|<|\mathbf{a}^{c}_{n}|+|\mathbf{a}^{d}_{n}|$, it can
employ a DNN with fewer trainable parameters, thereby reducing the complexity
of the training process. The joint DDPG-DQN algorithm employs a pair of agents
having different capabilities for jointly solving the problem having a hybrid
action space. Since an extra DQN agent has to be harnessed, given the
complexity of the DDPG and the DQN algorithm [39], the total complexity
becomes significantly higher than that of the hybrid DDPG scheme.

The complexity of the DDPG algorithm depends on the specification of the
employed DNN. Assuming that an actor network having $I$ layers is employed,
while each layer contains $\bm{\omega}^{\mu}_{i}$ nodes, the complexity of
propagation is given by
$\sum\limits_{i=0}^{I}\bm{\omega}^{\mu}_{i}\bm{\omega}^{\mu}_{i+1}$. Given the
number of nodes in the actor network as $\bm{\omega}^{\mu}_{\text{b}}$ for BN
layers, $\bm{\omega}^{\mu}_{\text{r}}$ for ’relu’ layers and
$\bm{\omega}^{\mu}_{\text{t}}$ for ’tanh’ layers, according to [39], the
required number of floating-point operations is given by
$5\bm{\omega}^{\mu}_{\text{b}}+\bm{\omega}^{\mu}_{\text{r}}+6\bm{\omega}^{\mu}_{\text{t}}$.
Applying the same theory for the critic networks, the complexity of a single
prediction and training step can be formulated by
$\mathcal{O}(\sum\limits_{i=0}^{I}\bm{\omega}^{\mu}_{i}\bm{\omega}^{\mu}_{i+1}+\sum\limits_{i=0}^{I}\bm{\omega}^{Q}_{i}\bm{\omega}^{Q}_{i+1}+5\bm{\omega}^{\mu}_{\text{b}}+\bm{\omega}^{\mu}_{\text{r}}+6\bm{\omega}^{\mu}_{\text{t}}+5\bm{\omega}^{Q}_{\text{b}}+\bm{\omega}^{Q}_{\text{r}}+6\bm{\omega}^{Q}_{\text{t}})$.
Since we usually have $\bm{\omega}_{i}\gg 5$, the computational complexity can
be approximated by
$\mathcal{O}(\sum\limits_{i=0}^{I}\bm{\omega}^{\mu}_{i}\bm{\omega}^{\mu}_{i+1}+\sum\limits_{i=0}^{I}\bm{\omega}^{Q}_{i}\bm{\omega}^{Q}_{i+1})$.
As mentioned, since the hybrid DDPG algorithm has smaller output and DNN
scale, we have $\bm{\omega}^{\mu,h}_{i}<\bm{\omega}^{\mu}_{i}$. Thus, the
hybrid DDPG algorithm has lower complexity than the conventional DDPG
algorithm.

The complexity of the joint DDPG-DQN algorithm is partially due to the DDPG
agent and partially from the DQN agent. The complexity of the DQN agent is
given by
$\mathcal{O}(3\sum\limits_{i=0}^{I}\bm{\omega}^{\mathcal{Q}}_{i}\bm{\omega}^{\mathcal{Q}}_{i+1})$
as suggested in [40]. The overall complexity of the joint DDPG-DQN algorithm
is
$\mathcal{O}(\sum\limits_{i=0}^{I}\bm{\omega}^{\mu}_{i}\bm{\omega}^{\mu}_{i+1}+\sum\limits_{i=0}^{I}\bm{\omega}^{Q}_{i}\bm{\omega}^{Q}_{i+1}+3\sum\limits_{i=0}^{I}\bm{\omega}^{\mathcal{Q}}_{i}\bm{\omega}^{\mathcal{Q}}_{i+1})$.
Therefore, the joint DDPG-DQN algorithm only has an increased complexity
compared to the hybrid DDPG algorithm if the DQN agent has a significantly
smaller $\bm{\omega}^{\mathcal{Q}}_{i}$.

##  V Numerical Results and Analysis

Let us now consider the performance of the proposed approaches. The
performance of the STAR-RIS is compared to that of the reflecting-only RIS and
double spliced RIS. The reflecting-only RIS can only serve $\mathcal{R}$ users
and fails to provide signal enhancements for the $\mathcal{T}$ users. As for
the other benchmark, the double spliced RIS is formed by splicing a pair of
RISs facing in opposite directions, where the total number of elements is set
to $N$. This also ensures the fairness of comparisons. The double spliced RIS
can also be equivalently regarded as a STAR-RIS employing the ’Mode Switching’
scheme [12], but the proportion of transmission elements and reflection
elements is fixed to 1. In terms of optimization algorithms, the DDPG
algorithm having a partial discrete action space is employed as the baseline.
The performance of the hybrid DDPG algorithm and DDPG-DQN algorithm is
compared.

As for the simulation parameters, the (STAR) RIS located 4km away from the BS,
where the $\mathcal{T}$ users and $\mathcal{R}$ users are randomly distributed
on both sides of the (STAR) RIS. The wireless channel model was introduced in
Section II. The simulated transmission process is 30 seconds and we assume
that the block fading envelope of the channel varies once per second. For the
intelligent agent, we employ the ’Adam’ optimizer for both the hybrid DDPG and
for the DDPG-DQN algorithms. The default learning rates for all agents,
including the learning rate for the DQN, the actor/critic learning rates for
the DDPG agent are set to be $3\times 10^{-4}$. For the DDPG agent, the actor
network has a single activation layer, and the critic network has two
activation layers. As for the DQN agent, the deep Q network has 1-2 activation
layer(s). The activation function of the hidden layers is ’relu’ in both
algorithms. As mentioned, the size of the activation layers has to be
determined by the complexity of the communication system. Empirically, in the
majority of realizations, we equip the actor and critic network with hidden
layer(s) containing 256-512 neuron nodes, and the DQN agent employs hidden
layer(s) with 200-300 neuron nodes. The default parameters used in the
simulations are listed in Table II.

TABLE II: Default Parameters Parameter | Description | Value | Parameter | Description | Value
---|---|---|---|---|---
$M$ | antenna number | 4 | $N$ | (STAR) element number | 12
$f_{\text{c}}$ | carrier frequency | 5GHz | $B$ | bandwidth | 1 MHz
$K$ | number of users | 4 | $P_{\text{max}}$ | maximum power per antenna | 29 dBm
${K_{AR},K_{RU}}$ | Rician factors | 3dB | $\sigma$ | noise power density | -95.2 dBm/MHz
$r$ | replay buffer size | 10000 | $\gamma$ | discount factor | 1
$e_{\text{DDPG}}$ | batch size | 32 samples | $e_{\text{DQN}}$ | batch size | 32 samples
$\tau_{\text{DDPG}}$ | target update rate | 0.002 | $\tau_{\text{DQN}}$ | target update rate | 0.003
Figure 5: Obtained reward of hybrid DDPG algorithm and DDPG-DQN algorithm with
different learning rate

Fig. 5 presents the reward obtained by the proposed algorithms having
different learning rates. Based on the rewards exhibited by the different
learning rates, we can draw two main conclusions. On the one hand, both the
hybrid DDPG and the DDPG-DQN schemes converge if the appropriate learning rate
is selected. It has to be clarified that, in general, a higher learning rate
leads to faster convergence. However, the curves $\text{lr}=0.001$ and
$\text{lr}=0.0003$ of Fig. 5 exhibit similar convergence rates, since we
invoked decaying action noise and exploration rate in the (hybrid) DDPG and
DQN agents, which affects the reward obtained. On the flip side, the joint
DDPG-DQN approach has an excellent capability of obtaining rewards. According
to (34), the rewards obtained by the agents represent the overall performance
of the wireless links, which translate into superior user satisfaction or
reduced energy consumption.

Figure 6: Performances of different algorithms for STAR-RIS

The rewards obtained by the different algorithms are plotted in Fig. 6. After
a training process, the conventional DDPG algorithm used as the baseline has
achieved an average reward of about 590, which is inferior to the proposed
algorithms. The DDPG-DQN algorithm having optimal action output achieved a
slight advantage of about $3-6\%$ compared to that of the hybrid DDPG
algorithm, but the price of this performance gain is that the DDQP-DQN
algorithm has higher complexity, since two agents are employed. Although the
hybrid DDPG scheme is slightly inferior in terms of optimality, it provides a
low-complexity rapidly converging solution. Observing the two branches of the
DDPG-DQN scheme, it can be observed that compared to the scheme in [35], the
separated action policy of inner and outer environments has achieved superior
performance, which supports the arguments in Remark 2.

Figure 7: Power consumption of different algorithms for STAR-RIS

Fig. 7 shows the performance of the proposed algorithm from the perspective of
power minimization. In the early stage of training, the agent executes fairly
random actions, which can be considered as a chaotic system and its transmit
power is relatively high. By contrast, for well-trained scenarios, the joint
DDPG-DQN scheme has achieved superior power minimization effects regardless of
the action policy, which is in line with their trends in the reward analyses.
The hybrid DDPG algorithm consumes more power than the DDPG-DQN algorithm,
which is why it obtains less rewards. What is worth noting is that the
baseline algorithm requires lower consumption than the hybrid DDPG algorithm.
Recalling reward function (34) reveals that the solution provided by the DDPG
agent does not satisfy (17d) even in some feasible conditions. Therefore, we
can speculate that the DDPG algorithm relying on discretized actions can
result in completely different actions on both sides of the threshold, which
may lead to some action errors and may result in violations of the QoS
constraint.

Figure 8: Performance comparison between STAR-RIS, reflecting-only RIS, and
double spliced RIS

The comparison between STAR-RIS, double spliced RIS, and reflecting-only RIS
is presented in Fig. 8. STAR-RIS has achieved the best overall performance in
all training stages, and the double spliced RIS has about $7\%$ performance
disadvantages over the STAR-RIS. This phenomenon confirms the conclusion that
even if the transmitted and reflected signals of the STAR-RIS are constrained
by each other, the STAR-RIS can provide further signal enhancement for users
than double spliced RISs, since the STAR-RIS has higher multipath gain. The
reflecting-only RIS has achieved a lower reward than the scheme having double-
sides coverage, since the reflecting-only RIS is not capable of serving
$\mathcal{T}$ users. In terms of the number of antennas, the simulation
results prove that the proposed model and algorithm have general
applicability. Furthermore, the multi-antenna gain of the STAR-RIS is higher
than that of the reflecting-only RIS.

Figure 9: Power consumption of STAR-RIS, reflecting-only RIS, and double
spliced RIS

The supporting evidence for Fig. 8 is provided by Fig. 9, which reveals the
transmission power consumption of different types of RISs. The energy
consumption characteristics shown by the STAR-RIS and double spliced RIS
follow their reward trend. It can be observed that since the STAR-RIS can have
a higher number of STAR elements to serve users, it obtained higher gains than
the double spliced RIS. Thus, the BS dissipates less transmission power to
meet the data rate requirements of users. It is worth paying attention to the
fact that the reflecting-only RIS has the lowest power consumption in Fig. 9.
However, this does not suggest that it has an overall favorable energy
efficiency, since it only serves about half the users. Based on similar logic,
in the case of $M=8$, the power consumption seen in Fig. 9 is higher than for
$M=4$, since the feasibility of the constraint (17d) has changed due to the
increased number of antennas.

Figure 10: Performances against the number of STAR/reflecting elements

Fig. 10 plots the reward against the number of STAR or reflection elements of
(STAR) RISs. Upon increasing the number of elements, the rewards have also
been improved to varying degrees, which is in line with the theoretical
expectations of the diversity gain. The result indicates that the performance
of the proposed algorithm is not significantly affected by the action
dimension (element number). Finally, it can be observed in Fig. 10 that the
gains obtained by the double-sided coverage of RISs are more significant than
these of the single-sided RISs.

##  VI Conclusions

A STAR-RIS assisted downlink network model was proposed and the effects of
coupled transmission and reflection phase-shift model were considered for the
STAR-RIS. Although the STAR-RIS expanded the service range of the reflecting-
only RIS, optimizing the beamforming of the coupled transmission and
reflection became a challenging problem, which required both continuous-valued
and discrete-valued control. Thus, we developed a hybrid DDPG algorithm and a
joint DDPG-DQN algorithm for jointly optimizing the active and passive
beamforming to minimize the energy consumption. The analysis and simulation
results indicated that 1) STAR-RIS exhibited superiority over the double
spliced and the reflecting-only RISs in terms of consuming transmission
energy; 2) The proposed hybrid DDPG algorithm and the DDPG-DQN algorithm have
outperformed the conventional DDPG algorithm; 3) The DDPG-DQN algorithm
achieved superior performance compared to the hybrid DDPG algorithm albeit at
an increased complexity.

## References

  * [1] C. Xu, N. Ishikawa, R. Rajashekar, S. Sugiura, R. G. Maunder, Z. Wang, L.-L. Yang, and L. Hanzo, “Sixty years of coherent versus non-coherent tradeoffs and the road from 5G to wireless futures,” _IEEE Access_ , vol. 7, pp. 178 246–178 299, Dec. 2019.
  * [2] X. Mu, Y. Liu, L. Guo, J. Lin, and N. Al-Dhahir, “Exploiting intelligent reflecting surfaces in NOMA networks: Joint beamforming optimization,” _IEEE Trans. Wirel. Commun.s_ , vol. 19, no. 10, pp. 6884–6898, Oct. 2020.
  * [3] M. A. ElMossallamy, H. Zhang, L. Song, K. G. Seddik, Z. Han, and G. Y. Li, “Reconfigurable intelligent surfaces for wireless communications: Principles, challenges, and opportunities,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 6, no. 3, pp. 990–1002, Sept. 2020.
  * [4] W. Zhang and W. P. Tay, “Cost-efficient RIS-aided channel estimation via rank-one matrix factorization,” _IEEE Wirel. Commun. Lett._ , vol. 10, no. 11, pp. 2562–2566, Nov. 2021.
  * [5] M. Naderi Soorki, W. Saad, M. Bennis, and C. S. Hong, “Ultra-reliable indoor millimeter wave communications using multiple artificial intelligence-powered intelligent surfaces,” _IEEE Trans. Commun._ , vol. 69, no. 11, pp. 7444–7457, Nov. 2021.
  * [6] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen, “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,” _IEEE Trans. Wirel. Commun._ , vol. 18, no. 8, pp. 4157–4170, Aug. 2019.
  * [7] I. Alamzadeh, G. C. Alexandropoulos, N. Shlezinger, and M. F. Imani, “A reconfigurable intelligent surface with integrated sensing capability,” _Sci. rep._ , vol. 11, no. 1, pp. 1–10, 2021.
  * [8] H. Zhang, N. Shlezinger, I. Alamzadeh, G. C. Alexandropoulos, M. F. Imani, and Y. C. Eldar, “Channel estimation with simultaneous reflecting and sensing reconfigurable intelligent metasurfaces,” in _IEEE 22nd International Workshop SPAWC_ , Sep. 2021, pp. 536–540.
  * [9] K. Keykhosravi, M. F. Keskin, S. Dwivedi, G. Seco-Granados, and H. Wymeersch, “Semi-passive 3D positioning of multiple RIS-enabled users,” _IEEE Trans. Vehi. Technol._ , vol. 70, no. 10, pp. 11 073–11 077, Oct. 2021.
  * [10] S. Basharat, S. Ali Hassan, H. Pervaiz, A. Mahmood, Z. Ding, and M. Gidlund, “Reconfigurable intelligent surfaces: Potentials, applications, and challenges for 6G wireless networks,” _IEEE Wirel. Commun._ , pp. 1–8, 2021.
  * [11] S. Gong, X. Lu, D. T. Hoang, D. Niyato, L. Shu, D. I. Kim, and Y.-C. Liang, “Toward smart wireless communications via intelligent reflecting surfaces: A contemporary survey,” _IEEE Commun. Surv. Tutor._ , vol. 22, no. 4, pp. 2283–2314, 2020.
  * [12] X. Mu, Y. Liu, L. Guo, J. Lin, and R. Schober, “Simultaneously transmitting and reflecting (STAR) RIS aided wireless communications,” _IEEE Trans. Wirel. Commun._ , vol. early access, Oct. 2021.
  * [13] S. Zeng, H. Zhang, B. Di, Z. Han, and L. Song, “Reconfigurable intelligent surface (RIS) assisted wireless coverage extension: RIS orientation and location optimization,” _IEEE Commun. Lett._ , vol. 25, no. 1, pp. 269–273, Jan. 2021.
  * [14] B. O. Zhu, K. Chen, N. Jia, L. Sun, J. Zhao, T. Jiang, and Y. Feng, “Dynamic control of electromagnetic wave propagation with the equivalent principle inspired tunable metasurface,” _Sci. rep._ , vol. 4, no. 1, pp. 1–7, May. 2014.
  * [15] B. O. Zhu and Y. Feng, “Passive metasurface for reflectionless and arbitary control of electromagnetic wave transmission,” _IEEE Trans. Antennas Propag._ , vol. 63, no. 12, pp. 5500–5511, Dec. 2015.
  * [16] J. Xu, Y. Liu, X. Mu, and O. A. Dobre, “STAR-RISs: Simultaneous transmitting and reflecting reconfigurable intelligent surfaces,” _IEEE Commun. Lett._ , vol. 25, no. 9, pp. 3134–3138, May. 2021.
  * [17] S. Zhang, H. Zhang, B. Di, Y. Tan, Z. Han, and L. Song, “Beyond intelligent reflecting surfaces: Reflective-transmissive metasurface aided communications for full-dimensional coverage extension,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 11, pp. 13 905–13 909, Nov. 2020.
  * [18] C. Wu, Y. Liu, X. Mu, X. Gu, and O. A. Dobre, “Coverage characterization of STAR-RIS networks: NOMA and OMA,” _IEEE Commun. Lett._ , vol. 25, no. 9, pp. 3036–3040, Sep. 2021.
  * [19] Y. Liu, X. Mu, R. Schober, and H. V. Poor, “Simultaneously transmitting and reflecting (STAR)-RISs: A coupled phase-shift model,” _arXiv preprint arXiv:2110.02374_ , 2021.
  * [20] S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” _IEEE Trans. Commun._ , vol. 68, no. 9, pp. 5849–5863, Sep. 2020.
  * [21] J. Wang, C. Jiang, H. Zhang, Y. Ren, K.-C. Chen, and L. Hanzo, “Thirty years of machine learning: The road to Pareto-optimal wireless networks,” _IEEE Commun. Surv. Tutor._ , vol. 22, no. 3, pp. 1472–1514, Jan. 2020.
  * [22] S. Gao, P. Dong, Z. Pan, and G. Y. Li, “Deep multi-stage CSI acquisition for reconfigurable intelligent surface aided MIMO systems,” _IEEE Commun. Lett._ , vol. 25, no. 6, pp. 2024–2028, June 2021.
  * [23] H. Song, M. Zhang, J. Gao, and C. Zhong, “Unsupervised learning-based joint active and passive beamforming design for reconfigurable intelligent surfaces aided wireless networks,” _IEEE Commun. Lett._ , vol. 25, no. 3, pp. 892–896, Mar. 2021.
  * [24] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, Aug. 2020.
  * [25] H. Yang, Z. Xiong, J. Zhao, D. Niyato, L. Xiao, and Q. Wu, “Deep reinforcement learning-based intelligent reflecting surface for secure wireless communications,” _IEEE Trans. Wirel. Commun._ , vol. 20, no. 1, pp. 375–388, Jan. 2021.
  * [26] X. Liu, Y. Liu, Y. Chen, and H. V. Poor, “RIS enhanced massive non-orthogonal multiple access networks: Deployment and passive beamforming design,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 4, pp. 1057–1071, Apr. 2021.
  * [27] M. Samir, M. Elhattab, C. Assi, S. Sharafeddine, and A. Ghrayeb, “Optimizing age of information through aerial reconfigurable intelligent surfaces: A deep reinforcement learning approach,” _IEEE Trans. Veh. Tech._ , vol. 70, no. 4, pp. 3978–3983, Mar. 2021.
  * [28] W. Ni, Y. Liu, Y. C. Eldar, Z. Yang, and H. Tian, “STAR-RIS enabled heterogeneous networks: Ubiquitous NOMA communication and pervasive federated learning,” _arXiv preprint arXiv:2106.08592_ , 2021.
  * [29] M. Neunert, A. Abdolmaleki, M. Wulfmeier, T. Lampe, T. Springenberg, R. Hafner, F. Romano, J. Buchli, N. Heess, and M. Riedmiller, “Continuous-discrete reinforcement learning for hybrid control in robotics,” in _Proceedings of the CoRL_ , vol. 100, 30 Oct–01 Nov 2020, pp. 735–751.
  * [30] O. Delalleau, M. Peter, E. Alonso, and A. Logut, “Discrete and continuous action representation for practical RL in video games,” _arXiv preprint arXiv:1912.11077_ , 2019.
  * [31] B. Li, H. Tang, Y. Zheng, J. Hao, P. Li, Z. Wang, Z. Meng, and L. Wang, “Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation,” _arXiv preprint arXiv:2109.05490_ , 2021.
  * [32] S. Zhang and R. Zhang, “Capacity characterization for intelligent reflecting surface aided MIMO communication,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1823–1838, Aug. 2020.
  * [33] “Study on 3d channel model for lte (3gpp tr 36.873 release 12),” _3rd Generation Partnership Project_ , Jan. 2018.
  * [34] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [35] Y. Li, A. H. Aghvami, and Y. Deng, “Joint resource block and beamforming optimization for cellular-connected UAV networks: A hybrid D3QN-DDPG approach,” _arXiv preprint arXiv:2102.13222_ , 2021.
  * [36] A. Feriani and E. Hossain, “Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial,” _IEEE Commun. Surv. Tutor._ , vol. 23, no. 2, pp. 1226–1252, Mar. 2021.
  * [37] I. Abraham, A. Prabhakar, and T. D. Murphey, “An ergodic measure for active learning from equilibrium,” _IEEE Trans. Autom. Sci. Eng._ , vol. 18, no. 3, pp. 917–931, July 2021.
  * [38] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Process. Mag._ , vol. 34, no. 6, pp. 26–38, Nov. 2017.
  * [39] C. Qiu, Y. Hu, Y. Chen, and B. Zeng, “Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications,” _IEEE Internet Things J._ , vol. 6, no. 5, pp. 8577–8588, Oct. 2019.
  * [40] C. Li, J. Xia, F. Liu, D. Li, L. Fan, G. K. Karagiannidis, and A. Nallanathan, “Dynamic offloading for multiuser muti-CAP MEC networks: A deep reinforcement learning approach,” _IEEE Trans. Vehi. Technol._ , vol. 70, no. 3, pp. 2922–2927, Mar. 2021.