# Deep Reinforcement Learning for RIS-aided Multiuser Full-Duplex Secure
Communications with Hardware Impairments

Zhangjie Peng, Zhibo Zhang, Lei Kong,
Cunhua Pan, Li Li, and Jiangzhou Wang,  This work was supported in part by the
Natural Science Foundation of Shanghai under Grant 22ZR1445600, in part by the
open research fund of National Mobile Communications Research Laboratory,
Southeast University under Grant 2018D14, and in part by the National Natural
Science Foundation of China under Grant 61701307. _(Corresponding authors:
Cunhua Pan and Zhibo Zhang.)_ Zhangjie Peng is with the College of
Information, Mechanical and Electrical Engineering, Shanghai Normal
University, Shanghai 200234, China, also with the National Mobile
Communications Research Laboratory, Southeast University, Nanjing 210096,
China, and also with the Shanghai Engineering Research Center of Intelligent
Education and Bigdata, Shanghai Normal University, Shanghai 200234, China
(e-mail: pengzhangjie@shnu.edu.cn). Zhibo Zhang and Li Li are with the College
of Information, Mechanical and Electrical Engineering, Shanghai Normal
University, Shanghai 200234, China (e-mail<EMAIL_ADDRESS>lilyxuan@shnu.edu.cn). Lei Kong is with New H3C Technologies Co., Limited,
Hangzhou 310052, China, and also with the National Mobile Communications
Research Laboratory, Southeast University, Nanjing 210096, China (e-mail:
konglei.seu@gmail.com). Cunhua Pan is with the National Mobile Communications
Research Laboratory, Southeast University, Nanjing 210096, China (e-mail:
cpan@seu.edu.cn). Jiangzhou Wang is with the School of Engineering, University
of Kent, Canterbury CT2 7NT, U.K. (e-mail: j.z.wang@kent.ac.uk). Copyright (c)
20xx IEEE. Personal use of this material is permitted. However, permission to
use this material for any other purposes must be obtained from the IEEE by
sending a request to<EMAIL_ADDRESS>
###### Abstract

In this paper, we investigate a reconfigurable intelligent surface (RIS)-aided
multiuser full-duplex secure communication system with hardware impairments at
transceivers and RIS, where multiple eavesdroppers overhear the two-way
transmitted signals simultaneously, and an RIS is applied to enhance the
secrecy performance. Aiming at maximizing the sum secrecy rate (SSR), a joint
optimization problem of the transmit beamforming at the base station (BS) and
the reflecting beamforming at the RIS is formulated under the transmit power
constraint of the BS and the unit modulus constraint of the phase shifters. As
the environment is time-varying and the system is high-dimensional, this non-
convex optimization problem is mathematically intractable. A deep
reinforcement learning (DRL)-based algorithm is explored to obtain the
satisfactory solution by repeatedly interacting with and learning from the
dynamic environment. Extensive simulation results illustrate that the DRL-
based secure beamforming algorithm is proved to be significantly effective in
improving the SSR. It is also found that the performance of the DRL-based
method can be greatly improved and the convergence speed of neural network can
be accelerated with appropriate neural network parameters.

###### Index Terms:

Reconfigurable intelligent surface (RIS), secure communication, full-duplex,
hardware impairment, deep reinforcement learning (DRL).

##  I Introduction

Reconfigurable intelligent surface (RIS), also known as intelligent reflecting
surface (IRS), is one of the most promising and disruptive technologies in the
sixth generation (6G) and beyond wireless communication [1, 2, 3, 4]. The RIS
is a uniform planar array made of numerous low-cost and nearly-passive
components. Every unit can reflect the incident electromagnetic wave
independently to the desired direction by dynamically adjusting the phase
shifts or amplitudes with the controller [5]. RIS can be expediently installed
on ceilings or billboards to establish a virtual line-of-sight (LoS) link
between the source and the destination when the direct link is blocked by
obstacles [3, 6].

Due to the inherent broadcast nature of wireless communication, the
transmitted information is easy to be eavesdropped. Thus, physical layer
security (PLS) is an indispensable part of the wireless communication, which
attracts a lot of attention. Recently, RIS has been introduced into secure
communication as an effective way with low power consumption and anti-
eavesdropping. With the assistance of RIS, the reflected signals received by
the legitimate users will be constructively enhanced while the signals leaked
to the eavesdroppers will be destructively weakened. Extensive research
attention has been devoted to RIS-assisted PLS. The initial contributions were
based on the simple system model with only one legitimate user in the presence
of an eavesdropper [7, 8, 9]. While in [10] and [11], a legitimate user was
wiretapped by multiple eavesdroppers. Different from these contributions
considering a single legitimate user [7, 8, 9, 10, 11], an RIS-aided secure
system including multiple legitimate users and eavesdroppers was considered in
[12, 13].

Most of the existing literatures on RIS-assisted PLS [7, 8, 9, 10, 11, 12, 13]
considered half-duplex (HD) communications, in which the signals cannot be
transmitted and received at the same time and over the same frequency.
However, it is less difficult for an eavesdropper to decode the desired signal
from the eavesdropped signal in an HD communication system [14]. The
development of full-duplex (FD) technology in wireless communication provides
the possibility of enhancing PLS [14]. Meanwhile, FD communications enable the
transceivers to exchange information simultaneously in the same frequency
band, which can effectively improve spectrum efficiency with the development
of residual self-interference cancellation technology. Recently, RIS-assisted
FD systems were studied in [15, 16], which highlighted the the superiority of
RIS-aided FD communications under proper residual self-interference
cancellation. Furthermore, in RIS-aided FD secure communication systems, the
bidirectional signals can be simultaneously received and transmitted in the
same frequency band, which can theoretically lead to the overlap of multiple
signals at the eavesdroppers to improve the secrecy performance [17].
Specifically, when the uplink/downlink signals are eavesdropped, the
downlink/uplink signals reflected by the RIS will be reused as interference
noise to degrade the eavesdropping performance of the eavesdroppers. To our
knowledge, the potential of deploying RIS to secure FD communications has
rarely been explored [17, 18]. The closed-form expression for the sum secrecy
rate (SSR) of an RIS-aided multi-pair two-way communication system was derived
in [17]. In [18], the SSR was maximized by optimizing the transmit power and
the phase shifts. However, there is still lack of research work on RIS-
assisted multiuser FD secure communications.

It is worth noting that the above contributions on RIS-assisted secure
communications [7, 8, 9, 10, 11, 12, 13, 18, 17] were based on the assumption
that the hardware is perfect. However, transceiver hardware impairment (T-HWI)
is generally inevitable considering the non-linearities and phase noise, $et\
al.$ [19, 20]. Even if the mitigation/compensation algorithm was proposed,
T-HWI cannot be completely eliminated [19]. A robust beamforming designed
algorithm was proposed in [21] to maximize the received signal-to-noise ratio
(SNR) considering the impact of T-HWI. The robust transmission optimization of
an RIS-assisted secure communication system with T-HWI was studied for the
first time in [22]. At the same time, the high-precision configuration of the
phase shifts at the RIS is not feasible [23], thus there is also the HWI at
the RIS (RIS-HWI). The achievable rate expression of RIS-assisted mmWave
communication system was derived, taking into account the RIS-HWI [24]. The
impact of RIS-HWI on the performance of a secure communication system was
investigated in [25]. However, there were only a few studies that analysed the
RIS-aided communication systems with both the T-HWI and the RIS-HWI. Both
analytical and experimental results in [26] showed that RIS-HWI as well as
T-HWI reduced the achievable rate. The spectral and energy efficiency was
analyzed in [27] considering an RIS-aided MISO communication system with both
T-HWI and RIS-HWI.

All the above-mentioned contributions [7, 8, 9, 10, 11, 12, 13, 15, 16, 18,
17, 19, 20, 21, 22, 23, 24, 25, 26, 14, 27] were based on traditional
mathematical optimization methods such as alternating optimization (AO) and
block coordinate descent (BCD) method to design the transmit beamforming
and/or phase shifts, which are model-based design paradigms that require
accurate mathematical models and expert knowledge. However, traditional model-
based wireless techniques are hard to meet the demanding requirements of
emerging applications in future 6G networks, including excessively complex
communication scenarios with unknown channel model and communication scenarios
that cannot be described mathematically such as the non-linearity due to the
inevitable HWI [28]. On the contrary, deep learning (DL) is a powerful
technique capable of learning complex interrelationships among variables,
especially those that are difficult to accurately describe with mathematical
models, which empowers us to design wireless communication systems without
knowledge of precise mathematical models [28]. In addition, dynamic 6G
networks can lead to uncertainty in the wireless environment (for example, due
to the changing of RIS configuration), which brings great difficulties to
real-time sensing. Meanwhile, artificial intelligence (AI) techniques are more
robust against the uncertainty in the systems, which can achieve accurate
real-time sensing. Therefore, the application of AI to 6G networks can
optimize the network structure and improve the communication performance [29].
Inspired by the advantages of model-free AI, extensive research attention has
been devoted to apply DL to the phase shifts design [30, 31].

However, DL methods that have been applied in the existing contributions [30,
31] to solve the sophisticated problems are supervised learning. Although
supervised learning has shown promising results in wireless problems such as
channel estimation, RIS phase shifts design and so on, it requires a large
number of prior labeled training and test dataset for offline training, which
relies on either the AO algorithms or the exhaustive search. As known, the
training dataset is difficult to obtain in many circumstances. Furthermore,
for an RIS-aided communication system, once the number of phase shifters
changes, numerous training samples need to be reacquired and the DL network
need to be retrained from scratch, which greatly limits the application of DL-
based method. In the considered sophisticated optimization problem, the
artificially labeled training dataset is unable to be acquired. On the
contrary, as a branch of machine learning (ML), reinforcement learning (RL)
has been considered to be an effective approach to deal with complex control
tasks, which possesses the characteristics of online learning and sample
generation, and does not require the training labels with prior knowledge.
Furthermore, deep reinforcement learning (DRL), which combines the advantages
of DL in function fitting and RL in policy-making, has been leveraged to
optimize the phase shifts [32, 33, 34]. Recently, DRL-based methods have also
been applied to solve the optimization problem for RIS-aided secure
communications [35, 36]. On the other hand, due to the time-varying channels
and the high-dimensional system, the proposed non-convex optimization problem
is mathematically intractable. It is challenging to derive the explicit
mathematical optimal solution for this problem. Fortunately, no complicated
mathematical formulations is necessary for DRL, which can significantly reduce
the computational complexity and computing time. Meanwhile, DRL is able to
learn the implicit knowledge about the radio channels through the trial-and-
error interactions without knowing the channel model and mobility pattern.
Therefore, DRL is particularly beneficial to deal with the problem in a high-
dimensional wireless communication system where the radio channel varies over
time. Motivated by the aforementioned facts, we explore a DRL-based method to
address the proposed sophisticated optimization problem.

In this paper, we present an RIS-assisted FD two-way wiretap communication
system with multiple legitimate users as well as multiple eavesdroppers. Both
T-HWI and RIS-HWI are considered. The transmit beamforming at the BS and phase
shifts at the RIS are jointly optimized to maximize the SSR. This complex
optimization problem is non-convex and the optimal solution is unknown. A
novel DRL-based method is developed to obtain the satisfactory solution, in
contrast to numerical optimization algorithms by utilizing complicated
mathematical formulations. The main contributions of this paper are summarized
as follows:

  * •

To the best of our knowledge, this is the first attempt to study the
performance of an RIS-assisted FD secure communication system with both T-HWI
and RIS-HWI. In order to improve the secrecy performance, an optimization
problem for jointly designing the transmit and reflecting beamforming is
formulated. In the complex and time-varying environment, it is a
mathematically intractable non-convex optimization problem, which is modeled
as a Markov decision process (MDP).

  * •

A DRL-based secure beamforming algorithm is developed to jointly design the
transmit and reflecting beamforming to maximize the SSR. Specifically, the SSR
is used as the instant reward to train the DRL network. The training data is
generated online through the trial-and-error interactions between the agent
and the environment. The network parameters are adjusted accordingly, so as to
design the transmit and reflecting beamforming simultaneously by gradually
maximizing the SSR through repeated iterations. Since the action space
containing the transmit beamforming matrix and the phase shifts matrix is
continuous, DDPG technique is adopted.

  * •

Dispensing with explicit mathematical calculation, the proposed DRL-based
algorithm has a standard framework, as well as low complexity in its
implementation, which can be conveniently deployed to the communication system
with different settings. On the other hand, compared with the DL-based
algorithm that requires artificially labeled training datasets, the DRL-based
algorithm learns the knowledge of the environment to adapt.

  * •

Extensive simulation results illustrate that the proposed DRL-based algorithm
is significantly effective in improving the SSR. Specifically, the agent
improves its action policy step by step according to the reward fed back from
the environment, to obtain the satisfactory transmit beamforming and phase
shifts to improve the SSR. It is also found that the performance of the DRL-
based method can be greatly improved and the convergence speed of neural
network can be accelerated with appropriate neural network parameters.

The remainder of this paper is organized as follows. In Section II, we
introduce the RIS-aided FD secure communication system with HWI and formulate
the SSR maximization problem. In Section III, we present the proposed DRL-
based algorithm for joint optimization of the transmit beamforming and the
phase shifts. In Section IV, we provide extensive simulation results to
elaborate the performance of the developed DRL-based algorithm. In Section V,
we draw a brief conclusion.

##  II System Model and Problem Formulation

###  II-A System Model

Consider an RIS-assisted FD two-way wiretap communication system, as
illustrated in Fig. 1. Both the BS and the legitimate users work in the FD
mode. The BS transmits independent confidential information to $K$ legitimate
users, while each legitimate user sends a data stream to the BS. At the same
time, $L$ unauthorized eavesdroppers that are distributed around the BS and
legitimate users are trying to wiretap the transmission information from the
BS and the legitimate users independently. An RIS is applied to enhance the
secrecy performance. Moreover, due to the existence of many obstacles, the
signals are only transmitted via the RIS-assisted reflecting links. We denote
the $k$-th legitimate user as $\text{B}_{k}$ and the $\ell$-th eavesdropper as
$\text{E}_{\ell}$, for $k=1,\ldots,K$ and $\ell=1,\ldots,L$. Each legitimate
user has a single transmit antenna and a single receive antenna, while each
eavesdropper is equipped with one antenna [37]. The BS is equipped with
$N_{t}$ transmit antennas and $N_{r}$ receive antennas. The RIS consists of
$M$ passive reflecting elements. The phase shift of each reflecting element
can be dynamically adjusted by a smart controller coordinated with the BS. The
phase shifts matrix $\bm{\Theta}$ is denoted by
$\bm{\Theta}={\rm{diag}}({\chi_{1}}{e^{j{\theta_{1}}}},\ldots,{\chi_{m}}{e^{j{\theta_{m}}}},\ldots,{\chi_{M}}{e^{j{\theta_{M}}}})$,
where ${\chi_{m}}\in\left[{0,1}\right]$ and
${\theta_{m}}\in\left[{0,2\pi}\right]$ represent the amplitude and phase shift
of the $m$-th reflecting element, respectively.

Figure 1: System model for RIS-aided multiuser FD two-way secure communication
system with multi-eavesdropper.

Channel state information (CSI) is crucial for achieving the performance gain
of the RIS-assisted secure communication. We suppose that the CSI of all
channels is perfectly known at the BS before the data transmission. On the one
hand, the acquisition of CSI for the legitimate users has been widely explored
[38, 39, 40]. On the other hand, for the CSI of the eavesdroppers, if the
eavesdroppers are active in the secure transmission system111For example, the
eavesdroppers can be disguised as active users in the secure communication
system trusted by the BS but untrusted by the legitimate users., then their
CSI acquisition can be acquired by regarding them as the legitimate users. If
the eavesdroppers are passive222The eavesdroppers are passive, which means
that they never transmit., their CSI can also be obtained by some
technologies, such as detecting the local oscillator power accidentally
leakage from the eavesdropper receivers’ RF front end [41, 42].

According to [43], we assume that the power of the signal reflected twice or
more times by the RIS is imperceptible, and therefore is ignored. In addition,
RIS is designed to maximize the power of the incident signal so that there is
no energy loss during reflection.

###  II-B Channel Model

According to the above-mentioned eavesdroppers’ CSI acquisition method, we
suppose that the BS perfectly knows the eavesdroppers’ CSI. The baseband
equivalent channels BS $\to$ RIS, RIS $\to$ BS, RIS $\to$ $k$-th legitimate
user $\text{B}_{k}$, $\text{B}_{k}$ $\to$ RIS , the downlink RIS $\to$
$\ell$-th eavesdropper $\text{E}_{\ell}$, and the uplink RIS $\to$
$\text{E}_{\ell}$ are represented as ${{\bf{H}}_{d}}\in{{\mathbb{C}}^{M\times
N_{t}}}$, ${{\bf{H}}_{u}}\in{{\mathbb{C}}^{M\times N_{r}}}$,
${{\bf{h}}_{d,k}}\in{{\mathbb{C}}^{M\times 1}}$,
${{\bf{h}}_{u,k}}\in{{\mathbb{C}}^{M\times 1}}$,
${{\bf{g}}_{d,\ell}}\in{{\mathbb{C}}^{M\times 1}}$,
${{\bf{g}}_{u,\ell}}\in{{\mathbb{C}}^{M\times 1}}$, respectively, as shown in
Fig. 1.

It is assumed that the RIS is installed on the outer wall of a high building
with only a few scatters. Therefore, we adopt a Rician fading channel
model333Actually, the proposed algorithm is suitable for any channel model.
for all reflecting channels, which is composed of LoS components as well as
NLoS components. The channels can be expressed as

$\displaystyle{\bf{h}}=\sqrt{\alpha_{\bf{h}}}\left({\sqrt{\frac{\varepsilon_{\bf{h}}}{{\varepsilon_{\bf{h}}+1}}}\overline{\bf{h}}+\sqrt{\frac{1}{{\varepsilon_{\bf{h}}+1}}}\widetilde{\bf{h}}}\right),$
(1)

where
${\bf{h}}\in{\bf{\mathcal{H}}}=\left\\{{{{\bf{H}}_{d}},{{\bf{H}}_{u}},{{\bf{h}}_{d,k}},{{\bf{h}}_{u,k}},{{\bf{g}}_{d,\ell}},{{\bf{g}}_{u,\ell}}}\right\\}$,
$\alpha_{\bf{h}}$ denotes the large-scale fading coefficient,
$\varepsilon_{\bf{h}}$ denotes the Rician factor. ${\overline{\bf{h}}}$ is
deterministic LoS channel component. ${\widetilde{\bf{h}}}$ is the NLoS
channel component whose elements are independent and identical distribution
(i.i.d.) standard Gaussian random variables following the distribution of
$\mathcal{CN}(0,1)$.

The BS and RIS are both equipped with uniform linear array (ULA), the LoS
components are modeled as

$\displaystyle\overline{\bf{h}}={{\bf{a}}_{{\bf{h}},r}}\left({{\vartheta^{{\bf{h}},{\rm{AOA}}}}}\right){\bf{a}}_{{\bf{h}},t}^{\rm{H}}\left({{\vartheta^{{\bf{h}},{\rm{AOD}}}}}\right),$
(2)

where
${{\bf{a}}_{{\bf{h}},r}}\left({{\vartheta^{{\bf{h}},{\rm{AOA}}}}}\right)$ and
${\bf{a}}_{{\bf{h}},t}^{\rm{H}}\left({{\vartheta^{{\bf{h}},{\rm{AOD}}}}}\right)$
are the array responses of $W_{\bf{h}}$-element/antenna ULA, which are
respectively given by

$\displaystyle\\!\\!\\!{{\bf{a}}_{{\bf{h}},r}}\left({{\vartheta^{{\bf{h}},{\rm{AOA}}}}}\right)\\!\\!=\\!\\!\left[{1,\ldots,{e^{j2\pi\frac{d_{\bf{h}}}{\lambda_{\bf{h}}}\left({{W_{{\bf{h}},r}}-1}\right)\sin{\vartheta^{{\bf{h}},{\rm{AOA}}}}}}}\right]^{T},$
(3)
$\displaystyle\\!\\!\\!\\!{{\bf{a}}_{{\bf{h}},t}}\left({{\vartheta^{{\bf{h}},{\rm{AOD}}}}}\right)\\!\\!=\\!\\!\left[{1,\ldots,{e^{j2\pi\frac{d_{\bf{h}}}{\lambda_{\bf{h}}}\left({{W_{{\bf{h}},t}}-1}\right)\sin{\vartheta^{{\bf{h}},{\rm{AOD}}}}}}}\right]^{T},$
(4)

where $W_{{\bf{h}},r}$ and $W_{{\bf{h}},t}$ represent the numbers of
antennas/elements at the receiver and transmitter, respectively[16],
${\vartheta^{{\bf{h}},{\rm{AOA}}}}$ and ${\vartheta^{{\bf{h}},{\rm{AOD}}}}$
are the angle of arrival (AoA) and angle of departure (AoD), respectively,
$d_{\bf{h}}$ is the inter-element spacing at the ULA, $\lambda_{\bf{h}}$ is
the carrier wavelength.

###  II-C Hardware Impairment Model

In practice, due to the non-ideality of hardware, the transmitted and received
signals are affected by HWI, which generally exists in practical communication
systems, including RIS-aided systems with RIS-HWI [26]. In the considered
system, there are two kinds of HWI with different mathematical models,
including RIS-HWI and T-HWI.

Firstly, RIS-HWI is caused by the inherent hardware imperfections of the
reflecting elements or the imperfect channel estimations, which is also called
phase noise [23]. The phase noise matrix is expressed as
${\bf{\Phi}}={\rm{diag}}\left({{e^{j\Delta{\theta_{1}}}},\ldots,{e^{j\Delta{\theta_{m}}}},\ldots,{e^{j\Delta{\theta_{M}}}}}\right)$,
where $\Delta{\theta_{m}}$ is random phase noise uniformly distributed within
$\left[{{{-\pi}\mathord{\left/{\vphantom{{-\pi}{2,{\pi\mathord{\left/{\vphantom{\pi
2}}\right.\kern-1.2pt}2}}}}\right.\kern-1.2pt}{2,{\pi\mathord{\left/{\vphantom{\pi
2}}\right.\kern-1.2pt}2}}}}\right]$ for $m=1,2,\ldots,M$, according to [26].

Then, another kind of HWI, namely T-HWI, is modeled as an independent Gaussian
distortion noise, which will cause that the received signal dose not match the
expected signal or produce distortion on the received signal during signal
processing [26]. The distortion power of the transmit antenna increases with
the increase of the signal power allocated to the antenna [19].

$\displaystyle\Gamma_{k}^{\rm B}\left({{\bf{W}},{\bf{\Theta}}}\right)=$
$\displaystyle\sum\limits_{i=1,i\neq
k}^{K}{{\left|{{\bf{h}}_{d,k}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{i}}}\right|}^{2}}+\kappa_{d}^{\rm{S}}{\bf{h}}_{d,k}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{{d}}}\widetilde{{\rm{diag}}}\left\\{{\sum\limits_{i=1}^{K}{{{\bf{w}}_{i}}}{\bf{w}}_{{i}}^{\rm{H}}}\right\\}{\bf{H}}_{{d}}^{\rm{H}}{{\bf{\Phi}}^{\rm{H}}}{{\bf{\Theta}}^{\rm{H}}}{{\bf{h}}_{d,k}}+\sigma_{d,k}^{2}$
$\displaystyle+\sum\limits_{i=1}^{K}{\left({1+\kappa_{u,i}^{\rm{B}}}\right)\rho{P_{i}}{{\left|{{\bf{h}}_{{}_{u,k}}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}}\right|}^{2}}}.$
(11)

###  II-D Signal Transmission Model

At the BS side, let ${\bf{w}}_{k}\in{{\mathbb{C}}^{N_{t}\times 1}}$ be the
beamforming vector for $\text{B}_{k}$ and ${{s}}_{d,k}$ be the confidential
information sent to $\text{B}_{k}$. Therefore, the signal transmitted from the
BS is modeled as

$\displaystyle{{\bf{x}}_{d}}=$
$\displaystyle~{}{\widehat{\bf{x}}_{d}}+{\bm{\eta}}_{d}^{\rm{S}},$ (5a)
$\displaystyle{\widehat{\bf{x}}_{d}}=$
$\displaystyle~{}\sum\limits_{k=1}^{K}{{{\bf{w}}_{k}}{s_{d,k}}},$ (5b)

where ${{s}}_{d,k}$ is modeled as i.i.d. random variable with zero mean and
unit variance, i.e. ${{s}}_{d,k}\sim\mathcal{CN}(0,1)$ and
$\mathbb{E}\\{{\left|s_{d,k}\right|^{2}}\\}=1$. According to [44],
${\bm{\eta}}_{d}^{\rm{S}}\in{{\mathbb{C}}^{N_{t}\times 1}}$ is the independent
Gaussian transmit distortion noise. The power of the distortion noise at each
antenna is proportional to the transmit power at that antenna, which can be
modeled as
$\bm{\eta}_{d}^{\rm{S}}\sim\mathcal{CN}\left({\bm{0},\bm{\Upsilon}_{d}^{\rm{S}}}\right)$,
where [21]

$\displaystyle\bm{\Upsilon}_{d}^{\rm{S}}=\kappa_{d}^{\rm{S}}\widetilde{\rm{diag}}\left\\{\sum\limits_{k=1}^{\rm{K}}{{{{{\bf{w}}_{k}}}{{{\bf{w}}_{k}^{\rm
H}}}}}\right\\},$ (6)

with $\kappa_{d}^{\rm{S}}\geq 0$ being a scale factor that describes the
severity of HWI at the BS’s transmitter, $\widetilde{\rm{diag}}{\\{{\bf
X}\\}}$ denotes a diagonal matrix whose diagonal entries are the diagonal
elements of matrix $\bf{X}$. The transmit power at the BS is subject to the
maximum power constraint

$\displaystyle{\rm{Tr}}\left({{\bf{W}}{{\bf{W}}^{\rm
H}}}\right)\leq{P_{max}},$ (7)

where
${\bf{W}}\buildrel\Delta\over{=}\left[{{{\bf{w}}_{1}},{{\bf{w}}_{2}},\ldots,{{\bf{w}}_{K}}}\right]\in{{\mathbb{C}}^{N_{t}\times
K}}$, and $P_{max}$ is maximum transmit power of the BS.

Similarly, at the legitimate users side, the transmitted signal from $B_{k}$
is

$\displaystyle{x_{u,k}}=$ $\displaystyle~{}\sqrt{{P_{k}}}{{{s}}_{u,k}},$ (8a)
$\displaystyle{{{s}}_{u,k}}=$
$\displaystyle~{}{\widehat{{s}}_{u,k}}+\eta_{u,k}^{\rm{B}},$ (8b)

where ${\widehat{{{s}}}_{u,k}}\sim\mathcal{CN}(0,1)$ represents the
independent Gaussian uplink information symbol sent by $B_{k}$, ${P_{k}}$ is
the transmit power of $B_{k}$, $\eta_{u,k}^{\rm{B}}$ is an independent
Gaussian transmit distortion noise and
$\eta_{u,k}^{\rm{B}}\sim\mathcal{CN}(0,\kappa_{u,k}^{\rm{B}}\mathbb{E}\left\\{{{{\widehat{{s}}}_{u,k}}\widehat{{s}}_{u,k}^{\rm{H}}}\right\\})$
, where $\kappa_{u,k}^{\rm{B}}\geq 0$ represents a scale factor that describes
the severity of HWI at the $k$-th legitimate user’s transmitter.

For downlink, the received signal at $B_{k}$ is written as

$\displaystyle y_{k}^{\rm{B}}=$
$\displaystyle\underbrace{{\bf{h}}_{d,k}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{{\bf{w}}_{k}}{{{s}}_{d,k}}}}_{\text{Desired\
signal}}+\\!\\!\\!\underbrace{\sum\limits_{i=1,i\neq
k}^{K}{{\bf{h}}_{{}_{d,k}}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{{\bf{w}}_{i}}{{{s}}_{d,i}}}}}_{\text{Multiuser\
interference}}$
$\displaystyle+\underbrace{{\bf{h}}_{d,k}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bm{\eta}}_{d}^{\rm{S}}}}_{\text{Transmitter\
HWI}}+\underbrace{\sqrt{{\rho_{L}}}\sqrt{{P_{k}}}{h_{kk}}\left({{{\widehat{{s}}}_{u,k}}+\eta_{u,k}^{\rm{B}}}\right)}_{\text{Loop-
interference}}$
$\displaystyle+\underbrace{\sqrt{{\rho_{S}}}\sqrt{{P_{k}}}{\bf{h}}_{{}_{d,k}}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\left({{{\widehat{{s}}}_{u,k}}+\\!\eta_{u,k}^{\rm{B}}}\right)}_{\text{Self-
interference}}~{}+\\!\\!\\!\\!\\!\underbrace{\eta_{d,k}^{\rm{B}}}_{\text{Receiver
HWI}}$ $\displaystyle+\\!\\!\\!\\!\underbrace{\sum\limits_{i=1,i\neq
k}^{K}{\sqrt{{P_{i}}}{\bf{h}}_{{}_{d,k}}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\left({{{\widehat{{s}}}_{u,i}}+\eta_{u,i}^{\rm{B}}}\right)}}_{\text{Co-
channel\ interference}}+\underbrace{{{n}}_{k}^{\rm{B}}}_{\text{Noise}},$ (9)

where $h_{kk}$ is the loop channel between the transmit antenna and receive
antenna of the $k$-th legitimate user $B_{k}$, ${{{n}}_{k}^{\rm{B}}}$ is
additive white Gaussian noise (AWGN) following the distribution of
$\mathcal{CN}(0,\sigma_{k}^{2})$, and ${\eta_{d,k}^{\rm{B}}}$ is an
independent additional distortion noise term at the $k$-th legitimate user’s
receiver. Similar to [22], we denote the received signal as
$y_{k}^{\rm{B}}=\widetilde{y}_{k}^{\rm{B}}+\eta_{d,k}^{\rm{B}}$, then
${\eta_{d,k}^{\rm{B}}}$ follows the distribution of
$\mathcal{CN}(0,\kappa_{d,k}^{\rm{B}}\mathbb{E}\left\\{{{\left|\widetilde{y}_{k}^{\rm{B}}\right|}^{2}}\right\\})$,
where $\kappa_{d,k}^{\rm B}$ is a proportionality coefficient that describes
the severity of HWI at the $k$-th legitimate user’s receiver. The coefficients
$\rho_{L}$ and $\rho_{S}$ with $0\leq{\rho_{L}},{\rho_{S}}\leq 1$ are the
coefficients of loop-interference (LI) and self-interference (SI),
respectively. $\rho_{L}$ and $\rho_{S}$ are introduced to characterize the LI
and SI which cannot be completely eliminated by some interference elimination
techniques, respectively. According to [16], LI term and noise term can be
combined, the sum of which is denoted as ${{m}}_{d,k}$, whose average power is
written as
$\sigma_{d,k}^{2}={\left|{{m_{d,k}}}\right|^{2}}={\rho_{L}}{P_{k}}{\left|{{h_{kk}}}\right|^{2}}+\sigma_{k}^{2}$.
Accordingly, the signal-to-interference-plus-noise ratio (SINR) at $B_{k}$ is

$\displaystyle{\rm{SINR}}_{k}^{\rm{B}}=\frac{{{{\left|{{\bf{h}}_{d,k}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{k}}}\right|}^{2}}}}{{\Gamma_{k}^{\rm
B}\left({{\bf{W}},{\bf{\Theta}}}\right)+{{\left|{\eta_{d,k}^{\rm{B}}}\right|}^{2}}}},$
(10)

where $\Gamma_{k}^{\rm B}\left({{\bf{W}},{\bf{\Theta}}}\right)$ is given by
(II-C) at the bottom of this page, where the coefficient $\rho$ is defined as
$\rho=\left\\{{\begin{array}[]{*{20}{c}}{{\rho_{\rm{S}}},\ \ {\text{if }}\
i=k;}\\\ {1,\ \ {\text{otherwise}}{.}}\end{array}}\right.$

$\displaystyle{\Gamma_{k}^{\rm
S}}\left({{\bf{W}},{\bf{\Theta}}}\right)={\sum\limits_{i=1,i\neq
k}^{K}{{P_{i}}{{\left|{{\bf{f}}_{u,k}^{\rm{H}}{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}}\right|}^{2}}}+\sum\limits_{i=1}^{K}{\kappa_{u,i}^{\rm{B}}{P_{i}}{{\left|{{\bf{f}}_{u,k}^{\rm{H}}{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}}\right|}^{2}}}+{{\left\|{{{\bf{f}}_{u,k}^{\rm{H}}}}\right\|}^{2}}{\delta_{u,k}^{2}}}.$
(15)
$\displaystyle\Gamma_{d,k,\ell}^{\rm{E}}\left({{\bf{W}},{\bf{\Theta}}}\right)=$
$\displaystyle\sum\limits_{i=1}^{K}{{{\left|{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}}\right|}^{2}}}+\sum\limits_{i=1}^{K}{\kappa_{u,i}^{\rm{B}}{{\left|{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}}\right|}^{2}}}+\sum\limits_{i=1,i\neq
k}^{K}{{{\left|{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{i}}}\right|}^{2}}}$
$\displaystyle+\kappa_{d}^{\rm{S}}{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}\widetilde{{\rm{diag}}}\left({\sum\limits_{i=1}^{K}{{{\bf{w}}_{i}}}{\bf{w}}_{{i}}^{\rm{H}}}\right){\bf{H}}_{d}^{\rm{H}}{{\bf{\Phi}}^{\rm{H}}}{{\bf{\Theta}}^{\rm{H}}}{{\bf{g}}_{d,\ell}}.$
(18)
$\displaystyle\Gamma_{u,k,\ell}^{\rm{E}}\left({{\bf{W}},{\bf{\Theta}}}\right)=$
$\displaystyle\sum\limits_{i=1,i\neq
k}^{K}{{{\left|{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}}\right|}^{2}}}+\sum\limits_{i=1}^{K}{\kappa_{u,i}^{\rm{B}}{{\left|{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}}\right|}^{2}}}+\sum\limits_{i=1}^{K}{{{\left|{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{i}}}\right|}^{2}}}$
$\displaystyle+\kappa_{d}^{\rm{S}}{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}\widetilde{{\rm{diag}}}\left({\sum\limits_{i=1}^{K}{{{\bf{w}}_{i}}}{\bf{w}}_{{i}}^{\rm{H}}}\right){\bf{H}}_{d}^{\rm{H}}{{\bf{\Phi}}^{\rm{H}}}{{\bf{\Theta}}^{\rm{H}}}{{\bf{g}}_{d,\ell}}.$
(19)

Then, we consider the uplink, the received signal at the BS can be written as

$\displaystyle{\bf{y}}_{k}^{\rm{S}}=$
$\displaystyle\underbrace{{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}{{{\widehat{{s}}}_{u,k}}}}_{{\rm{Desired\
signal}}}+\underbrace{{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}{\eta_{u,k}^{\rm{B}}}}_{{\rm{Transmitter\
HWI}}}$ $\displaystyle+\underbrace{\sum\limits_{i=1,i\neq
k}^{K}{{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}\left({{{\widehat{{s}}}_{u,i}}+\eta_{u,i}^{\rm{B}}}\right)}}_{{\rm{Multiuser\
interference}}}$
$\displaystyle+\underbrace{{{\bf{H}}_{S}}\sum\limits_{i=1}^{K}{\left({{{\bf{w}}_{i}}{{{s}}_{d,i}}+{\bm{\eta}}_{d}^{\rm{S}}}\right)}}_{{\rm{Loop-
interference}}}~{}+\\!\\!\\!\\!\\!\\!\underbrace{{\bm{\eta}}_{u}^{\rm{S}}}_{{\rm{Receiver\
HWI}}}$
$\displaystyle+\underbrace{{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}\sum\limits_{i=1}^{K}{\left({{{\bf{w}}_{i}}{{{s}}_{d,i}}+{\bm{\eta}}_{d}^{\rm{S}}}\right)}}_{{\rm{Self-
interfenrence}}}+\underbrace{{{\bf{n}}^{\rm{S}}}}_{{\rm{Noise}}},$ (12)

where ${\bf{H}}_{S}$ is the loop channel between the transmit antennas and
receive antennas of the BS, ${\bf{n}}^{\rm{S}}$ is AWGN vector, whose elements
are independent and follow the same distribution of
$\mathcal{CN}(0,{\bf{\delta}}_{k}^{2}$). ${{\bm{\eta}}_{u}^{\rm{S}}}$ is an
independent additional distortion noise term at the BS’s receiver. Similar to
the downlink, we express the received signal as
${\bf{y}}_{k}^{\rm{S}}=\widetilde{\bf{y}}_{k}^{\rm{S}}+{\bm{\eta}}_{u}^{\rm{S}}$,
then ${{\bm{\eta}}_{u}^{\rm{S}}}$ follows the distribution of
$\mathcal{CN}(0,\kappa_{u}^{\rm{S}}\mathbb{E}\left\\{{{\widetilde{\rm{diag}}\left\\{\widetilde{\bf{y}}_{k}^{\rm{S}}{{\mathop{\widetilde{\bf{y}}_{k}^{\rm{S}}}\nolimits^{\rm{H}}}}\right\\}}}\right\\})$,
where $\kappa_{u}^{\rm S}$ is a proportionality coefficient that describes the
severity of HWI at the BS’s receiver. According to [45] and [46], we assume
that the BS is equipped with the SI and LI cancellation capabilities, thus LI
and SI can be effectively cancelled. The residual noise generated in the
interference cancellation process can be modeled as i.i.d. AWGN. Therefore,
the total noise can be denoted as
${\bf{m}}_{\rm{S}}\buildrel\Delta\over{=}\left[{{m_{1}},{m_{2}},\ldots,{m_{{N_{r}}}}}\right]^{T}$,
where ${m_{i}}\sim\mathcal{CN}(0,{\bf{\delta}}_{u,k}^{2}),i=1,2,\ldots,N_{r}$.

Note that the decoding matrix at the BS, which is denoted by
${{\bf{F}}}{\buildrel\Delta\over{=}}[{\bf{f}}_{u,1},\ldots,{\bf{f}}_{u,k},\ldots,{\bf{f}}_{u,K}]\in{{\mathbb{C}}^{N_{r}\times
K}}$, where ${\bf{f}}_{u,k}\in{{\mathbb{C}}^{N_{r}\times 1}}$ denotes the
combining vector at the BS for $\text{B}_{k}$. Therefore, the recovered signal
at the BS is obtained by [16]

$\displaystyle{{y}}_{u,k}^{\bf{f}}=\ $
$\displaystyle{\bf{f}}_{u,k}^{\rm{H}}{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}{\widehat{{s}}_{u,k}}+{\bf{f}}_{u,k}^{\rm{H}}{\bm{\eta}}_{u}^{\rm{S}}+{\bf{f}}_{u,k}^{\rm{H}}{{\bf{m}}_{\rm{S}}}$
$\displaystyle+{\bf{f}}_{u,k}^{\rm{H}}\sum\limits_{i=1,i\neq
k}^{K}{{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}\left({{{\widehat{{s}}}_{u,i}}+\eta_{u,i}^{\rm{B}}}\right)}$
$\displaystyle+{\bf{f}}_{u,k}^{\rm{H}}{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}\eta_{u,k}^{\rm{B}}.$
(13)

Accordingly, the SINR of the recovered signal at the BS is obtained by

$\displaystyle{\rm{SINR}}_{k}^{\rm{S}}=\frac{{{P_{k}}{{\left|{{\bf{f}}_{u,k}^{\rm{H}}{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}}\right|}^{2}}}}{{{\Gamma_{k}^{\rm{S}}}\left({{\bf{W}},{\bf{\Theta}}}\right)+{{{\bf{f}}_{u,k}^{\rm{H}}{{\bm{\eta}}_{u}^{\rm{S}}{\mathop{\bm{\eta}_{u}^{\rm{S}}}\nolimits^{\rm{H}}}}{\bf{f}}_{u,k}}}}},$
(14)

where ${\Gamma_{k}^{\rm S}}\left({{\bf{W}},{\bf{\Theta}}}\right)$ is given by
(15) at the bottom of the previous page.

Since the full knowledge about the eavesdroppers cannot be well acquired by
the BS, we make the worst-case assumptions that the signal processing
capabilities of the eavesdroppers are significantly strong and the hardware of
the eavesdroppers are perfect. Thus there is no residual HWIs on the
eavesdroppers[47]. Correspondingly, the signal eavesdropped by the $\ell$-th
eavesdropper $E_{\ell}$ can be written as

$\displaystyle y_{k,\ell}^{\rm{E}}\\!=$
$\displaystyle\underbrace{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}{{\widehat{s}}_{u,k}}}_{{\rm{Uplink\
desired\ signal}}}+\\!\\!\underbrace{\sum\limits_{i=1,i\neq
k}^{K}{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}{{\widehat{s}}_{u,i}}}}_{{\rm{Uplink\
multiuser\ interference}}}$
$\displaystyle+\underbrace{\sum\limits_{i=1}^{K}{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,i}}\sqrt{{P_{i}}}\eta_{u,i}^{\rm{B}}}}_{{\rm{Uplink\
HWI}}}+\underbrace{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{k}}{{{s}}_{d,k}}}_{{\rm{Downlink\
desired\ signal}}}$ $\displaystyle+\underbrace{\sum\limits_{i=1,i\neq
k}^{K}{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{i}}{{{s}}_{d,i}}}}_{{\rm{Downlink\
multiuser\
interference}}}+\\!\\!\underbrace{{{n}}_{u,\ell}^{\rm{E}}}_{{\rm{Uplink\
noise}}}$
$\displaystyle+\underbrace{{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}\bm{\eta}_{d}^{\rm{S}}}}_{{\rm{Downlink\
HWI}}}~{}+\\!\\!\\!\\!\underbrace{{{n}}_{d,\ell}^{\rm{E}}}_{{\rm{Downlink\
noise}}},$ (16)

where ${n}_{u,\ell}^{\rm{E}}\sim\mathcal{CN}(0,\mu_{{u,\ell}}^{2})$ and
${n}_{d,\ell}^{\rm{E}}\sim\mathcal{CN}(0,\mu_{{d,\ell}}^{2})$ denote the AWGN.
The SINR at $E_{\ell}$ are respectively written as [48]

$\displaystyle{\rm{SINR}}_{d,k,\ell}^{\rm{E}}=\frac{{{{\left|{{\bf{g}}_{d,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{\bf{w}}_{k}}}\right|}^{2}}}}{{\Gamma_{d,k,\ell}^{\rm{E}}\left({{\bf{W}},{\bf{\Theta}}}\right)+\mu_{{\rm{d,\ell}}}^{2}}},$
(17a)
$\displaystyle{\rm{SINR}}_{u,k,\ell}^{\rm{E}}=\frac{{{{\left|{{\bf{g}}_{u,\ell}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u,k}}\sqrt{{P_{k}}}}\right|}^{2}}}}{{\Gamma_{u,k,\ell}^{\rm{E}}\left({{\bf{W}},{\bf{\Theta}}}\right)+\mu_{{\rm{u,\ell}}}^{2}}},$
(17b)

where $\Gamma_{d,k,\ell}^{\rm E}\left({{\bf{W}},{\bf{\Theta}}}\right)$ and
$\Gamma_{u,k,\ell}^{\rm E}\left({{\bf{W}},{\bf{\Theta}}}\right)$ are
respectively given by (II-D) and (II-D) at the bottom of this page.

Furthermore, the information rate for the $k$-th legitimate user $B_{k}$, the
BS and the $\ell$-th eavesdropper $E_{\ell}$ can be written as

$\displaystyle{R}_{k}^{\rm{B}}={\log_{2}}\left({1+{\rm{SINR}}_{k}^{\rm{B}}}\right),$
(20)
$\displaystyle{R}_{k}^{\rm{S}}={\log_{2}}\left({1+{\rm{SINR}}_{k}^{\rm{S}}}\right),$
(21)
$\displaystyle{R}_{k,\ell}^{\rm{E}}={\log_{2}}\left({1+{\rm{SINR}}_{d,k,\ell}^{\rm{E}}}\right)\\!+\\!{\log_{2}}\left({1+{\rm{SINR}}_{u,k,\ell}^{\rm{E}}}\right).$
(22)

Since every eavesdropper has the ability to wiretap any signal of $K$
legitimate users and the BS, according to [12][35], the sum of the secrecy
rate BS $\to$ $B_{k}$ and the secrecy rate $B_{k}$ $\to$ BS can be represented
as

$\displaystyle
R_{k}^{\sec}={\left[{R_{k}^{\rm{B}}+{R}_{k}^{\rm{S}}-\mathop{\max}\limits_{\forall\ell}R_{k,\ell}^{\rm{E}}}\right]^{+}},$
(23)

where ${\left[x\right]^{+}}=\max(0,x)$, and the SSR can be written as

$\displaystyle C=\sum\limits_{k=1}^{K}{R_{k}^{\sec}}.$ (24)

###  II-E Problem Formulation

In this work, our objective is to jointly optimize the transmit beamforming
matrix $\bf{W}$ and phase shifts matrix $\bf{\Theta}$ to maximize the SSR
under the transmit power constraint of the BS and the unit modulus constraint
of the phase shifters. The optimization problem is formulated as

$\displaystyle\mathop{\max}\limits_{{\bf{W,\Theta}}}\ $ $\displaystyle\
C\left({{\bf{W,\Theta,\Phi}},{\bf{\mathcal{H}}}}\right)$ s.t. $\displaystyle\
{\rm{Tr}}\left({{\bf{W}}{{\bf{W}}^{\rm H}}}\right)\leq{P_{max}},$ (25)
$\displaystyle\left|{{\chi_{m}}{e^{j{\theta_{m}}}}}\right|=1,0\leq{\theta_{m}}\leq
2\pi,1\leq m\leq M.$

It can be seen that Problem (II-E) is a non-convex problem because of the non-
convexity of the objective function and the unit modulus constraint. In the
traditional mathematical optimization algorithm, the exhaustive algorithm can
be used to obtain the optimal solution, but it is difficult to be practically
implemented due to the large amount of optimization variables in our problem.
We may also explore an AO algorithm to develop the suboptimal solution of the
optimization problem, like BCD algorithm in [16], but this also requires
complex mathematical derivations, and the optimal solution is not guaranteed.
Moreover, the AO algorithm cannot be directly applied to solve the considered
problem due to its complicated SINR expression. Furthermore, the artificially
labeled training dataset is unable to be acquired for DL technology in dealing
with this problem. The aforementioned facts motivate us to adopt the DRL-based
method, which is different from other traditional mathematical methods as well
as DL-based algorithms. Therefore, we utilize the advanced DRL-based algorithm
to solve the joint optimization problem without complex mathematical
derivations to achieve a satisfactory transmit beamforming matrix $\bf{W}$ and
phase shifts matrix $\bf{\Theta}$.

##  III DRL-Based Joint Optimization of Transmit Beamforming and Phase Shifts

The optimization problem formulated in (II-E) is mathematically intractable as
it is a non-linear and non-convex problem. Furthermore, in a practical RIS-
assisted FD secure communication system with both T-HWI and RIS-HWI, the
hardware quality is unknown, the capabilities of the legitimate users and the
channel quality are time-varying. The traditional optimization algorithms
(such as AO and BCD) can address single time slot optimization problem, while
they ignore the historical information and long-term benefits of the system
and may achieve sub-optimal solutions or performances similar to the greedy-
search. Therefore, it is usually not feasible to apply the traditional
optimization techniques to obtain the satisfactory secure beamforming in an
uncertain dynamic environment [35].

Model-free RL is a dynamic programming technique which learns the knowledge of
the uncertain dynamic environment to deal with decision-making problems.
Therefore, the Problem (II-E) is modeled as a RL problem in this paper, and a
DRL-based secure beamforming algorithm is developed to jointly design the
transmit and reflecting beamforming to maximize the SSR.

###  III-A Optimization Problem Transformation Based on RL

In DRL, the RIS-aided multiuser FD secure communication system is regarded as
environment, the smart controller coordinating with the BS is assumed to be an
agent, which can collect CSI, such as ${\bf{H}}_{u}$ and all other channels.
The agent interacts with the environment in discrete time slots. In time slot
$t$, the agent gets the current state $s_{t}$ of the environment, and chooses
an action $a_{t}$ based on the policy $\pi(s_{t},a_{t})$. After receiving the
action $a_{t}$, the environment will update the state to $s_{t+1}$ and feed
back a reward $r_{t}$, which represents the performance of the action $a_{t}$
under current state $s_{t}$, and then the agent chooses action $a_{t+1}$ for
the new state $s_{t+1}$. The agent exploits feedback to learn the policy to
maximize cumulative rewards. The interaction process between the agent and the
environment can be modeled as an MDP, which consists of a tuple denoted by
$({\mathcal{S}},{\mathcal{A}},{\mathcal{P}},r,\gamma)$. The key elements of
the proposed DRL-based algorithm are defined in the following.

1) State space: ${s_{t}\in\mathcal{S}}$ represents the state observed from the
environment at time slot t. The state $s_{t}$ at time slot $t$ is composed of
the rate part at the $(t-1)$-th time slot, the cascaded channel part at the
$t$-th time slot, the phase noise part at the $t$-th time slot, the action at
the $(t-1)$-th time slot, the transmit power of the BS and the received power
of the legitimate users at the $(t-1)$-th time slot [33]. The rate part
consists of the sum rate at the legitimate users, the sum rate at the BS, the
sum rate at the eavesdroppers and the SSR. The cascaded channel part contains
the BS-RIS-legitimate users channel
${\bf{G}_{1d}}={\bf{h}}_{d}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{{\bf{W}}}}\in\mathbb{C}^{K\times
K}$, the BS-RIS-eavesdroppers channel
${\bf{G}_{2d}}={\bf{g}}_{d}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{H}}_{d}}{{{\bf{W}}}}\in\mathbb{C}^{L\times
K}$, the legitimate users-RIS-BS channel
${\bf{G}_{1u}}={{\bf{H}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u}}}\in\mathbb{C}^{N_{r}\times
K}$ and the legitimate users-RIS-eavesdroppers channel
${\bf{G}_{2u}}={{\bf{g}}_{u}^{\rm{H}}{\bf{\Theta\Phi}}{{\bf{h}}_{u}}}\in\mathbb{C}^{L\times
K}$. The phase noise part consists of the random phase noise
$\Delta{\theta_{m}}$. In order to reduce the dimension of the state space and
the computational complexity, we only take the cascaded channels as the inputs
instead of all the channels. Note that neural network is only capable of
taking real number as input instead of complex number, so if there is complex
number in state $s$, which needs to be divided into real part and imaginary
part as independent input items. In total, there are $2K^{2}+4L{K}+2{N_{r}}K$
entries of state $s$ made up of the cascaded channel part. The transmit power
for $B_{k}$ is given by
${\left\|{{{\bf{w}}_{k}}}\right\|^{2}}={\left|{{\mathop{\rm\Re}\nolimits}\left\\{{{\bf{w}}_{k}^{\rm{H}}{{\bf{w}}_{k}}}\right\\}}\right|^{2}}+{\left|{{\mathop{\rm\Im}\nolimits}\left\\{{{\bf{w}}_{k}^{\rm{H}}{{\bf{w}}_{k}}}\right\\}}\right|^{2}}$.
The received power for $B_{k}$ is given by
$\left|\bf{G}_{1d,k}\right|^{2}=\left|\Re\left\\{\bf{G}_{1d,k}\right\\}\right|^{2}+\left|\Im\left\\{\bf{G}_{1d,k}\right\\}\right|^{2}$.
The number of entries of the action at the $(t-1)$-th time slot is
$2M+2N_{t}K$. To sum up, the dimension of the state space is
$D_{s}=4+2K^{2}+4L{K}+2{N_{r}}K+4K+3M+2N_{t}K$.

2) Action space: Action space includes the transmit beamforming matrix
$\bf{W}$ and the phase shifts matrix $\bm{\Theta}$. The diagonal elements of
phase shifts matrix is regarded as entries of the action. Specifically, the
real part and imaginary part,
$\bm{\Theta}=\Re\left\\{\bm{\Theta}\right\\}+\Im\left\\{\bm{\Theta}\right\\}$
and $\bf{W}=\Re\left\\{\bf{W}\right\\}+\Im\left\\{\bf{W}\right\\}$ make up the
action. Then, the dimension of the action space is $D_{a}=2M+2N_{t}K$.
Therefore, the action vector $\bm{a}_{t}$ is defined as

$\displaystyle\bm{a}_{t}=\left[\\{{\bf{w}}^{(t)}_{k}\\}_{k\in
K},\\{{\bm{\Theta}}^{(t)}_{m,m}\\}_{m\in M}\right].$ (26)

The action $\bm{a}_{t}$ will be reformed into a transmit beamforming matrix
${\bf{W}}^{(t)}$ and an RIS phase shifts matrix ${\bm{\Theta}}^{(t)}$.

3) State transition function: ${\mathcal{P}}$ is the transition probability
matrix, where ${\mathcal{P}}(s_{t},a_{t},s_{t+1})\in[0,1]$ represents the
probability that state $s_{t}$ changes to the new state $s_{t+1}$ when the
agent chooses an action $a_{t}$.

4) Reward function: The objective of the optimization problem is to maximize
the SSR. Thus, the reward function is defined as

$\displaystyle r_{t}=C^{(t)}.$ (27)

5) Discount factor: $\gamma\in[0,1]$ is applied to discount future rewards and
represents the uncertainty of future rewards.

6) Policy: the policy $\pi(s_{t},a_{t})$ denotes probability distribution of
choosing an action $a_{t}$ over the state $s_{t}$, which satisfies
$\sum\nolimits_{{a_{t}}\in\mathcal{A}}{\pi\left({{s_{t}},{a_{t}}}\right)}=1$.

###  III-B DDPG-Based Joint Transmit and Reflecting Beamforming Optimization

In DRL, whether transition probability is known or not is a big difference
between model-based and model-free algorithms. The proposed multiuser FD PLS
problem is built as an MDP model, once the transition probability set is
given, the problem can be effectively solved by dynamic programming techniques
[49]. However, this is not the case, as the transition probability set is
difficult to obtain in most cases. Therefore, some standard RL algorithms,
such as Q-learning, are considered to be instrumental in solving the model-
free problems, in which the transition probability is not required. However,
Q-learning is unable to deal with the high-dimensional input problem and the
continuous state-action space. To put it in practical terms, the channel
matrix and beamforming matrix in the considered RIS-aided FD secure
communication system, which are difficult to enumerate or even to discretize.

In order to overcome the shortcomings of tabular RL algorithms as well as to
tackle the continuous action and state space, we proposed DDPG-based
algorithm, which is based on actor-critic framework and shown in Fig. 2, to
jointly design the BS transmit beamforming and the phase shifts for the RIS-
aided FD secure communication system. There are four neural networks in the
proposed DDPG-based secure beamforming algorithm, named actor network, target
actor network, critic network and target critic network, the function of each
network is listed as below [50]:

Figure 2: The framework of the proposed DDPG algorithm for RIS-aided multiuser
FD two-way communication system with multi-eavesdropper.

1) Actor network: It is also known as policy network, takes state $s$ as input
and outputs an continuous action $a$, update network parameter $\theta_{\mu}$.
The actor network is trained by the equation (37), which aims to maximize the
state-value function. The update on the actor network is expressed as

$\displaystyle{\theta_{\mu}}={\theta_{\mu}}-{\alpha_{\mu}}\nabla
Q\left({{\theta_{ct}}|s,a}\right)\nabla\pi\left({{\theta_{\mu}}|s}\right),$
(28)

where $\alpha_{\mu}$ is the learning rate for the actor network,
$\nabla\left(\cdot\right)$ denotes the gradient and $\theta_{ct}$ is the
parameter of the target critic network.

2) Target actor network: It is parameterized by $\theta_{\mu_{t}}$, and
$\theta_{\mu_{t}}$ is periodically soft updated by

$\displaystyle{\theta_{\mu}}_{t}\leftarrow{\beta_{\mu}}{\theta_{\mu}}+\left({1-{\beta_{\mu}}}\right){\theta_{\mu}}_{t},$
(29)

where ${\beta_{\mu}}$ is the learning rate for the target actor network.

3) Critic network: It is also called Q-network, parameterized by $\theta_{c}$,
takes state $s$ and action $a$ as input and outputs the Q-function defined in
(35). The loss function $L(\theta)$ is a difference between the output Q-value
of the target neural network and the predict neural network, therefore, the
loss function is defined by

$\displaystyle
L\left({{\theta_{c}}}\right)={\left({{y}-Q\left({{\theta_{c}|s_{t},a_{t}}}\right)}\right)^{2}},$
(30)

where ${y}$ is the target value, which is estimated by

$\displaystyle{y}={r_{t+1}}+\gamma\mathop{\max}\limits_{{a_{t+1}}}Q\left({{\theta_{c}}|{s_{t+1}},{a_{t+1}}}\right).$
(31)

The gradient update of the loss function (30) with respect to $\theta_{c}$ is
updated by

$\displaystyle{\theta_{c}}={\theta_{c}}-{\alpha_{c}}\nabla
L\left({{\theta_{c}}}\right),$ (32)

where $\alpha_{c}$ is the learning rate for the critic network.

4) Target critic network: It is parameterized by $\theta_{ct}$, and outputs
${Q_{t}}\left({{\theta_{ct}}|{s_{t+1}},{a_{t+1}}}\right)$, $\theta_{ct}$ is
soft updated by

$\displaystyle{\theta_{c}}_{t}\leftarrow{\beta_{c}}{\theta_{c}}+\left({1-{\beta_{c}}}\right){\theta_{c}}_{t},$
(33)

where ${\beta_{c}}$ is the learning rate for the target critic network.

In the process of interaction between agent and environment, the goal of the
agent is to search the optimal policy $\pi^{*}$ to maximize the long-term
expected discounted reward, and the cumulative discounted reward function is
defined as

$\displaystyle{R_{t}}=\sum\limits_{\tau=0}^{\infty}{{\gamma_{\tau}}{r_{t+\tau+1}}},$
(34)

where $\tau$ is the number of time slots. When given a certain policy
$\pi_{t}$ and a state-action pair $(s_{t},a_{t})$, the cumulative discounted
reward can be approximately calculated by

$\displaystyle{Q_{\pi}}\left({{s_{t}},{a_{t}}}\right)=\mathbb{E}_{\pi}\left[{{R_{t}}|{s_{t}}=s,{a_{t}}=a}\right],$
(35)

which is called the action-value function and is also known as the Q-function.
It can be updated by Bellman expectation equation[35]. The optimal Q-function
in (35) can be solved by Bellman optimality equation as follows:

$\displaystyle{Q^{*}}\left({{s_{t}},{a_{t}}}\right)={r_{t}}+\gamma\mathop{\max}\limits_{{a_{t+1}}\in\mathcal{A}}{Q^{*}}\left({{s_{t+1}},{a_{t+1}}}\right),$
(36)

the corresponding optimal action $a^{*}$ can be obtained by

$\displaystyle{a^{*}}=\arg\mathop{\max}\limits_{a\in\mathcal{A}}Q^{*}\left({s,a}\right).$
(37)

The details of the proposed DRL-based method for joint optimization of the
transmit beamforming and the phase shifts are shown in Algorithm 1. The
algorithm runs over $N^{epi}$ episodes and each episode iterates $T$ steps. At
the beginning of each episode, the parameters of the four networks are
initialized, including $\theta_{\mu}$ in the actor network, $\theta_{\mu t}$
in the target actor network, $\theta_{c}$ in the critic network and
$\theta_{ct}$ in the target critic network, all of which are uniformly
distributed. In addition, the experience replay memory $\mathcal{M}$ with size
$\mathcal{D}$ will be emptied. For the sake of encouraging the agent to fully
explore the environment, exploration noise $\mathcal{N}$ is added to the
output of the actor network. It implies that the action to be chosen for state
$s$ is given by $\widetilde{a_{t}}=a_{t}+\mathcal{N}$, where $a_{t}$ is the
output of the actor network at time slot $t$, and the exploration noise
$\mathcal{N}$ can be chosen appropriately to suit the environment and will
decrease with the increase of the number of iterations. We adopt the identity
matrix to initialize the action $a_{0}$, including the transmit beamforming
matrix $\bf{W}_{0}$ and the phase shifts matrix $\bm{\Theta}_{0}$. In
addition, all the channels and the exploration noise $\mathcal{N}$ will be
initialized at the beginning of each episode.

Algorithm 1 DDPG-based joint optimization of transmit beamforming and phase
shifts

Input: All the channels, ${{\bf{H}}_{d}}$, ${{\bf{H}}_{u}}$,
${{\bf{h}}_{d,k}}$, ${{\bf{h}}_{u,k}}$, ${{\bf{g}}_{d,\ell}}$,
${{\bf{g}}_{u,\ell}}$; the coefficients of HWI, $\kappa_{u,k}^{\rm{B}}$,
$\kappa_{u}^{\rm{S}}$, $\kappa_{d,k}^{\rm{B}}$, $\kappa_{d}^{\rm{S}}$.

Output: The optimal phase shifts matrix $\bm{\Theta}^{*}$, the optimal
transmit beamforming matrix $\bf{W}^{*}$, the maximized SSR $C^{*}$ under
current channel state.

Initialize: actor network parameter $\theta_{\mu}$, target actor network
parameter $\theta_{\mu t}$, critic network parameter $\theta_{c}$, target
critic network parameter $\theta_{ct}$, learning rate $\alpha_{\mu}$,
$\alpha_{c}$, $\beta_{\mu}$ and $\beta_{c}$, mini-batch size $H$, experience
replay memory $\mathcal{M}$ with size $D$.

1:for episode = $1,2,\ldots,N^{epi}$ do

2: Initialize the exploration noise $\mathcal{N}$;

3: Initialize the phase shifts matrix $\bm{\Theta}_{0}$ and the beam-

4: forming matrix $\bf{W}_{0}$;

5: Obtain the initial state $s_{0}$;

6: for time slot $t=1,2,\ldots,T$ do

7: Obtain action $a_{t}$ from the actor network;

8: Add exploration noise to $a_{t}$ as $\widetilde{a_{t}}=a_{t}+\mathcal{N}$;

9: Calculate the instant reward $r_{t}$ using (27);

10: Observe the new state $s_{t+1}$;

11: Store transition $\\{s_{t},a_{t},r_{t},s_{t+1}\\}$ into $\mathcal{M}$;

12: Sample a $H$ mini-batch transitions from $\mathcal{M}$;

13: Set target function $y$ according to (31);

14: Minimize the loss function $L\left({{\theta_{c}}}\right)$ by equation
(30);

15: Update the actor network ${\theta_{\mu}}$ by (28) and the critic

16: network ${\theta_{c}}$ by (32);

17: Update the target actor network ${\theta_{\mu t}}$ by (29) and

18: the target critic network ${\theta_{ct}}$ by (33);

19: Update the sate $s_{t}=s_{t+1}$;

20: Reduce the exploration noise $\mathcal{N}$.

21: end for

22:end for

All the proposed neural network structures are based on fully connected layer.
The actor network structure is composed of five layers, including an input
layer, an output layer and three hidden layers. The critic network shares
almost the same network structure as the actor network. The input and output
dimensions of the actor network are equal to state and action dimensions,
respectively, and the input dimension of critic network is equal to the sum of
action and state dimensions. Due to the negative inputs, the activation
function $tanh$ is utilized for the network. The optimizer applied for the
network is Adam, which is with adaptive learning rate. More details are shown
in TABLE II. Furthermore, the correlation between input data will adversely
affect the learning of neural networks. In order to reduce the correlation
among the data in the state, the state needs to go through the whitening
process before being input into the actor network and critic network [33],
which is conducive to the learning of neural network.

Note that the transmit beamforming matrix $\bf{W}$ and the phase shifts matrix
$\bm{\Theta}$ need to meet the power constraint defined in (7) and the unit
modulus constraints, respectively. In order to satisfy these two constraints,
we add a batch normalization layer after the output layer of actor network,
where ${\rm{Tr}}\left({{\bf{W}}{{\bf{W}}^{\rm H}}}\right)\leq{P_{max}}$ and
the phase shifts satisfy $\left|{{\chi_{m}}{e^{j{\theta_{m}}}}}\right|=1$, the
later constraint means the transmission signal will change the direction
without power loss when going through RIS. Specifically, to satisfy the BS
maximum power constraint, the following projection operator is applied.

$\prod{\left\\{{\bf{W}}\right\\}}=\left\\{{\begin{array}[]{*{20}{c}}~{}{{\bf{W}},~{}~{}~{}~{}~{}~{}~{}{\rm{Tr}}\left({{\bf{W}}{{\bf{W}}^{\rm
H}}}\right)\leq{P_{\max}}},\\\
\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!{\frac{{\sqrt{{P_{\max}}}}}{{\left\|{\bf{W}}\right\|}_{F}}{\bf{W}},~{}~{}{\text{otherwise}}}.\end{array}}\right.$
(38)

After the normalization operation, we can obtain and reformulate the
normalized transmit beamforming
${{\bf{W}}_{{\rm{nor}}}}=\prod{\left\\{{\bf{W}}\right\\}}$, and then
${\rm{Tr}}({{\bf{W}}_{{\rm{nor}}}}{{\bf{W}}^{\rm
H}_{{\rm{nor}}}})\leq{P_{\max}}$, which always satisfies the BS maximum power
constraint in Problem (II-E).

Figure 3: Simulation setup for RIS-aided multiuser FD two-way communication system with multi-eavesdropper. TABLE I: Parameters for Simulation Parameters | Description | settings
---|---|---
$P_{max}$ | transmit power at BS |  $10$ dBm $\sim$ 40 dBm
$P_{k}$ | transmit power at users | 100 mW
$\alpha$ | path loss exponent | 2.0
$\varepsilon_{\bf{h}}$ | Rician factor | 10
$BW$ | channel bandwidth | 10 MHz
$\delta^{2}$ | noise power density |  $-174$ dBm/Hz
$\rho_{S}$ | SI coefficient | 1

Our objective is to take advantage of DRL to find satisfactory $\bf{W}$ and
$\bm{\Theta}$ by maximizing the SSR under given a particular CSI, according to
[33], rather than offline training and online deployment such as those in
[35]. In the presented DRL-based algorithm, each CSI is utilized to construct
the state, then the DDPG algorithm is run to find the optimal solutions. In
this paper, the optimal transmit beamforming matrix $\bf{W}^{*}$ and the phase
shifts matrix $\bm{\Theta}^{*}$ are obtained by the action with the largest
instant reward.

##  IV Simulation Results

In this section, extensive simulation results are presented to evaluate the
performance of the DRL-based algorithm in the RIS-aided multiuser FD secure
communication system with multiple eavesdroppers.

###  IV-A Simulation Setup

Figure 4: Average reward and instant reward performance versus time steps at $P_{max}=20$ dBm and $P_{max}=10$ dBm, respectively. TABLE II: Settings for DDPG Networks Network | Layer | Neurons | Activation
---|---|---|---
| Input | $D_{s}+D_{a}$ | /
| Hidden 1 | 128 | ReLU
critic network | Hidden 2 | 128 | ReLU
| Output | $1$ | ReLU
| Input | $D_{s}$ | /
| Hidden 1 | 128 | ReLU
actor network | Hidden 2 | 128 | ReLU
| Hidden 3 | 128 | ReLU
| Output | $D_{a}$ | tanh

Fig. 3 depicts a two-dimensional plane of the proposed RIS-assisted FD secure
communication system, the default unit in the figure is meter (m). As shown in
the figure, $K$ legitimate users and $L$ eavesdroppers are randomly
distributed in the light blue square in the right half of the figure, with the
size of 100 m × 100 m. The BS and RIS are located at (0,0) and (20,100),
respectively.

Unless stated otherwise, the following parameters are employed in the
simulations. The numbers of antennas at the BS are set to $N_{t}=N_{r}=4$, the
number of the RIS reflecting elements is set to $M=8$, the number of the
legitimate users is set to $K=2$ and the number of the eavesdroppers is set to
$L=2$. The large-scale path loss in equation (1) is defined by
$PL=PL_{0}-10\alpha\log_{10}(d/d_{0})$, where $PL_{0}=-30$ dB is the path loss
at $d_{0}=1~{}m$, $d$ is the distance between the transmitter and the
receiver. All channels follow the Rician distribution. Let
${\bf{\delta}}_{u,k}^{2}=1.1{\bf{\delta}}_{k}^{2}$,
$\sigma_{d,k}^{2}=1.1\sigma_{k}^{2}$, the coefficients of HWI
$\kappa_{u,k}^{\rm{B}}=\kappa_{u}^{\rm{S}}=\kappa_{d,k}^{\rm{B}}=\kappa_{d}^{\rm{S}}=\kappa=0.01$,
the random phase errors are uniformly distributed within
$\left[{{{-\pi}\mathord{\left/{\vphantom{{-\pi}{2,{\pi\mathord{\left/{\vphantom{\pi
2}}\right.\kern-1.2pt}2}}}}\right.\kern-1.2pt}{2,{\pi\mathord{\left/{\vphantom{\pi
2}}\right.\kern-1.2pt}2}}}}\right]$. The other parameters are given in TABLE
I.

In the proposed DRL algorithm, the discount factor is set to $\gamma=0.9$, the
learning rate is set to $\alpha_{\mu}=0.0005,\alpha_{c}=0.001$, the batch size
of the mini-batch is set to $H=128$, the experience replay capacity is set to
$D=100000$, the number of steps in each episode is set to $T=20000$. Other
neural network parameter settings, such as activation function, are given in
TABLE II.

###  IV-B Convergence

Figure 5: Average reward performance for 10 random initializations.

Before showing the simulation results, the convergence performance of the
presented DDPG algorithm is verified. Fig. 4. shows the convergence behavior
of the proposed method. In order to better demonstrate the convergence
performance of the proposed algorithm, instant reward and average reward are
considered, among which the average reward is defined as follows[51]:

$\displaystyle r^{{\text{Average}}}_{t+1}=\varpi
r^{{\text{Average}}}_{t}+\left({1-\varpi}\right)r^{{\text{Instant}}}_{t+1},$
(39)

where $\varpi$ is the smoothing factor, $t$ is the time step. In this paper,
the reward function represents the optimization objective, and our objective
is to maximize the SSR. Thus, the reward function is defined as the SSR. The
convergence behavior of the DDPG algorithm under different BS transmit power
is shown in Fig. 4. It is obvious from the figure that both the instant
rewards and the average rewards will converge with the increase of time step
$t$. This result indicates that the DRL-based algorithm explores the
environment through interactions between agent and environment, then learns
from the exploration experience to converge at about $t=8000$ steps and
obtains a satisfactory solution.

Figure 6: SSR versus $P_{max}$ with $K=2$, $L=2$, $\kappa=0.01$ under
different schemes.

Consider the non-convexity of Problem (II-E), different initial points may
lead to different local optimal solutions of the developed algorithm. Fig. 5
investigates the impact of different initializations on the performance of
proposed algorithm. The initialization strategy of Algorithm 1 is that the
identity matrix is adopted to initialize the action $a_{0}$, including the
transmit beamforming matrix $\bf{W}_{0}$ and the phase shifts matrix
$\bm{\Theta}_{0}$. “Random initialization” denotes that the initial point of
Algorithm 1 is randomly selected under the constraints in Problem (II-E). As
can be seen from the figure that although the initial points of each curve are
different, the convergence points are almost the same. Firstly, it shows that
the initialization strategy adopted by the proposed algorithm is a good
choice. In addition, it is also proved that the proposed algorithm is robust
to the initial points. In other words, the proposed DRL-based algorithm can
converge to a satisfactory solution that is almost the same even though the
initial points are different.

Figure 7: SSR versus $\kappa$ with $K=2$, $L=2$, $P_{max}=30$ dBm under
different schemes.

###  IV-C Comparisons with Benchmarks

Fig. 6 illustrates the SSR versus $P_{max}$ under several different schemes.
We consider six schemes, “Proposed DRL method, FD” and “Proposed DRL method,
HD” denote Algorithm 1 with FD and HD, i.e., $P_{k}=0$ mW. For the sake of
comparison, classical TD3 [52] is applied to the joint design of transmit
beamforming and RIS phase shifts as the performance benchmark, which are
denoted by “Classical TD3, FD” and “Classical TD3, HD”. Moreover, “Fixed phase
RIS, FD” and “Fixed phase RIS, HD” denote that the phase shift for each
reflecting unit is fixed while the transmit beamforming is optimized by
Algorithm 1, which serves as another performance benchmark. From the figure,
firstly, the proposed DRL scheme achieves much higher SSR than both classical
TD3 and fixed phase RIS scheme. Furthermore, with the increase of BS maximum
transmit power $P_{max}$, the SSRs of all schemes increase as expected, either
in FD mode or HD mode.

Fig. 7 shows the SSR performance versus $\kappa$ under different schemes. It
is obvious from the figure that the performance of various schemes
deteriorates with the increase of $\kappa$, owing to the fact that adverse
effect of hardware impairments is more prominent. In addition, it can be found
that the SSR in FD mode decreases faster than the SSR in HD mode with the
increase of $\kappa$. This is because the distortion noise power in FD mode
increases much more than the distortion noise power in HD mode with the
increase of $\kappa$. In other words, for a relatively small $\kappa$, a
greater performance gain will be achieved by FD mode in contract with HD mode.

###  IV-D Impact of the System Settings

In order to further verify the effectiveness of our proposed DRL-based method,
we evaluate the performance under various system settings. We consider three
cases, i.e., $\\{N_{t}=2,N_{r}=2,M=8\\}$, $\\{N_{t}=4,N_{r}=4,M=8\\}$ and
$\\{N_{t}=4,N_{r}=4,M=16\\}$, which are shown in Fig. 8. From this figure, we
see that, the SSR increases with $P_{max}$. As more transmit power is
allocated to BS, higher SSR can be obtained. This observation is consistent
with that of conventional secure systems. In addition, larger size of RIS and
more antennas at the BS lead to higher SSR.

Figure 8: SSR as a function of $P_{max}$ under different system settings.
Figure 9: CDF of SSR for various system settings. Figure 10: SSR versus $M$
with $P_{max}=10$ dBm, $K=2$, $L=2$ and different values of $\kappa$.

Furthermore, we also compare the cumulative distribution function (CDF) of
SSRs under different system settings, which are shown in Fig. 9. It is obvious
from the figure that the CDF curves demonstrate the observations from Fig. 8,
where the SSR improves with the increase of the BS transmit power $P_{max}$,
the number of antennas at the BS and the number of RIS reflecting elements
$M$. The performance gain of the proposed DRL-based algorithms is stable in
the light of the CDF curves, which means that the presented algorithm is
robust to the system settings and can perform well with high probability. On
the other hand, the CDF curves demonstrate the convergence of proposed
algorithm shown in Fig. 4. For example, focus on the curve with
$\\{N_{t}=2,N_{r}=2,P_{\max}=10~{}{\rm{dBm}},M=8\\}$, there is a $40\%$
probability that the abscissa of this curve is less than 4.5, and $60\%$ is
stable at about 4.5. This proves that the agent in proposed DRL-based method
explores in the environment in the first 8000 steps, then learns from the
exploration experience to converge and obtains a satisfactory solution in
8000-20000 steps.

###  IV-E Impact of Number of Reflecting Elements at RIS

Fig. 10 depicts the impact of the number of the reflecting elements $M$ on the
SSR performance with $K=2,L=2$,
$\kappa_{r}=\kappa^{\rm{B}}_{d,k}=\kappa^{\rm{S}}_{u}=\\{0.01,0.05\\}$ and
$\kappa_{t}=\kappa^{\rm{B}}_{u,k}=\kappa^{\rm{S}}_{d}=\\{0.01,0.05\\}$ when
${P_{max}=10}$ dBm. On the one hand, it can be seen that increasing the number
of RIS’s reflecting elements will increase the SSR of the system. However, the
performance gain decreases while the level of the HWI increases. On the other
hand, the SSR in the system with $\\{\kappa_{r}=0.01,\kappa_{t}=0.05\\}$ is
much higher than the SSR in the system with
$\\{\kappa_{r}=0.05,\kappa_{t}=0.01\\}$. This observation means that the HWI
on receivers has a greater negative impact on the SSR than that of the HWI on
the transmitters. This is because the proportion of the receive distortion
noise power in the total received noise is much larger than that in the
transmit distortion noise power.

###  IV-F Impact of the Path Loss Exponent

In the aforementioned simulations, the path loss exponent is set as 2.0 due to
the assumption that the RIS is deployed in an unobstructed location so that
the link of BS-RIS-Users is established. However, in some practical
communication scenarios, this ideal place may not exist. Therefore, the impact
of large-scale fading channel on the performance of secure system is
investigated. Fig. 11 shows the SSR versus the path loss exponent. The SSR
obtained by the proposed DRL algorithm decreases with the increase of $\alpha$
as expected. This is because larger-scale fading will lead to a weaker signal
reflected from the RIS, which weakens the benefits of RIS. Fortunately, this
provides practical engineering guidance that RIS should be carefully deployed
in a place with less obstacles in the legitimate link and more obstacles in
the eavesdropping link to improve the system’s secrecy performance.

Furthermore, we also compare the CDF of SSR for various $\alpha$, $P_{max}$ as
well as $M$, i.e., $\alpha=\\{2.0,2.8\\}$, $P_{max}=\\{10,20\\}$ dBm and
$M=\\{8,16\\}$, which are shown in Fig. 12. It is obvious from the figure that
the CDF curves are consistent with the results in Fig. 11. In addition, the
performance gain obtained by the proposed DRL method is stable according to
the CDF curves.

Figure 11: SSR versus pass loss exponent $\alpha$ with $P_{max}=\\{10,20\\}$
dBm and $M=\\{8,16\\}$. Figure 12: CDF of SSR for various $\alpha$, $P_{max}$
as well as $M$.

###  IV-G Impact of Hyper-Parameters of DRL

Network parameters play an important role in DRL, which determine the
convergence speed and efficiency of learning. By choosing appropriate network
parameters, the DRL can achieve the desired results. Here, we take the
learning rate and the discount factor as examples to illustrate that selecting
network parameter appropriately is crucial to DRL. Fig. 13 exhibits the
average rewards versus time steps under different learning rates, i.e.,
$\\{0.1,0.01,0.001,0.0001,0.00001\\}$. It is seen from the figure that
learning rates with different levels have a great impact on the performance of
DRL. Specifically, compared with other learning rates in the figure, the DRL
with learning rate $\alpha_{c}=0.001$ achieves the best performance. If the
learning rate is too large, it will increase the oscillation, which will lead
to poor performance or even decline in performance. If the learning rate is
too small, it will take a long time to achieve convergence or it is difficult
to converge to a good result. Therefore, the learning rate should be set
appropriately, neither too large nor too small. Hence, in the proposed DRL
algorithm, the learning rate is set to 0.001, which can achieve better
performance. Fig. 14 compares the average rewards performance versus time
steps under different discount factors, i.e.,{0.3, 0.5, 0.7, 0.9}. It shares
similar conclusions with the learning rate, but it has less effect on the
performance and convergence speed of DRL. It can be seen from the figure that
the discount factor with 0.9 achieves the best performance.

Figure 13: Average rewards performance versus time steps under different
learning rates, i.e.,{0.1, 0.01, 0.001, 0.0001, 0.00001}. Figure 14: Average
rewards performance versus time steps under different discount factors,
i.e.,{0.3, 0.5, 0.7, 0.9}.

Finally, we can conclude that DRL is a complex dynamic learning process, and
the selection of the network parameters determines its performance,
convergence speed and efficiency of learning. These parameters include not
only learning rate, discount factor, delaying rate but also activation
function, etc. Choosing appropriate parameters can improve the performance of
DRL-based algorithm and accelerate the convergence speed of neural network.

##  V Conclusion

In this paper, we studied an RIS-assisted FD secure communication system in
the presence of HWI. A SSR maximization problem was formulated by jointly
optimizing both the transmit beamforming at the BS and the phase shifts at the
RIS. To tackle this mathematically intractable NP-hard problem, a DRL-based
algorithm was proposed to obtain the satisfactory solution. Moreover, the
proposed DRL method had a standard framework and low complexity in
implementation, without explicit mathematical calculation. Therefore, it was
conveniently deployed to communication systems with different settings.
Extensive simulation results confirmed the effectiveness of the developed DRL-
based algorithm in improving the SSR. It was also found that appropriate
neural network parameters can improve the performance of DRL-based algorithm
and accelerate the convergence speed of the neural network.

## References

  * [1] E. Basar, M. Di Renzo, J. De Rosny, M. Debbah, M.-S. Alouini, and R. Zhang, “Wireless communications through reconfigurable intelligent surfaces,” _IEEE Access_ , vol. 7, pp. 116 753–116 773, 2019.
  * [2] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network,” _IEEE Commun. Mag._ , vol. 58, no. 1, pp. 106–112, Jan. 2020.
  * [3] C. Pan _et al._ , “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” _IEEE Commun. Mag._ , vol. 59, no. 6, pp. 14–20, Jun. 2021.
  * [4] Y. Zhou _et al._ , “Service-aware 6G: An intelligent and open network based on the convergence of communication, computing and caching,” _Digit. Commun. Netw._ , vol. 6, no. 3, pp. 253–260, 2020.
  * [5] Z. Yang, W. Xu, C. Huang, J. Shi, and M. Shikh-Bahaei, “Beamforming design for multiuser transmission through reconfigurable intelligent surface,” _IEEE Trans. Commun._ , vol. 69, no. 1, pp. 589–601, Jan. 2021.
  * [6] C. Pan _et al._ , “Multicell MIMO communications relying on intelligent reflecting surfaces,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp. 5218–5233, May 2020.
  * [7] M. Cui, G. Zhang, and R. Zhang, “Secure wireless communication via intelligent reflecting surface,” _IEEE Wireless Commun. Lett._ , vol. 8, no. 5, pp. 1410–1414, Oct. 2019.
  * [8] H. Shen, W. Xu, S. Gong, Z. He, and C. Zhao, “Secrecy rate maximization for intelligent reflecting surface assisted multi-antenna communications,” _IEEE Commun. Lett._ , vol. 23, no. 9, pp. 1488–1492, Sep. 2019.
  * [9] Z. Chu, W. Hao, P. Xiao, and J. Shi, “Intelligent reflecting surface aided multi-antenna secure transmission,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 1, pp. 108–112, Jan. 2020.
  * [10] S. Hong, C. Pan, H. Ren, K. Wang, K. K. Chai, and A. Nallanathan, “Robust transmission design for intelligent reflecting surface-aided secure communication systems with imperfect cascaded CSI,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 4, pp. 2487–2501, Apr. 2021.
  * [11] P. Xu, G. Chen, G. Pan, and M. D. Renzo, “Ergodic secrecy rate of RIS-assisted communication systems in the presence of discrete phase shifts and multiple eavesdroppers,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 3, pp. 629–633, Mar. 2021.
  * [12] X. Yu, D. Xu, Y. Sun, D. W. K. Ng, and R. Schober, “Robust and secure wireless communications via intelligent reflecting surfaces,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2637–2652, Nov. 2020\.
  * [13] J. Chen, Y.-C. Liang, Y. Pei, and H. Guo, “Intelligent reflecting surface: A programmable wireless environment for physical layer security,” _IEEE Access_ , vol. 7, pp. 82 599–82 612, 2019.
  * [14] F. Liu, J. Li, S. Li, and Y. Liu, “Physical layer security of full-duplex two-way AF relaying networks with optimal relay selection,” in _2018 IEEE Globecom Workshops (GC Wkshps)_ , 2018, pp. 1–6.
  * [15] H. Shen, T. Ding, W. Xu, and C. Zhao, “Beamformig design with fast convergence for IRS-aided full-duplex communication,” _IEEE Commun. Lett._ , vol. 24, no. 12, pp. 2849–2853, Dec. 2020.
  * [16] Z. Peng, Z. Zhang, C. Pan, L. Li, and A. L. Swindlehurst, “Multiuser full-duplex two-way communications via intelligent reflecting surface,” _IEEE Trans. Signal Process._ , vol. 69, pp. 837–851, 2021.
  * [17] L. Lv, Q. Wu, Z. Li, N. Al-Dhahir, and J. Chen, “Secure two-way communications via intelligent reflecting surfaces,” _IEEE Commun. Lett._ , vol. 25, no. 3, pp. 744–748, Mar. 2021.
  * [18] M. Wijewardena, T. Samarasinghe, K. T. Hemachandra, S. Atapattu, and J. S. Evans, “Physical layer security for intelligent reflecting surface assisted two–way communications,” _IEEE Commun. Lett._ , vol. 25, no. 7, pp. 2156–2160, Jul. 2021.
  * [19] E. Björnson, J. Hoydis, M. Kountouris, and M. Debbah, “Massive MIMO systems with non-ideal hardware: Energy efficiency, estimation, and capacity limits,” _IEEE Trans. Inf. Theory_ , vol. 60, no. 11, pp. 7112–7139, Nov. 2014.
  * [20] B. Sun, Y. Zhou, J. Yuan, Y.-F. Liu, L. Wang, and L. Liu, “High order psk modulation in massive MIMO systems with 1-bit adcs,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 4, pp. 2652–2669, Apr. 2021.
  * [21] H. Shen, W. Xu, S. Gong, C. Zhao, and D. W. K. Ng, “Beamforming optimization for IRS-aided communications with transceiver hardware impairments,” _IEEE Trans. Commun._ , vol. 69, no. 2, pp. 1214–1227, Feb. 2021.
  * [22] G. Zhou, C. Pan, H. Ren, K. Wang, and Z. Peng, “Secure wireless communication in RIS-aided MISO system with hardware impairments,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 6, pp. 1309–1313, Jun. 2021.
  * [23] M. Badiu and J. P. Coon, “Communication through a large reflecting surface with phase errors,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 2, pp. 184–188, Feb. 2020.
  * [24] K. Zhi, C. Pan, H. Ren, and K. Wang, “Uplink achievable rate of intelligent reflecting surface-aided millimeter-wave communications with low-resolution ADC and phase noise,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 3, pp. 654–658, Mar. 2021.
  * [25] J. D. Vega Sánchez, P. Ramírez-Espinosa, and F. J. López-Martínez, “Physical layer security of large reflecting surface aided communications with phase errors,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 2, pp. 325–329, Feb. 2021.
  * [26] Z. Xing, R. Wang, J. Wu, and E. Liu, “Achievable rate analysis and phase shift optimization on intelligent reflecting surface with hardware impairments,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 9, pp. 5514–5530, Sep. 2021\.
  * [27] S. Zhou, W. Xu, K. Wang, M. Di Renzo, and M.-S. Alouini, “Spectral and energy efficiency of IRS-assisted MISO communication with hardware impairments,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1366–1369, Sep. 2020.
  * [28] L. Dai, R. Jiao, F. Adachi, H. V. Poor, and L. Hanzo, “Deep learning for wireless communications: An emerging interdisciplinary paradigm,” _IEEE Wireless Commun._ , vol. 27, no. 4, pp. 133–139, Aug. 2020.
  * [29] H. Yang, A. Alphones, Z. Xiong, D. Niyato, J. Zhao, and K. Wu, “Artificial-intelligence-enabled intelligent 6G networks,” _IEEE Netw._ , vol. 34, no. 6, pp. 272–280, Oct. 2020.
  * [30] C. Huang, G. C. Alexandropoulos, C. Yuen, and M. Debbah, “Indoor signal focusing with deep learning designed reconfigurable intelligent surfaces,” in _Proc. IEEE 20th Int. Workshop on Signal Process. Adv. Wireless Commun. (SPAWC)_ , Cannes, France, Jul. 2019, pp. 1–5.
  * [31] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” _IEEE Access_ , vol. 9, pp. 44 304–44 321, 2021.
  * [32] K. Feng, Q. Wang, X. Li, and C.-K. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745–749, May 2020.
  * [33] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, Jun. 2020.
  * [34] C. Huang _et al._ , “Multi-hop RIS-empowered terahertz communications: A DRL-based hybrid beamforming design,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 6, pp. 1663–1677, Jun. 2021.
  * [35] H. Yang, Z. Xiong, J. Zhao, D. Niyato, L. Xiao, and Q. Wu, “Deep reinforcement learning-based intelligent reflecting surface for secure wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 375–388, Jan. 2021.
  * [36] H. Yang _et al._ , “Intelligent reflecting surface assisted anti-jamming communications: A fast reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 1963–1974, Mar. 2021.
  * [37] Q. Li, Y. Zhang, J. Lin, and S. X. Wu, “Full-duplex bidirectional secure communications under perfect and distributionally ambiguous eavesdropper’s CSI,” _IEEE Trans. Signal Process._ , vol. 65, no. 17, pp. 4684–4697, Sep. 2017.
  * [38] L. Wei, C. Huang, G. C. Alexandropoulos, C. Yuen, Z. Zhang, and M. Debbah, “Channel estimation for RIS-empowered multi-user MISO wireless communications,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 4144–4157, Jun. 2021.
  * [39] B. Zheng and R. Zhang, “Intelligent reflecting surface-enhanced OFDM: Channel estimation and reflection optimization,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 4, pp. 518–522, Apr. 2020.
  * [40] C. Hu, L. Dai, S. Han, and X. Wang, “Two-timescale channel estimation for reconfigurable intelligent surface aided wireless communications,” _IEEE Trans. Commun._ , vol. 69, no. 11, pp. 7736–7747, Nov. 2021.
  * [41] A. Mukherjee and A. L. Swindlehurst, “Detecting passive eavesdroppers in the MIMO wiretap channel,” in _Proc. IEEE Int. Conf. Acoust., Speech Signal Processing (ICASSP)_ , Kyoto, Japan, Mar. 2012, pp. 2809–2812.
  * [42] Z. Chu _et al._ , “Secrecy rate optimization for intelligent reflecting surface assisted MIMO system,” _IEEE Trans. Inf. Forensics Sec._ , vol. 16, pp. 1655–1669, Nov. 2021.
  * [43] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 11, pp. 5394–5409, Nov. 2019.
  * [44] M. A. Saeidi, M. J. Emadi, H. Masoumi, M. R. Mili, D. W. K. Ng, and I. Krikidis, “Weighted sum-rate maximization for multi-IRS-assisted full-duplex systems with hardware impairments,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 7, no. 2, pp. 466–481, Jun. 2021.
  * [45] T. Riihonen, S. Werner, and R. Wichman, “Mitigation of loopback self-interference in full-duplex MIMO relays,” _IEEE Trans. Signal Process._ , vol. 59, no. 12, pp. 5983–5993, Dec. 2011.
  * [46] H. Iimori, G. T. F. de Abreu, and G. C. Alexandropoulos, “MIMO beamforming schemes for hybrid SIC FD radios with imperfect hardware and CSI,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 10, pp. 4816–4830, Oct. 2019\.
  * [47] J. Zhu, D. W. K. Ng, N. Wang, R. Schober, and V. K. Bhargava, “Analysis and design of secure massive MIMO systems in the presence of hardware impairments,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 3, pp. 2001–2016, Mar. 2017.
  * [48] V.-D. Nguyen, H. V. Nguyen, O. A. Dobre, and O.-S. Shin, “A new design paradigm for secure full-duplex multiuser systems,” _IEEE J. Sel. Areas Commun._ , vol. 36, no. 7, pp. 1480–1498, Jul. 2018.
  * [49] T. Chen and W. Su, “Local energy trading behavior modeling with deep reinforcement learning,” _IEEE Access_ , vol. 6, pp. 62 806–62 814, 2018\.
  * [50] Z. Yang, Y. Liu, Y. Chen, and J. T. Zhou, “Deep reinforcement learning for RIS-aided non-orthogonal multiple access downlink networks,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM)_ , Taipei, Taiwan, Dec. 2020, pp. 1–6.
  * [51] H. Ren, C. Pan, L. Wang, W. Liu, Z. Kou, and K. Wang, “Long-term CSI-based design for RIS-aided multiuser MISO systems exploiting deep reinforcement learning,” _IEEE Commun. Lett._ , vol. 26, no. 3, pp. 567–571, Mar. 2022\.
  * [52] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” 2018. [Online]. Available: https://arxiv.org/abs/1802.09477

|  Zhangjie Peng received the B.S. degree from Southwest Jiaotong University,
Chengdu, China, in 2004, and the M.S. and Ph.D. degrees from Southeast
University, Southeast University, Nanjing, China, in 2007, and 2016,
respectively, all in Communication and Information Engineering. He is
currently an Associate Professor at the College of Information, Mechanical and
Electrical Engineering, Shanghai Normal University, Shanghai 200234, China.
His research interests include reconfigurable intelligent surface (RIS),
cooperative communications, information theory, physical layer security, and
machine learning for wireless communications.
---|---
|  Zhibo Zhang received the B.S. degree from the College of Internet of Things
Engineering, Hohai University, Changzhou, China, in 2020. He is currently
pursuing the M.S. degree at the College of Information, Mechanical and
Electrical Engineering, Shanghai Normal University, Shanghai, China. His major
research interests lie in the areas of communication and signal processing,
including reconfigurable intelligent surface (RIS), physical layer security
and machine learning for wireless communications.
---|---
|  Lei Kong received the Ph.D. degree in the School of Information Science and
Engineering, National Mobile Communications Research Laboratory, Southeast
University, Nanjing, China in 2016. From 2016 to 2021, he worked as System
Engineer at Nokia in Hangzhou, China. He is now a Post Doctor researcher in
New H3C Corporation and Southeast University. His research interests span on
the URLLC, Deterministic Networks, Full Duplex Communication, Network Energy
Saving and RIS.
---|---
|  Cunhua Pan (Member, IEEE) received the B.S. and Ph.D. degrees from the
School of Information Science and Engineering, Southeast University, Nanjing,
China, in 2010 and 2015, respectively. From 2015 to 2016, he was a Research
Associate at the University of Kent, U.K. He held a post-doctoral position at
Queen Mary University of London, U.K., from 2016 and 2019.From 2019 to 2021,
he was a Lecturer in the same university. From 2021, he is a full professor in
Southeast University. His research interests mainly include reconfigurable
intelligent surfaces (RIS), intelligent reflection surface (IRS), ultra-
reliable low latency communication (URLLC) , machine learning, UAV, Internet
of Things, and mobile edge computing. He has published over 120 IEEE journal
papers. He is currently an Editor of IEEE Wireless Communication Letters, IEEE
Communications Letters and IEEE ACCESS. He serves as the guest editor for IEEE
Journal on Selected Areas in Communications on the special issue on xURLLC in
6G: Next Generation Ultra-Reliable and Low-Latency Communications. He also
serves as a leading guest editor of IEEE Journal of Selected Topics in Signal
Processing (JSTSP) Special Issue on Advanced Signal Processing for
Reconfigurable Intelligent Surface-aided 6G Networks, leading guest editor of
IEEE Vehicular Technology Magazine on the special issue on Backscatter and
Reconfigurable Intelligent Surface Empowered Wireless Communications in 6G,
leading guest editor of IEEE Open Journal of Vehicular Technology on the
special issue of Reconfigurable Intelligent Surface Empowered Wireless
Communications in 6G and Beyond, and leading guest editor of IEEE ACCESS
Special Issue on Reconfigurable Intelligent Surface Aided Communications for
6G and Beyond. He is Workshop organizer in IEEE ICCC 2021 on the topic of
Reconfigurable Intelligent Surfaces for Next Generation Wireless
Communications (RIS for 6G Networks), and workshop organizer in IEEE Globecom
2021 on the topic of Reconfigurable Intelligent Surfaces for future wireless
communications. He is currently the Workshops and Symposia officer for
Reconfigurable Intelligent Surfaces Emerging Technology Initiative. He is
workshop chair for IEEE WCNC 2024, and TPC co-chair for IEEE ICCT 2022. He
serves as a TPC member for numerous conferences, such as ICC and GLOBECOM, and
the Student Travel Grant Chair for ICC 2019. He received the IEEE ComSoc
Leonard G. Abraham Prize in 2022.
---|---
|  Li Li received the B.S. EE degree from Tsinghua University in 1985, the
M.S. EE degree from China Research Institute of Radiowave Propagation in 1988,
and Ph.D degree in science from Peking University in 1997. Her research
interest is currently at the adaptive digital signal processing, spectrum
allocation and interference alignment algorithms in heterogeneous cognitive
radio network; network modeling of ultra-dense wireless mobile communication.
---|---
|  Jiangzhou Wang (Fellow, IEEE) is a Professor at the University of Kent,
U.K. His research interest is in the area of mobile communications. He has
published over 400 papers and 4 books. He was a recipient of the 2022 IEEE
Communications Society Leonard G. Abraham Prize and the 2012 IEEE Globecom
Best Paper Award. Professor Wang is a Fellow of the Royal Academy of
Engineering, U.K., Fellow of the IEEE, and Fellow of the IET. He was the
Technical Program Chair of the 2019 IEEE International Conference on
Communications (ICC2019), Shanghai, the Executive Chair of the IEEE ICC2015,
London, and the Technical Program Chair of the IEEE WCNC2013.
---|---