# Proximal Policy Optimization-based Transmit Beamforming and Phase-shift
Design in an IRS-aided ISAC System for the THz Band

Xiangnan Liu, Haijun Zhang, , Keping Long, , Mingyu Zhou, Yonghui Li, , and H.
Vincent Poor This work is supported in part by the National Key Research and
Development Program of China (Grant No. 2020YFB1708800), in part by Beijing
Natural Science Foundation (L212004), in part by the National Natural Science
Foundation of China under Grants 61822104 and 61771044, in part by the State
Key Laboratory of Advanced Metallurgy under Grant KF20-04, in part by the
Fundamental Research Funds for the Central Universities under Grants FRFTP-
19-002C1 and RC1631, and in part by Beijing Top Discipline for Artificial
Intelligent Science and Engineering, University of Science and Technology
Beijing (_Corresponding author: Haijun Zhang_.) Xiangnan Liu, Haijun Zhang,
and Keping Long are with Institute of Artificial Intelligence, Beijing
Advanced Innovation Center for Materials Genome Engineering, Beijing
Engineering and Technology Research Center for Convergence Networks and
Ubiquitous Services, University of Science and Technology Beijing, Beijing
100083, China (email<EMAIL_ADDRESS><EMAIL_ADDRESS>longkeping@ustb.edu.cn). Mingyu Zhou is with BaiCells Technologies Company,
Beijing, China, 100094 (email: Zhoumingyu@baicells.com). Yonghui Li is with
the School of Electrical and Information Engineering, The University of
Sydney, Sydney, NSW 2006, Australia (email: yonghui.li@sydney.edu.au). H.
Vincent Poor is with the Department of Electrical and Computer, Princeton
University, NJ 08544 USA (e-mail: poor@princeton.edu).

###### Abstract

In this paper, an IRS-aided integrated sensing and communications (ISAC)
system operating in the terahertz (THz) band is proposed to maximize the
system capacity. Transmit beamforming and phase-shift design are transformed
into a universal optimization problem with ergodic constraints. Then the joint
optimization of transmit beamforming and phase-shift design is achieved by
gradient-based, primal-dual proximal policy optimization (PPO) in the multi-
user multiple-input single-output (MISO) scenario. Specifically, the actor
part generates continuous transmit beamforming and the critic part takes
charge of discrete phase shift design. Based on the MISO scenario, we
investigate a distributed PPO (DPPO) framework with the concept of multi-
threading learning in the multi-user multiple-input multiple-output (MIMO)
scenario. Simulation results demonstrate the effectiveness of the primal-dual
PPO algorithm and its multi-threading version in terms of transmit beamforming
and phase-shift design.

###### Index Terms:

Integrated sensing and communications, transmit beamforming, phase shift
design, intelligent reflecting surface, distributed reinforcement learning.

##  I Introduction

Rrecently, the integrated sensing and communications (ISAC) has been proposed
to enhance the sensing capability in the location/environment-aware scenarios.
Many scenarios require ISAC to obtain high-rate transmission and high-
resolution target detection, such as autonomous vehicles, indoor localization,
and extended reality (XR) [1]. ISAC is considered to be a promising technique
for the next generation wireless systems to support high-accuracy sensing
services [2]. Following the communication signal processing, the millimeter
wave (mmWave) band is considered in the ISAC system. Previous discussions of
ISAC systems have considered the use of the mmWave bands for use in automotive
radars and high-resolution imaging radars [3]. Therefore, higher bandwidth and
transmission rate is a technique development in the envisioned sixth
generation (6G) networks.

In this context, terahertz (THz) communication is expected to enable ultra-
high-speed communications in the era of 6G. The 6G networks will require
higher data rates and capacity to maintain quality of service, economical
operation, and flexible resource management [4]. The THz band can provide
wider communication bandwidth than the current wireless communication band.
THz communication can obtain Gbps wireless transmission rate, which will
enable new sensing applications such as miniaturized radars for gesture
detection and touchless smartphones, spectrometers for explosive detection and
gas sensing. THz communication can provide higher transmission rates and wider
bandwidths than communication in lower-frequency bands. However, the molecular
absorption in the THz band is severe. And the THz waves with strong
directivity and poor diffraction are blocked by many obstacles more easily.

To tackle these challenges, the intelligent reflecting surface (IRS) has
recently emerged as a promising technique to address the issue of excessive
pathloss by creating better propagation environments [5]. IRSs are composed of
passive antenna elements with adjustable phase shifts. Combined with a phase-
shift design in an IRS, transmit beamforming can realize better ISAC
performance. On the one hand, the transmit beamforming can be utilized to
synthesize multiple beams towards existing users and targets [6]. On the other
hand, the IRS can boost the ISAC signal by changing the phase optimization
dynamically [7]. Compared with a conventional relay, the IRS does not employ a
radio frequency unit and so can save energy. Thus an IRS-aided wireless
communication system can obtain higher spectrum and energy efficiencies.

By utilizing the IRS technique, the communication performance of the ISAC
system can be enhanced. Meanwhile, high-data and high-resolution sensing
performance can be achieved in the THz band. While IRS-aided systems open new
possibilities, they also bring new design challenges, such as passive
beamforming, channel realization, and deployment of the IRS. Moreover, there
have been few studies of transmit beamforming and phase shift design for the
IRS-aided ISAC system. Therefore. we focus on the joint optimization of
transmit beamforming and phase-shift design in this paper.

###  I-A Related Works

Transmit beamforming and phase-shift design in general IRS-aided system have
attracted extensive attention in recent years. They have been studied in
various systems, such as multi-user multiple-input single-output (MISO) [8],
multiple-input multiple-output (MIMO) communication [9], and simultaneous
wireless information and power transfer (SWIPT) systems [10]. To reduce the
power consumption, Abeywickrama et.al. proposed a practical method by jointly
designing the access point transmit beamforming and IRS passive phase
optimization, subject to constraints on users’ individual signal-to-
interference-plus-noise ratios (SINRs) [8]. Transmit beamforming was studied
in an IRS-assisted SWIPT system, with the constraints of quality-of-service
(QoS) constraints at all users, by applying a penalty-based optimization
method [10]. In [11], Shen et al. first derived a closed-form solution for the
base station (BS)’s transmit beamforming to maximize the signal-to-noise ratio
(SNR) of a radar signal. Furthermore, transmit beamforming has been designed
to optimize various objective functions, such as power consumption [12],
achievable rate [13], and energy efficiency [14]. On the one hand, THz will
enable new sensing applications such as miniaturized radars for gesture
detection and touchless smartphones, spectrometers for explosive detection and
gas sensing. On the other hand, IRS is a powerful relay with unique properties
to enhance the radar and communication signal strength in the THz band.
Therefore, the joint optimization of transmit beamforming and phase-shift
design is meaningful, in terms of the communication and radar performance.

Deep reinforcement learning (DRL) is a state-of-the-art method that can be
used to optimize radio resource allocation [15]. Despite the fact that
excellent performances of many optimization algorithms have been observed
through numerical simulations and theoretical analysis, implementing them in
real systems still faces many serious difficulties. In particular, the high
computational cost incurred by these algorithms has been one of the most
challenging issues. Deep learning can reduce the computational cost but it
acquires the training set, resulting in poor flexibility of deep learning.
Furthermore, DRL-based methods do not need training labels and possess the
property of online learning and sample generation, which is more storage-
efficient. Recently, it has been leveraged to optimize performance in wireless
systems extensively, for problems such as power control [16], bandwidth
scheduling [17], computation offloading [18], and caching deployment [19].
Mismar et al. proposed joint optimization of analog beamforming, power
allocation, and interference coordination in the sub-6 GHz and mmWave band. A
deep Q-learning network (DQN) was applied to tackle this joint optimization
[20]. Recently DRL has been applied to solve optimization problems in IRS
assisted communications [21, 22, 23]. In [21], a DRL framework is designed to
investigate the passive phase optimization for the IRS-aided MISO scenario to
maximize the SNR. The results demonstrated a considerable gain compared to
existing schemes. In [22], the transmit beamforming and passive phase were
jointly optimized by using a DRL framework. The developed framework can
support large-dimensional optimization problems in massive MIMO systems. The
transmit beamforming in the BS and passive beamforming in the IRS were jointly
designed to optimize the system transmission rate in [23] by using
prioritized. Towards the complicated practical scenario, the DRL with a single
thread is intractable because of its lower utilization of computing process.

Furthermore, distributed DRL is a promising technique to solve more
complicated wireless networks problems. Distributed DRL is composed of a
central controller and a group of learners. It can be classified as multi-
agent DRL or multi-threading DRL. The latter uses the multi-threading method
to train agents. Interactive learning takes place simultaneously in multiple
threads and each thread summarizes the learning results, sorting and saving
them in a public place. Compared with multi-agent DRL, multi-threading DRL can
save power consumption. Additionally, multi-threading DRL can save training
time and keep the training process stable [24]. Du et al. proposed an
asynchronous advantage actor-critic (A3C) algorithm to solve joint
optimization of viewport rendering offloading and power allocation [25]. Zhang
et al. solved the optimization problem of power control in the cognitive radio
network by utilizing distributed DRL methods, such as A3C and distributed
proximal policy optimization (DPPO) [26]. Dinh et al. designed a distributed
model-free algorithm DeepPool to optimize ride-sharing platforms [27].
Compared with multi-agent reinforcement learning, there are relatively few
studies of multi-threaded DRL, especially for transmit beamforming and passive
phase optimization.

On the other hand, ergodic stochastic optimization, which uses term averages
to reflect system performance, is suitable for wireless resource allocation.
This idea was originated by Ribeiro [28], who proposed the average variable
scheme for radio resource management. This ergodic system performance metric
is valid for both independent channels and correlated channels. A further
study [29] considered optimizing primal and dual variables by learning
resource allocation policy gradients. Lee et al. in [30] proposed a
distributed scheme involving average variables to capture system performance,
and two deep neural networks (DNNs) were introduced to approximate the
allocation policy. Combining this scheme of average variables with a DRL
framework is a promising approach, which also becomes one of motivations of
this paper.

###  I-B Contributions

In this context, we study an IRS-aided ISAC system in the THz band. In
particular, transmit beamforming and phase-shift design in an IRS-aided
device-based ISAC system with multiple users are studied. The capacity with
ergodic constraints is set as the utility function to evaluate the IRS-aided
ISAC system performance. This problem is challenging because of the joint
optimization of transmit beamforming and phase-shift design. To tackle this
problem, proximal policy optimization (PPO) and its distributed forms are
developed in different scenarios, including MISO and MIMO. The main
contributions of this paper are summarized below.

  * •

_The IRS-aided ISAC system in the THz band:_ The IRS is explored to compensate
for the high path loss caused by molecular absorption in the THz band. We
consider an IRS-aided ISAC system consisting one BS equipped with several
antennas, several users and an IRS. The capacity of the ISAC system is
optimized through transmit beamforming and passive phase-shift design of the
IRS.

  * •

_The primal-dual model-free PPO with ergodic constraints:_ The optimization
problem of capacity maximization is formulated as a universal optimization
problem with ergodic constraints and is transformed into the dual domain. A
gradient-based, primal-dual PPO algorithm is designed to solve the problem of
capacity maximization with ergodic constraints.

  * •

_The joint optimization via the actor-critic structure:_ PPO is leveraged to
achieve joint optimization on continuous transmit beamforming of the ISAC BS
and discrete phase design of the IRS. The actor part optimizes transmit
beamforming and the critic part designs the phase shifts, to maximize the
total capacity in the multi-user MISO scenario.

  * •

_The optimization of multi-user MIMO scenario through distributed DRL:_
Distributed PPO (DPPO) is introduced in the established model to solve this
joint optimization. The concept of multi-threading applied to the more
complicated matrix processing in the MIMO scenario. Each worker docks with one
user to collect observations and transmit them to the global PPO. The global
PPO then broadcasts the optimized gradients to each worker.

The DRL-based method to achieve the joint optimization of transmit beamforming
and phase-shift design is different from the current studies of transmit
beamforming in ISAC system. Utilizing the IRS to counteract the high pathloss
in the THz band, this emerging technique still lacks in-depth studies studies
for ISAC system. Towards the joint optimization in the multi-user MISO
scenario, the proposed primal-dual PPO algorithm is novel and effective.
Beyond the multi-user MISO scenario, the primal-dual PPO algorithm’s
distributed version can be applied to the multi-user MIMO scenario. Simulation
results will demonstrate the effectiveness of primal-dual PPO-based algorithm
in both the multi-user MISO scenario and multi-user MIMO scenario.

The rest of this paper is organized as follows. Section II introduces the
system model and formulates the optimization problem of transmit beamforming
and phase-shift design. Section III presents the primal-dual PPO algorithm to
realize joint optimization of transmit beamforming and passive phase
optimization in the multi-user MISO scenario. In Section IV, the primal-dual
DPPO is extended to the MIMO scenario. Section V shows the pseudo codes and
implementations of these two algorithms. The proposed algorithms are verified
by simulations results in VI, and Section VII concludes the paper.

##  II System Model and Problem Formulation

###  II-A System Model

Figure 1: Primal-dual PPO learning beamforming and phase-shift design in the
IRS-aided ISAC system.

As shown in Fig. 1, a multi-user MISO model is considered in the IRS-aided
ISAC system. It consists of one ISAC BS with $M$ antennas and $K$ users
equipped with single antenna. Let ${\cal M}=\left\\{{1,2,...,M}\right\\}$ and
${\cal K}=\left\\{{1,2,...,K}\right\\}$ denote the set of the BS’s antennas
and served users.
${\bf{s}}={\left[{{s_{1}},s_{2},...,s_{k}}\right]^{T}}\in{{\mathbb{C}}^{K\times
1}}$ and ${\bf{W}}\in{{\mathbb{C}}^{M\times K}}$ are the information-bearing
symbol and the transmit beamforming matrix, respectively. The transmit signal
${\bf{X}}\in{{\mathbb{C}}^{M\times 1}}$ after transmit beamforming is given by

${\bf{X}}={\bf{W}}\cdot{\bf{s}}.$ (1)

_1) Communication Signal:_

Let ${{\bf{H}}_{1}}\in{{\mathbb{C}}^{N\times M}}$,
${{\bf{H}}_{0}}\in{\mathbb{C}}^{{M\times K}}$, and
${{\bf{H}}_{2}}\in{\mathbb{C}}^{{N\times K}}$ represent the channel gain from
the ISAC BS to the IRS, and the channel gain from the ISAC BS to user $k$, the
channel gain from the IRS’s element $n$ to user $k$, respectively. The
communication signal ${{\bf{y}}_{c}}\in{{\mathbb{C}}^{K\times 1}}$ received by
users can be written as

${\bf{y}}_{c}=\left({{{\bf{H}}_{0}^{H}}+{{\bf{H}}_{2}^{H}}{\bf{V}}{{\bf{H}}_{1}}}\right){{\bf{X}}}+{\bf{n}}_{c},$
(2)

${\bf{n}}_{c}\in{{\mathbb{C}}^{K\times 1}}$ is the independent identically
distributed complex Gaussian stochastic process. The effective diagonal phase
matrix $\bf V$ is
$diag\left\\{{{\gamma_{1}}{e^{j{\beta_{1}}}},{\gamma_{2}}{e^{j{\beta_{2}}}},...,{\gamma_{N}}{e^{j{\beta_{N}}}}}\right\\}$,
where ${\gamma_{n}}\in\left[{0,1}\right]$ and
${\beta_{n}}\in\left[{0,2\pi}\right]$ are amplitude and phase, respectively.
Let ${\cal N}=\left\\{{1,2,...,N}\right\\}$ denote the set of the IRS’s
elements. Define $\gamma_{n}$ as the amplitude reflection coefficient of the
IRS and it can be calculated as [8]

${\gamma_{n}}\left({{\beta_{n}}}\right)=\left({1-{\gamma_{\min}}}\right){\left({\frac{{\sin\left({{\beta_{n}}-\varphi}\right)+1}}{2}}\right)^{\varepsilon}}+{\gamma_{\min}}.$
(3)

${\gamma_{\min}}$ is the minimum amplitude, $\varphi$ is the horizontal
distance between $-\pi/2$ and ${\gamma_{\min}}$. $\varphi\geq 0$ and
$\varepsilon\geq 0$ are the constants depending on the IRS’s circuit.

All of them are modeled as the Rician channel [31],

${{\bf{H}}_{i}}=\sqrt{\frac{\gamma_{AI}}{{1+\gamma_{AI}}}}{{\bf{H}}_{LOS}}+\sqrt{\frac{1}{{1+\gamma_{AI}}}}{{\bf{H}}_{NLOS}},i=0,1,2,$
(4)

where $\gamma_{AI}$ is the Rician factor, ${\bf{H}}_{LOS}$ is the line-of-
sight (LoS) component and it can be expressed as
${{\bf{H}}_{LOS}}={{\alpha}_{LOS}\left({f,l}\right){G_{t}}{G_{r}}{{\bf{b}}_{r}{\left({{\theta_{AoA}}}\right)}}{\bf{a}}_{t}^{H}\left({{\theta_{AoD}}}\right)}$,
where ${{\bf{b}}_{r}}\left(\theta_{AoA}\right)$ and
${{\bf{a}}_{t}}\left(\theta_{AoD}\right)$ is the receive response vector and
transmit response vector, respectively.
${{\mathbf{a}}_{t}^{H}}\left(\cdot\right)$ denotes the conjugate array
steering vector. Additionally, $\theta_{AoD}$ and $\theta_{AoA}$ are the angle
of departure (AoD) and the angle of arrival (AoA), respectively.

We consider a uniform linear antenna (ULA) to be the structure of the ISAC
BS’s antennas, so the steering vector
${{\bf{a}}}\left(\theta\right)\in{{\mathbb{C}}^{M\times 1}}$ is given by

${{{\bf{a}}}\left(\theta\right)=\sqrt{\frac{1}{M}}{\left[{1,{e^{-j{{2\pi
d}\mathord{\left/{\vphantom{{2\pi
d}\lambda}}\right.\kern-1.2pt}\lambda}\cos\theta}},...,{e^{-j{{2\pi
d}\mathord{\left/{\vphantom{{2\pi
d}\lambda}}\right.\kern-1.2pt}\lambda}\left({M-1}\right)\cos\theta}}}\right]^{T}}.}$
(5)

$d$ is the antenna element spacing and $\lambda$ is the wavelength.

For the IRS, an $N\times N$ element uniform planar antenna (UPA) is
considered. The array steering vector
${{\mathbf{v}}}\left({\phi,{\mathbf{\theta}}}\right)\in{{\mathbb{C}}^{N\times
1}}$ is given by

$\begin{array}[]{*{20}{l}}{{{\mathbf{v}}}\left({\phi,{\mathbf{\theta}}}\right)=\frac{1}{N}\left[{1,...,{e^{j\frac{{2\pi}}{\lambda}d\left({n\cos{\phi}\cos{\theta}+n\sin{\phi}\sin{\theta}}\right)}},...{\kern
1.0pt}{\kern 1.0pt},}\right.}\\\ {{{\left.{{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern
1.0pt}{e^{j\frac{{2\pi}}{\lambda}d\left({\left({\sqrt{N}-1}\right)\cos{\phi}\cos{\theta}+\left({\sqrt{N}-1}\right)\sin{\phi}\sin{\theta}}\right)}}}\right]}^{T}}}.\end{array}$
(6)

$n$ is the antenna element index with $0\leq n\leq N-1$.

In the THz band, there is not only free space pathloss $L_{spread}$, but also
the module pathloss $L_{medium}$. The pathloss of LoS
${\left|{{\alpha_{L}}\left({f,l}\right)}\right|^{2}}$ is expressed by

${\left|{{\alpha_{L}}\left({f,l}\right)}\right|^{2}}={L_{spread}}\left({f,l}\right){L_{medium}}\left({f,l}\right).$
(7)

$f$ is the carrier frequency and $l$ is the transmission distance. The free
space pathoss $L_{spread}$ can be calculated by

${L_{spread}}\left({f,l}\right){\text{ = }}{\left({\frac{c}{{4\pi
fl}}}\right)^{2}}.$ (8)

Similar to the channel state of the LoS path ${{\bf{H}}_{LOS}}$,
${{\bf{H}}_{NLOS}}={\sum\limits_{i=1}^{{n_{NL}}}{{\alpha_{i}}\left({f,l}\right){G_{t}}{G_{r}}{{\bf{b}}_{r}}\left({{\theta_{AoA,i}}}\right){\bf{a}}_{t}^{H}\left({{\theta_{AoD,i}}}\right)}}$
is set as the non-line-of-sight (NLoS) component. Let $G_{t}$ and $G_{r}$
represent channel gain from transmit antenna and received antenna,
respectively. ${n_{NL}}$ is the number of NLoS rays and
$\alpha\left({f,l}\right)$ is the complex gain of the path.

The large-scale channel gain $\alpha_{LOS}$ and $\alpha_{NLOS}$ can be given
by

$\displaystyle\begin{array}[]{*{20}{l}}{\alpha_{LOS}\left({f,l}\right)}={L_{spread}}\left({f,l}\right){L_{abs}}\left({f,l}\right)\\\
{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}={\left({\frac{c}{{4\pi
fl}}}\right)^{2}}{e^{-{k_{abs}}\left(f\right)l}},\end{array}$ (9c)
$\displaystyle\begin{array}[]{*{20}{l}}{\alpha_{NLOS}\left({f,l}\right)}={\Gamma^{2}\left(f\right)}{L_{spread}}\left({f,l}\right){L_{abs}}\left({f,l}\right)\\\
{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}={\left({\frac{c}{{4\pi
fl}}}\right)^{2}}{e^{-{k_{abs}}\left(f\right)l}},\end{array}$ (9f)

the speed of light is
$c=3\times{10^{8}}\;{m\mathord{\left/{\vphantom{ms}}\right.\kern-1.2pt}s}$.
Let $l$ denote the transmission distance in free space. The absorption
coefficient $k_{abs}\left(f\right)$ can be calculated by the medium available
at a molecular level [25]. The reflecting coefficient ${\Gamma}\left(f\right)$
is the product of the Fresnel reflection coefficient $\iota$ and the Rayleigh
roughness factor $\xi$. The high reflection loss (i.e. up to second order
reflections) is only considered in the THz band. The Fresnel reflection
coefficient $\iota$ can describe the behavior of light in medium with
different refractive indices. It can be calculated by

$\iota\left(f\right)=\frac{{Z\left(f\right)\cos{\varphi_{in}}-{Z_{0}}\cos{\varphi_{ref}}}}{{Z\left(f\right)\cos{\varphi_{in}}+{Z_{0}}\cos{\varphi_{ref}}}},$
(10)

where $Z\left(f\right)$ is the reflecting material ’s frequency-dependent wave
impedance and ${{\rm{Z}}_{0}}{\rm{=377}}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}\Omega{\rm{}}$ is the free space wave impedance. Let
$\varphi_{in}$ denote the angle of incidence and reflection. Additionally,
${\varphi_{ref}}=\arcsin\left({{\textstyle{Z\over{{Z_{0}}}}}\sin{\varphi_{in}}}\right)$
represents the angle of refraction.

The Rayleigh roughness factor $\xi$ is calculated by

$\zeta\left(f\right)={e^{-{\textstyle{1\over 2}}{{\left({{\textstyle{{4\pi
f\sigma\cos{\varphi_{in}}}\over c}}}\right)}^{2}}}}.$ (11)

They are composed of reflection and scattering, so each user $k$’s channel
state can be calculated by [32]

${\bf{y}}_{c,k}=\left({{{\bf{H}}_{0,k}^{H}}+{{\bf{H}}_{2,k}^{H}}{\bf{V}}{{\bf{H}}_{1,k}}}\right){{\bf{X}}}+{\bf{n}}_{c,k}={\bf
H}_{k}{\bf X}_{k}+{\bf{n}}_{c,k},$ (12)

Furthermore, the user $k$ ’s signal-to-interference-to-noise ratio (SINR) can
be calculated by

${SINR}_{k}=\frac{{{{\left|{{\bf{H}}_{k}^{H}{{\bf{W}}_{k}}}\right|}^{2}}}}{{\sum\limits_{l\neq
k}{{{\left|{{\bf{H}}_{k}^{H}{{\bf{W}}_{l}}}\right|}^{2}}}+\sigma_{c}^{2}}}.$
(13)

_2) Radar Signal:_

The radar signal received at the ISAC receiver is given by [33]

${\bf{y}}_{r}=\left({{\bf{H}}_{1}^{H}{\bf{VB}}{{\bf{V}}^{H}}{{\bf{H}}_{1}}{\bf{+A}}}\right){\bf{X}}+{\bf{n}}_{r},$
(14)

It is assumed that IRS is a mono-static MIMO radar[2]. Thus,
${\bf{A}}\in{{\mathbb{C}}^{M\times M}}$ is the target response matrix of the
ISAC BS and ${\bf{B}}\in{{\mathbb{C}}^{N\times N}}$ is the target response
matrix based on parameters of the IRS’s elements in the device-based ISAC
system. They can be calculated by,

$\displaystyle{\bf{A}}=\sum\limits_{k=1}^{K}{{\alpha_{k}}{{\bf{a}}_{r}}\left({{\theta_{k}}}\right){{\bf{a}}_{t}^{H}}\left({{\theta_{k}}}\right)},$
(15a)
$\displaystyle{\bf{B}}=\sum\limits_{k=1}^{K}{{\alpha_{k}}{{\bf{v}}_{r}}\left({{\upsilon_{k},\theta_{k}}}\right){\bf{v}}_{t}^{H}\left({{\upsilon_{k},\theta_{k}}}\right)}.$
(15b)

$\alpha_{k}$ is the complex pathloss that include the coefficient of the
pathloss, reflection, and complex radar cross of the target. Vector
${{\mathbf{a}}_{t}}\left(\theta\right)$ and
${{\mathbf{a}}_{r}}\left(\theta\right)$ are array steering vectors from
transmit and receive antennas. For radar signals, beam pattern error is
considered in the proposed system [6]. MIMO radar transmit beamforming design
aims to optimize the transmit power at given directions, or generally match a
desired beam pattern. The beam pattern error is the mean square error (MSE)
between the obtained beam pattern and the ideal beam pattern. It can be
calculated by the following formula

${L_{r}}\left({\bf{R}}\right)=\frac{1}{L}\sum\limits_{l=1}^{L}{{{\left|{d\left({{\psi_{l}}}\right)-P\left({{\psi_{l}}}\right)}\right|}^{2}}},$
(16)

where ${\bf{R}}={\bf{W}}{{\bf{W}}^{H}}$ is covariance matrix. The direction
grids $\left\\{{{\psi_{l}}}\right\\}_{l=1}^{L}$ are sampled in the range of
$-90^{\circ}$ to $90^{\circ}$ with resolution of $0.1^{\circ}$.
$P\left(\psi\right)={{\bf{a}}^{H}}\left(\psi\right){\bf{Ra}}\left(\psi\right)$
is power consumption in direction $\psi$ and $d\left({{\psi}}\right)$ is
desired beam pattern, its calculating principle is

$d\left(\psi\right)=\left\\{{\begin{array}[]{*{20}{l}}{1,}&{{\psi_{p}}-{\Delta\mathord{\left/{\vphantom{\Delta
2}}\right.\kern-1.2pt}2}\leq\psi\leq{\psi_{p}}+{\Delta\mathord{\left/{\vphantom{\Delta
2}}\right.\kern-1.2pt}2},p=1,2,3,}\\\ {0,}&{otherwise},\end{array}}\right.$
(17)

where the ideal beam pattern $\psi_{p}$ consists of three main beams and
$\Delta=10^{\circ}$ is each ideal beam width.

The beam pattern error is set as a constraint and it can be expressed as

${L_{r}}\left({\bf{R}}\right)\leq\ell,$ (18)

The above constraint implies that the beam pattern error is not allowed to
exceed the threshold $\ell$. The threshold symbolizes the accuracy of the
sensing performance.

###  II-B Problem Formulation

The problem of transmit beamforming and phase shift design can be transformed
into a long-term instantaneous performance function using the ergodic average
value $\bf x$ to reflect the system performance [28], and this solution can be
applied to optimize other systems, including optimization of frequency
division multiplexing, power control, and random access. It can be calculated
by

${\bf{x}}\leq{\mathbb{E}}\left[{{{\bf
f}_{1}}\left({{\bf{h}},{\bf{p}}\left({\bf{h}}\right)}\right)}\right],$ (19)

where ${{\bf f}_{1}}\left({{\bf{h}},{\bf{p}}\left({\bf{h}}\right)}\right)$ is
an instantaneous performance function. The design is to choose a resource
allocation ${\bf{p}}\left({\bf{h}}\right)$ to maximize the ergodic variable
${\bf{x}}$. The average variable ${\bf{x}}$ reflects the performance of
wireless communication systems in a considerably long period and is influenced
by instantaneous resource allocation ${\bf{p}}\left({\bf{h}}\right)$.

$\begin{array}[]{*{20}{l}}{\max}&{{f_{0}}\left({\mathbf{x}}\right)}\\\
{s.t.}&{{\bf
x}\leqslant\mathbb{E}\left[{{{\mathbf{f}}_{1}}\left({{\mathbf{h}},{\mathbf{p}}\left({\mathbf{h}}\right)}\right)}\right]}\\\
{}&{{{\mathbf{f}}_{2}}\left({\mathbf{x}}\right)\geqslant
0,{\mathbf{x}}\in\chi,{\mathbf{p}}\in\mathcal{P}.}\end{array}$ (20)

In the proposed system model, ${\bf{h}}$ is the channel state $\bf H$,
${\bf{p}}\left({\bf{h}}\right)$ is the transmit beamforming matrix $\bf W$ and
phase optimization $\bf V$.
${{\bf{f}}_{1}}\left({{\bf{h}},{\bf{p}}\left({\bf{h}}\right)}\right)$ is the
corresponding data transmission rate. ${{\bf{f}}_{2}}\left(\bf x\right)$
indicates the vector utility function. The design goal is to maximize the
average vector $\bf{x}$ for transmit beamforming
${\bf{p}}\left({\bf{h}}\right)$ under constraints,

${x_{k}}\leqslant{{\mathbb{E}}_{\bf{H}}}\left[{\log\left({1+SIN{R_{k}}}\right)}\right].$
(21)

Its design goal is to minimize radar signal loss performance with long-term
constraints for transmit beamforming and phase-shift design. It is assumed
that the element of covariance matrix ${\left[\bf{R}\right]_{mm}}$ has power
budget:

${{\left[\bf{R}\right]_{mm}}}\leqslant{{{P_{max}}}\mathord{\left/{\vphantom{{{P_{max}}}M}}\right.\kern-1.2pt}M},m=1,2,...,M,$
(22)

where $P_{max}$ is the total transmit power. The utility
${f_{0}}\left({\bf{x}}\right)$ is set as the sum rate function.

${f_{0}}\left({\bf{x}}\right)=\sum\limits_{k=1}^{K}{{x_{k}}}.$ (23)

Finally, the optimization problem of MISO scenario in the IRS-aided ISAC
system can be formulated as

$\begin{array}[]{*{20}{l}}{\mathop{\max}\limits_{{\bf{V,W}}}{f_{0}}\left({\bf{x}}\right)}\\\
{\begin{array}[]{*{20}{l}}{s.t.{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{x_{k}}\leqslant{{\mathbb{E}}_{\bf{H}}}\left[{\log\left({1+SIN{R_{k}}}\right)}\right]},\\\
{{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern
1.0pt}{{\left[\bf{R}\right]_{mm}}}\leq{{{P_{max}}}\mathord{\left/{\vphantom{{{P_{max}}}M}}\right.\kern-1.2pt}M},m=1,2,...,M,}\\\
{{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern
1.0pt}{L_{r}}\left({\bf{R}}\right)\leq\ell}.\end{array}}\end{array}$ (24)

##  III Primal-dual Proximal Policy Optimization Scheme in Multi-user MISO
Scenario

In this section, we investigate multi-user MISO scenario in the IRS-aided ISAC
system in the THz band, and use the primal-dual optimization to solve the
problem (24).

###  III-A Transmit Beamforming and Phase-shift Design via PPO

For the problem (24), the value-based method in DRL is not able to deal with
continuous actions. The value-based method is unable to obtain the optimal
solution with the constraints. Therefore, the research is mainly based on
policy optimization. The traditional policy-based solution is expressed by
[29]:

${L^{PG}}\left({\mbox{\boldmath{$\omega$}}}\right)={\mathbb{E}_{t}}\left[{\log{\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right){\mathbf{A}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}\right].$
(25)

The advantage function can be calculated by
${\bf{A}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)=\sum\nolimits_{t^{\prime}>t}{{\gamma^{t^{\prime}-t}}{r^{t^{\prime}}}}-{v_{\upsilon}}\left({{{\bf{H}}^{t}},{\mbox{\boldmath{$\upsilon$}}}^{t}}\right)$.
The shortcoming of this conventional method is the need to update the step
size. Inappropriate setting of the step size will lead to a degraded
performance. We need find a step size to ensure that the reward function
increases monotonically for each iteration. On the other hand, the
conventional policy gradient methods are sensitive to their hyper parameters
and have high variance. Employing a trust region constraint can be regarded as
an effective choice. The popular solution is trust region policy optimization
(TRPO). It can be given by

$\begin{array}[]{*{20}{l}}{\mathop{\max}\limits_{\mbox{\boldmath{$\omega$}}}}&{{\mathbb{E}_{{\rho_{\omega}}\left(\tau\right)}}\left[{\sum\nolimits_{t}{{\gamma^{t-1}}\frac{{{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{\mathbf{A}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}\right]},\\\
{s.t.}&{{D_{KL}}\left[{\frac{{{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}}\right]\leqslant\delta}.\end{array}$
(26)

More detailed classifications on TRPO are presented in Appendix A. The PPO
algorithm is an approximate version of TRPO based on the first order
gradients, utilizing DNNs to a large-scale distributed setting. PPO solves
this problem by introducing relative entropy in policy update [34].

$\begin{array}[]{*{20}{l}}{{L^{PPO}}\left({\mbox{\boldmath{$\omega$}}}\right)=\sum\nolimits_{t}{\left({{{{\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}\mathord{\left/{\vphantom{{{\mathbf{\pi}}\left({{\mathbf{H,\omega}}}\right)}{{{\mathbf{\pi}}_{old}}\left({{\mathbf{H,\omega}}}\right)}}}\right.\kern-1.2pt}{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}}\right){\mathbf{A}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}\\\
{{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern
1.0pt}-\lambda{D_{KL}}\left[{{{{\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}\mathord{\left/{\vphantom{{{\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}}\right.\kern-1.2pt}{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}}\right]}.\end{array}$
(27)

Generally speaking, PPO is based on the actor-critic structure, and the actor
part’s goal is to maximize ${L^{PPO}}\left(\mbox{\boldmath{$\omega$}}\right)$,
which can be divided into two types, the one is
${L^{KLPEN}}\left(\mbox{\boldmath{$\omega$}}\right)$ in PPO1 and the other is
${L^{CLIP}}\left(\mbox{\boldmath{$\omega$}}\right)$ in PPO2. A penalty on
Kullback-Leibler divergence (KL divergence) is used in PPO1, and to adapt the
penalty coefficient so that we achieve some target values of the KL divergence
target each policy update. PPO2 has no KL divergence term in the target, nor
any constraints. Instead, it relies on tailoring the objective function to
eliminate the incentives of the new policy and the old policy.

$\begin{array}[]{*{20}{l}}{{L^{CLIP}}\left({\mbox{\boldmath{$\omega$}}}\right)=}&{{\mathbb{E}_{t}}\left[{\min\left({{\textstyle{{\pi\left({H,{\mbox{\boldmath{$\omega$}}}}\right)}\over{{\pi_{old}}\left({\bf
H,{\mbox{\boldmath{$\omega$}}}}\right)}}}{\bf A}\left({{\bf
H},{\mbox{\boldmath{$\omega$}}}}\right),}\right.}\right.}\\\
{}&{\left.clip\left({{\textstyle{{\pi\left({{\bf
H},{\mbox{\boldmath{$\omega$}}}}\right)}\over{{\pi_{old}}\left({{\bf
H},\omega}\right)}}},1-\epsilon,1+\epsilon}\right){\bf A}\left({{\bf
H},{\mbox{\boldmath{$\omega$}}}}\right)\right]}.\end{array}$ (28)

To get the similarity degree of the probability distribution of actions, KL
divergence $D_{KL}$ can be used to calculate. Under the setting of
hyperparameter is $\epsilon=2$, the second term
$clip\left({\frac{{\pi\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}{{{\pi_{old}}\left({{\bf{H,\mbox{\boldmath{$\omega$}}}}}\right)}},1-\epsilon,1+\epsilon}\right)$
modifies the surrogate objective by clipping the probability ratio. In the
formula (28),
$\frac{{\pi\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)}}{{{\pi_{old}}\left({{\bf{H,\mbox{\boldmath{$\omega$}}}}}\right)}}$
can ensure that the distribution gap between the two updates is small. This
clipped operation can avoid unnecessary samples to save training time.

To obtain optimization of transmit beamforming and phase-shift design, the
actor part takes charge of continuous transmit beamforming in the established
PPO framework. Meanwhile, the critic part undertakes the discrete phase-shift
design.

The policy
${\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)$
is introduced to act as the transmit beamforming $\bf W$. For trained
parameters $\omega$, we make
${\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)={\bf
W}$. And the problem (20) is transformed into the following problem.

$\begin{array}[]{*{20}{l}}{\mathop{\max}\limits_{{{\mathbf{H}}},{{\mbox{\boldmath{$\omega$}}}}}{f_{0}}\left({{{\mathbf{x}}}}\right)}\\\
{\begin{array}[]{*{20}{l}}{s.t.{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{{\mathbf{x}}}\leqslant\mathbb{E}\left[{{{{\bf
f}_{1}}}\left({{\bf{H}},{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}\right)}\right]},\\\
{{\mathbf{f}}_{2}}\left({\mathbf{x}}\right)\geqslant
0.\par\end{array}}\end{array}$ (29)

The problem (29) is the general form of the problem (24). The constraint
${{\left[\bf{R}\right]_{mm}}}\leq{{{P_{max}}}\mathord{\left/{\vphantom{{{P_{max}}}M}}\right.\kern-1.2pt}M}$
is clipped by the action output of policy
${\mbox{\boldmath{$\pi$}}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)$.
We assume that
${{\mathbf{f}}_{2}}\left({\mathbf{x}}\right)=\ell-{L_{r}}\left({\bf{R}}\right)$.
To convenience of understanding, the following formulation is based on the
general form (29).

For the critic part, the phase shift of the instantaneous performance function
can be calculated by the output action. The critic part adopts the design
concept of DQN, it needs to be discrete,
${i_{n}}\in\left\\{{0,1,...,{2^{b}}-1}\right\\}$, the corresponding phase
degree ${\beta_{n}}$ is given by

${\beta_{n}}=\frac{{{i_{n}}2\pi}}{{{2^{b}}-1}}.$ (30)

Accordingly, its Q-value function produces a discrete action ${i_{n}}$ as

${i_{n}}=\mathop{\arg\max}\limits_{{{\bf{i}}_{n}}}{\bf Q}^{t}\left({{{{\bf
H}_{k}^{t}}},{{\bf{i}}_{n}}},{{\mbox{\boldmath{$\upsilon$}}}_{k}^{t}}\right).$
(31)

And the amplitude $\gamma_{n}$ also can be calculated through the formula (3).
Therefore, the phase shift design of IRS can be obtained by the critic part.

###  III-B Primal-dual Learning Optimization

The primal-dual optimization is introduced to tackle the problem (29). The
Lagrangian of the problem (29) is utility and constraints weighted by their
multipliers. For convenience, its Lagrangian function
$\mathcal{L}\left(\cdot\right)$ for the problem (29) is given by

$\begin{array}[]{*{20}{l}}{\mathcal{L}\left({{\mbox{\boldmath{$\omega$}}},{\bf{x}},{\mbox{\boldmath{$\lambda$}}},{\mbox{\boldmath{$\mu$}}}}\right)={f_{0}}\left({\bf{x}}\right)+{{\mbox{\boldmath{$\mu$}}}^{T}}{{{\bf
f}_{2}}}\left({\bf{x}}\right)}\\\ {{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}+{{\mbox{\boldmath{$\lambda$}}}^{T}}\left({{\mathbb{E}}\left[{{{{\bf
f}_{1}}}\left({{\bf{H}},{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}\right)}\right]-{\bf{x}}}\right)}.\end{array}$
(32)

Find the gradient of the four parameters in sequence,

${{\mbox{\boldmath{$\omega$}}}^{t+1}}={{\mbox{\boldmath{$\omega$}}}^{t}}+{\tau_{1}}{\nabla_{\omega}}{\mathbb{E}}\left[{{{\bf{f}}_{1}}\left({{\bf{H}},{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}\right){{\mbox{\boldmath{$\lambda$}}}^{t}}}\right],$
(33)
${{\bf{x}}^{t+1}}={{\bf{x}}^{t}}+{\tau_{2}}\left({\nabla{f_{0}}\left({{{\bf{x}}^{t}}}\right)+\nabla{{\bf{f}}_{2}}\left({{{\bf{x}}^{t}}}\right){{\mbox{\boldmath{$\mu$}}}^{t}}-1}\right),$
(34)

where $\tau_{1},\tau_{2}>0$ denote scalar step sizes. A gradient update is
performed on current dual iterates $\lambda_{k},\mu_{k}$ in a similar manner
by subtracting the partial stochastic gradients $\nabla_{\lambda}L$,
$\nabla_{\mu}L$ and projecting onto the positive orthant to obtain

${{\mbox{\boldmath{$\lambda$}}}^{t+1}}={{\mbox{\boldmath{$\lambda$}}}^{t}}-{\tau_{3}}\left({{\mathbb{E}_{\bf
H}}{{\bf{f}}_{1}}\left({\bf{H}},{{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{{\mbox{\boldmath{$\omega$}}}^{t+1}}}\right)}\right)-{{\bf{x}}^{t+1}}}\right).$
(35)
${{\mbox{\boldmath{$\mu$}}}^{t+1}}={{\mbox{\boldmath{$\mu$}}}^{t}}-{\tau_{4}}{{\bf{f}}_{2}}\left({{{\bf
x}^{t+1}}}\right).$ (36)

with associated step sizes $\tau_{3},\tau_{4}>0$. The gradient primal-dual
updates in (33)–(36) successively optimize the primal and dual variables to
the maximum and minimum points of the Lagrangian function, respectively.

The updates in the formula (33) and (34) can be replaced with the zeroth-
ordered gradient updates. The gradients estimates with finite difference can
be established using function observations in the given initial variables
${{\mbox{\boldmath{$x$}}}_{0},{\mbox{\boldmath{$\omega$}}}_{0}}$ and the
sampled points
${{\bf\widehat{x}}_{1},{\bf\widehat{x}}_{2},{\mbox{\boldmath{$\widehat{\omega}$}}}}$
as,

$\widehat{\nabla}{{{f}}_{0}}\left({{{\bf{x}}_{0}}}\right):=\frac{{{{\widehat{f}}_{0}}\left({{{\bf{x}}_{0}}+{\alpha_{1}}{{{\bf{\hat{x}}}}_{1}}}\right)-{{\widehat{f}}_{0}}\left({{{\bf{x}}_{0}}}\right)}}{{{\alpha_{1}}}}{{\bf{\hat{x}}}_{1}},$
(37)
$\widehat{\nabla}{\bf{f}_{2}}\left({{{\bf{x}}_{0}}}\right):=\frac{{{{\widehat{\bf{f}}}_{2}}\left({{{\bf{x}}_{0}}+{\alpha_{2}}{{{\bf{\hat{x}}}}_{2}}}\right)-{{\widehat{\bf{f}}}_{2}}\left({{{\bf{x}}_{0}}}\right)}}{{{\alpha_{2}}}}\widehat{\bf{x}}_{2}^{T},$
(38)
$\begin{array}[]{*{20}{l}}{{{\widehat{\nabla}}_{\mbox{\boldmath{$\omega$}}}}\mathbb{E}\left[{{\bf{f}_{1}}\left({{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right),{\bf{H}}}\right)}\right]}\\\
{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{:=\frac{{\widehat{\bf{f}_{1}}\left({{\bf{H}},{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{{\mbox{\boldmath{$\omega$}}}_{0}}+{\alpha_{3}}\widehat{\mbox{\boldmath{$\omega$}}}}\right)}\right)-\widehat{{\bf
f}_{1}}\left({{\bf{H}},{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{{\mbox{\boldmath{$\omega$}}_{0}}}}\right)}\right)}}{{{\alpha_{3}}}}{{\widehat{\mbox{\boldmath{$\omega$}}}}^{T}}},\end{array}$
(39)

where $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ are the gradient estimated
upgrading step factors. Furthermore, the $\tau_{1}$, $\tau_{2}$, $\tau_{3}$,
$\tau_{4}$ are the primal-dual upgrading steps.

##  IV Distributed Proximal Policy Optimization Scheme for Multi-user MIMO
Scenario

In this section, we will explore how to obtain joint optimization of transmit
beamforming and phase shift design in the multi-user MIMO scenario.
Distributed DRL will be introduced in this section.

###  IV-A System Model and Formulation in Multi-user MIMO Scenario

In the multi-user MIMO ISAC system, it is assumed that there exists a BS with
$M$ transmit antennas and $K$ users with $R$ receive antennas. The transmit
signal to the user $k$ through beamforming
${{\mathbf{X}}_{k}}\in{\mathbb{C}^{M\times R}}$ is

${{\mathbf{X}}_{k}}={{\mathbf{W}}_{k}}\cdot{{\mathbf{s}}_{k}},$ (40)

where
${{\mathbf{s}}_{k}}={\left[{{s_{1}},{s_{2}},...,{s_{R}}}\right]^{T}}\in{\mathbb{C}^{R\times
1}}$. The responding transmit beamforming matrix is
${{\mathbf{W}}_{k}}\in{\mathbb{C}^{M\times R}}$ and radar signal is expressed
by:

${{\mathbf{y}}_{r,k}}=\left({\mathbf{H}}_{1}^{H}{{\mathbf{V}}}{\mathbf{B}}{{\mathbf{V}}^{H}}{{\mathbf{H}}_{1}+{\mathbf{A}_{k}}}\right){{\mathbf{X}}_{k}}+{\mathbf{n}}_{r,k}.$
(41)

Similar to the formula (2), the communication signal ${\mathbf{y}}_{c,k}$ is
derived from

${\mathbf{y}}_{c,k}=\left({{{\mathbf{H}}_{0,k}}+{{\mathbf{H}}_{2,k}}{\mathbf{V}}{{\mathbf{H}}_{1}}}\right){{\mathbf{X}}_{k}}+{\mathbf{n}}_{c,k}.$
(42)

Compared with ${{\bf{H}}_{0}}\in{\mathbb{C}}^{{M\times K}}$ and
${{\bf{H}}_{2}}\in{\mathbb{C}}^{{N\times K}}$ in multi-user MISO scenario, the
channel gains are transformed into
${{\mathbf{H}}_{0,k}}\in{\mathbb{C}^{M\times R}}$ and
${{\mathbf{H}}_{2,k}}\in{\mathbb{C}^{N\times R}}$, respectively. And the
corresponding parameterization problem is expressed as follows:

$\begin{array}[]{*{20}{l}}{\mathop{\max}\limits_{{{\mathbf{H}}_{k}},{{\mathbf{\omega}}_{k}}}{f_{0}}\left({{{\mathbf{x}}_{k}}}\right)}\\\
{\begin{array}[]{*{20}{l}}{s.t.{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{{\mathbf{x}}_{k}}\leqslant\mathbb{E}\left[{{\text{log}}\left({1+SIN{R_{k}}\left({{{\mathbf{H}}_{k}},{{{\mbox{\boldmath{$\pi$}}}}_{k}}\left({{{\mathbf{H}}_{k}},{{{\mbox{\boldmath{$\omega$}}}}_{k}}}\right)}\right)}\right)}\right]},\\\
{{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{{\left[{\bf
R}_{k}\right]_{mm}}}\leqslant{{{P_{max}}}\mathord{\left/{\vphantom{{{P_{max}}}M}}\right.\kern-1.2pt}M},m=1,2,...,M},\\\
{{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern
1.0pt}{L_{r}}\left({{\mathbf{R}}_{k}}\right)\leqslant\ell.}\end{array}}\end{array}$
(43)

Similar to the problem (29), ${f_{0}}\left({{{\mathbf{x}}_{k}}}\right)$ in the
problem (43) is set as the sum rate function. Typically, the instantaneous
system performance in practical MIMO transmission designs can not reflect the
system performance well. One of solutions adopts the long term average such as
ergodic capacity for transmission optimization [35].

The channel gain ${\bf H}_{k}$ is the state in the environment, what we
optimize is the transmit beamforming and phase-shift design. In this paper,
the transmit beamforming ${\bf W}_{k}$ and phase-shift design ${\bf V}$ is
policy derived form the state, i.e. channel gain ${\bf H}_{k}$. Similar to the
multi-user MISO scenario, the ${{\mbox{\boldmath{$\pi$}}}_{k}\left({\bf
H}_{k},{\mbox{\boldmath{$\omega$}}}_{k}\right)}$ form actor part realizes the
transmit beamforming and the critic part takes charge of the phase-shift
design. When the scenario extended to the MIMO one, the number of receive
antennas $R$ takes place of the number of users $K$. The channel ${\bf
H}\in{\mathbb{C}}^{{M\times R\times K}}$ is 3 dimensions, compared with
2-dimension ${\bf H}$ in MISO scenario. How to solve higher dimension in
multi-user MIMO IRS-aided ISAC system is worthy of to be studied, so DPPO,
this multi-threading DRL technique is introduced in the following subsection.

###  IV-B DPPO Optimization via multi-threading DRL

DPPO with multiple threads can be adapted to more complex environments. DPPO
is not a specific algorithm like PG or DQN, it conveys the concept of multi-
threading learning. Its core is that in the training process of DRL, we can
train multiple agents in parallel threading. In the training process, each
worker takes charge of data collection locally. Multiple workers can enrich
the experiences form the environment. After the same episodes, distributed
learning can accelerate the learning speed further because of its richer
experience replay and its higher utilization rate of CPU.

The DPPO is suitable for the proposed multi-user MIMO scenario in IRS-aided
ISAC system. Every user can act as each worker in multi-threading DRL. It can
collect different channel realization from its own situation. Accordingly,
each worker (i.e. user) takes the corresponding policy to optimize its
capacity.

Through the total rewards from different workers (i.e. users), the original
PPO algorithm can be achieved by estimated advantage functions ${{\bf
A}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)$. To facilitate the use
of DNNs with batch updates while also supporting variable length episodes,
L-step returns is proposed to estimate the advantage, i.e. we sum the rewards
over the same L-step windows and bootstrap from the value function after
L-steps:

${{\bf
A}^{t}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)=\sum\nolimits_{l=1}^{L}{{\gamma^{l-1}}{r^{t}}+}{\gamma^{L-1}}{v}\left({{{\bf{H}}^{t+L}}},{{\mbox{\boldmath{$\upsilon$}}}^{t+L}}\right)-{v}\left({{\bf{H}}^{t}},{{\mbox{\boldmath{$\upsilon$}}}^{t}}\right).$
(44)

Workers share a global PPO and the gradients of PPO would not be calculated by
workers. Gradients will not be transformed to the chief like A3C. Workers only
push the data collection by themselves to the global PPO. Each worker
calculates the local $L^{CLIP}_{k}\left(\mbox{\boldmath{$\omega$}}\right)$ by
the formula (29), computes gradient ${\nabla_{\omega}}{L^{CLIP}}$ and sends to
the global PPO. The global PPO updates a certain batch of data from multiple
workers (workers stop collecting when the chief is updating). After updating,
workers collect data with the latest policy ${\mbox{\boldmath{$\pi$}}}^{t}$.

###  IV-C Transmit Beamforming and Phase-shift Design in Multi-user MIMO
Scenario

Figure 2: Primal-dual DPPO beamforming and phase-shift design in the multi-
user MIMO scenario.

In the design of the proposed multi-user MIMO scenario, the dimension of
channel state between the ISAC BS and users consists of $M\times R\times K$.
It is quite difficult to realize transmit beamforming and passive phase
optimization directly. With the help of DPPO’s multi-threading training, each
user will be allocated to each worker. The workers can collect observation and
choose actions in parallel. Compared with multi-user MISO scenario, each
worker processes the channel state ${\bf H}_{k}\in{\mathbb{C}}^{{M\times R}}$,
which is similar to ${\bf H}\in{\mathbb{C}}^{{M\times K}}$. The global PPO
will allocate actions to different workers according to the experience memory.

##  V Transmit Beamforming and Phase-Shift Design Optimization Algorithm
Design

###  V-A Primal-dual PPO Algorithm for Multi-user MISO Scenario

In the proposed algorithm, we utilize the PPO’s architecture to learn two
kinds of actions. Firstly, we allocate the equal initial beamforming to each
user, and initial policy parameters including mean
$\mbox{\boldmath{$\omega$}}^{0}$ and Langrange multipliers
$\mbox{\boldmath{$\lambda$}}^{0}$ and $\mbox{\boldmath{$\mu$}}^{0}$. And then,
each of the ISAC BS’s antennas deploys PPO method in turn, and the sum of
antennas’ iteration is $M$. Next, beginning with the actor part’s and the
critic part’s training, we compute the gradient estimate from replay
experience. The estimated gradient integrates into the primal-dual
optimization. The primal-dual PPO algorithm is used to iteratively calculate
the transmit beamforming matrix ${\bf W}^{\rm*}$ and the phase-shift design
matrix ${\bf V}^{\rm*}$.

In Algorithm 1, episode begins with channel state ${\bf
H}^{0}\in{\mathbb{C}}^{{M\times K}}$ between the ISAC BS and users. It can be
calculated by the formula (4) and the first phase shift ${\bf V}^{0}$ in IRS
is sampled in domain between $\left[{-\pi/2,\pi/2}\right]$. Then, the
iteration begins and the channel state ${\bf
H}_{k}^{0}\in{\mathbb{C}}^{{M\times 1}}$ is transited into the DNN of policy.
Accordingly, the actor part outputs the policy
${\mbox{\boldmath{$\pi$}}}_{k}^{t}\left({\bf
H}_{k}^{t},{\mbox{\boldmath{$\omega$}}}_{k}^{t}\right)$ to realize transmit
beamforming. Simultaneously, the critic part outputs Q-value function ${\bf
Q}^{t}\left({\bf H}_{k}^{t},{\mbox{\boldmath{$\upsilon$}}}_{k}^{t}\right)$.
The choice of ${i_{n}^{t}}$ follows the greedy policy. The algorithm selects a
random discrete action $i_{n}$ with the probability $\varepsilon$ and selects
${i_{n}}=\mathop{\arg\max}\limits_{{{\bf{i}}_{n}}}{\bf Q}^{t}\left({{{{\bf
H}_{k}^{t}}},{{\bf{i}}_{n}}},{{\mbox{\boldmath{$\upsilon$}}}_{k}^{t}}\right)$
with probability $1-\varepsilon$. The reward ${\bf
r}^{t}_{k}={{{f}}_{0}^{t}}\left({{{\bf{x}}}}\right)$ can be calculated by the
obtained channel state ${\bf H}_{k}^{t}$ and transmit beamforming ${\bf
W}_{k}^{t}$. At the same time, the index $i_{n}^{t}$ of the IRS derived from
the critic part affects the next state ${\bf H}_{k}^{t+1}$ through new ${\bf
V}^{t+1}$. Therefore, the batch of
$\left\\{{{\bf{H}}_{k}^{t},{{\mbox{\boldmath{$\pi$}}}}_{k}^{t},{\bf{r}}_{k}^{t}},{{\bf
H}_{k}^{t+1}}\right\\}$ is obtained. Then batches are collected into the
experience memory and parameters of the actor part $\omega$ and the critic
part $\upsilon$ can be updated by the experience memory.

Algorithm 1 Transmit beamforming and phase-shift design via primal-dual PPO
algorithm

Input Initial transmit beamforming ${\bf W}^{0}$, policy parameters
${\mbox{\boldmath{$\omega$}}^{0}}$, Lagrange multipliers
${{\mbox{\boldmath{$\lambda$}}}^{0}},{{\mbox{\boldmath{$\upsilon$}}}^{0}}$ can
be allocated for users. channel states ${\bf H}_{0}$, ${\bf H}_{1}$, and ${\bf
H}_{2}$ are fixed in each episode.

Episode begins

For $i=1,2,...,I$ do:

For users $k=1,2,...,K$ do:

a) Draw samples ${{{\bf\hat{x}}_{1}}}$ and ${{{\bf\hat{x}}_{2}}}$ from a
truncated standard normal distribution;

b) Calculate the channel state ${{\bf H}_{k}^{t}}\in{{\mathbb{C}}^{M\times
1}}$, ${{\bf H}^{t}=}{{\bf{H}}_{0}}{\bf{+}}{{\bf{H}}_{2}}{{\bf
V}^{t}}{{\bf{H}}_{1}}$;

c) ${{\bf H}_{k}^{t}}$ is fed into the actor part, then the actor part outputs
strategy function ${\bf
W}_{k}^{t}={\mbox{\boldmath{$\pi$}}_{k}^{t}}\left({{{\bf
H}_{k}^{t}},{\mbox{\boldmath{$\omega$}}_{k}^{t}}}\right)$; fed into the critic
part, then the critic part outputs Q-value function ${\bf Q}^{t}\left({{{\bf
H}_{k}^{t}},{{\mbox{\boldmath{$\upsilon$}}}_{k}^{t}}}\right)$;

d) Calculate the reward ${\bf
r}^{t}_{k}={{{f}}_{0}^{t}}\left({{{\bf{x}}}}\right)$ via the formula (23);

e) With the probability $\varepsilon$ select a random discrete action $i_{r}$
otherwise select ${i_{r}}=\mathop{\arg\max}\limits_{{{\bf{i}}_{r}}}{\bf
Q}^{t}\left({{{{\bf
H}_{k}^{t}}},{{\bf{i}}_{r}}},{{\mbox{\boldmath{$\upsilon$}}}_{k}^{t}}\right)$;

f) Calculate the amplitude $\gamma_{n}$ and phase $\beta_{n}$ through the
formula (3) and (29). Subsequently, the phase-shift design ${\bf V}^{t+1}$ can
be obtained;

g) The next state ${{\bf H}_{k}^{t+1}}$ can be obtained by ${{\bf
H}^{t+1}=}{{\bf{H}}_{0}}{\bf{+}}{{\bf{H}}_{2}}{{\bf V}^{t+1}}{{\bf{H}}_{1}}$;

h) Collect
$\left\\{{{\bf{H}}_{k}^{t},{{\mbox{\boldmath{$\pi$}}}}_{k}^{t},{\bf{r}}_{k}^{t}},{{\bf
H}_{k}^{t+1}}\right\\}$ into the experience memory to estimate discounted
reward and advantages, then update the actor part and the critic part;

i) Compute the gradient estimate
$\widehat{\nabla}{{{f}}_{0}}\left({{{\bf{x}}}}\right)$, $\widehat{\nabla}{{\bf
f}_{2}}\left({{{\bf{x}}}}\right)$,
${{{\widehat{\nabla}}_{\mbox{\boldmath{$\omega$}}}}\mathbb{E}\left[{{\bf{f}_{1}}\left({{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right),{\bf{h}}}\right)}\right]}$:
Execute the formulas (37)–(39);

j) Update the primal-dual variable ${\mbox{\boldmath{$\omega$}}}^{t+1}_{k}$,
${{\bf{x}}^{t+1}_{k}}$, ${{\mbox{\boldmath{$\lambda$}}}^{t+1}_{k}}$,
${{\mbox{\boldmath{$\mu$}}}^{t+1}_{k}}$: Execute the formulas (33)–(36);

Execute step b) to h) until convergence, and an optimized strategy is obtained
after the PPO training ${{\mbox{\boldmath{$\pi$}}}_{k}^{\rm{*}}}\left({{{\bf
H}_{k}},{{\mbox{\boldmath{$\omega$}}}_{k}}}\right)$;

End for

End for

Episode ends

Output Transmit beamforming ${\bf{W}^{\rm{*}}}$, phase-shift design
$\bf{V^{\rm{*}}}$, and system performances
${{f}_{0}^{\rm{*}}}\left({{{\bf{x}}}^{\rm{*}}}\right)$.

Subsequently, the gradient estimate
$\widehat{\nabla}{{{f}}_{0}}\left({{{\bf{x}}}}\right)$, $\widehat{\nabla}{{\bf
f}_{2}}\left({{{\bf{x}}}}\right)$,
${{{\widehat{\nabla}}_{\mbox{\boldmath{$\omega$}}}}\mathbb{E}\left[{{\bf{f}_{1}}\left({{\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right),{\bf{h}}}\right)}\right]}$
can be obtained from the formulas (37)–(39). And primal-dual variables can be
obtained through the formulas (33)–(36). After primal-dual PPO training
performance converges, the optimized transmit beamforming ${\bf{W}^{\rm{*}}}$,
the phase-shift design $\bf{V^{\rm{*}}}$, and the system performances
${{f}_{0}^{\rm{*}}}\left({{{\bf{x}}}^{\rm{*}}}\right)$ can be achieved. For
the sake of clarity, a full notation list is included in TABLE I.

TABLE I: List of Notations. Notations | Explanations
---|---
$\cal M$ | The set of ISAC BS’s antennas
$\cal K$ | The set of served users
$\cal N$ | The set of IRSs’ elements
$\cal R$ | The set of user’s antennas
$\bf A$ | The target response matrix of the ISAC BS
$\bf B$ | The target response matrix of the IRS’s elements
${\bf V}$ | The effective diagonal phase matrix
$i_{n}$ | The discrete action of the IRS’s element index
$\gamma$ | The amplitude reflection coefficient of the IRS
$\beta$ | The phase-shift coefficient of the IRS
$\theta$ | The azimuth angle
$\theta_{AoD}$ | The angle of departure
$\theta_{AoA}$ | The angle of arrival
$\psi_{l}$ | The sampled angle
$d\left({{\psi}}\right)$ | The desired beam pattern in direction $\psi$
$d\left({{\psi_{p}}}\right)$ | The ideal beam pattern in direction $\psi_{p}$
$\phi$ | The elevation angle
${\bf{a}}\left({{\theta}}\right)$ | Steering vector at the ISAC BS
${{\mathbf{v}}}\left({\phi,{\mathbf{\theta}}}\right)$ | Steering vector at the IRSs
${\bf{b}}\left({{\theta}}\right)$ | Steering vector at receive users
$P\left(\psi\right)$ | The power consumption in direction $\psi$
$P_{max}$ | The total transmit power
$\bf R$ | The covariance matrix
${\bf H}_{0}$ | The channel gain from the ISAC BS to user $k$
${\bf H}_{1}$ | The channel gain from the ISAC BS to the IRS
${\bf H}_{2}$ | The channel gain from the IRS’s element $n$ to user $k$
${\bf H}_{LOS}$ | The LoS component
${\bf H}_{NLOS}$ | The NLoS component
$\bf x$ | The ergodic average value
${{f}_{0}}\left(\cdot\right)$ | The sum function
${{\bf f}_{1}}\left(\cdot\right)$ | The instantaneous performance function
${{\bf f}_{2}}\left(\cdot\right)$ | The constraint function
$\omega$ | The parameters of the actor neural network
$\upsilon$ | The parameters of the critic neural network
$\ell$ | The threshold at the target
${\bf{A}}\left({{\bf{H},\mbox{\boldmath{$\omega$}}}}\right)$ | The advantage function
${v}\left({{{\bf{H}},{\mbox{\boldmath{$\upsilon$}}}}}\right)$ | The value function of critic part
${\bf Q}\left({{{{\bf H}}},{{\bf{i}}}},{{\mbox{\boldmath{$\upsilon$}}}}\right)$ | The Q-value function
${\mbox{\boldmath{$\pi$}}}\left({\bf H},{\mbox{\boldmath{$\omega$}}}\right)$ | The policy function in the current episode
${\mbox{\boldmath{$\pi$}}}_{old}\left({\bf H},{\mbox{\boldmath{$\omega$}}}\right)$ | The policy function in the former episode
${L^{PG}}\left({\mbox{\boldmath{$\omega$}}}\right)$ | The loss function of policy gradient method
${L^{PPO}}\left({\mbox{\boldmath{$\omega$}}}\right)$ | The loss function of PPO method
${L^{CLIP}}\left({\mbox{\boldmath{$\omega$}}}\right)$ | The loss function of simplified PPO method
$\lambda$, $\mu$ | The Lagrange multipliers
$\tau$ | The size of updating step length
$\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ | The gradient estumated upgrading step factors
$\epsilon$ | The hyperparameter of PPO
$\varepsilon$ | The exploring rate of greedy policy of DQN

###  V-B Primal-dual DPPO Algorithm for Multi-user MIMO Scenario

Algorithm 2 Transmit beamforming and phase-shift design via primal-dual DPPO
algorithm

Input Initial beamforming $\bf{W}^{0}$, Lagrange multipliers
${\mbox{\boldmath{$\lambda$}}}^{0},{\mbox{\boldmath{$\mu$}}}^{0}$, policy
parameters ${{\mbox{\boldmath{$\omega$}}}^{0}}$ and DQN’s parameters
${{\mbox{\boldmath{$\upsilon$}}}^{0}}$. Channel states ${\bf H}_{0,k}$, ${\bf
H}_{1,k}$, and ${\bf H}_{2,k}$ are fixed in each episode.

Episode begins

For $i=1,2,...,I$ do:

Wait until a certain amount of gradients $\omega$ are available average
gradients and update global $\omega$. Wait until a certain amount of
$\upsilon$ are available average gradients and update global $\upsilon$
(workers stop collecting data when the chief is updating).

For the worker (user) $k=1,2,...,K$ do:

For received antennas $n=1,2,...,N$ do:

a) Draw samples ${{{\bf\hat{x}}_{1}}}$ and ${{{\bf\hat{x}}_{2}}}$ from a
truncated standard normal distribution;

b) Channel state ${{\bf H}_{n}^{t}}\in{{\mathbb{C}}^{M\times 1}}$, obtained
from the matrix of ${{\bf H}^{t}=}{{\bf{H}}_{0}}{\bf{+}}{{\bf{H}}_{2}}{{\bf
V}^{t}}{{\bf{H}}_{1}}$;

c) Calculate the reward ${\bf
r}^{t}_{k}={{{f}}_{0}^{t}}\left({{{\bf{x}}}}\right)$ via the formula (23);

d) Each worker $k$ repeats step d) to step j) in Algorithm 1;

End for

Worker (user) ends

End for

Episode ends

Output Transmit beamforming ${\bf{W}^{\rm{*}}}$, phase-shift design
$\bf{V^{\rm{*}}}$, and the system performances
${{f}_{0}^{\rm{*}}}\left({{{\bf{x}}}^{\rm{*}}}\right)$.

For multi-user MIMO scenario, the pseudo code of the proposed primal-dual DPPO
is concluded in Algorithm 2. As discussed in Section IV. B, each worker takes
charge of each user. They transmit respective gradients of DNNs to the global
PPO. The global PPO updates policy parameters
${{\mbox{\boldmath{$\omega$}}}^{0}}$ and DQN’s parameters
${{\mbox{\boldmath{$\upsilon$}}}^{0}}$ based on the estimated advantage
functions ${\hat{\bf A}}^{t}\left({\bf
H},{\mbox{\boldmath{$\omega$}}}\right)$. Each worker (i.e. user) utilizes the
original primal-dual PPO algorithm in Algorithm 1.

##  VI Numerical Simulation and Analysis

The simulation results in this section. We simulated in an x64 workstation
with Intel i7 CPU and the Microsoft Windows 10 Operation System. The version
of Python and TensorFlow is v3.7.9 and v2.1.0, respectively. According to the
work in [36], the carrier frequency is set as 0.55 THz and the factor of
molecular absorption loss ${k_{abs}}$ is $6.7141\times{10^{-4}}$. The paper
simulated IRS-aided ISAC system where several users are randomly distributed
in need. The targets’ azimuth angles $\theta_{k}$ are assumed in
${\text{-40}}^{\circ}$, ${\text{0}}^{\circ}$, and ${\text{40}}^{\circ}$. The
$\Delta$ in the formula (17) is set as ${\text{5}}^{\circ}$ and the ideal beam
pattern $\theta_{p}$ consists of three directions, including
$\theta_{1}=-40^{\circ}$, $\theta_{2}=0^{\circ}$, and $\theta_{3}=40^{\circ}$
[37].

The PPO’s structure is divided into two DNNs, including the actor part and the
critic part. The primal-dual PPO optimization is performed with Adam optimizer
for these two DNNs parameters update. The layer of actor part is a single
dense layer with 30 units and its activation function is ReLU. The policy
${\mbox{\boldmath{$\pi$}}}_{\omega}$ is derived from the truncated Gaussian
distribution. In the truncated Gaussian distribution, the mean is the output
of dense layer with tanh activation function and the standard deviation is the
output of dense layer with softplus activation function. The action of policy
${\bf W}_{k}\in{{\mathbb{C}}^{M\times 1}}$ is fixed on the domain
$\left[0,P_{max}\right]$. The maximum time step in one episode is set as 50.

There exists a single dense layer with 20 units in the critic part. And the
activation function of the dense layer is ReLU. Besides output of value
function, the critic part also outputs the Q-value function. It has the
function of adjusting the IRS’elements and it can generate the indexes of
phases shift. The action choice of Q-value function follows
$\varepsilon\text{-greedy}$ strategy. The parameter of
$\varepsilon\text{-greedy}$ strategy is set as $\varepsilon=0.95$. Thus, the
critic part outputs two values, including the baseline function [35] and
Q-value function. For updating the primal and dual variables, the batch size
of replay experience is set to 32. And the primal dual upgrade steps are set
as
${\gamma_{\rm{1}}}{\rm{=}}{\gamma_{\rm{2}}}{\rm{=}}{\gamma_{\rm{3}}}{\rm{=}}{\gamma_{\rm{4}}}{\rm{=0}}{\rm{.001}}$,
and the gradient upgrades are set as
${\alpha_{\rm{1}}}{\rm{=}}{\alpha_{\rm{2}}}{\rm{=}}{\alpha_{\rm{3}}}{\rm{=}}{\rm{0}}{\rm{.001}}$.

###  VI-A Multi-user MISO Scenario

In the simulation of multi-user MISO scenario, we deploy a ISAC BS equipped
with 5 antennas and 4 users with single antenna in IRS-aided ISAC system. In
this case, the proposed primal-dual PPO algorithm obtains each user’s channel
state ${\bf H}_{k}\in{{\mathbb{C}}^{M\times 1}}$ in turn within each episode.

Figure 3: The reward of primal-dual PPO algorithm with different learning
rates.

In Fig. 3, the reward of the proposed primal-dual PPO is simulated in
different learning rates. The appropriate learning rate can avoid unnecessary
training and achieve quicker optimization. As shown in Fig. 3, when the
learning rate of actor equals to
${\text{1}}\times{\text{1}}{{\text{0}}^{{\text{-3}}}}$ and the learning rate
of critic equals to ${\text{2}}\times{\text{1}}{{\text{0}}^{{\text{-3}}}}$,
the moving averaged episode reward can convergence after 1200 episodes.
However, when the order of magnitude is $-\text{2}$ in learning rate (the
learning rate of the actor part is
${\text{1}}\times{\text{1}}{{\text{0}}^{{\text{-2}}}}$ and the learning rate
of the critic part is ${\text{2}}\times{\text{1}}{{\text{0}}^{{\text{-2}}}}$),
it remains in around 90. Similarly, $-\text{5}$ order of magnitude (the
learning rate of the actor part is
${\text{1}}\times{\text{1}}{{\text{0}}^{{\text{-5}}}}$ and the learning rate
of the critic part is ${\text{2}}\times{\text{1}}{{\text{0}}^{{\text{-5}}}}$)
also keeps in around 70 due to too small learning rate. Although $-\text{4}$
order of magnitude (the learning rate of the actor part is
${\text{1}}\times{\text{1}}{{\text{0}}^{{\text{-4}}}}$ and the learning rate
of the critic part is ${\text{2}}\times{\text{1}}{{\text{0}}^{{\text{-4}}}}$)
has tendency to converge, it still requires more unnecessary episodes to
converge. The step size of learning rate is so big that the agent does not
learn the suitable action. Using so many learning parameters, it can be seen
that the learning rate of actor equals to
${\text{1}}\times{\text{1}}{{\text{0}}^{{\text{-3}}}}$ and the learning rate
of critic equals to ${\text{2}}\times{\text{1}}{{\text{0}}^{{\text{-3}}}}$,
the moving averaged episode reward can convergence after 1200 episodes. Too
high or too small learning rate is not suitable in the designed model.

Figure 4: Comparison of performance using primal-dual PPO algorithm and other
algorithms.

Fig. 4 shows the capacity of IRS-aided ISAC system based on different
algorithms. The zero force (ZF) transmit beamforming and maximum ratio
transmission (MRT) are introduced to compare with the proposed primal dual PPO
algorithm [38]. Initially, the performance function $f_{0}\left({\bf
x}\right)$ (i.e. total capacity) with the primal-dual PPO algorithm is lower
than the previous two methods slightly. Nevertheless, the obtained system
capacity converges after 200 episodes. Due to the shifting environment in each
episode, it can be seen that ZF and MRT can only obtain around 1 bps capacity
in the same training conditions. These traditional optimization occupy too
high computational cost so that is not suitable the practical IRS-aided ISAC
system. Referring to the primal-dual PPO, a primal-dual advantage actor-critic
(A2C) algorithm is designed. Although primal-dual A2C can obtain the similar
convergence, there still exists around 0.5 bps gap between two different
primal-dual learning algorithms. This is because PPO’s important sampling
ensures new policy can not deviate too far. The new improved policy can be
modified by the old policy constantly. Compared with the primal-dual A2C
algorithm, the proposed primal-dual PPO method is more efficient and robust.

Figure 5: Comparison of capacity versus transmit power under different users.

Fig. 5 studies the capacity comparison versus different transmit powers for
different users. It is clear that as the ISAC BS transmit power increases, the
performance function increases as well. It reflects that more transmit power
contributes to realizing higher capacity of IRS-aided ISAC system. Meanwhile,
the gap between different users does not change significantly. The simulation
result reveals the more receive users obtain higher capacity on the condition
that equal transmit power. When transmit power is big enough, the increasing
speed of capacity tends to slow. The reason is that the logarithmic sum
relationship between ${f_{0}}\left({{{\mathbf{x}}}}\right)$ and policy
${\mbox{\boldmath{$\pi$}}}\left({{\bf{H}},{\mbox{\boldmath{$\omega$}}}}\right)$.

Figure 6: Comparison of beam patterns using primal-dual PPO algorithm and ZF
algorithm.

Fig. 6 shows the beam pattern is optimized by the primal-dual PPO algorithm
and the ZF algorithm. The transmit beam pattern is calculated by the
covariance of transmit waveform $\bf R$. It shows beam pattern against
different angle spaces, where the angle ranges from ${-\rm{90}}^{\circ}$ to
${\rm{90}}^{\circ}$. It can be seen that the performance of detecting targets
is focused on the desired angles, including ${-\rm{40}}^{\circ}$,
${\rm{0}}^{\circ}$, and ${\rm{40}}^{\circ}$. Compared to the ZF beam pattern,
the beam pattern optimized by the primal-dual PPO algorithm has a better
detecting performance in the proposed multi-user MISO scenario.

Figure 7: Beam pattern error versus different thresholds $\varepsilon$.

Fig. 7 shows the beam pattern error $L_{r}\left({\bf R}\right)$ under
different thresholds $\ell$. The different thresholds are compared among 1, 2,
and 3. It can be seen that the lower threshold $\ell$ can obtain the lower
beam pattern error $L_{r}\left({\bf R}\right)$. When $L_{r}\left({\bf
R}\right)\leq 1$, the beam pattern error $L_{r}\left({\bf R}\right)$
fluctuates among 0.2. However, the lower threshold $\ell$, the learning time
requested is longer. This is because the higher sensing accuracy need longer
time to learning.

Figure 8: Radar-communication trade-off performance.

Fig. 8 indicates the radar-communication trade-off performance in the proposed
ISAC system. The different thresholds are increasing form 2 to 10. It is
obvious that the larger given threshold, the obtained system capacity will be
larger in the proposed algorithm. This is because the tolerance of radar
performance, MSE, relieve the pressure from sensing function. The higher
communication performance will be obtained when $\ell$ equals 10.

Figure 9: System capacity versus the number of reflective elements.

Fig. 9 reveals the system capacity versus the number of reflective elements.
We compared with two different methods, including with IRS relaying and
without IRS relaying. It can be seen that the system capacity increases with
the increasing of the elements of IRS, under the condition of IRS relaying.
Obviously, when the IRS is not aided in the ISAC system, the capacity does not
change. It suggests the function of IRS is significant in the field of
enhancing capacity.

###  VI-B Multi-user MIMO Scenario

After discussing the simulation results in multi-user MISO scenario in IRS-
aided ISAC system. The figures about multi-user MIMO scenario are shown in
this subsection. The number of the transmit antennas of the ISAC BS is set as
5 and the received antennas of each user is set as 4. The number of DPPO’s
workers equals to the number of severed users. The learning rate of the actor
part and the critic part is $1\times 10^{-3}$ and $2\times 10^{-3}$,
respectively. The primal dual upgrade steps $\gamma_{1}$, $\gamma_{2}$,
$\gamma_{3}$, $\gamma_{4}$ and gradient upgrades $\alpha_{1}$, $\alpha_{2}$,
$\alpha_{3}$ are the same as the case of multi-user MISO scenario.

Fig. 10 shows that the moving averaged episode reward with the growing of
episode. To verify the effectiveness of the proposed primal-dual DPPO
algorithm, A3C algorithm and weighted minimum mean squared error (WMMSE)
algorithm [29] are introduced in this simulation. More detailed discussion
about these two algorithms are introduced in the Appendix B. As shown in Fig.
10, the A3C algorithm also has a tendency to increase, but the speed of
learning is too slow under the same learning rate. This is the reason that the
learning rate is not suitable in A3C method, resulting in the learning effect
is slow. Meanwhile, we also introduce the WMMSE algorithm as an
unparameterized method and calculate its reward in each episode. Obviously,
the proposed primal-dual DPPO converges after 800 episodes, and achieves a
higher reward than two other algorithms’.

Figure 10: Convergence of moving averaged episode reward using different
algorithms.

Fig. 11 compares the capacity of different distributed DRL algorithms.
Obviously, the proposed primal-dual DPPO algorithm can converge after 90
episodes and it shows more stable performance. This is because the global PPO
collects local data rather than policy gradients from different workers.
Compared with the parallel optimization in the A3C algorithm, the primal-dual
DPPO algorithm can save computation time and realize quicker optimization.

Figure 11: Comparison of performance using primal-dual DPPO algorithm and A3C
algorithm.

Fig. 12 illustrates the beam patterns of 4 users in the multi-user MIMO
scenario. The number of transmit antennas is set as 5. The targets are also
focused on the desired angles, including ${-\rm{40}}^{\circ}$,
${\rm{0}}^{\circ}$, and ${\rm{40}}^{\circ}$. It can be seen that each user
realizes the desired performance of detecting targets via the proposed primal-
dual DPPO algorithm. The amplitudes of 4 users may be different. The reason is
that the different users may face with different channel states observations.
Different workers take charges of different users, it implies that the
utilization rate of process is fully scheduled.

Figure 12: Beam patterns in different users via primal-dual DPPO algorithm.

##  VII Conclusion

In this paper, we have investigated capacity maximization in the IRS-aided
ISAC system and considered the joint transmit beamforming and passive phase
optimization. The optimization problem was transformed into an ergodic form to
capture the long-term ISAC system performance. Considering the power budget
and target response, a beamforming optimization scheme combining PPO with
primal-dual was adopted in the multi-user MISO-based scenario. Furthermore, we
proposed to utilize the concept of multi-threading derived from the DPPO, to
realize the joint optimization of transmit beamforming and phase-shift design
in the multi-user MIMO scenario. Simulations results have verified the
effectiveness of the primal-dual PPO algorithm and the primal-dual DPPO
algorithm. With the aid of the actor-critic structure in DRL, the joint
optimization transmit beamforming and phase-shift design achieved higher
capacity.

##  Appendix A Proof of the Trust Region Policy Optimization

The goal of intensive learning is to obtain the best strategy that maximizes
the desired reward function.

$\eta\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)={\mathbb{E}_{{{\mathbf{H}}_{0}},{p_{0}},...,{{\mbox{\boldmath{$\pi$}}}^{t}}}}\left[{\sum\limits_{t=0}^{\infty}{{\gamma^{t}}{{\bf
A}^{t}}\left({{{\mathbf{H}}^{t}},{\mbox{\boldmath{$\omega$}}}}\right)}}\right]$
(45)

The discounted visitation frequencies
${\rho_{\mbox{\boldmath{$\pi$}}}}\left({\mathbf{H}}\right)=\sum\limits_{t=0}^{\infty}{{\gamma^{t}}P\left({{{\mathbf{H}}^{t}}={\mathbf{H}}}\right)}$
are introduced to the formula (45), it can be transformed into

$\eta\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)=\sum\limits_{\mathbf{H}}{{\rho_{{{\mbox{\boldmath{$\pi$}}}^{t+1}}}}\left({\mathbf{H}}\right)}\sum\limits_{p}{{{\mbox{\boldmath{$\pi$}}}^{t+1}}\left({p\left|{\mathbf{H}}\right.}\right)}{{\bf
A}^{t}}\left({{{\mathbf{H}}^{t}},{\mbox{\boldmath{$\omega$}}}}\right)$ (46)

However, the discounted visitation frequencies
${\rho_{{\mbox{\boldmath{$\pi$}}}}}\left({\mathbf{H}}\right)$ of new policy
${\mbox{\boldmath{$\pi$}}}^{t+1}$ in the formula (46) has to be calculated,
which requires too high computation cost. The surrogate function
${L_{\mbox{\boldmath{$\pi$}}}}\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$
is introduced to approximate the
$\eta\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$:

${L_{\mbox{\boldmath{$\pi$}}}}\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)=\sum\limits_{\mathbf{H}}{{\rho_{\mbox{\boldmath{$\pi$}}}}\left({\mathbf{H}}\right)}\sum\limits_{p}{{{\mbox{\boldmath{$\pi$}}}^{t+1}}\left({p\left|{\mathbf{H}}\right.}\right)}{{\bf
A}^{t}}\left({{{\mathbf{H}}^{t}},{\mbox{\boldmath{$\omega$}}}}\right)$ (47)

The difference between
${L_{\mbox{\boldmath{$\pi$}}}}\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$
and $\eta\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$ is different
${\rho_{\mbox{\boldmath{$\pi$}}}}\left({\mathbf{H}}\right)$. The discounted
visitation frequencies
${\rho_{\mbox{\boldmath{$\pi$}}}}\left({\mathbf{H}}\right)$ of the surrogate
function
${L_{\mbox{\boldmath{$\pi$}}}}\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$
can be obtained by the old policy ${\mbox{\boldmath{$\pi$}}}^{t}$. When
${\mbox{\boldmath{$\pi$}}}^{t}$ and ${\mbox{\boldmath{$\pi$}}}^{t+1}$ satisfy
the constraints, they can be equal.

Herein, the total variation (TV) divergence $D_{TV}$ is utilized to deign the
constraints. For two discrete probabilities $p$ and $q$, the TV divergence
${D_{TV}}\left({p\left\|q\right.}\right)=\frac{1}{2}\sum\limits_{i}{\left|{{p_{i}}-{q_{i}}}\right|}$.
It is assumed that
$D_{TV}^{\max}\left({{{\mathbf{\pi}}^{t}},{{\mathbf{\pi}}^{t+1}}}\right)=\mathop{\max}\limits_{\mathbf{H}}{D_{TV}}\left({{{{\mbox{\boldmath{$\pi$}}}}^{t}}\left({\mathbf{H}}\right)\left\|{{{{\mbox{\boldmath{$\pi$}}}}^{t+1}}\left({\mathbf{H}}\right)}\right.}\right)$,
the difference of the surrogate function
${L_{\mbox{\boldmath{$\pi$}}}}\left({{{\mbox{\boldmath{$\pi$}}}^{t+1}}}\right)$
and the cumulative reward $\eta\left({\mbox{\boldmath{$\pi$}}}\right)$ is
given by

$\eta\left({{\mbox{\boldmath{$\pi$}}}}\right)\geqslant{L_{\pi}}\left({{\mbox{\boldmath{$\pi$}}}}\right)-\frac{{4\varepsilon\gamma}}{{{{\left({1-\gamma}\right)}^{2}}}}D_{TV}^{\max}\left({{{{\mbox{\boldmath{$\pi$}}}}^{t}},{{{\mbox{\boldmath{$\pi$}}}}^{t+1}}}\right)$
(48)

The relationship between TV divergence and KL divergence satisfy
${D_{TV}}{\left({p\left\|q\right.}\right)^{2}}\leqslant{D_{KL}}\left({p\left\|q\right.}\right)$,
the constraint (48) can be reformulated by the DL divergence.

$\eta\left({{{{\mbox{\boldmath{$\pi$}}}}^{t+1}}}\right)\geqslant{L_{\pi}}\left({{{{\mbox{\boldmath{$\pi$}}}}^{t+1}}}\right)-\frac{{4\varepsilon\gamma}}{{{{\left({1-\gamma}\right)}^{2}}}}D_{KL}^{\max}\left({{{{\mbox{\boldmath{$\pi$}}}}^{t}},{{{\mbox{\boldmath{$\pi$}}}}^{t+1}}}\right)$
(49)

With the constraint of the KL divergence, the lower bound of the cumulative
reward $\eta\left({\mbox{\boldmath{$\pi$}}}\right)$ can be obtained.
Additionally, the true objective $\eta\left({\mbox{\boldmath{$\pi$}}}\right)$
is non-decreasing. The following constraint also can guarantee the
optimization of the cumulative reward
$\eta\left({\mbox{\boldmath{$\pi$}}}\right)$.

$\mathop{\max}\limits_{\theta}{L_{{{\mathbf{\pi}}_{{\theta_{0}}}}}}\left({{{\mathbf{\pi}}_{\theta}}}\right)-\frac{{4\varepsilon\gamma}}{{{{\left({1-\gamma}\right)}^{2}}}}D_{KL}^{\max}\left({{{\mathbf{\pi}}_{\theta}},{{\mathbf{\pi}}_{{\theta_{0}}}}}\right)$
(50)

In practical, the penalty coefficient
$\frac{{4\varepsilon\gamma}}{{{{\left({1-\gamma}\right)}^{2}}}}$ will cause
the step length of policy iteration small. One way to take larger steps is to
use a constraint based on the KL divergence between the new policy and the old
policy, i.e., a trust region constraint.

$\begin{array}[]{*{20}{l}}{\mathop{\max}\limits_{\mbox{\boldmath{$\omega$}}}}&{{\mathbb{E}_{{\rho_{\omega}}\left(\tau\right)}}\left[{\sum\nolimits_{t}{{\gamma^{t-1}}\frac{{{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{\mathbf{A}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}\right]},\\\
{s.t.}&{{D_{KL}}\left[{\frac{{{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}{{{{\mbox{\boldmath{$\pi$}}}_{old}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}}}\right]\leqslant\delta}.\end{array}$
(51)

##  Appendix B Introduction of A3C algorithm and WMMSE algorithm

###  B-A A3C Algorithm

The A3C algorithm is one of the methods for asynchronous DRL in a multi-
threading mode. The A3C algorithm essentially puts actor-critic algorithm into
multiple threads for synchronous training. On the one hand, it is based on the
policy-based method, so it can deal with continuous state and action spaces.
On the other hand, A3C algorithm employs asynchronous method to relieve data
correlation, in which data is not generated at the same time. Compared with
the DQN algorithm, the A3C algorithm does not need to use the experience pool
to store historical samples. This asynchronous method can save storage space
and accelerate the sampling speed of data. All the actor learners update the
policy
${\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)$
and the advantage function
${\mathbf{A}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)$
according to gradient loss. Based on the advantage function, the gradient loss
function is given by

${L^{A3C}}\left({\mbox{\boldmath{$\omega$}}}\right){\text{ =
}}\mathbb{E}\left[{{\nabla_{\mbox{\boldmath{$\omega$}}}}\log{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right){\mathbf{A}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)}\right]$
(52)

The update formulas for $\omega$ and $\upsilon$ are

$\begin{array}[]{*{20}{l}}{{{\mbox{\boldmath{$\omega$}}}^{t+1}}={{\mbox{\boldmath{$\omega$}}}^{t}}+{\nabla_{\mbox{\boldmath{$\omega$}}}}\log{\mbox{\boldmath{$\pi$}}}\left({{\mathbf{H}},{\mbox{\boldmath{$\omega$}}}}\right)\left({r-v\left({{\mathbf{H}},{{\mbox{\boldmath{$\upsilon$}}}^{t}}}\right)}\right)}\\\
{{{\mbox{\boldmath{$\upsilon$}}}^{t+1}}={{\mbox{\boldmath{$\upsilon$}}}^{t}}+\partial{{\left({r-v\left({{\mathbf{H}},{{\mbox{\boldmath{$\upsilon$}}}^{t}}}\right)}\right)}\mathord{\left/{\vphantom{{\left({r-v\left({{\mathbf{H}},{{\mbox{\boldmath{$\upsilon$}}}^{t}}}\right)}\right)}{\partial{\mbox{\boldmath{$\upsilon$}}}}}}\right.\kern-1.2pt}{\partial{\mbox{\boldmath{$\upsilon$}}}}}}\end{array}$
(53)

###  B-B WMMSE Algorithm

This algorithm transforms the weighted sum-rate maximization problem into a
higher dimensional space, using the well-known minimum mean squared error. The
sum-rate maximization is equivalent to the following weighted sum-MSE
minimization,

$\begin{array}[]{*{20}{l}}{\mathop{\min}\limits_{\varsigma,\vartheta,W}}&{\sum\limits_{k=1}^{K}{\left({{\varsigma_{k}}{e_{k}}-\log\left({{\varsigma_{k}}}\right)}\right)}}\\\
{s.t.}&{0\leqslant{W_{k}}\leqslant\sqrt{{P_{\max}}},k=1,2,...,K.}\end{array}$
(54)

where $e_{k}$ is the meansquare estimation error and $\vartheta_{k}$ equals
the ${SINR}_{k}$.

${e_{k}}={\left({{\vartheta_{k}}{{\mathbf{H}}_{k}}{{\mathbf{W}}_{k}}-1}\right)^{2}}+\sum\limits_{l\neq
k}{{{\left({{\vartheta_{k}}{{\mathbf{H}}_{l}}{{\mathbf{W}}_{l}}}\right)}^{2}}+\sigma_{c}^{2}\vartheta_{k}^{2}}$
(55)

The $\varsigma_{k}=e_{k}^{-1}$. Here by equivalent we meant that all
stationary solutions of the sum-MSE minimization is identical to maximizing
sum rate.

## References

  * [1] Y. Cui, F. Liu, X. Jing, and J. Mu, “Integrating sensing and communications for ubiquitous IoT: Applications, trends, and challenges,” _IEEE Network_ , vol. 35, no. 5, pp. 158–167, Sep. 2021.
  * [2] A. Liu et al., “A survey on fundamental limits of integrated sensing and communication,” arXiv: 2104.09954, 2021. [Online]. Available: http://arxiv.org/abs/2104.09954
  * [3] F. Liu, C. Masouros, A. P. Petropulu, H. Griffiths, and L. Hanzo, “Joint radar and communication design: Applications, state-of-the-art, and the road ahead,” _IEEE Trans. Commun._ , vol. 68, no. 6, pp. 3834–3862, Jun. 2020.
  * [4] M. T. Barros, R. Mullins, and S. Balasubramaniam, “Integrated terahertz communication with reflectors for 5G small-cell networks,” _IEEE Trans. Veh. Technol._ , vol. 66, no. 7, pp. 5647–5657, Jul. 2017.
  * [5] H. Xie, J. Xu, and Y. -F. Liu, “Max-min fairness in IRS-aided multi-cell MISO systems with joint transmit and reflective beamforming,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 2, pp. 1379–1393, Feb. 2021.
  * [6] X. Liu, T. Huang, N. Shlezinger, Y. Liu, J. Zhou, and Y. C. Eldar, “Joint transmit beamforming for multiuser MIMO communications and MIMO radar,” _IEEE Trans. Signal Process._ , vol. 68, pp. 3929–3944, 2020.
  * [7] F. Liu et al., “Integrated sensing and communications: Towards dual-functional wireless networks for 6G and beyond,” arXiv: 2108.07165, 2021. [Online]. Available: http://arxiv.org/abs/2108.07165
  * [8] S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” _IEEE Trans. Commun._ , vol. 68, no. 9, pp. 5849–5863, Sep. 2020.
  * [9] S. Zhang and R. Zhang, “Capacity characterization for intelligent reflecting surface aided MIMO communication,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1823–1838, Aug. 2020.
  * [10] Q. Wu and R. Zhang, “Joint active and passive beamforming optimization for intelligent reflecting surface assisted SWIPT under QoS constraints,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1735–1748, Aug. 2020.
  * [11] H. Shen, W. Xu, S. Gong, C. Zhao, and D. W. K. Ng, “Beamforming optimization for IRS-aided communications with transceiver hardware impairments,” _IEEE Trans. Commun._ , vol. 69, no. 2, pp. 1214–1227, Feb. 2021.
  * [12] H. Wang, C. Liu, Z. Shi, Y. Fu, and R. Song, “On power minimization for IRS-aided downlink NOMA systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 11, pp. 1808–1811, Nov. 2020.
  * [13] D. L. Dampahalage, K. B. S. Manosha, N. Rajatheva, and M. Latva-Aho, “Weighted-sum-rate maximization for an reconfigurable intelligent surface aided vehicular network,” _IEEE Open J. Commun. Soc._ , vol. 2, pp. 687–703, 2021.
  * [14] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen, “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,” _IEEE Trans. Wirel. Commun._ , vol. 18, no. 8, pp. 4157–4170, Aug. 2019.
  * [15] C. Jiang, H. Zhang, Y. Ren, Z. Han, K. Chen, and L. Hanzo, “Machine learning paradigms for next-generation wireless networks,” _IEEE Wireless Commun._ , vol. 24, no. 2, pp. 98–105, Apr. 2017.
  * [16] P. Zhou, Y. Chang, and J. A. Copeland, “Reinforcement learning for repeated power control game in cognitive radio networks,” _IEEE J. Sel. Areas Commun._ , vol. 30, no. 1, pp. 54–69, Jan. 2012.
  * [17] S. Jang, K. G. Shin, and S. Bahk, “Post-CCA and reinforcement learning based bandwidth adaptation in 802.11ac networks,” _IEEE Trans. Mob. Comput._ , vol. 17, no. 2, pp. 419–432, Feb. 2018.
  * [18] L. Huang, S. Bi, and Y.-J. A. Zhang, “Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks,” _IEEE Trans. Mob. Comput._ , vol. 19, no. 11, pp. 2581–2593, Nov. 2020.
  * [19] A. Sadeghi, F. Sheikholeslami, and G. B. Giannakis, “Optimal and scalable caching for 5G using reinforcement learning of space-time popularities,” _IEEE J. Sel. Top. Signal Process._ , vol. 12, no. 1, pp. 180–190, Feb. 2018.
  * [20] F. B. Mismar, B. L. Evans, and A. Alkhateeb, “Deep reinforcement learning for 5G networks: Joint beamforming, power control, and interference coordination,” _IEEE Trans. Commun._ , vol. 68, no. 3, pp. 1581–1592, Mar. 2020.
  * [21] K. Feng, Q. Wang, X. Li, and C. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745–749, May 2020.
  * [22] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, Aug. 2020.
  * [23] H. Yang, Z. Xiong, J. Zhao, D. Niyato, L. Xiao, and Q. Wu, “Deep reinforcement learning-based intelligent reflecting surface for secure wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 375–388, Jan. 2021.
  * [24] T. Chen, K. Zhang, G. B. Giannakis, and T. Basar, “Communication-efficient policy gradient methods for distributed reinforcement learning,” _IEEE Trans. Control Network Syst._ , May 2021.
  * [25] J. Du, F. R. Yu, G. Lu, J. Wang, J. Jiang, and X. Chu, “MEC-assisted immersive VR video streaming over terahertz wireless networks: A deep reinforcement learning approach,” _IEEE Internet Things J._ , vol. 7, no. 10, pp. 9517–9529, Oct. 2020.
  * [26] H. Zhang, N. Yang, W. Huangfu, K. Long, and V. C. M. Leung, “Power control based on deep reinforcement learning for spectrum sharing,” _IEEE Trans. Wirel. Commun._ , vol. 19, no. 6, pp. 4209–4219, Jun. 2020.
  * [27] C. T. Dinh et al., “DeepPool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning,” _IEEE Trans. Intell. Transp. Syst._ , vol. 20, no. 12, pp. 4714–4727, Dec. 2019.
  * [28] A. Ribeiro, “Ergodic stochastic optimization algorithms for wireless communication and networking,” _IEEE Trans. Signal Process._ , vol. 58, no. 12, pp. 6369–6386, Dec. 2010.
  * [29] M. Eisen, C. Zhang, L. F. O. Chamon, D. D. Lee, and A. Ribeiro, “Learning optimal resource allocations in wireless systems,” _IEEE Trans. Signal Process._ , vol. 67, no. 10, pp. 2775–2790, May 2019.
  * [30] H. Lee, S. H. Lee, and T. Q. S. Quek, “Deep learning for distributed optimization: Applications to wireless resource management,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 10, pp. 2251–2266, Oct. 2019.
  * [31] C. Lin and G. Y. Li, “Energy-efficient design of indoor mmWave and sub-THz systems with antenna arrays,” _IEEE Trans. Wirel. Commun._ , vol. 15, no. 7, pp. 4660–4672, Jul. 2016.
  * [32] W. Lu, B. Deng, Q. Fang, X. Wen, and S. Peng, “Intelligent reflecting surface-enhanced target detection in MIMO radar,” _IEEE Sens. Lett._ , vol. 5, no. 2, pp. 1–4, Feb. 2021.
  * [33] Z.-M. Jiang et al., “Intelligent reflecting surface aided dual-function radar and communication system,” _IEEE Syst. J._ , pp. 1–12, 2021.
  * [34] N. Heess et al., “Emergence of locomotion behaviours in rich environments,” arXiv: 1707.02286, 2017. [Online]. Available: https://arxiv.org/abs/1707.02286
  * [35] D. A. Basnayaka, P. J. Smith, and P. A. Martin, “Ergodic sum capacity of macrodiversity MIMO systems in flat Rayleigh fading,” _IEEE Trans. Inf. Theory_ , vol. 59, no. 9, pp. 5257–5270, Sep. 2013.
  * [36] C. Han, A. O. Bicen, and I. F. Akyildiz, “Multi-ray channel modeling and wideband characterization for wireless communications in the terahertz band,” _IEEE Trans. Wirel. Commun._ , vol. 14, no. 5, pp. 2402–2412, 2014.
  * [37] P. Stoica, Jian Li, and Yao Xie, “On probing signal design for MIMO radar,” _IEEE Trans. Signal Process._ , vol. 55, no. 8, pp. 4151–4161, Aug. 2007.
  * [38] F. Liu and C. Masouros, “A tutorial on joint radar and communication transmission for vehicular networks—part I: Background and fundamentals,” _IEEE Commun. Lett._ , vol. 25, no. 2, pp. 322–326, Feb. 2021.

|  Xiangnan Liu received the B.S. degree from the School of Computer and
Communication Engineering, University of Science and Technology of Beijing,
Beijing, China, in 2019. He is currently pursuing his Ph.D. degree at
University of Science and Technology Beijing, China. His research interests
include access control, beamforming, and resource allocation in 6G wireless
communication.
---|---
|  Haijun Zhang (M’13, SM’17) is currently a Full Professor and Associate Dean
at University of Science and Technology Beijing, China. He was a Postdoctoral
Research Fellow in Department of Electrical and Computer Engineering, the
University of British Columbia (UBC), Canada. He serves/served as Track Co-
Chair of WCNC 2020, Symposium Chair of Globecom’19, TPC Co-Chair of INFOCOM
2018 Workshop on Integrating Edge Computing, Caching, and Offloading in Next
Generation Networks, and General Co-Chair of GameNets’16. He serves as an
Editor of IEEE Transactions on Communications, IEEE Transactions on Network
Science and Engineering, and IEEE Transactions on Vehicular Technology. He
received the IEEE CSIM Technical Committee Best Journal Paper Award in 2018,
IEEE ComSoc Young Author Best Paper Award in 2017, and IEEE ComSoc Asia-
Pacific Best Young Researcher Award in 2019.
---|---
|  Keping Long (SM’06) received the M.S. and Ph.D. degrees from the University
of Electronic Science and Technology of China, Chengdu, in 1995 and 1998,
respectively. From September 1998 to August 2000, he was a Postdoctoral
Research Fellow at the National Laboratory of Switching Technology and
Telecommunication Networks, Beijing University of Posts and Telecommunications
(BUPT), China. From September 2000 to June 2001, he was an Associate Professor
at BUPT. From July 2001 to November 2002, he was a Research Fellow with the
ARC Special Research Centre for Ultra Broadband Information Networks (CUBIN),
University of Melbourne, Australia. He is currently a professor and Dean at
the School of Computer and Communication Engineering, University of Science
and Technology Beijing. He has published more than 200 papers, 20 keynote
speeches, and invited talks at international and local conferences. His
research interests are optical Internet technology, new generation network
technology, wireless information networks, value added services, and secure
technology of networks. Dr. Long has been aTPC or ISC member of COIN
2003/04/05/06/07/08/09/10, IEEE IWCN2010, ICON2004/06, APOC2004/06/08, Co-
Chair of the organization Committee for IWCMC2006, TPC Chair of COIN 2005/08,
and TPC Co-Chair of COIN 2008/10. He was awarded by the National Science Fund
for Distinguished Young Scholars of China in 2007 and selected as the Chang
Jiang Scholars Program Professor of China in 2008. He is a member of the
Editorial Committees of Sciences in China Series F and China Communications.
---|---
|  Mingyu Zhou received the Ph.D. degree from the Beijing University of Posts
and Telecommunications (BUPT) as a focus on research into key technologies of
wireless communications. After graduation in 2008, he became a Senior Engineer
in Huawei Technologies, dedicated in 3GPP standardization and patent
application. In 2014, he joined Baicells Technologies as the Research
Director. Until now, he has applied for more than 100 patents (tens of them
are PCTs), published more than 20 academic papers, and finished more than 100
standardization proposals. He was qualified in the Beijing Nova Program and
has participated in several science and technology projects, targeting to
bring more innovation to the 5G system.
---|---
|  Yonghui Li (M’04, SM’09, F’19) received his PhD degree in November 2002
from Beijing University of Aeronautics and Astronautics. Since 2003, he has
been with the Centre of Excellence in Telecommunications, the University of
Sydney, Australia. He is now a Professor and Director of Wireless Engineering
Laboratory in School of Electrical and Information Engineering, University of
Sydney. He is the recipient of the Australian Queen Elizabeth II Fellowship in
2008 and the Australian Future Fellowship in 2012. He is a Fellow of IEEE. His
current research interests are in the area of wireless communications, with a
particular focus on MIMO, millimeter wave communications, machine to machine
communications, coding techniques and cooperative communications. He holds a
number of patents granted and pending in these fields. He is now an editor for
IEEE transactions on communications, IEEE transactions on vehicular
technology. He also served as the guest editor for several IEEE journals, such
as IEEE JSAC, IEEE Communications Magazine, IEEE IoT journal, IEEE Access. He
received the best paper awards from IEEE International Conference on
Communications (ICC) 2014, IEEE PIRMC 2017 and IEEE Wireless Days Conferences
(WD) 2014.
---|---
|  H. Vincent Poor (S’72, M’77, SM’82, F’87) received the Ph.D. degree in EECS
from Princeton University in 1977. From 1977 until 1990, he was on the faculty
of the University of Illinois at Urbana-Champaign. Since 1990 he has been on
the faculty at Princeton, where he is currently the Michael Henry Strater
University Professor. During 2006 to 2016, he served as the dean of
Princeton’s School of Engineering and Applied Science. He has also held
visiting appointments at several other universities, including most recently
at Berkeley and Cambridge. His research interests are in the areas of
information theory, machine learning and network science, and their
applications in wireless networks, energy systems and related fields. Among
his publications in these areas is the forthcoming book Machine Learning and
Wireless Communications. (Cambridge University Press). Dr. Poor is a member of
the National Academy of Engineering and the National Academy of Sciences and
is a foreign member of the Chinese Academy of Sciences, the Royal Society, and
other national and international academies. He received the IEEE Alexander
Graham Bell Medal in 2017.
---|---