#  AI-based Radio Resource Management and Trajectory Design for PD-NOMA
Communication in IRS-UAV Assisted Networks

Hussein M. Hariz, Saeed Sheikhzadeh, Nader Mokari, Mohammad R. Javan, B.
Abbasi-Arand and Eduard A. Jorswieck Hussein M. Hariz, Saeed Sheikhzadeh,
Nader Mokari and B. Abbasi-Arand are with the Department of Electrical and
Computer Engineering, Tarbiat Modares University, Tehran, Iran (e-mail:
<EMAIL_ADDRESS>abbasi@modares.ac.ir). Mohammad R. Javan is with
Department of Electrical and Computer Engineering, Shahrood University of
Technology, Iran (e-mail: javan@shahroodut.ac.ir). Eduard A. Jorswieck is with
the Department of Information Theory and Communication Systems, Technische
Universität Braunschweig, Braunschweig, Germany (e-mail<EMAIL_ADDRESS>bs.de). This work was supported by the joint Iran national science foundation
(INSF) and German research foundation (DFG) under grant No. 96007867.

###### Abstract

In this paper, we consider that the unmanned aerial vehicles (UAVs) with
attached intelligent reflecting surfaces (IRSs) play the role of flying
reflectors that reflect the signal of users to the destination, and utilize
the power-domain non-orthogonal multiple access (PD-NOMA) scheme in the
uplink. We investigate the benefits of the UAV-IRS on the internet of things
(IoT) networks that improve the freshness of collected data of the IoT devices
via optimizing power, sub-carrier, and trajectory variables, as well as, the
phase shift matrix elements. We consider minimizing the average age-of-
information (AAoI) of users subject to the maximum transmit power limitations,
PD-NOMA-related restriction, and the constraints related to UAV’s movement.
The optimization problem consists of discrete and continuous variables. Hence,
we divide the resource allocation problem into two sub-problems and use two
different reinforcement learning (RL) based algorithms to solve them, namely
the double deep Q-network (DDQN) and a proximal policy optimization (PPO). Our
numerical results illustrate the performance gains that can be achieved for
IRS enabled UAV communication systems. Moreover, we compare our deep RL (DRL)
based algorithm with matching algorithm and random trajectory, showing the
combination of DDQN and PPO algorithm proposed in this paper performs 10% and
15% better than matching algorithm and random-trajectory algorithm,
respectively.
_Index Terms—_ Unmanned aerial vehicles, intelligent reflecting surface,
internet of things, age of information, trajectory design, 6G, non-orthogonal
multiple access, proximal policy optimization, double deep Q-learning.

##  I Introduction

###  I-A State of the art

Unmanned aerial vehicles (UAVs) play an important role in enhancing spectral
efficiency [1]. The UAVs can improve the telecommunication system from
different aspects, e.g., they can improve the coverage area and rate [2],
increase secure communications[3], achieve low cost and high mobility[4],
achieve long-range control in real time [3], they can also decrease outage
probability [5], or provide on-demand deployment [6]. Another major
application is to employ UAVs as mobile relays to provide reliable
connectivity for remote users in locations where there are no established
wireless networks (e.g., an empty desert) [7] and [8].

Recently, power-domain non-orthogonal multiple access (PD-NOMA) has been
considered as a promising solution to improve the communication throughput
[9]. The basic idea behind the PD-NOMA technique is that more than one user
can simultaneously use a sub-carrier. The combination of the PD-NOMA and UAVs
was considered as a solution to improve the coverage and spectral efficiency
(SE) [10]. Moreover, the intelligent reflecting surface (IRS), known as a
large intelligent surface (LIS), is an artificial radio structure which
reflects the incident radio-frequency (RF) waves into specified directions
[11] and [12]. These surfaces can be manufactured at appropriate cost and as
such they can be deployed universally providing an unprecedented opportunity
to control the wireless multi-path radio environment. The combination of UAV
and IRS is another way of improving the efficiency of UAV application [13].

In order to characterize the freshness of information, the concept of age of
information (AoI) has been introduced. The AoI is defined as the amount of
time that has elapsed since the source generated the last successfully
received update packet at the monitor [14]. In other words, the AoI is a
metric to quantify the freshness of information at the destination.

With the growth of the industrial field, the industrial internet of things
(IIoT) plays an important role in connecting the physical objects in the
industry environment with the internet [15]. We can also benefit the UAVs in
the IIoT areas [16]. The combination of UAV and IIoT seems to be essential for
the future industry, due to the highly changing environment, as well as, need
for immediate response and time-restricted data.

###  I-B Related works

In the following part, with consider the UAV as an effective technique to
improve the performance of the network, firstly, we investigate the
applications of UAV in the IoT and IIoT area. Then, we summarize the works
that existed in the UAV and AoI sectors. After that, the combination of the
IoT and AoI with the presence of the UAVs is considered. Finally, the
applications of artificial intelligence in those areas are argued.

  * •

UAV, IoT, and IIoT: The authors in [17] utilize UAVs to assist with emergency
communication in a heterogeneous IoT (Het-IoT) and considered the NOMA scheme
without the necessity of successive interference cancellation (SIC). The
authors in[18] study the three-dimensional (3D) placement and resource
allocation of multiple UAV-mounted base stations (BSs) in an uplink IoT
network and proposed a clustering algorithm to achieve high reliability, and
effectiveness.

  * •

UAV and AoI: The wireless network assisted by the AoI-aware UAVs has received
increasing attention, due to the importance of the AoI and the advantages of
the UAVs in IoT systems [19, 20, 21]. In [19], the authors proposed the UAV’s
trajectory to keep the sensed data fresh in wireless sensor networks. This
paper optimizes the minimization of the average AoI (AAoI) of various wireless
sensors, where the UAV is used to collect the ground sensor nodes (SNs) data.
The authors in [20] proposed joint optimization of the UAV trajectory and data
collection based on dynamic programming. The paper try to minimize AoI of the
SNs, independent form the UAV flight mode. The authors in [21] proposed an
efficient iterative algorithm to minimize the total AoI of the system for the
UAV’s flight trajectory and service time allocation jointly.

  * •

UAV, IoT, and AoI: The authors in [22] investigate the UAV-assisted wireless
powered IoT system and study a joint energy transfer and data collection time
allocation and a UAV’s trajectory planning problem. In [21], the authors
investigate the role of a UAV as a mobile relay to minimize the average peak
age-of-information (PAoI) for a source-destination pair and jointly optimize
the UAV’s flight trajectory as well as energy and service time allocations for
packet transmissions.

  * •

UAV, IoT, and AI: In [23], the authors proposed a cooperative UAV-enabled
mobile edge (MEC) network structure in which the UAVs are able to support
other UAVs to perform the computation tasks, while considering the
interference reduction from UAVs to devices, where the deep reinforcement
learning (DRL) based algorithms to achieve better performance than that of the
non-cooperative UAV edge computing methods. The authors in [24] study an UAV-
enabled system architecture consisting of multiple IIoTs, in which the
collected data by sensors in IIoTs can be delivered to UAVs for processing
directly and also proposed a learning-based cooperative particle swarm
optimization algorithm (LCPSO) with Markov random field (MRF) based
decomposition strategy.

  * •

UAV, AoI, and AI: The authors in [25] considered UAV-assisted single-hop
vehicular network, DRL and leverage Deep Deterministic Policy Gradient (DDPG)
to learn UAVs trajectory and scheduling policy under minimum throughput
constraints, and also the deployed UAVs adapt their velocities during the data
collection mission in order to minimize the AoI.

We summarize the main differences between our paper and other related works in
Table I.

Table I:  Related work and contribution Ref. |  Location of IRS |  Access technology |  Number of UAV |  Duty of UAV |  Objective function |  Constraints |  Machine learning based solution
---|---|---|---|---|---|---|---
[19] |  No IRS |  OMA |  One |  Collecting and transmitting. |  Minimize the maximum AoI and Minimize the AAoI. |  Trajectory |  DP and GA.
[22] |  No IRS |  OMA |  One |  Collecting and transmitting. |  Minimize the AAoI. |  Trajectory, allocate the time for EH and data collection for all SNs. |  KKT, DP, and AC.
[26] |  On UAV. |  OMA |  One |  Relaying |  Minimize the EAoI. |  UAV altitude and the IoTDs scheduling. |  DRL, MDP, and PPO.
[25] |  No IRS |  TDMA |  Multiple-UAVs. |  Collecting and processing. |  Minimize the EWSA. |  Trajectories and scheduling policies. |  DRL and DDPG.
[27] |  On fixed point. |  FDMA |  One |  Collecting and transmitting. |  Maximize the total number of served devices.  |  UAV trajectory, IoTDs scheduling and RIS phase-shift. |  MDP, PPO, and BCD.
[28] |  On fixed point. |  OMA |  One |  UAV-supported THz communication . |  Maximize the minimum average achievable rate. |  UAV’s trajectory, the phase shift of IRS, the allocation of THz band and the power control. |  No.
[29] |  No IRS |  TDMA |  One |  Collecting and transmitting. |  Minimize the maximum and AAoIs. |  Trajectory and design of SN association. |  APC, DP, and GA.
[30] |  No IRS |  OMA |  One |  Collecting and transmitting. |  Minimize the total number of expired packets and minimize the maximum AoI. |  Trajectory and expired packets. |  RL.
[31] |  No IRS |  OMA |  One |  Collecting and transmitting. |  Minimize the AoI. |  Sensing time, transmission time, UAV trajectory and task scheduling optimization problem. |  RL and DP.
[32] |  No IRS |  OMA |  Multiple-UAVs. |  Collecting and transmitting |  Minimize the AoI. |  The UAV trajectory design. |  CA2C, DDPG, and DQN.
[33] |  No IRS |  OMA |  Multiple-UAVs. |  Collecting and transmitting. |  Minimize the AAoI. |  Trajectory of the UAV in the sensing stage and transmission stage. |  DRL and DDQN.
[34] |  On fixed point |  OMA |  One |  Improve the throughput of wireless networks. |  Improve the average achievable rate of the system. |  Trajectory design and Passive beamforming. |  No
Our Paper |  On UAV |  PD-NOMA |  Multiple-UAVs. |  Relaying |  Minimizing the AAoI. |  Trajectory, reflection amplitude and phase shift, transmit power of users and sub-carrier assignment. |  Combination of PPO and DDQN methods.

###  I-C Contribution and motivation

The most related works to our paper are [26], [35], and [36]. In [26], the
authors determined the scheduling and altitude. However, the trajectory is not
considered, where we consider the sub-carrier allocation and trajectory. The
authors considered just one UAV, but we use several UAVs to improve the
converge of users. Moreover, the authors in [26] neglect the direct channel
from the BS to users and use orthogonal multiple access (OMA) techniques.
However, besides the non-direct link, we consider a direct link between users
and the receiver, as well as, the PD-NOMA technique. In the solution part, the
authors use the RL method to solve the proposed problem. Furthermore, we
propose combinatorial machine learning methods where discrete and continuous
actions are derived separately.

The authors in [35] considered the joint IRS phase shift and scheduling, but
there is no UAV and the location of IRS is fixed. In comparison, we use the
IRS attached UAVs to optimize the channels. The IRS-UAVs can fly vertically
and find better situations to improve the performance of the network. Also, in
[35], the authors proposed algorithm based on semi-definite relaxation (SDR)
to solve the problem. As mentioned before, we adopt two mixed DRLs based
methods.

In [36], the authors considered joint scheduling policy and dynamic UAV
altitude control in the UAV-assisted IoT networks without considering an IRS.
It is worth mentioning that the combination of UAV and IRS is a good method to
improve the coverage and capacity of the mobile networks that we consider in
our paper. The authors in [36] neglect the direct channel between the IoT and
BS, and also the OMA technique is adopted.

The main contributions of our paper can be summarized as follows points:

  * •

We propose a novel model in the IRS which is embedded in the UAVs. The IRS-UAV
assisted networks improve the uplink channel reliability between the IoT
devices and a central unit (CU) as a receiver and enhances the freshness of
information. Also, we propose the AAoI formulation for the IRS-UAV assisted
networks under different practical constraints.

  * •

We formulate the optimization problem to find the optimal variables such as
trajectory, the configuration of IRS (reflection amplitude and phase shift),
power allocation, and sub-carrier assignment. The objective function of our
problem is minimizing the AAoI of users over time slots subject to the maximum
transmit power, PD-NOMA-related constraints, and restrictions related to UAVs
trajectory.

  * •

We formulate the status update framework for IoT users in an aerial IRS
problem as a Markov Decision Process (MDP) and adopt the DRL based learning
method to learn environment dynamics in order to handle the trajectory of the
UAVs, power allocation, and sub-carrier assignment. The proposed problem
consists of discrete and continuous variables, for that we leverage the DDQN
for discrete actions and the PPO algorithm for continuous actions. This
solution is a highly stable state-of-the-art model-free DRL and it can find
the best policy that minimizes the AAoI.

  * •

Numerical results reveal significant improvement by comparing with other
algorithms and a benchmark method. The performance gains achieved by our
proposed designs provide insights into the practical deployment of IRS-UAV-
assisted IoT-enabled networks.

Notation: Matrices are shown by bold uppercase letter, such as $\bm{H}$ and
vectors are denoted by lowercase letters, such as $\bm{h}$.
$\mathcal{C}^{M\times N}$ and $\mathcal{C}^{M\times 1}$ represent the set of
all $M\times N$ matrices with complex elements and the set of all $M\times 1$
vectors with complex arrays, respectively. $(\cdot)^{T}$ and $(\cdot)^{H}$
show the transpose and conjugate transpose, respectively. The notation
$\text{diag}(\bm{h})$ represents a diagonal matrix with elements in vector
$\bm{h}$. $\mathbb{E}$ and $\parallel\cdot\parallel$ denote the expectation
and Euclidean norm operations, respectively. $\mathcal{CN}(\mu,\sigma^{2})$
shows the complex circular Gaussian random variable with mean $\mu$ and
variance $\sigma^{2}$. We show the abbreviations in Table I-C.

Table II: List of abbreviations in alphabetical order.

Item |  Description
---|---
AAoI |  Avarge Age of Information.
 AI |  Artificial Intelligence.
AoI |  Age of Information.
AP |  Access Point.
APC |  Affinity Propagation Clustering.
BCD |  Block Coordinate Descent.
 CA2C |  Compounded-Action Actor-Critic.
CU |  Central Unit.
DDPG |  Deep Deterministic Policy Gradient.
DDQN |  Double Deep Q-Network.
DNNs |  Deep Neural Networks.
DP |  Dynamic Programming.
DQN |  Deep Q-Network.
DRL |  Deep Reinforcement Learning.
EAoI |  Expected Sum AoI.
EH |  Energy Harvesting.
EWSA |  The Expected Weighted Sum AoI.
GA |  Genetic Algorithm.
IIoT |  Industrial Internet of Things.
IRS |  Intelligent Reflecting Surface.
IoT |  Internet of Things.
KKT |  Karush-Kuhn-Tucker.
LCPSO |  Learning Based Cooperative Particle Swarm Optimization Algorithm.
LIS |  Large Intelligent Surface.
MDP |  Markov Decision Process.
MEC |  Mobile Edge Computing.
MRF |  Markov Random Field.
NLoS |  Link is Non-Line of Sight.
NNs |  Neural Networks.
NOMA |  Non Orthogonal Multiple Access.
OMA |  Orthogonal Multiple Access.
PD-NOMA |  Power Domain Non Orthogonal Multiple Access.
PPO |  Proximal Policy Optimization.
RF |  Radio-Frequency.
RL |  Reinforcement Learning .
SDR |  Semi-Definite Relaxation.
SE |  Spectral Efficiency.
SIMO |  Single Input Multiple Output.
SINR |  Signal To Noise Plus Interference Ration.
SIC |  Successive Interference Cancellation.
UAV |  Unmanned Aerial Vehicle.
UE |  User Equipment.

###  I-D Paper Organization

This paper is arranged as follows. In Section II, the system model and problem
formulation are proposed. Section III presents the solution method of the
formulating problem. Simulation results are provided in Section IV. In the
end, concluding remarks of this paper are stated in Section V.

##  II System Model and Problem Formulation

In this section, the considered network, related parameters, and channel
models of uplink PD-NOMA IRS-based UAV communication system are discussed.
Then, the resource allocation problem is introduced.

###  II-A System Model and Parameters

Figure 1: System model consisting of $J$ UAVs equipped with $L$ IRS’s
components and help IoT devices to transmit their own data toward the CU.

We consider the uplink transmission of IRS-assisted UAVs communication system
to serve IoT user equipments (UEs) on the ground. Each UE sends the measured
data to a central unit (CU) through SIMO network.

As shown in Fig. 1, we consider $U$ single antenna of UEs are denoted by set
$\mathcal{U}=\\{1,\ldots,U\\}$, and indexed by $u\in\mathcal{U}$. The location
of UE $u$ is denoted as $\bm{\alpha}_{u}=[x_{u},y_{u},0]^{T}$. The users are
stable in their location. The total bandwidth is divided into $N$ orthogonal
sub-carriers denoted by $n\in\mathcal{N}=\\{1,\cdots,N\\}$.

It is assumed that there is a single CU as a macro base station (MBS), with
$M$ antennas indexed by $m$ and locates at the origin. In our model, all UEs
send their own signals to the CU directly and/or via reflection from UAVs. We
suppose that there are $J$ flying UAVs in the network. The set of UAVs is
indicated as $\mathcal{J}=\\{1,\dots,J\\}$ and each one is indexed by $j$.

Each UAV is equipped with IRS consisting of $L$ reflecting elements with an
equal distance $d$(mm). It is worth mentioning that these elements are active.
Hence, they can change the value of the reflecting amplitude and phase shift.
Here, the UAV-IRS plays the role of a flying reflector that reflects the
signal of UEs to the CU. Let
$\bm{\Theta}_{j}=\operatorname{diag}\left(\beta_{1,j}e^{i\theta_{1,j}},\cdots,\beta_{L,j}e^{i\theta_{L,j}}\right)$
denotes the reflection-coefficient matrix in UAV $j$, where
$\beta_{l,j}\in[0,1]$ and $\theta_{l,j}\in[0,2\pi]$ denote the reflection
amplitude and phase shift of $l$-th element, respectively [37]. Here, we
consider both of them as the optimization variables.

We consider that the time is divided into $K$ discrete time slots with equal
duration $\delta$. The location of flying UAV $j$ at time slot $k$ can be
denoted as $\bm{q}_{j}[k]=[x_{j}[k],y_{j}[k],z_{j}]^{T}$. We assume that the
location of UAVs changes with respect to the X-axis and Y-axis and is fixed
related to Z-axis. The UAV trajectory at each time slot needs to meet the
following constraints:

C1: $\displaystyle\|\bm{q}_{j}[k]-\bm{q}_{j}[k-1]\|^{2}\leq
D,\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \forall
k,\forall j\in\mathcal{J},$ (1) C2:
$\displaystyle\|\bm{q}_{j}[k]-\bm{q}_{j^{\prime}}[k]\|^{2}\geq
D_{\min},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \forall k,\forall j^{\prime},j\in\mathcal{J},$ (2) C3:
$\displaystyle\|\bm{q}_{j}[k]\|^{2}\leq r_{\max},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \forall k,\forall j\in\mathcal{J},$
(3)

where $D_{\min}$ is the minimum distance between any pair of UAVs, $r_{\max}$
is the maximum radius of coverage CU, and $D=v_{\max}\delta$ is the distance
that UAV $j$ with maximum speed $v_{\max}$ can move during each time slot. (1)
is related to the maximum distance that UAV $j$ during time slot $k$ can
horizontally move, (2) ensures UAVs never have collision, and (3) guarantees
that the UAV stays in the coverage area of CU [34].

In the following, we will discuss the broadband channel in the link between
the UEs, UAVs, and CU. Here, we have three kinds of channels: 1) channel
between the CU and UEs, 2) channel between the UEs and UAV, and 3) channel
between the UAV and CU. The channel coefficient between UE $u$ and the CU on
sub-carrier $n$ in time slot $k$ is denoted by
$\mathbb{h}_{u,n,(BS)}[k]\in\mathcal{C}^{M\times 1}$. We assume that this link
is non-line of sight (NLoS) and experiences an independently distributed
Rayleigh fading. The channel gain can be modeled as follows [34]:

$\displaystyle\mathbb{h}_{u,n,(BS)}[k]=\sqrt{\rho
d_{u}^{-\alpha}}\bm{g}_{u,n}[k],$ (4)

where $\rho$ is the path loss at the reference distance $D_{0}=1$m, $\alpha$
is the path loss exponent, and $d_{u}$ indicates the distance between the CU
and UE $u$ at slot $k$. $\bm{g}_{u,n}[k]$ is the small scale fading and is
generated with complex Gaussian distribution with zero mean and unit variance,
and the location of the CU and UEs are fixed over time slots.

The channel gain between UAV $j$ and UE $u$ over sub-carrier $n$ at time slot
$k$ is denoted by $\mathbb{h}_{u,j,n}[k]\in\mathcal{C}^{L\times 1}$. The UAVs
fly at the higher altitude of $z_{u}$ meters from the ground level, hence, we
consider that there is LoS-dominated propagation in the link between the UEs
and UAVs, as well as the UAVs and CU. As described before, each UE is in a
specific position $\bm{\alpha}_{u}$, far from the CU. Therefore, the time-
dependent distance between UE $u$ and UAV $j$ denoted by $d_{u,j}[k]$ and can
be calculated by

$\displaystyle d_{u,j}[k]={\|\bm{\alpha}_{u}-\bm{q}_{j}[k]\|}.$ (5)

The channel gain includes path loss, small-scale fading, and IRS array
response and can be given by [34]

$\displaystyle\mathbb{h}_{u,j,n}[k]=$ $\displaystyle\underbrace{\sqrt{\rho
d_{u,j}[k]^{-\alpha}}}_{\text{path loss
}}\underbrace{\left(\sqrt{\frac{\kappa_{n}}{1+\kappa_{n}}}\mathbb{h}_{u,j}^{\mathrm{LoS}}[k]+\sqrt{\frac{1}{\kappa_{n}+1}}\mathbb{h}_{u,j}^{\mathrm{NLoS}}[k]\right)}_{\mathrm{\text{aray
response \& small-scale fading}}},$ (6)

where $\kappa_{n}$ is the Rician factor. $\mathbb{h}_{u,j}^{\mathrm{NLoS}}[k]$
is NLOS term and can be complex Gaussian distribution with zero mean and unit
variance. $\mathbb{h}_{u,j}^{\mathrm{LoS}}[k]$ is the LoS term and is given by

$\displaystyle\mathbb{h}_{u,j}^{\mathrm{LoS}}[k]=\left[1,e^{-i\frac{2\pi}{\lambda}d\psi_{u,j}[k]},\ldots,e^{-i\frac{2\pi}{\lambda}(L-1)d\psi_{u,j}[k]}\right]^{T}\in\mathcal{C}^{L\times
1},$ (7)

where $d$ is the distance between antennas arrays, $\lambda$ is wavelength,
and $\psi_{u,j}[k]=\frac{x_{u}-x_{j}[k]}{d_{u,j}[k]}$ is the cosine of the
angle of departure [34].

The signals of UAVs can propagate in obstacle-free airspace. In other words,
the link between the UAVs and CU is existing, hence, the effect of reflection
or obstruction can be ignored [38]. The channel gain between the CU and IRS
$l$ of UAV $j$ at time slot $k$ is given by;

$\displaystyle g_{m,j_{l}}[k]={\sqrt{\rho
d_{\text{BS},j}[k]^{-\alpha}}}\exp\left(-i\frac{2\pi}{\lambda}(l-1)d\psi_{\text{BS},j}[k]\right),$
(8)

where $d_{\text{BS},j}[k]={\|\mathbb{q}_{j}[k]\|}$ and
$\psi_{\text{BS},j}[k]=\frac{x_{j}[k]}{d_{\text{BS},j}[k]}$. Let’s denote
$\mathbb{G}_{j}[k]\in\mathcal{C}^{M\times L},j\in\mathcal{J}$ as the channel
gain matrix between UAV $j$ and the CU with elements $[g_{m,j_{l}}[k]]$.

Now, the effective channel between UE $u$ and CU assisted by UAV $j$ over sub-
carrier $n$ denoted by $\mathbb{h}_{(u,j,n),\text{BS}}^{H}[k]$ is equal to
$\mathbb{h}_{u,j,n}^{H}[k]\bm{\Theta}_{j}[k]\mathbb{G}_{j}[k]+\mathbb{h}_{u,n,(BS)}[k]$.
Finally, the channel coefficient between each UE $u$ on sub-carrier $n$
through all the UAVs can be formulated by

$\displaystyle\mathbb{h}_{u,n}[k]=\sum_{j=1}^{J}\mathbb{h}_{u,j,n}^{H}[k]\bm{\Theta}_{j}[k]\mathbb{G}_{j}[k]+\mathbb{h}_{u,n,(BS)}[k].$
(9)

The sub-carriers are assigned to the UEs by utilizing the PD-NOMA schemes.
Generally, in the PD-NOMA technique, each sub-carrier can be assigned to more
than one UE in the service coverage area. Let us define sub-carrier assignment
variable by $\rho_{u}^{n}[k]\in\\{0,1\\}$. If sub-carrier $n$ is assigned to
UE $u$ at time slot $k$, we denote $\rho_{u}^{n}[k]=1$, otherwise, its value
equals $0$. Based on the PD-NOMA technique, each sub-carrier can be utilized
at most $Q$ times in the network. In other words, $Q$ is the cluster size of
PD-NOMA and is guaranteed by the following:

C4: $\displaystyle\sum_{u=1}^{U}\rho_{u}^{n}[k]\leq Q,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \forall
k,\forall n\in\mathcal{N}.$ (10)

The power allocation variable of user $u$ on sub-carrier $n$ at slot $k$ is
$p_{u}^{n}[k]$. Here, we consider that the transmit power related to each user
over the assigned sub-carrier has to remain in the predefined limitation.
Moreover, there is a restriction related to the maximum power that each user
can transmit. The power allocation constraints can be formulated as

C5: $\displaystyle p_{u}^{n}[k]\rho_{u}^{n}[k]\leq
P_{\text{mask}}^{n},\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \forall
n\in\mathcal{N},\forall u\in\mathcal{U},\forall k,$ (11) C6:
$\displaystyle\sum\limits_{n=1}^{N}p_{u}^{n}[k]\rho_{u}^{n}[k]\leq
P^{u}_{\max},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\forall u\in\mathcal{U},\forall k,$ (12)

where $P_{\text{mask}}^{n}$ and $P^{u}_{\max}$ are the maximum allowable
transmit power of user $u$ over sub-carrier $n$ and total transmit power of
user $u$ over all sub-carriers, respectively.

On the receiver side, the received signal is sorted based on the received
signal power at the given time slot to determine the decoding order. In this
order, the received signal is sorted from the highest value to the lowest
value. The receiver decodes and removes the UE’s signal from higher received
signal power to lower one. We suppose that the decoding order of UE $u$ on
sub-carrier $n$ is shown by ${O}_{n}^{u}(k)=m$ and it means that the CU will
decode the signal of user $u$ on sub-carrier $n$ after decoding the UEs that
place at $\\{m+1,\ldots,U\\}$ order. The received signal to noise plus
interference ratio (SINR) of UE $u$ on sub-carrier $n$ at time slot $k$ in the
CU is denoted by $\gamma_{u}^{n}[k]$ and can be formulated as

$\displaystyle\gamma_{u}^{n}[k]=\frac{\rho_{u}^{n}[k]p_{u}^{n}[k]\big{|}\mathbb{|h\|}^{2}}{\sum\limits_{O_{n}^{u^{\prime}}(k)<O_{n}^{u}(k)}\rho_{u^{\prime}}^{n}[k]p_{u^{\prime}}^{n}[k]\big{|}\mathbb{h}_{u^{\prime},n}^{H}[k]\big{|}^{2}+\sigma^{2}},$
(13)

where $O_{n}^{u^{\prime}}(k)<O_{n}^{u}(k)$ denotes the users that have higher
decoding order to user $u$ and the users in this set can successfully adopt
the SIC technique. $\sigma^{2}$ is the variance of zero-mean additive white
Gaussian noise. The decoding order can be defined as

$\displaystyle\rho_{u^{\prime}}^{n}[k]\big{|}\mathbb{h}_{u^{\prime},n}^{H}[k]\big{|}^{2}p_{u^{\prime}}^{n}[k]\leq\rho_{u^{\prime}}^{n}[k]\rho_{u}^{n}[k]\big{|}\mathbb{h}_{u,n}^{H}[k]\big{|}^{2}p_{u}^{n}[k]\
.$ (14)

By using (13), the achieved data rate of UE $u$ at time slot $t$ is formulated
as

$\displaystyle r_{u}[k]=\sum\limits_{n=1}^{N}\log(1+\gamma_{u}^{n}[k]).$ (15)

###  II-B AoI Modeling

In our network, in each time slot, the users send the measured information to
the serving gateway, which is CU in our considered network. The data of each
user is delivered to the CU via the direct link and the UAVs. Actually, the
UAVs play the role of mobile relay for a source-destination pair [21].

UE $u$ generates $d_{u}[k]$ (in bits) data at time slot $k$ according to the
Poisson distributions with parameter $\lambda$ [22]. Then, the UEs attempt to
transmit their own data as soon as the resources are assigned. Data can be
transmitted successfully by UE $u$ with given data rate $r_{u}[k]$ within the
time duration $\delta$. In this case, the transmission delay of the AoI can be
formulated as $\frac{d_{u}[k]}{r_{u}[k]}$ in which
$\frac{d_{u}[k]}{r_{u}[k]}\leq\delta$. Otherwise, the AoI is increased by one
time slot duration.

In addition, we consider that the received side can successfully decode the
received signal. Moreover, it is supposed that the signal processing time
which is needed to process the received data was neglected. If the generation
time of data is defined as $k^{\prime}\in\\{1,2,\ldots,k\\}$, the AoI
evolution of UE $u$ at time slot $k$ can be expressed as:

$\displaystyle\Delta_{u}[k]=\begin{cases}(k-k^{\prime})\delta+\delta,&\text{if}\leavevmode\nobreak\
\leavevmode\nobreak\
\delta\geq\frac{d_{u}[k^{\prime}]}{r_{u}[k]},k^{\prime}\in\mathcal{K^{\prime}},\\\
\Delta_{u}[k-1]+\delta,&\text{otherwise.}\end{cases}$ (16)

Note that we assume the differences between the propagation delay between
relay link, UE-UAV-CU, and direct link, UE-CU, are negligible [39]. Also, we
assume that when the UE’s data can not be transmitted, the data are remains
and waits until the next attempt. Moreover, it is possible that some UEs do
not have any data for transmission, hence, there is no need for resource
allocation for that UE in that time slot.

###  II-C Problem Formulation

In this paper, our objective is to minimize the AAoI of users over $K$ time
slots by jointly optimizing the transmitting power
$\bm{P}\triangleq\\{p_{u}^{n}[k],u\in\mathcal{U},n\in\mathcal{N},\forall
k\\}$, the UAVs’ trajectory
$\bm{Q}\triangleq\\{q_{j}[k],j\in\mathcal{J},\forall k\\}$, the reflection-
coefficient elements
$\bm{\Theta}\triangleq\\{\beta_{l,j}[k],\theta_{l,j}[k],l\in\mathcal{L},j\in\mathcal{J},\forall
k\\}$, and the sub-carrier assignment
$\bm{\rho}\triangleq\\{\rho_{u}^{n}[k],u\in\mathcal{U},n\in\mathcal{N},\forall
k\\}$ subject to the mobility restrictions of the UAVs, limitations on
transmit power, and constraints related to the PD-NOMA technique. The
objective function can be formulated as follows:

$\displaystyle\Psi=\max\limits_{u\in\mathcal{U}}\lim_{K\rightarrow\infty}\frac{1}{K}\sum_{k=1}^{K}\Delta_{u}[k],$
(17)

and the optimization problem can be defined as

$\displaystyle\min\limits_{\bm{P},\bm{Q},\bm{\Theta},\bm{\rho}}\;\leavevmode\nobreak\
\Psi,$ (18a) s.t. C1, C2, C3, C4, C5, C6, C7:
$\displaystyle\rho_{u^{\prime}}^{n}[k]\big{|}\mathbb{h}_{(u^{\prime},n)}^{H}[k]\big{|}^{2}p_{u^{\prime}}^{n}[k]\leq\rho_{u^{\prime}}^{n}[k]\rho_{u}^{n}[k]\big{|}\mathbb{h}_{(u,n)}^{H}[k]\big{|}^{2}p_{u}^{n}[k]$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{\leavevmode\nobreak\ \leavevmode\nobreak\
if\leavevmode\nobreak\ \leavevmode\nobreak\
}O_{n}^{u^{\prime}}(k)<O_{n}^{u}(k),\forall u^{\prime},u,n,k,$ (18b) C8:
$\displaystyle 0\leq\theta_{1,j}\leq 2\pi,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \forall j,$ (18c) C9:
$\displaystyle\beta_{l,j}\in(0,1]\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \forall j,$ (18d) C10:
$\displaystyle\rho_{u}^{n}[k]\in\\{0,1\\},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \forall n,j,$ (18e)

where (18) is SIC-related constraint. The range of phase shift and amplitude
of the reflecting elements are defined in (18c) and (18d). Finally, (18e)
defines the sub-carrier allocation variable as a distinct variable.

##  III Solution Method

Problem (18) is a non-convex optimization problem and incorporates both
integer and continuous variables which is difficult to solve. Therefore,
finding a low complex and efficient solution is complicated. To solve (18), we
propose a deep reinforcement learning (DRL) that has shown high potential in
solving a variety of complex problems in the context of UAV communications. To
apply the DRL-based solution, we transform the problem (18) in the form of
MDP, then devise a DRL-based technique to solve it.

Firstly, we explain the elements of the RL, and the MDP, in the following [40,
41]. Generally, the environment and the agent are two main parts of the RL.
The agent during numerous time slots/epochs learns environment. We suppose
that the CU is the agent that performs the actions in a central manner.

We define the states, actions, and reward function of the problem (18) as
follows: States: The state of the environment $s_{k}\in\mathcal{S}$ at the
beginning of the $k$-th time slot is defined as

$\displaystyle{s}_{k}=\\{\bm{q}[k-1],\bm{d}[k],\bm{\Delta}[k]\\},$ (19)

where $\bm{q}[k-1]\triangleq[q_{1}[k-1],\ldots,q_{J}[k-1]]^{T}$,
$\bm{d}[k]\triangleq[d_{1}[k],\ldots,d_{U}[k]]^{T}$, and
$\bm{\Delta}[k]\triangleq[\Delta_{1}[k],\ldots,\Delta_{U}[k]]^{T}$. The state
includes the current position of the UAVs, the amount of data of all users,
the value of the AAoI related to all users. Actions: Actions are the
optimization variables, i.e. the position of UAVs, the configuration of IRS
(phase shift and angle), power allocation, and sub-carrier assignment. Let
$a_{k}=\\{\mathbb{q}[k],\bm{\theta}[k],\mathbb{p}[k],\bm{\rho}[k]\\}$ be the
actions at time slot $k$. Let us define
$\bm{\theta}[k]\triangleq\\{\beta_{l,j}[k],\theta_{l,j}[k],l\in\mathcal{L},j\in\mathcal{J}\\}$,
$\mathbb{p}[k]\triangleq\\{p_{u}^{n}[k],u\in\mathcal{U},n\in\mathcal{N}\\}$,
and $\bm{\rho}[k]\triangleq\\{q_{j}[k],j\in\mathcal{J}\\}$. The action
includes both integer, i.e. $\bm{\rho}[k]$, and continuous variables, i.e.
$\bm{\theta}[k]$ and $\mathbb{p}[k]$. We note that the action space includes
all valid actions which fulfill the constraints of (18). Reward Function: In
general, the reward function is a numerical value obtained by the agent from
the environment and amount the impact of current action. Since the main
purpose is to minimize the AAoI, we consider the value of (17) as the reward.
The maximum freshness (as negative of AAoI) is defined as reward function as
follows:

$\displaystyle
R_{k}=-\max\limits_{u\in\mathcal{U}}\sum_{k=1}^{K}\Delta_{u}[k].$ (20)

The reward is calculated based on ${a}_{k}$ which is decided by observing
state $s_{k}$, then the environment transits to the next state, $s_{k+1}$. It
is worthwhile to mention that the state transition is a random process, with
transition probability denoted by $\Pr\big{(}s_{k+1}|s_{k},a_{k}\big{)}$. The
randomness in our model comes from the probability of taking actions and the
randomness related to the network, i.e, changing the traffic of users as well
as their channel gains.

###  III-A MDP and Policy Algorithm

Generally, MDP describes an environment that is perfectly observable, i.e.,
the observed state can completely characterize the features of the
environment. The main aspect of the MDP is that the action is chosen based on
the current state and the history of states has no effect on decisions [32].

The goal in the solution of the MDP is to find the optimal policy that teaches
the agent how to maximize the reward function by choosing the next action
based on the current state. Basically, the policy is a mapping between states
to the actions that are determined by a probability distribution over the
action space. Considering the action, state, and the reward function, let
$\pi(s_{k}),s_{k}\in\mathcal{S}$ denotes the policy function.

Due to the fact that the transition probability is unspecified, we adopt the
Q-learning method. Which is a subset of the RL methods. The Q-function is
defined as the value to taking action $a_{k}$ in the state $s_{k}$ under given
policy $\pi(s_{k})$ and indicates the expected cumulative reward after taking
action $a_{k}$ under state $s_{k}$ as follows:

$\displaystyle
Q^{\pi}(s_{k},a_{k})=\mathbb{E}\Big{[}\sum\limits_{i=0}^{\infty}\gamma^{i}R_{k+i}|s_{k}=s,a_{k}=a\Big{]}.$
(21)

By using the Bellman equation, which can be formulated as

$\displaystyle Q^{\pi}(s_{k},a_{k})=\mathbb{E}\big{[}R_{k}+\gamma
Q^{\pi}(s_{k+1},a_{k+1})\big{]}.$ (22)

Let’s denote $Q^{\star}(s,a)$ as the optimal $Q$-function by following
$\pi^{\star}$ where
$Q^{\star}(s_{k},a_{k})=Q^{\pi^{\star}}(s_{k},a_{k})=\max_{\pi}Q^{\pi}(s_{k},a_{k})$.

Thanks to recent achievements on the neural networks, deep neural networks
(DNNs) are introduced to mitigate the issue of large action state spaces and
can be used as function approximators to predict the optimal Q-function as
$Q^{\star}(s_{k},a_{k})\approx Q(s_{k},a_{k};\bm{w})$, where $\bm{w}$ is the
vector of the weights of the neural (deep neural) [42]. While DQN works well
on discrete action spaces, for continuous there are some reductions in the
performance [42]. Meanwhile, as mentioned before, our action space includes
both discrete and continuous. Therefore, we have challenges in finding a DRL-
based method that handles both discrete and continuous, and scales well with
the numbers of users. In the following subsection, we will discuss our devised
DRL-based algorithm.

###  III-B Proposed DRL-based Algorithm

As discussed before, our original problem incorporates both integer and
continuous high dimensional variables. Actually, the DRL methods do not have
appropriate functionality in a problem with mixed-integer and continuous
variables.

Inspired by [32], we propose the neural combinatorial method where discrete
action $a_{k}^{\text{Dis}}$ and continuous actions $a_{k}^{\text{Con}}$ are
derived separately. Actually, this concept is based on the well-known
iterative optimization which has low complexity and can deal with more
complicated problems. The rationale behind this method is dividing the main
problem into two sub-problems, then optimizing each of them with corresponding
variables.

At first, we divide the action space into two action spaces, discrete and
continuous. Then, we perform DRL-based optimization to find the corresponding
actions, applied distinct methods DRL-based for each separate actions have a
distinct method DRL-based. Thanks to the advanced results on the RL [32, 43],
we propose a double DQN (DDQN) method for our discrete action space and a
proximal policy optimization (PPO) for the continuous action space [44, 26].

$\bullet\leavevmode\nobreak\ $DDQN for Discrete Actions: In DDQN, there are
two neural networks (NNs) named as main and target networks with weights
$\bm{w}$ and $\bm{w}^{\prime}$, respectively. The function of the main network
approximates the Q-function and performs the action selection. The target
network stabilizes the performance of the learning network. At each time slot
$k$, the agent chooses action $a^{\text{Dis}}_{k}$ based on state
$s^{\text{Dis}}_{k}$ and perceives the reward $R^{\text{}}_{k}$ and new state
$s^{\text{Dis}}_{k+1}$ which represent changes taking place in the environment
after taking the action. The set
$(s^{\text{Dis}}_{k},a^{\text{Dis}}_{k},R^{\text{}}_{k},s^{\text{Dis}}_{k+1})$
which is known as an experience sample is stored in an experience memory
related to the DDQN algorithm with capacity $D_{\text{DDQN}}$. When the number
of samples reaches the adequate amount, $D_{s}$ samples (mini-batch) are
selected randomly to minimize the loss function and update the weight of main
network. The loss function is defined as [45]

$\displaystyle
L(\bm{w})^{\text{DDQN}}=\mathbb{E}\Big{[}\big{(}y^{\text{DDQN}}-Q(s^{\text{Dis}}_{k},a^{\text{Dis}}_{k};\bm{w})\big{)}^{2}\Big{]},$
(23)

where

$\displaystyle y^{\text{DDQN}}=R^{\text{}}_{k}+\gamma
Q\left(s^{\text{Dis}}_{k+1},\underset{a^{\text{Dis}}_{k+1}}{\operatorname{argmax}}Q\left(s^{\text{Dis}}_{k+1},a^{\text{Dis}}_{k+1};\bm{w}\right),\bm{w}^{\prime}\right)^{2}.$
(24)

In our case, the output of DDQN is the discrete action when the continuous
actions have been known (obtained via PPO). The discrete action selection is
done via taking the maximization operator over the estimated Q-function.

$\bullet\leavevmode\nobreak\ $PPO for Continuous Actions: Recently focus on
the RL has shown that the PPO is a powerful tool to solve continuous actions.
The PPO uses a new objective function and the implementation and tuning tasks
are more convenient than other policy gradient methods [44, 46]. These are a
few works that applied PPO in the optimization, for example, the AoI
optimization in aerial RIS via the PPO are studied in [26]. These results
motivate us to customize the PPO for our continuous actions and analyze its
performance in the context of the wireless networks. In the PPO, the learning
process includes iteration phases in which each iteration includes $K_{\max}$
time slots. In each learning iteration, the PPO agent alternates between the
sampling and optimizing phases over $K_{\max}$ time slots and performs
optimization on its own action and neural networks parameter $\bm{\varpi}$.
During each time slot, after observing the current state of the network and
choosing the appropriate action, an advantage estimate is computed to improve
the training efficiency of the policy [44].

To update the policy $\pi_{\bm{\varpi}}$, the objective function defined by
(25) (at top of next page) is computed,

$\displaystyle
L^{\text{CLIP}}(\bm{\varpi})=\mathbb{E}_{k}\left[\min\left(\frac{\pi_{\bm{\varpi}}(a^{\text{Con}}_{k}\mid
s^{\text{Con}}_{k})}{\pi_{\bm{\varpi}_{\text{old}}}(a^{\text{Con}}_{k}|s^{\text{Con}}_{k})}\mathfrak{A}_{\pi_{\bm{\varpi}_{\text{old}}}}(s^{\text{Con}}_{k},a^{\text{Con}}_{k}),\right.\right.\left.\left.\operatorname{clip}\left(\frac{\pi_{\bm{\varpi}}(a^{\text{Con}}_{k}\mid
s^{\text{Con}}_{k})}{\pi_{\bm{\varpi}_{\text{old}}}(a^{\text{Con}}_{k}|s^{\text{Con}}_{k})},1+\epsilon,1-\epsilon\right)\mathfrak{A}_{\pi_{\bm{\varpi}_{\text{old}}}}(s^{\text{Con}}_{k},a^{\text{Con}}_{k})\right)\right],$
(25)

where $\epsilon$ is the clip fraction used to control the clip range and
$\mathfrak{A}(s^{\text{Con}}_{k},a^{\text{Con}}_{k})$ is the estimated
advantage function in episode $k$. Moreover, the term
$\operatorname{clip}\left(\frac{\pi_{\bm{\varpi}}(a^{\text{Con}}_{k}\mid
s^{\text{Con}}_{k})}{\pi_{\bm{\varpi}_{\text{old}}}(a^{\text{Con}}_{k}\mid
s^{\text{Con}}_{k})},1+\epsilon,1-\epsilon\right)$ modifies the objective by
clipping the probability ratio, which limits the variation of policy into the
interval $[1−\epsilon,1+\epsilon]$. For more details about the structure and
mechanism of PPO agent, please refer to the [44].

The main steps of the solution are illustrated in Fig. (2) and Algorithm 1.

Figure 2: The schematic illustration of the proposed DRL-based solution.

1 Initialize the experience memory for DDQL and PPO with capacity
$D_{\text{DDQL}}$ and $D_{\text{PPO}}$;

2 Initialize $\bm{w}$ and $\bm{\varpi}$ randomly and set
$\bm{w}^{\prime}\leftarrow\bm{w}$ and
$\bm{\varpi}_{old}\leftarrow\bm{\varpi}$;

3 Set $D=1$ as number of experience in experience memories;

4 for  _$k=1:K_{\max}$_ do

5 Calculate current network state $s^{\text{Dis}}_{k}$ based on
$a_{k-1}^{\text{Con}}$;

6 Based on $\epsilon$-greedy select action $a^{\text{Dis}}_{k}$;

7 Observe next state $s^{\text{Dis}}_{k+1}$;

8 Set $s^{\text{Con}}_{k}=s^{\text{Dis}}_{k+1}$;

9 Sample action $a^{\text{Con}}_{k}$ via policy $\pi_{\bm{\varpi}}$;

10 Get relevant reward $R^{\text{}}_{k}$ and $s^{\text{Con}}_{k+1}$ based on
$a^{\text{Dis}}_{k}$ and $a^{\text{Con}}_{k}$;

11 Store experience
$(s^{\text{Dis}}_{k},a^{\text{Dis}}_{k},R^{\text{}}_{k},s^{\text{Dis}}_{k+1})$
and
$(s^{\text{Con}}_{k},a^{\text{Con}}_{k},R^{\text{}}_{k},s^{\text{Con}}_{k+1})$
in DDQL and PPO memories, respectively;

12 Set $D\leftarrow D+1$;

13 if  _$D >{D_{s}}$_ then

14 Sample a mini-batch from DDQL memory experiences ramdomly;

15 Calculate (23) and minimize it to update $w$;

16 Sample a mini-batch from PPO memory experiences ramdomly;

17 Compute and optimize PPO’s objective function given (25);

18 Update weight of target network $w^{\prime}$ by;

19 $w^{\prime}=\tau w+(1-\tau)w^{\prime}$ every predefined time slot;

20 Update $\bm{\varpi}_{old}$ by every predefined time slot:;

21 $\bm{\varpi}_{old}\leftarrow\tau\bm{\varpi}+(1-\tau)\bm{\varpi}$;

22

23

Algorithm 1 DDQN-PPO mixed algorithm

###  III-C Complexity and Convergence

The complexity analysis is described in this subsection. In practice, the PPO
frequently is implemented in an Actor-Critic framework. More objective
functions are supplied to the surrogate goal. The amount of multiplications in
each iteration is commonly used to calculate the complexity of DRL techniques
like the Actor-Critic framework. We suppose that a learning network consists
of $L_{\text{Hid}}$ hidden layers where the neurons of hidden layer $l$ can be
denoted by $n_{l}$. Actually, the number of operations at each iteration
determines the complexity of learning networks. The numbers of neurons are
denoted by $n_{l}$, $l=1,\dots,{L_{\text{Hid}}}$. The complexity of each
machine learning network is
$\mathcal{O}\Big{(}\sum\limits_{l=1}^{L_{\text{Hid}}-1}n_{l}n_{l+1}$). The
convergence of learning algorithms highly depends on the hyperparameters in
the training process. Hence, finding a theoretical analysis is hard. The
convergence of our learning algorithm is shown via numerical simulation
results in Fig. 3.

##  IV SIMULATION AND NUMERICAL ANALYSIS

In this section, we numerically evaluate our proposed algorithm in the studied
framework. 111The implementation of simulation is provided on: http://ieee-
dataport.org/8593  For the simulation setup, we consider a square area of
$400$ m $\times$ $400$ m, where $U=20$ users are randomly distributed with
uniform distribution. The initial location of UAVs is selected randomly inside
the allowed area. The UAVs fly at height $50$ m above the ground. Moreover, we
consider that the CU is located at point $(0,0,10\textrm{m})$ 222Actually the
CU with 10m antenna height is located at point (0,0,10). Hence, it is obvious
that we can consider that the location of the CU is $(0,0,10\textrm{ m})$..
The minimum distance between two UAVs is set to $D_{\min}=8$ m. The maximum
speed of each UAV is $10$ m/s and they only can serve within the area with a
distance of $400$ m from CU.

The number of orthogonal sub-carriers is $N=4$, and the number of antennas
$M=2$. We assume all users have the same maximum transmit power which is
configured to $20$ dBm. Other parameters are listed in Table III. Results are
obtained by using the PyTorch library, and the parameters of the learning
algorithm are listed in Table IV.

Table III: Simulation Parameters Parameter |  Value
---|---
The number of reflector elements ($L$) |  100
$d/\lambda$ |  0.5
$z_{u}^{n}$ |  $10{e}^{-}15$ dBm
$\delta$ |  100 s
$P_{\text{mask}}^{n}$ |  5 dBm
Band-width |  200 KHz
$\kappa_{n}$ |  3
Table IV: Learning parameters Parameter |  Value
---|---
Reward discount ($\gamma$) |  0.8
Layer1-size |  400
Layer2-size |  30
Episode |  4000
Number of time slot |  600
Learning rate for actor |  0.00001
Learning rate for critic |  0.0001
Soft-replacement$(\tau)$ |  0.01
Memory size of DDQN and PPO |  10000
Batch size of DDQN and PPO |  128

First, we verify the convergence behavior of the proposed algorithm shown in
Fig. 3. The negative reward is because of considering AoI in the reward
function. It can be seen that the adapted agent improves the reward as the
learning time grows, and convergence to a stable point with acceptable speed
(i.e., the learning time is not much) is achieved. Thus, our algorithm can
deal with environments with no exact knowledge of its dynamic, where in some
cases obtaining exact knowledge of the underlying dynamic is quite complex.

In Fig. 4, we illustrate an exemplary trajectory, denoted by solid lines, of
UAVs in the coverage area with the given user’s locations. It is observed that
optimizing trajectory is in the direction of making UAVs near to users that
are far from the CU, to improve the wireless channel, resulting in the
improving date rate and the AAoI reduction.

Figure 3: An illustration of the convergence behavior of the proposed
algorithm. Figure 4: An example of trajectory design of UAVs.

To compare our algorithm and considered system model, we consider the
following baselines: 1) No-RIS scheme, where we consider standalone UAV
without RIS elements [29],[26] 2) Random-trajectory, where the trajectory of
UAV is determined randomly regardless of the current state; which is similar
to random walk policy in [26], 3) Matching algorithm, where sub-carriers are
assigned to users according to the matching algorithm [47]. Moreover, we
schedule users with highest AoI, i.e., greedy policy, so that the overall size
of status data does not exceed the throughput capacity. We plot the AAoI with
respect to the number of total users in Fig. 5 for different algorithms.
First, we can observe that the AAoI grow as the number of users increase, due
to the limited wireless capacity of the considered network under given
available resources, i.e., only a limited number of users can be scheduled to
send their own status data at each time. Consequently, the waiting time to
send new updates is increased generally; hence, the AAoI is also increased. In
this figure, we observe that the proposed algorithm outperforms other baseline
algorithms, e.g., the No-RIS scheme and random trajectory. This is because, in
our scheme, we both exploit the RIS technology and optimize trajectory,
gaining an increased throughput of the network; hence, it reduces the waiting
time of update for all users at all. However, increasing throughput does not
necessarily improve the AAoI; because it is also important to optimize the
users who are scheduled, where we use a greedy-based algorithm.

Figure 5: AoI versus the number of total users.

Fig. 6 plots the sum of AAoI versus the maximum transmission power per user
for different algorithms. This figure shows the impact of the amount of
transmit power on the AAoI reduction, for some specific ranges of the power.
The reason is that by increasing the amount of transmit power, the SINR value
will be increased, hence the throughput of the network improves and the AAoI
reduces. On the other hand, increasing power leads to more interference in the
network which reduces wireless capacity. An important result from Fig. 6, when
the power budget is small, employing the RIS is important, where in many
applications such as power-limited sensor networks power is a pivotal resource
it is not allowable to use much powers. It means in such cases, we can use
some alternative advanced technologies, such as RIS to enhance network
performance from the AAoI perspective.

Figure 6: Sum of AAoI versus the maximum transmit power (dBm).

##  V Conclusion

In this paper, we studied a status update framework for IoT users in an aerial
RIS communication systems. To this end, we focused on generated updates of
deliver users randomly to the intended destination timely, with help of aerial
RIS-embedded stations, where they relay information to the destination over
wireless channels. In this regard, we formulated an optimization problem
aiming to minimize the AAoI by optimizing trajectory, sub-carrier, and power
variables under system-wide and budget constraints. As a solution for our
mixed-integer non-convex problem, we adopted a state-of-the-art learning
method based on the policy optimization called PPO-agent. Numerical analysis
was provided to investigate our algorithm and compare its performance with
other baseline algorithms. Simulations results showed the effectiveness of our
algorithm and system compared to baselines algorithms.

## References

  * [1] L. Gupta, R. Jain, and G. Vaszkun, “Survey of important issues in UAV communication networks,” _IEEE Communications Surveys Tutorials_ , vol. 18, no. 2, pp. 1123–1152, 2016.
  * [2] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage,” _IEEE Communications Letters_ , vol. 20, no. 8, pp. 1647–1650, 2016.
  * [3] G. Zhang, Q. Wu, M. Cui, and R. Zhang, “Securing UAV communications via joint trajectory and power control,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 2, pp. 1376–1389, 2019.
  * [4] S. Sotheara, K. Aso, N. Aomi, and S. Shimamoto, “Effective data gathering and energy efficient communication protocol in wireless sensor networks employing UAV,” in _Proc. IEEE Wireless Communications and Networking Conference (WCNC)_ , 2014, pp. 2342–2347.
  * [5] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Wireless communication using unmanned aerial vehicles (UAVs): Optimal transport theory for hover time optimization,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 12, pp. 8052–8066, 2017.
  * [6] Y. Zeng, R. Zhang, and T. J. Lim, “Wireless communications with unmanned aerial vehicles: opportunities and challenges,” _IEEE Communications Magazine_ , vol. 54, no. 5, pp. 36–42, 2016.
  * [7] ——, “Throughput maximization for UAV-enabled mobile relaying systems,” _IEEE Transactions on communications_ , vol. 64, no. 12, pp. 4983–4996, 2016\.
  * [8] T. A. Johansen, A. Zolich, T. Hansen, and A. J. Sørensen, “Unmanned aerial vehicle as communication relay for autonomous underwater vehicle — field tests,” in _Proc. IEEE Globecom Workshops (GC Wkshps)_ , 2014, pp. 1469–1474.
  * [9] O. Maraqa, A. S. Rajasekaran, S. Al-Ahmadi, H. Yanikomeroglu, and S. M. Sait, “A survey of rate-optimal power domain NOMA with enabling technologies of future wireless networks,” _IEEE Communications Surveys and Tutorials_ , vol. 22, no. 4, pp. 2192–2235, 2020.
  * [10] D. Hu, Q. Zhang, Q. Li, and J. Qin, “Joint position, decoding order, and power allocation optimization in UAV-based NOMA downlink communications,” _IEEE Systems Journal_ , vol. 14, no. 2, pp. 2949–2960, 2020.
  * [11] X. Tan, Z. Sun, D. Koutsonikolas, and J. M. Jornet, “Enabling indoor mobile millimeter-wave networks based on smart reflect-arrays,” in _Proc. IEEE INFOCOM 2018 - IEEE Conference on Computer Communications_ , 2018, pp. 270–278.
  * [12] F. Liu, O. Tsilipakos, A. Pitilakis, A. C. Tasolamprou, M. S. Mirmoosa, N. V. Kantartzis, D.-H. Kwon, J. Georgiou, K. Kossifos, M. A. Antoniades, M. Kafesaki, C. M. Soukoulis, and S. A. Tretyakov, “Intelligent metasurfaces with continuously tunable local surface impedance for multiple reconfigurable functions,” _Phys. Rev. Applied_ , vol. 11, p. 044024, Apr 2019. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevApplied.11.044024
  * [13] T. Shafique, H. Tabassum, and E. Hossain, “Optimization of wireless relaying with flexible UAV-borne reflecting surfaces,” _IEEE Transactions on Communications_ , vol. 69, no. 1, pp. 309–325, 2021.
  * [14] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in _Proc. IEEE INFOCOM_ , 2012, pp. 2731–2735.
  * [15] T. Qiu, X. Wang, C. Chen, M. Atiquzzaman, and L. Liu, “TMED: A spider-web-like transmission mechanism for emergency data in vehicular Ad Hoc networks,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 9, pp. 8682–8694, 2018.
  * [16] S. Khisa and S. Moh, “Priority-aware fast MAC protocol for UAV-assisted industrial IoT systems,” _IEEE Access_ , vol. 9, pp. 57 089–57 106, 2021\.
  * [17] M. Liu, J. Yang, and G. Gui, “DSF-NOMA: UAV-assisted emergency communication technology in a heterogeneous internet of things,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 5508–5519, 2019.
  * [18] Y. Liu, K. Liu, J. Han, L. Zhu, Z. Xiao, and X.-G. Xia, “Resource allocation and 3-D placement for UAV-enabled energy-efficient IoT communications,” _IEEE Internet of Things Journal_ , vol. 8, no. 3, pp. 1322–1333, 2021.
  * [19] J. Liu, X. Wang, B. Bai, and H. Dai, “Age-optimal trajectory planning for UAV-assisted data collection,” in _Proc. IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)_ , 2018, pp. 553–558.
  * [20] Z. Jia, X. Qin, Z. Wang, and B. Liu, “Age-based path planning and data acquisition in UAV-assisted IoT networks,” in _Proc. IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2019, pp. 1–6.
  * [21] M. A. Abd-Elmagid and H. S. Dhillon, “Average peak age-of-information minimization in UAV-assisted IoT networks,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 2, pp. 2003–2008, 2019.
  * [22] H. Hu, K. Xiong, G. Qu, Q. Ni, P. Fan, and K. B. Letaief, “AoI-minimal trajectory planning and data collection in UAV-assisted wireless powered IoT networks,” _IEEE Internet of Things Journal_ , vol. 8, no. 2, pp. 1211–1223, 2021.
  * [23] Y. Liu, S. Xie, and Y. Zhang, “Cooperative offloading and resource management for UAV-enabled mobile edge computing in power IoT system,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 10, pp. 12 229–12 239, 2020\.
  * [24] L. Sun, L. Wan, and X. Wang, “Learning-based resource allocation strategy for industrial IoT in UAV-enabled MEC systems,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 7, pp. 5031–5040, 2021.
  * [25] M. Samir, C. Assi, S. Sharafeddine, D. Ebrahimi, and A. Ghrayeb, “Age of information aware trajectory planning of UAVs in intelligent transportation systems: A deep learning approach,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 11, pp. 12 382–12 395, 2020.
  * [26] M. Samir, M. Elhattab, C. Assi, S. Sharafeddine, and A. Ghrayeb, “Optimizing age of information through aerial reconfigurable intelligent surfaces: A deep reinforcement learning approach,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 4, pp. 3978–3983, 2021.
  * [27] A. Al-Hilo, M. Samir, M. Elhattab, C. Assi, and S. Sharafeddine, “Ris-assisted UAV for timely data collection in iot networks,” _arXiv preprint arXiv:2103.17162_ , 2021.
  * [28] Y. Pan, K. Wang, C. Pan, H. Zhu, and J. Wang, “UAV-assisted and intelligent reflecting surfaces-supported terahertz communications,” _IEEE Wireless Communications Letters_ , vol. 10, no. 6, pp. 1256–1260, 2021.
  * [29] J. Liu, P. Tong, X. Wang, B. Bai, and H. Dai, “UAV-aided data collection for information freshness in wireless sensor networks,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 4, pp. 2368–2382, 2021.
  * [30] W. Li, L. Wang, and A. Fei, “Minimizing packet expiration loss with path planning in UAV-assisted data sensing,” _IEEE Wireless Communications Letters_ , vol. 8, no. 6, pp. 1520–1523, 2019.
  * [31] S. Zhang, H. Zhang, Z. Han, H. V. Poor, and L. Song, “Age of information in a cellular internet of UAVs: Sensing and communication trade-off design,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 10, pp. 6578–6592, 2020.
  * [32] J. Hu, H. Zhang, L. Song, R. Schober, and H. V. Poor, “Cooperative internet of UAVs: Distributed trajectory design by multi-agent deep reinforcement learning,” _IEEE Transactions on Communications_ , vol. 68, no. 11, pp. 6807–6821, 2020.
  * [33] F. Wu, H. Zhang, J. Wu, Z. Han, H. V. Poor, and L. Song, “UAV-to-device underlay communications: Age of information minimization by multi-agent deep reinforcement learning,” _IEEE Transactions on Communications_ , vol. 69, no. 7, pp. 4461–4475, 2021.
  * [34] S. Li, B. Duo, X. Yuan, Y.-C. Liang, and M. Di Renzo, “Reconfigurable intelligent surface assisted UAV communication: Joint trajectory design and passive beamforming,” _IEEE Wireless Communications Letters_ , vol. 9, no. 5, pp. 716–720, 2020.
  * [35] A. Muhammad, M. Elhattab, M. Shokry, and C. Assi, “Age of information optimization in a ris-assisted wireless network,” _arXiv preprint arXiv:2103.06405_ , 2021.
  * [36] M. Samir, C. Assi, S. Sharafeddine, and A. Ghrayeb, “Online altitude control and scheduling policy for minimizing aoi in UAV-assisted iot wireless networks,” _IEEE Transactions on Mobile Computing_ , pp. 1–1, 2020.
  * [37] Q. Wu and R. Zhang, “Weighted sum power maximization for intelligent reflecting surface aided SWIPT,” _IEEE Wireless Communications Letters_ , vol. 9, no. 5, pp. 586–590, 2020.
  * [38] Z. Wei, Y. Cai, Z. Sun, D. W. K. Ng, J. Yuan, M. Zhou, and L. Sun, “Sum-rate maximization for IRS-assisted UAV OFDMA communication systems,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 4, pp. 2530–2550, 2021.
  * [39] C.-F. Liu and M. Bennis, “Taming the tail of maximal information age in wireless industrial networks,” _IEEE Communications Letters_ , vol. 23, no. 12, pp. 2442–2446, 2019.
  * [40] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT press, 2018.
  * [41] B. Khamidehi and E. S. Sousa, “Reinforcement learning-based trajectory design for the aerial base stations,” in _Proc. IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2019, pp. 1–6.
  * [42] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [43] X. Wang and M. Cenk Gursoy, “Multi-agent double deep Q-Learning for beamforming in mmWave MIMO networks,” in _Proc. IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications_ , 2020, pp. 1–6.
  * [44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [45] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proc. AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.
  * [46] Y. Sun, X. Yuan, W. Liu, and C. Sun, “Model-based reinforcement learning via proximal policy optimization,” in _Proc. Chinese Automation Congress (CAC)_ , 2019, pp. 4736–4740.
  * [47] A. Zakeri, M. Moltafet, and N. Mokari, “Joint radio resource allocation and SIC ordering in NOMA-based networks using submodularity and matching theory,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 10, pp. 9761–9773, 2019.