# Fairness Based Energy-Efficient 3D Path Planning of a Portable Access Point:
A Deep Reinforcement Learning Approach

Nithin Babu, Igor Donevski,
Alvaro Valcarce, , Petar Popovski, ,
Jimmy Jessen Nielsen, , and Constantinos B. Papadias ©2022 IEEE. Personal use
of this material is permitted. Permission from IEEE must be obtained for all
other uses, in any current or future media, including reprinting/republishing
this material for advertising or promotional purposes, creating new collective
works, for resale or redistribution to servers or lists, or reuse of any
copyrighted component of this work in other works. This version of the work
has been submitted in the IEEE OJCOMS. This work is supported by the project
PAINLESS which has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 812991.N. Babu and
C. B. Papadias are with Research, Technology and Innovation Network (RTIN),
Alba, The American College of Greece, Greece (e-mail<EMAIL_ADDRESS>cpapdias@acg.edu).N. Babu, I. Donevski, P. Popovski, J. J. Nielsen, and C. B.
Papadias, and with Department of Electronic Systems, Aalborg University,
Denmark (e-mail: {niba, igordonevski, petarp, jjn, copa,}@es.aau.dk).A.
Valcarce is with Nokia Bell-Labs France (e-mail: alvaro.valcarce_rial@
nokia$-$bell$-$labs.com).

###### Abstract

In this work, we optimize the 3D trajectory of an unmanned aerial vehicle
(UAV)-based portable access point (PAP) that provides wireless services to a
set of ground nodes (GNs). Moreover, as per the Peukert effect, we consider
pragmatic non-linear battery discharge for UAV’s battery. Thus, we formulate
the problem in a novel manner that represents the maximization of a fairness-
based energy efficiency metric and is named fair energy efficiency (FEE). The
FEE metric defines a system that lays importance on both the per-user service
fairness and the PAP’s energy efficiency. The formulated problem takes the
form of a non-convex problem with non-tractable constraints. To obtain a
solution we represent the problem as a Markov Decision Process (MDP) with
continuous state and action spaces. Considering the complexity of the solution
space, we use the twin delayed deep deterministic policy gradient (TD3) actor-
critic deep reinforcement learning (DRL) framework to learn a policy that
maximizes the FEE of the system. We perform two types of RL training to
exhibit the effectiveness of our approach: the first (offline) approach keeps
the positions of the GNs the same throughout the training phase; the second
approach generalizes the learned policy to any arrangement of GNs by changing
the positions of GNs after each training episode. Numerical evaluations show
that neglecting the Peukert effect overestimates the air-time of the PAP and
can be addressed by optimally selecting the PAP’s flying speed. Moreover, the
user fairness, energy efficiency, and hence the FEE value of the system can be
improved by efficiently moving the PAP above the GNs. As such, we notice
massive FEE improvements over baseline scenarios of up to 88.31%, 272.34%, and
318.13% for suburban, urban, and dense urban environments, respectively.

###### Index Terms:

UAV communication, Energy-Efficiency, TD3, 3D Trajectory Optimization,
Reinforcement Learning, User Fairness

##  I Introduction

To provide seamless network connectivity, it is expected that future radio
access networks implement much denser deployments of small cells, that imply
very high deployment costs. A more cost-efficient solution for serving a set
of ground nodes (GNs) is to use an unmanned aerial vehicle (UAV) that carries
a radio access node, hereafter referred to as a portable access point (PAP)
[1]. The third generation partnership project (3GPP) item [2] proposes the
architecture and Quality-of-Service (QoS) requirements for such a system. The
ability to have a controllable maneuver and the presence of line-of-sight
(LoS) dominant air to ground channels [3] make it appropriate for applications
such as data collection from wireless sensor networks (WSNs), enhancing the
cellular coverage, remote sensing, emergency deployments, and so on [1]. The
main drawback of the PAP system is its limited air-time which is a function of
the capacity of the onboard battery unit and its power consumption profile.
The air-time of a PAP is defined as the duration it remains aloft. The power
consumed by a PAP varies with its mode of flying; for instance, a PAP consumes
the maximum amount of power when it climbs vertically up, whereas the power
consumption can be the least during a horizontal flight at an certain non-zero
velocity [4]. Hence, the air-time of a PAP can be increased by suitably
selecting its flying mode and velocity. Moreover, the available capacity of a
PAP battery unit is a non-linear function of the power-draw profile of the PAP
[5]. Additionally, the air to ground channel LoS probability and the path loss
between a PAP and a GN are proportional functions of the elevation angle and
3D distance between them, respectively [3]. Consequently, the trajectory of a
PAP can be used as a tool to increase its air-time and improve the channel to
a GN. Hence, in this work, we design a 3D trajectory for a PAP that maximizes
the number of bits transmitted per Joule of energy consumed while guaranteeing
a fair service to the GNs measured in terms of fair energy efficiency (FEE) of
the system.

Figure 1: Trajectory determination scenario.

###  I-A Related Works

The works in [3]-[6] consider UAV placement optimization and trajectory design
problems with main objectives as maximizing coverage area, throughput, air-
time, energy efficiency, and minimizing mission time, power consumption,
e.t.c. The authors of [3]-[7] consider the 3D placement of UAV(s) to maximize
the coverage area. In [3], the authors propose a probabilistic LoS-non-LoS
(NLoS) air to ground channel model and use it to find the optimal hovering
altitude of a stationary UAV that maximizes the coverage area. [8] and [9]
find the optimal altitude that maximizes the coverage area of a multi- and
single-UAV system using circle packing theory. [10] uses a combination of
exhaustive search and maximal weighted area algorithm to propose an optimal
UAV placement method that maximizes the number of users covered, whereas [7]
considers the placement optimization of a dynamic standalone drone equipped
with a steerable antenna. The work in [11] proposes a power-efficient
deployment of multiple UAVs which are used as aerial base stations to collect
data from ground Internet of Things (IoT) devices, whereas [12] and [13]
consider minimizing the total transmit power of a drone base station by
considering a downlink communication scenario.

The authors of [14]-[15] consider the average throughput of a UAV-based aerial
communication system as the performance metric. [14] and [16] maximize the
minimum average throughput by considering an uplink communication between a
set of GNs and a UAV, whereas [17], [18], [15], and [19] consider a downlink
communication scenario. In [20]-[21], the authors consider the aerial
vehicle’s energy consumption while proposing an energy-efficient UAV(s)
deployment policy. In [20], the authors propose a tractable power consumption
model for a single-rotor rotary-wing UAV and use it to design a 2D trajectory
that consumes the least amount of energy. In [21], we extend the model to a
multi-rotor UAV and propose a 2D trajectory for a PAP that maximizes the
number of bits transmitted per Joule of energy consumed while following a fly-
hover-communicate protocol to serve the users. The algorithm given in [22]
designs an energy-efficient 2D trajectory for a fixed-wing UAV, while in [23],
we determine a set of energy-efficient hovering points using circle packing
theory. The works [24]-[25] use a deep reinforcement learning (DRL) technique
to design a UAV(s) placement policy that guarantees fair service to the users.
[24] uses the UAV trajectory as a tool to achieve fairness in terms of
learning staleness, which reflects the learning time discrepancy among the
users. The proposed policies of [26] and [25] achieve fairness in terms of
coverage and throughput, respectively. Comprehensive lists of works that
consider placement optimization of a UAV-based system are available in [27]
and [6].

###  I-B Main Contributions and Paper Organization

The works in [3]-[18] propose UAV trajectory design algorithms that either
maximize communication-related parameters such as the coverage area and sum or
average throughput or minimize the transmit power. The works mainly design a
2D trajectory or represent the 3D optimal UAV(s) positioning problems as two
subproblems that optimize the vertical and horizontal positioning of the
UAV(s) recurrently. Even though the problem formulations to maximize the
energy-efficiency in [20]-[21] consider the aerial vehicle’s power
consumption, the solutions are again 2D flight trajectories. Please note that
for an energy-limited system such as a PAP, maximizing the number of bits
transmitted per Joule of energy consumed while ensuring user fairness is
paramount. Maximizing the throughput for a given energy budget is different
from maximizing the energy efficiency since each movement of the UAV should
maximize the throughput and minimizes the energy consumption simultaneously.
Moreover, a UAV consumes different power during its axial climb and forward
flight modes. Neglecting this, as in [15], [19], and [26], falsely
overestimates the air-time of a UAV resulting in the initiation of the early-
landing procedure before completing the planned trajectory. Furthermore, [15]
and [26] propose trajectory planning and resource allocation schemes for high-
mobility users in which the trajectory parameters and the resources are
allocated to guarantee high instantaneous throughput fairness between all
users. Even though the proposed fairness metric is ideal for analyzing the
performance of the considered scenario, it might be sub-optimal for an IoT
application such as data collection from an IoT network. For such
applications, long-term fairness metrics are more suitable. For instance,
consider a scenario in which the PAP is deployed to deliver a file of a given
size to all the users by the end of the trajectory. In this case, the service
fairness could be measured at the end of the trajectory; if all the users are
delivered with an equal amount of bits on an average by the end of the
trajectory, the fairness between the users will be high.

Suppose the PAP flies near to a user in a given time instant. In that case, it
is more efficient to allocate more resources to that user since the
communication channel to the user, as well as the throughput, will improve.
However, to guarantee a high long-term user fairness, the later segments of
the trajectory should be closer to the remaining users. Finally, none of the
above works consider the Peukert effect seen in Li-ion batteries that are
typically used in UAVs. Neglecting the Peukert effect overestimates the air-
time of the PAP, resulting in initiating the early-landing procedure before
completing the planned trajectory. In practise, the PAP will be flying at
different velocities resulting in different power consumption; hence the
remaining air-time of the PAP varies after each action as a non-linear
function of the power consumption. This affects the system’s energy efficiency
since the number of trajectory segments varies as a non-linear function of the
power consumption profile. In essence, the 3D trajectory design of a PAP that
maximizes the fairness-based energy efficiency while factoring in the UAV
power consumption and the Peukert effect has, to the best of our knowledge,
not yet been considered in the literature. The main contributions of this work
are summarized as follows:

  * •

We propose a method to model the non-linear Peukert effect of the PAP battery
using the data points from a data sheet;

  * •

We additionally propose an algorithm to estimate the air-time of a PAP by
considering the Peukert effect and the PAP power consumption profile;

  * •

We introduce a user fairness-based energy efficiency metric called the fair
energy efficiency that considers user fairness, sum throughput, and the PAP
propulsion power consumption;

  * •

Finally, we implement a twin delayed deep deterministic policy gradient (TD3)-
based 3D path planning algorithm to design a 3D trajectory for the PAP that
maximizes the FEE value of the system.

This paper is structured as follows: Section II explains the system setup,
propagation environment, the 3D power consumption model of the PAP, and the
FEE metric. In the section, we also detail the Peukert effect of the PAP
battery and propose an algorithm to estimate the air-time of the PAP. Section
III includes the problem formulation to maximize the FEE of the system and the
solving methodology. Section IV presents the main findings through numerical
evaluations, and elaborates the significance of the results. Finally, Section
V summarizes the main findings of this work. All the quantities are in SI
units unless otherwise specified.

##  II System Model

In this work, we consider a PAP deployed to serve a set of $N$ GNs. Each GN
$n\in\mathcal{N}=\\{1,2,3,..,N\\}$ is located at
$\mathbf{g}_{\mathrm{n}}=[\mathbf{g}_{\mathrm{h},{n}},0]$ of Cartesian space
$(x,y,z)$ with
$\mathbf{g}_{\mathrm{h},{n}}=[g_{\mathrm{x},{n}},g_{\mathrm{y},{n}}]$, as
shown in Fig. 1. The PAP flies along a 3D path to serve the set of GNs. Both
the PAP and the GNs are assumed to be equipped with omni-directional antennas.

###  II-A PAP Trajectory Model

The optimal flying path of the PAP is obtained by dividing the total air time
$T$ into $M$ time segments of length $\delta_{t}$ each such that
$T=M\delta_{t}$ [20]. The value of $\delta_{t}$ is chosen so that within each
segment the PAP can be assumed to fly with a constant velocity, and the change
in path loss values between the PAP and each GN is insignificant, i.e.,
$\delta_{t}v_{\text{max}}\leq\Delta$ where $v_{\text{max}}$ is the maximum
speed of the UAV and $\Delta$ is the maximum change in distance below which
the path loss values between the PAP and each GN remain stationary.
Consequently, the path of the PAP can be represented using $M+1$ points, whose
locations are denoted as
$\mathbf{u}_{m}=[\mathbf{u}_{\mathrm{h},m},u_{\mathrm{z},{m}}]$,
$m\in\mathcal{M}=\\{1,2,3,..,M+1\\}$ where
$\mathbf{u}_{\mathrm{h},m}=[u_{\mathrm{x},{m}},u_{\mathrm{y},{m}}]$ is the
projection of the PAP location on the horizontal plane. The length of each
segment and the maximum PAP velocity are constrained as,

$\displaystyle\|\mathbf{u}_{m+1}-\mathbf{u}_{m}\|$ $\displaystyle=$
$\displaystyle\delta_{t}v_{m}\leq\delta_{t}v_{\text{max}}\leq\Delta\,\,\forall
m\in\mathcal{M}^{{}^{\prime}},$ (1)

where $\mathcal{M}^{{}^{\prime}}=\\{1,2,..M\\}$. In a particular segment, the
PAP follows a time-division multiple access (TDMA) scheme to serve the GNs:
let $T_{m,n}$ be the time allocated to the $n^{\text{th}}$ GN while the PAP is
flying in the $m^{\text{th}}$ path segment with a speed of $v_{m}$ m/s, then,

$\displaystyle\Sigma_{n=1}^{N}T_{m,n}=\delta_{t}\quad\forall
m\in\mathcal{M}^{{}^{\prime}}.$ (2)

###  II-B Propagation Environment

The communication channel between the PAP and a GN at a given time can be
either LoS or NLoS depending on the relative position of the GN with respect
to the PAP and the blockage profile of the environment. The LoS and NLoS path
loss values can be expressed as [3]-[11],

$\displaystyle L^{\text{los}}_{m,n}$ $\displaystyle=$ $\displaystyle
20\text{log}d^{\text{3D}}_{m,n}+20\text{log}f_{c}+20\text{log}\left(\dfrac{4\pi}{c}\right)+\eta^{\text{los}},$
(3) $\displaystyle L^{\text{nlos}}_{m,n}$ $\displaystyle=$ $\displaystyle
20\text{log}d^{\text{3D}}_{m,n}+20\text{log}f_{c}+20\text{log}\left(\dfrac{4\pi}{c}\right)+\eta^{\text{nlos}},$
(4)

with
$d^{\text{3D}}_{m,n}=\sqrt{{d^{\text{2D}}_{m,n}}^{2}+u^{2}_{\mathrm{z},m}}$
and
$d^{\text{2D}}_{m,n}=\|\mathbf{u}_{\mathrm{h},m}-\mathbf{g}_{\mathrm{h},n}\|$.
$f_{c}$ and $c$ are the carrier frequency and the velocity of light,
respectively. The corresponding probability of existence of a LoS link between
the PAP and the $n^{\text{th}}$ GN while the PAP is in the $m^{\text{th}}$
path segment can be expressed as,

$\displaystyle P^{\text{los}}_{m,n}$ $\displaystyle=$
$\displaystyle\dfrac{1}{1+a\exp{\left[-b(\beta_{m,n}-a)\right]}},$ (5)

with
$\beta_{m,n}=\text{arctand}\left(\dfrac{u_{\mathrm{z},m}}{d^{\text{2D}}_{m,n}}\right)$;
$a$ and $b$ are the environment dependent parameters; $\eta^{\text{los}}$ and
$\eta^{\text{nlos}}$ are the mean values of the respective additional path
loss values due to long-term channel variations. For a given elevation angle,
this additional path loss has a Gaussian distribution [28], and we use its
mean value in this work[3]-[9]. The mean value depends on the building profile
of the region and it is noticed that the change in the additional path loss
within a particular propagation group (LoS/NLoS) is insignificant compared to
the change in path loss value from one group to the other [28],[3]. This
allows us to model the path loss with a constant gap between the two
propagation groups as given in (3) and (4). Hence, the expected spectral
efficiency to the $n^{\text{th}}$ GN is given by,

$\displaystyle\overline{R}_{m,n}$ $\displaystyle=$ $\displaystyle
P^{\text{los}}_{m,n}{R}_{m,n}^{\text{los}}+(1-P^{\text{los}}_{m,n}){R}_{m,n}^{\text{nlos}},$
(6)

where
${R}_{m,n}^{x}=\text{log}_{2}\left(1+\dfrac{P_{t}}{\sigma^{2}10^{L^{x}_{m,n}/10}}\right)\,\,\forall
x\in\\{\text{los},\text{nlos}\\}$; $P_{t}$ and $\sigma^{2}$ are the respective
transmitted signal and noise power values.

###  II-C UAV power consumption model

TABLE I: UAV’s physical properties [21]. Label | Definition | Value
---|---|---
$W$ | Weight of the UAV in Newton | 24.5 N
$N_{\text{R}}$ | Number of rotors | 4
$v_{m}$ | UAV’s horizontal flying velocity | -
$v_{\text{tip}}$ | Tip speed of the rotor | 102 m/s
$A_{\text{f}}$ | Fuselage area |  0.038 $\text{m}^{2}$
$\rho(u_{\mathrm{z},m})$ |  Air density at $u_{\mathrm{z},m}$ | -
$C_{D}$ | Drag Co-efficient | 0.9
$A_{r}$ | Rotor disc area |  0.06 $\text{m}^{2}$
$\Delta_{p}$ | Profile drag coefficient | 0.002
$s$ | Rotor solidity | 0.05

h

Figure 2: Velocity Vector.

In this section, we provide the general expressions to calculate the total
power consumed by the PAP during a considered time slot. The definitions and
values111https://dl.djicdn.com/downloads/m100/M100_User_Manual_EN.pdf of all
the variables used in this section are given in Table I. In the
$m^{\text{th}}$ time slot, the PAP moves from $\mathbf{u}_{m}$ to
$\mathbf{u}_{m+1}$ in $\delta_{t}$ seconds. Then, as shown in Fig. 2, the PAP
velocity vector in the spherical coordinates system can be represented as
$\mathbf{v}_{m}=(v_{m},\alpha_{m},\epsilon_{m})$, in which
$v_{m}=\|\mathbf{u}_{m+1}-\mathbf{u}_{m}\|/\delta_{t}$ is the speed of the PAP
at which it travels from $\mathbf{u}_{m}$ to $\mathbf{u}_{m+1}$,
$\alpha_{m}=\text{arctan}\left[({u}_{\mathrm{y},m+1}-{u}_{\mathrm{y},m})/({u}_{\mathrm{x},m+1}-{u}_{\mathrm{x},m})\right]$
and
$\epsilon_{m}=\text{arctan}\left[\|\mathbf{u}_{\mathrm{h},m+1}-\mathbf{u}_{\mathrm{h},m}\|/({u}_{\mathrm{z},m+1}-{u}_{\mathrm{z},m})\right]$
are the azimuth and elevation angles of $\mathbf{u}_{m+1}$ with respect to the
axes located at $\mathbf{u}_{m}$. In each time slot, the PAP is in one of the
following flight conditions:

####  II-C1 Forward flight ($v_{m}\neq 0,$ $\epsilon_{m}\neq 0$)

The forward flight condition contains the following PAP flying modes: 1) the
PAP moves along a plane that is parallel to the horizontal plane ($v_{m}\neq
0,$ $\epsilon_{m}=90^{\circ}$) commonly called as level forward flight; 2)
forward (inclined) ascent or descent mode in which the PAP moves in the 3D
space thereby changing all the 3 coordinates of its position ($v_{m}\neq 0,$
$\epsilon_{m}\not\in\\{90^{\circ},0^{\circ}$}). The amount of power required
to maintain this flight condition can be determined using [21], [4],

$\displaystyle
P^{\text{fwd}}_{\text{uav}}(\mathbf{v}_{m})=\underbrace{N_{\mathrm{R}}P_{\mathrm{b}}\left(1+\dfrac{3v^{2}_{m}}{v_{\text{tip}}^{2}}\right)}_{P_{\text{blade}}}+\underbrace{\dfrac{1}{2}C_{D}A_{\text{f}}\rho(u_{\mathrm{z},m})v^{3}_{m}}_{P_{\text{fuselage}}}$
$\displaystyle+\underbrace{W\left[\sqrt{\left(\sqrt{\dfrac{W^{2}}{4N_{\mathrm{R}}^{2}\rho^{2}(u_{\mathrm{z},m})A_{\mathrm{r}}^{2}}+\dfrac{v_{m}^{4}}{4}}-\dfrac{v_{m}^{2}}{2}\right)}+\text{cos}\epsilon_{m}\right]}_{P_{\text{induce}}}$

where
$P_{\text{b}}=\dfrac{\Delta_{p}}{8}\rho(u_{\mathrm{z},m})sA_{\mathrm{r}}v^{3}_{\text{tip}}$
and $\rho(u_{\mathrm{z},m})=(1-2.2558.10^{-5}u_{\mathrm{z},m})^{4.2577}$.
$P_{\text{blade}}$ and $P_{\text{fuselage}}$ are the powers required to
overcome the profile drag forces of the rotor blades and the fuselage of the
aerial vehicle that oppose its forward movement, respectively, while
$P_{\text{induce}}$ represents the induced power from the rotation of rotors.

####  II-C2 Hover ($v_{m}=0$)

In this mode, the PAP is static and its position is the same as that in the
previous time slot. From [21], the hovering power consumption of a PAP is
estimated using,

$\displaystyle P^{\text{hov}}_{\text{uav}}$ $\displaystyle=$ $\displaystyle
N_{\mathrm{R}}P_{\mathrm{b}}+\dfrac{W^{3/2}}{\sqrt{4N_{\mathrm{R}}\rho(u_{\mathrm{z},m})A_{\mathrm{r}}}}.$
(8)

####  II-C3 Axial climb or descent ($v_{m}\neq 0,$ $\epsilon_{m}=0$)

Here, the PAP moves along the $+/-$ z-direction. Using (12.35) of [4], the
power required by the PAP to climb vertically ($\epsilon_{m}=0$) is expressed
as,

$\displaystyle P^{\text{vert}}_{\text{uav}}(v_{m})$ $\displaystyle=$
$\displaystyle\dfrac{W}{2}\left(v_{m}+\sqrt{v_{m}^{2}+\dfrac{2W}{N_{\mathrm{R}}\rho(u_{\mathrm{z},m})A_{\mathrm{r}}}}\right)+N_{\mathrm{R}}P_{\mathrm{b}}.$

Hence, the total power consumed by the PAP while it flies along the
$m^{\text{th}}$ path segment is calculated as,

$\displaystyle P_{\text{uav}}(\mathbf{v}_{m})$ $\displaystyle=$
$\displaystyle\left\\{\begin{matrix}P^{\text{fwd}}_{\text{uav}}(\mathbf{v}_{m})&\quad\text{if}\,\,v_{m}\neq
0\,\&\,\epsilon_{m}\neq 0,\\\
P^{\text{hov}}_{\text{uav}}&\quad\text{if}\,\,v_{m}=0,\\\
P^{\text{vert}}_{\text{uav}}(v_{m})&\quad\text{if}\,\,v_{m}\neq
0\,\&\,\epsilon_{m}=0.\end{matrix}\right.$ (10)

###  II-D Fair Energy Efficiency

The fair energy efficiency (bits/Joule) of the system is expressed as,

$\displaystyle\color[rgb]{0,0,0}\text{FEE}\left(\mathcal{V}_{M}\right)$
$\displaystyle=$
$\displaystyle\dfrac{\color[rgb]{0,0,0}\text{FI}\left(\mathcal{V}_{M}\right)\sum_{m=1}^{M}\sum_{n=1}^{N}D_{m,n}\left(\mathbf{v}_{m}\right)}{\sum_{m=1}^{M}\delta_{t}P_{\text{uav}}(\mathbf{v}_{m})},$

where
$\color[rgb]{0,0,0}D_{m,n}\left(\mathbf{v}_{m}\right)=BT_{m,n}\overline{R}_{m,n}$
is the number of bits transmitted to the $n^{\text{th}}$ GN while the PAP is
in the $m^{\text{th}}$ segment, and

$\displaystyle\color[rgb]{0,0,0}{\text{FI}}\left(\mathcal{V}_{M}\right)$
$\displaystyle=$
$\displaystyle\dfrac{\left[\sum_{n=1}^{N}\color[rgb]{0,0,0}\overline{\text{D}}_{n}\left(\mathcal{V}_{M}\right)\right]^{2}}{N\sum_{n=1}^{N}\color[rgb]{0,0,0}\overline{\text{D}}_{n}^{2}\left(\mathcal{V}_{M}\right)},$
(12)

is the fairness index with
$\color[rgb]{0,0,0}\overline{\text{D}}_{n}\left(\mathcal{V}_{M}\right)=\sum_{m=1}^{M}\text{D}_{m,n}\left(\mathbf{v}_{m}\right)/M$
giving the average number of bits transmitted to the $n^{\text{th}}$ GN by the
end of the trajectory.
$\color[rgb]{0,0,0}\text{FI}\left(\mathcal{V}_{M}\right)=1$ means the PAP
sends equal number of data bits to the GNs when it completes the trajectory.
$B$ is the total available bandwidth;
$\mathcal{V}_{M}=\\{\mathbf{v}_{m},\forall m\in\mathcal{M}^{{}^{\prime}}\\}$.
Considering the energy efficiency metric alone could allow the PAP to fly
above a sub-set of GNs to maximize the energy efficiency by increasing the sum
rate. Furthermore, the fairness index can be maximized either by maximizing
the average number of bits transmitted to each GN or by minimizing it. The FEE
metric defined in (LABEL:eq_fee) is a weighted energy efficiency metric, where
the weight is the fairness index. This forces the PAP to follow a 3D
trajectory that maximizes energy efficiency and per-user fairness.

###  II-E The Peukert Effect

Figure 3: Air-time with and without considering the Peukert effect for
constant 200W power-draw until the battery discharges completely.

A usual approach to estimate the maximum air-time of a PAP is to find the
ratio of the initial onboard energy to the sum of instantaneous power
consumption values [20]-[21]. This calculation has the fundamental assumption
that the available discharge time of the PAP battery remains the same
irrespective of the power-draw profile of the PAP. But, in practice, the
battery discharge rate affects its available discharge time as shown in Fig.
3, called the Peukert effect [5].

Let $c_{o}$ be the rated capacity of a cell of the PAP battery unit in ampere-
hours (Ah) and $t_{o}$ be the rated discharge time in hours (h). This means,
if the PAP draws 1A of current from the cell, the cell will be completely
discharged after $t_{o}$ hours. However, in practice, the current drawn by the
PAP changes with time as a function of the power required and the terminal
voltage of the battery:

$\displaystyle P_{\text{uav}}(\mathbf{v}_{m})$
$\displaystyle=i^{\mathrm{b}}_{m}\cdot n^{\mathrm{b}}\cdot
V^{\mathrm{b}}_{m}\,\,\,\,\forall m\in\mathcal{M}^{{}^{\prime}},$ (13)

where $n^{\mathrm{b}}$ is the number of battery cells connected in series to
form the battery unit of the PAP with $V^{\mathrm{b}}_{m}$, the terminal
voltage of a battery cell at the beginning of the $m^{\text{th}}$ time slot;
also, $V^{\mathrm{b}}_{1}=V_{o}$ is the nominal voltage of the battery. Hence,
the current drawn by the PAP during the $m^{\text{th}}$ slot is
$i^{\mathrm{b}}_{m}=P_{\text{uav}}(\mathbf{v}_{m})/(n^{\mathrm{b}}\cdot
V^{\mathrm{b}}_{m})$. After the $m^{\text{th}}$ slot, the battery terminal
voltage drops according to,

$\displaystyle V^{\mathrm{b}}_{m+1}$
$\displaystyle=V^{\mathrm{b}}_{m}-s_{m}^{\mathrm{b}}(i^{\mathrm{b}}_{m})\cdot
i^{\mathrm{b}}_{m}\delta_{t}\,\,\forall m\in\mathcal{M}^{{}^{\prime}},$ (14)

where $s^{\mathrm{b}}_{m}(i^{\mathrm{b}}_{m})$ is the rate of change of
terminal voltage per Ah that changes as a function of $i^{\mathrm{b}}_{m}$. In
addition to the drop in the battery terminal voltage, the remaining discharge
time also changes after each time slot according to,

$\displaystyle t^{\mathrm{b}}_{m+1}$
$\displaystyle=t^{\mathrm{b}}_{m}\left(\dfrac{c^{\mathrm{b}}_{m}-i^{\mathrm{b}}_{m}\delta_{t}}{i^{\mathrm{b}}_{m+1}t^{\mathrm{b}}_{m}}\right)^{p^{\mathrm{b}}}\quad\,\,\forall
m\in\mathcal{M}^{{}^{\prime}},$ (15)

with

$\displaystyle t^{\mathrm{b}}_{1}$
$\displaystyle=t_{o}\left(\dfrac{c_{o}}{i^{\mathrm{b}}_{1}t_{o}}\right)^{p^{\mathrm{b}}};$
(16)

$c_{m}^{\mathrm{b}}=t^{\mathrm{b}}_{m}i_{m}^{\mathrm{b}}$ with
$c_{1}^{\mathrm{b}}=c_{o}$; $p^{\mathrm{b}}>1$ is the Peukert coefficient that
depends on the type of the battery used; $i^{\mathrm{b}}_{m+1}$ is determined
by substituting the voltage determined using (14) in (13) to guarantee a power
output of $P_{\text{uav}}(\mathbf{v}_{m+1})$. The PAP should reach the
destination either before the value of the terminal voltage reaches
$V_{\text{cutoff}}$: $V^{\mathrm{b}}_{M+1}\geq V_{\text{cutoff}}$ or
$t^{\mathrm{b}}_{M+1}\geq 0$.

Figure 4: Variation of battery discharge slope for different discharge
current.

The slope, $s_{m}^{\mathrm{b}}(i^{\mathrm{b}}_{m})$ depends on the type of the
battery used in the PAP. For a Li-ion battery with $c_{o}=4.5$ Ah, $t_{o}=3$
h, $i_{o}=1$ A, $V_{\text{cutoff}}=2.5$ V, and $V_{o}=3.7$ V, we perform a
curve fitting over the variation of $s_{m}^{\mathrm{b}}(i^{\mathrm{b}}_{m})$
with regards to $i^{\mathrm{b}}_{m}$ using the data points from the data sheet
[29] as shown in Fig. 4:

$\displaystyle s_{m}^{\mathrm{b}}(i^{\mathrm{b}}_{m})$
$\displaystyle=f^{\mathrm{b}}_{1}\cdot{i^{\mathrm{b}}_{m}}^{f^{\mathrm{b}}_{2}},$
(17)

where $f^{\mathrm{b}}_{1}=0.2941$, and $f^{\mathrm{b}}_{2}=0.06888$.

The Peukert effect is better explained in Fig. 3. The figure shows the voltage
drop and the remaining discharge time of a typical Li-ion battery (commonly
used in UAVs) during discharge when the PAP draws a power of 200 W
continuously. As shown in the figure, a battery is useful until the terminal
voltage or the remaining discharge time becomes lower than the corresponding
threshold values (2.9 V and 30 Seconds, respectively), whichever happens
first. The air-time of the PAP is defined as the time elapsed from the
beginning till the battery is useful. As seen in the figure, neglecting the
Peukert effect overestimates the air-time of the PAP. Thus a trajectory
planned considering the availability of air-time determined without
considering the Peukert effect will initiate the early-landing procedure
before completing the trajectory. This affects the system’s service fairness,
sum rate, and energy efficiency. The PAP air-time for a given power profile
considering the Peukert effect can be estimated using Algorithm 1.

1 Initialize $m=1$ $P_{\text{uav}}(\mathbf{v}_{m})$, $t_{o}$, $V_{o}$,
$c_{o}$, $V_{\text{cutoff}}$, $\delta_{t}$;

2 while  _1_ do

3 if  _( $m==1$)_ then

4 $V_{m}^{\mathrm{b}}=V_{o}$; find $i_{m}^{\mathrm{b}}$ from (13); find
$t_{m}^{\mathrm{b}}$ using (16);

5

6 if  _( $t_{m}^{\mathrm{b}}\leq 0$)_ then

7 break;

8 find $s_{m}^{\mathrm{b}}(i_{m}^{\mathrm{b}})$ using (17);

9 find $V^{\mathrm{b}}_{m+1}$ using (14);

10 if  _( $V^{\mathrm{b}}_{m+1}<V_{\text{cutoff}}$)_ then

11 break;

12

13 $m=m+1$; update $P_{\text{uav}}(\mathbf{v}_{m})$;

14 find $i_{m}^{\mathrm{b}}$ from (13); $t_{m}^{\mathrm{b}}$ using (15);

15

Output: Air-Time: $T_{\text{uav}}=m\delta_{t}$.

Algorithm 1 PAP Air-Time Estimation

##  III Trajectory Optimization Using DRL Method

In this section, we formulate the problem and use the deep reinforcement
learning technique to design an optimal trajectory for the PAP that maximizes
the FEE of the system.

###  III-A Problem Formulation

The FEE of the considered system can be increased by suitably designing the 3D
trajectory of the PAP. The corresponding problem can be formulated as,

(P1) $\displaystyle:$
$\displaystyle\underset{\color[rgb]{0,0,0}\mathcal{V}_{M}}{\text{maximize}}\,\,\,\,{\color[rgb]{0,0,0}\text{FEE}}\color[rgb]{0,0,0}\left(\mathcal{V}_{M}\right),$
s.t. $\displaystyle V^{\mathrm{b}}_{M+1}\geq
V_{\text{cutoff}};t_{M+1}^{\mathrm{b}}\geq 0,\,\,\,\,$ (19) $\displaystyle
u_{\mathrm{x},m+1}=u_{\mathrm{x},m}+\underbrace{\delta_{t}v_{m}\text{sin}\,\epsilon_{m}\text{cos}\,\alpha_{m}}_{\Delta^{\mathrm{x}}_{{m}}(\mathbf{v}_{m})}\,\forall
m\in\mathcal{M}^{{}^{\prime}},$ $\displaystyle
u_{\mathrm{y},m+1}=u_{\mathrm{y},m}+\underbrace{\delta_{t}v_{m}\text{sin}\,\epsilon_{m}\text{sin}\,\alpha_{m}}_{\Delta^{\mathrm{y}}_{{m}}(\mathbf{v}_{m})}\,\forall
m\in\mathcal{M}^{{}^{\prime}},$ (20) $\displaystyle
u_{\mathrm{z},m+1}=u_{\mathrm{z},m}+\underbrace{\delta_{t}v_{m}\text{cos}\,\epsilon_{m}}_{\Delta^{\mathrm{z}}_{{m}}(\mathbf{v}_{m})}\forall
m\in\mathcal{M}^{{}^{\prime}},$ (21) $\displaystyle
u_{\mathrm{z},\text{min}}\leq u_{\mathrm{z},m}\leq
u_{\mathrm{z},\text{max}}\quad\forall m\in\mathcal{M},$ (22)
$\displaystyle\Sigma_{n=1}^{N}T_{m,n}=\delta_{t}\quad\forall
m\in\mathcal{M}^{{}^{\prime}},$ (23)
$\displaystyle\mathbf{u}_{M+1}=\mathbf{u}_{\mathrm{F}};\mathbf{u}_{1}=\mathbf{u}_{\mathrm{I}},$
(24) $\displaystyle\quad T_{m,n}\geq 0\quad\forall
m\in\mathcal{M}^{{}^{\prime}},n\in\mathcal{N}.$ (25)

The objective function of (P1) maximizes the FEE; (19) ensures that the PAP
will not run out of onboard available battery capacity at any point of the
trajectory. The x, y, and z coordinates of the PAP position are changed
according to $\eqref{c5}-\eqref{c7}$, respectively. The flying region of the
PAP is limited in the z-direction using $\eqref{c10}$ with
$u_{\mathrm{z},\text{min}}$ and $u_{\mathrm{z},\text{max}}$ as the respective
minimum and maximum permitted flying altitudes. (23) is the TDMA scheduling
constraint. (24) constrains the initial and final positions of the PAP to be
$\mathbf{u}_{\mathrm{I}}$ and $\mathbf{u}_{\mathrm{F}}$, respectively. (P1) is
a non-convex optimization problem with a large number of optimization
variables restricting the use of conventional convex optimization methods such
as sequential convex programming[30]. Consequently, (P1) is equivalently
represented as a Markov Decision Process (MDP) with continuous state and
action spaces, and a DRL-based algorithm is proposed to design a 3D trajectory
for the PAP that maximizes the FEE of the system.

The PAP is considered as an agent of the DRL framework; the framework takes
the state observed by the PAP, $s_{m}$, and outputs an action, $a_{m}$. The
agent receives a reward $r_{m}$ after taking the action $a_{m}$ that moves it
from state $s_{m}$ to state $s_{m+1}$. The whole trajectory of the PAP is
considered as an episode of the DRL framework; an episode ends (i.e., $m=M+1$)
if it runs out of the onboard battery capacity. It should be noted that the
value of $M$ is not constant here, and it varies according to the profile of
the PAP power consumption. Additionally, to model the FEE solely as a function
of the PAP trajectory, we schedule the data transmission to each GN for a time
that is proportional to the respective expected spectral efficiency (i.e.
$T_{m,n}=\overline{R}_{m,n}/\sum_{n=1}^{N}\overline{R}_{m,n}$, $\forall
m\in\mathcal{M}^{{}^{\prime}},n\in\mathcal{N}$).

###  III-B PAP Trajectory as an MDP

Since the next state and action of the PAP depend only on the present state of
the PAP, we use a standard MDP representation as a 4-tuple
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$ with sets: state space
$\mathcal{S}$, action space $\mathcal{A}$, probability of transition
$\mathcal{P}$, and a state-action reward map
$\mathcal{S}\times\mathcal{A}->\mathcal{R}$.

####  III-B1 State Space, $\mathcal{S}=\\{s_{m}\\}$

The state of the PAP consists of 3D-coordinates of the PAP location written
relative to the destination, the PAP’s battery terminal voltage, the total
energy consumed, coordinates of the GNs written relative to the horizontal
projection of the PAP position, and the number of bits transmitted to each GN
until the end of the $m^{\text{th}}$ time slot:

$\displaystyle s_{m}$ $\displaystyle=$
$\displaystyle\\{\\{(\mathbf{u}_{m}-\mathbf{u}_{\mathrm{F}}),{V}^{\mathrm{b}}_{m},E_{m},\\{(\mathbf{g}_{\mathrm{h},n}-\mathbf{u}_{\mathrm{h},m})\\},\\{{D}^{\text{sum}}_{m,n}\\}\\},$

where ${D}^{\text{sum}}_{m,n}=\sum_{j=1}^{m}D_{j,n}(\mathbf{v}_{j},T_{jn})$ is
the total number of bits transmitted to the $n^{\text{th}}$ GN till the end of
the $m^{\text{th}}$ time slot;
$E_{m}=\sum_{j=1}^{m}\delta_{t}P_{\text{uav}}(\mathbf{v}_{j})$ is the total
energy consumed until the end of the $m^{\text{th}}$ time slot. Hence, $s_{m}$
has $5+3\cdot N$ dimensions.

####  III-B2 Action Space $\mathcal{A}=\\{a_{m}\\}$

Since all the state dimensions are functions of the 3D movement of the PAP,
the action $a_{m}$ taken by the PAP is velocity-steered and can be expressed
as a vector of dimension $3$:
$a_{m}=\\{c_{m}^{\mathrm{x}},c_{m}^{\mathrm{y}},c_{m}^{\mathrm{z}}\\}\in\left[-1,1\right]$
such that the components of the velocity vector are given by,

$\displaystyle v_{m}$ $\displaystyle=$
$\displaystyle\sqrt{{c_{m}^{\mathrm{x}}}^{2}+{c_{m}^{\mathrm{y}}}^{2}+{c_{m}^{\mathrm{z}}}^{2}}\cdot\dfrac{v_{\text{max}}}{3},$
(27) $\displaystyle\epsilon_{m}$ $\displaystyle=$
$\displaystyle\text{arctan}\left(\dfrac{\sqrt{{c_{m}^{\mathrm{x}}}^{2}+{c_{m}^{\mathrm{y}}}^{2}}}{c_{m}^{\mathrm{z}}}\right),$
(28) $\displaystyle\alpha_{m}$ $\displaystyle=$
$\displaystyle\text{arctan}\left(\dfrac{{{c_{m}^{\mathrm{y}}}}}{c_{m}^{\mathrm{x}}}\right).$
(29)

Moreover, if the action takes the PAP out of the altitude boundaries, the
z-coordinate of the next state is readjusted to the corresponding boundary
value.

####  III-B3 Reward Space $\mathcal{R}=\\{r_{m}\\}$

The reward function determines how fast the PAP finds the optimal trajectory.
Here, the primary objectives are to maximize the FEE and let the PAP reach the
specified destination before the battery becomes obsolete by satisfying all
the constraints of (P1). To efficiently map the above objectives, we leverage
the reward shaping technique [31]. Hence, the reward $r_{m}$ is expressed as,

$\displaystyle r_{m}=f_{m}+p_{m},$ (30)

where,

$\displaystyle p_{m}$ $\displaystyle=$
$\displaystyle\left\\{\begin{matrix}\color[rgb]{0,0,0}\text{FEE}\color[rgb]{0,0,0}(\mathcal{V}_{m+1})&\text{if}\,\,\text{FEE
improves},\\\ 0&\text{otherwise}/\text{if}\,$m=M+1$,\end{matrix}\right.$ (31)

is the position reward that encourages the PAP to move in a direction that
improves the FEE of the system, and

$\displaystyle f_{m}$ $\displaystyle=$
$\displaystyle\left\\{\begin{matrix}\kappa_{\mathrm{f}}\cdot\color[rgb]{0,0,0}\text{FEE}\color[rgb]{0,0,0}(\mathcal{V}_{M})&\text{if}\quad
m=M+1,\\\ 0&\text{else},\end{matrix}\right.$ (32)

is the terminal reward. The value of $\kappa_{\mathrm{f}}$ should be selected
in a way that ensures the sum of the position rewards is always less than or
equal to the terminal reward. $\kappa_{\mathrm{f}}$ is needed to balance the
position and terminal rewards. Otherwise, the position reward would dominate
over the terminal reward.

Since the defined MDP is deterministic, no randomness is considered and all
transitions follow the agent’s decisions [24]. Therefore, the next state is a
direct consequence of the current action of the agent.

####  III-B4 Episode Termination

The FEE of the system increases when the PAP spends the maximum time over the
air to serve the GNs. Hence, an episode is terminated when the remaining air-
time ($T_{\text{uav}}$) of the PAP is equal to the minimum time required by
the PAP to reach the destination from the current position with a speed of
$v_{\text{max}}$:
$T_{\text{uav},m}^{\text{min}}=\|\mathbf{u}_{m}-\mathbf{u}_{\mathrm{F}}\|/v_{\text{max}}$.
The remaining air-time of the PAP can be estimated using Algorithm 1.
Consequently, if an action takes the PAP to $\mathbf{u}_{m+1}$ and if
$T_{\text{uav}}<T_{\text{uav},m+1}^{\text{min}}$, the action is discarded and
the PAP moves to the destination from $\mathbf{u}_{m}$ with a speed of
$v_{\text{max}}$ m/s. Accordingly, the GNs are served for a maximum amount of
time while ensuring a safe landing of the PAP at the destination.

The safety check explained above satisfies (19) by ensuring sufficient energy
available at the PAP to fly back to the destination after each action. The 3D
coordinates of the PAP after taking an action are determined by substituting
(27)-(29) in (19)-(21), satisfying the PAP movement constraints. The altitude
constraint (22) is satisfied by limiting the action space if such an action
violates the constraint. The proposed heuristic time allocation in which the
data transmission to each GN is scheduled for a time proportional to the
respective expected spectral efficiency satisfies the TDMA constraints (23)
and (25). Finally, all the episodes start and end at $\mathbf{u}_{\mathrm{I}}$
and $\mathbf{u}_{\mathrm{F}}$, respectively satisfying constraint (24).

###  III-C TD3-Based PAP 3D Path Design

Here, the objective is to find the optimal policy $\pi$ that takes the current
state of the PAP (agent) and gives an action that maximizes the expected
return: $R_{m}=\sum_{i=m}^{M}\gamma^{i-m}r_{m}$, where $\gamma$ is a discount
factor determining the priority of short-term rewards. The action value of a
state, $Q_{\pi}(s_{m},a_{m})$, gives the expected return for starting in state
$s_{m}$, taking action $a_{m}$, and then acting according to the policy $\pi$
forever after. The optimal action-value function is given by the Bellman
equation as,

$\displaystyle
Q^{*}(s_{m},a_{m})=\left[r_{m}+\gamma\max_{a_{m+1}}Q^{*}(s_{m+1},a_{m+1})\right].$

1 Initialize the locations of GNs;

2 Initialize critic networks $Q_{\phi_{1}}$, $Q_{\phi_{2}}$, and actor network
$\mu_{\theta}$ with random parameters $\phi_{1}$, $\phi_{2}$, $\theta$;

3 Initialize target networks $\phi_{1,\text{tgt}}\leftarrow\phi_{1}$,
$\phi_{2,\text{tgt}}\leftarrow\phi_{2}$, $\theta_{\text{tgt}}\leftarrow\theta$

4 Initialize replay buffer $\mathcal{H}$;

5 for  _each episode_ do

6 Initialize the location of PAP to $\mathbf{u}_{I}$, $m=1$,
$V_{m}^{b}=V_{o}$, $t_{m}=t_{o}$; ${d}=1$;

7 Formulate the state of the PAP $s_{m}$;

8 while  _the episode is not over_ do

9 The agent takes an action with exploration noise:
$a_{m}=\mu_{\theta}(s_{m})+\epsilon$, observe the reward $r_{m}$ and new state
$s_{m+1}$;

10 store $(s_{m},a_{m},r_{m},s_{m+1})$ in the replay buffer;

11 if  _replay buffer is sufficient_ then

12 sample mini batch of $|\mathcal{H}^{{}^{\prime}}|$ transitions from
$\mathcal{H}$;

13 compute target actions using (III-C3);

14 compute target using (38);

15 update critic networks by one step gradient descent using,
$\nabla_{\phi_{j}}\frac{1}{|\mathcal{H}^{{}^{\prime}}|}\sum_{e_{i}\in\mathcal{H}^{{}^{\prime}}}\Bigg{(}Q_{\phi_{j}}(s_{i},\mu_{\theta}(s_{i}))-y_{i}\Bigg{)}^{2}$
for $j\in\\{1,2\\}$;

16 if  _it is time to update_ then

17 update policy network by one step gradient ascent using,
$\frac{\sum_{e_{i}\in\mathcal{H}^{{}^{\prime}}}\nabla_{\mu_{\theta}(s_{i})}Q_{\phi_{1}}(s_{i},\mu_{\theta}(s_{i}))\nabla_{{\theta}}\mu_{\theta}(s_{i})}{|\mathcal{H}^{{}^{\prime}}|}$;

18 update target networks using,
$\theta_{\text{tgt}}=\tau\theta_{\text{tgt}}+(1-\tau\theta)$;

19 $\phi_{j,\text{tgt}}=\tau\phi_{j,\text{tgt}}+(1-\tau\phi_{j})$ for
$j\in\\{1,2\\}$.

20

21 $m=m+1$;

22

23

Algorithm 2 Energy Efficient 3D Path Planning

In DRL, $Q^{*}(s_{m},a_{m})$ is approximated by a neural network
$Q_{\phi}(s_{m},a_{m})$ with parameters $\phi$. Then the closeness of
$Q_{\phi}(s_{m},a_{m})$ to $Q^{*}(s_{m},a_{m})$ is judged by evaluating the
mean-squared Bellman error (MSBE) function:

$\displaystyle L(\phi,{\mathcal{H}})$ $\displaystyle=$
$\displaystyle\underset{\\{e_{i}\\}\sim{\mathcal{H}}}{{\mathrm{E}}}\left[\Bigg{(}Q_{\phi}(s_{i},a_{i})-y_{i}\Bigg{)}^{2}\right],$
(34)

where

$\displaystyle y_{i}$ $\displaystyle=$ $\displaystyle
r_{i}+\gamma(1-d)\max_{a_{i+1}}Q_{\phi}(s_{i+1},a_{i+1}),$ (35)

is the target value and $d=1$ represents the terminal state. The expectation
in (34) is taken over a mini batch of experiences,
$\\{e_{i}\\}=\\{(s_{i},a_{i},r_{i},s_{i+1},d)\\}=\mathcal{H}^{{}^{\prime}}$
sampled from the experience replay buffer, $\mathcal{H}$. The parameter $\phi$
is updated to minimize the MSBE. Since the considered action space is
continuous, the evaluation of the MSBE is not trivial because of the
$\max_{a_{i+1}}Q_{\phi}(s_{i+1},a_{i+1})$ term in the target value where the
maximization has to be done over a continuous action space. To tackle this, we
use an actor-critic framework-based twin delayed deep deterministic policy
gradient (TD3) algorithm [32]. An actor-critic framework uses an actor network
that takes the state $s_{m}$ as input and outputs the action $a_{m}$, whereas
the Q-value of the taken action $a_{m}$ at state $s_{m}$ is estimated by the
critic network. At the end of the training, the actor network represents the
optimal policy, $\pi$. Hence, (34) and (35) can be rewritten as,

$\displaystyle L(\phi,{\mathcal{H}})$ $\displaystyle=$
$\displaystyle\underset{\\{e_{i}\\}\sim{\mathcal{H}}}{{\mathrm{E}}}\left[\Bigg{(}Q_{\phi}(s_{i},\mu_{\theta}(s_{i}))-y_{i}\Bigg{)}^{2}\right],$
(36) $\displaystyle y_{i}$ $\displaystyle=$ $\displaystyle
r_{i}+\gamma(1-d)Q_{\phi}(s_{i+1},\mu_{\theta}(s_{i+1})),$ (37)

where $\mu_{\theta}$ is the actor network with parameters $\theta$ and
$Q_{\phi}$ is the critic network with parameters $\phi$. From (36) and (37),
the target $y_{i}$ depends on the same parameters we are trying to train:
$\phi$ and $\theta$ which makes the MSBE minimization unstable. The solution
is to use target networks that have sets of parameters which come close to
$\phi$ and $\theta$, but with a time delay. The parameters of the target
network are denoted as $\phi_{\text{tgt}}$ and $\theta_{\text{tgt}}$,
respectively. In order to avoid the overestimation problem of the deep
deterministic policy gradient (DDPG) algorithm [33], the TD3 algorithm
proposed in [32] uses:

####  III-C1 Clipped Double-Q Learning

in which two critic networks are used instead of one, and uses the smaller of
the two Q-values to form the targets in the MSBE functions:

$\displaystyle y_{i}$ $\displaystyle=$ $\displaystyle
r_{i}+\gamma(1-d)\min_{j=1,2}Q_{\phi_{j,\text{tgt}}}(s_{i+1},\mu_{\theta_{\text{tgt}}}(s_{i+1})),$
(38)

where $Q_{\phi_{j,\text{tgt}}}$ for $j\in\\{1,2\\}$ are the corresponding
target critic networks. Both networks are then trained to minimize this
target:

$\displaystyle L(\phi_{1},{\mathcal{H}})$ $\displaystyle=$
$\displaystyle\underset{\\{e_{i}\\}\sim{\mathcal{H}}}{{\mathrm{E}}}{\Bigg{[}Q_{\phi_{1}}(s_{i},\mu_{\theta}(s_{i}))-y_{i}\Bigg{]}^{2}},$
(39) $\displaystyle L(\phi_{2},{\mathcal{H}})$ $\displaystyle=$
$\displaystyle\underset{\\{e_{i}\\}\sim{\mathcal{H}}}{{\mathrm{E}}}{\Bigg{[}Q_{\phi_{2}}(s_{i},\mu_{\theta}(s_{i}))-y_{i}\Bigg{]}^{2}},$
(40)

that avoids the overestimation problem;

####  III-C2 Delayed Policy Updates

through which the TD3 updates the policy ($\mu_{\theta}$) and target networks
less frequently than the critic networks (once every $K$ critic networks
update);

####  III-C3 Target Policy Smoothing

which adds a clipped noise on each dimension of the action produced by the
target policy network. After adding the clipped noise, the target action is
then clipped to lie in the valid action range:
$[a_{\text{min}},a_{\text{max}}]$,

$\displaystyle\mu_{\theta_{\text{tgt}}}(s_{i+1})=\text{clip}\left(\mu_{\theta_{\text{tgt}}}(s_{i+1})+\text{clip}(\epsilon,-c,c),a_{\text{min}},a_{\text{max}}\right),$

where $\epsilon\sim\mathcal{N}(0,\sigma)$ and
$\text{clip}(x,a,b)=\text{max}(\text{min}(x,b),a)$. This avoids the problem of
developing an incorrect sharp peak for some actions by the Q-function
approximator. The steps to design a fair energy-efficient 3D trajectory for
the PAP using the TD3 framework are given in Algorithm 2.

##  IV Numerical Evaluation

TABLE II: Testing environment settings Label | Definition | Value
---|---|---
$f_{c}$ | Channel carrier frequency | 5.8 GHz
$c$ | Velocity of light |  $3\cdot 10^{8}$ m/s
$B$ | Channel bandwidth for each GN | 40 MHz
$N_{0}$ | Noise spectral power | -174 dBm/Hz
$u_{z,\text{min}}$ | PAP’s minimum flying altitude | 20 m
$u_{z,\text{maz}}$ | PAP’s maximum flying altitude | 100 m
$v_{\text{max}}$ | Maximum achievable PAP speed | 24 m/s
$\delta_{t}$ | Time discretization Interval | 1 s
$P_{t}$ | Transmission Power | 23 dBm

In this section, we present our main findings obtained through numerical
evaluations. The evaluations consider a square area of 1000 $\times$ 1000
meters with 16 GNs. The values of all the environment-related parameters are
listed in Table II [24]. To the best of our knowledge, the same overall
setting has not been considered in the literature yet, hence we are comparing
our results with the following two baseline scenarios:

  1. 1.

Baseline 1: the first maneuver executed by the PAP is a diagonal climb from
$\mathbf{u}_{\mathrm{I}}$ up to the center of the region with coordinates
$(500,500,100)$; hovers there until it only has sufficient energy to reach its
destination; and flies reclined to the end of its trajectory
$\mathbf{u}_{\mathrm{F}}$;

  2. 2.

Baseline 2: the first maneuver executed by the PAP is a diagonal climb from
$\mathbf{u}_{\mathrm{I}}$ to (200,200,100). It then continues through the
shortest path between the locations of the GNs until it only has sufficient
energy to reach its destination. We determine the shortest path using the well
known travelling salesman algorithm.

The simulations are done considering 3 different environment scenarios namely,
suburban, urban and dense urban with
$(a,b,\eta^{\text{LoS}},\eta^{\text{NLoS}})$ parameters $(4.88,0.43,0.2,24)$,
$(9.61,0.16,1.2,23)$, and $(12.08,0.11,1.8,26)$, respectively [3].

Figure 5: Baseline1 scenario to compare the performance; the PAP starts at
(0,0,20), flies to the center of the geographical region (500,500,100), hovers
there until the battery capacity reaches the threshold value, flies back to
the destination (1000,1000,20).

The architecture of actor and critic networks used in the simulations are
listed in Table III. After an extensive experimentation, the values of various
hyper parameters associated with the networks that give the maximum FEE value
after training the networks for 1000 episodes are listed in Table IV.

TABLE III: Network Architecture Network(s) | Layer | Depth | Activation
---|---|---|---
Critic | Input Layer | 56 | $-$
Critic | Hidden Layer 1 | 256 | ReLu
Critic | Hidden Layer 2 | 512 | ReLu
Critic | Hidden Layer 3 | 512 | ReLu
Critic | Output Layer | 1 | ReLu
Actor | Input Layer | 53 | $-$
Actor | Hidden Layer 1 | 256 | ReLu
Actor | Hidden Layer 2 | 512 | ReLu
Actor | Hidden Layer 3 | 512 | ReLu
Actor | Output Layer | 3 | TanH
TABLE IV: Network Parameters. Label | Definition | Value
---|---|---
$\alpha$ | Actor learning rate | $10^{-4}$
$\beta$ | Critic learning rate | $10^{-3}$
$|\mathcal{H}^{{}^{\prime}}|$ | Batch size | 64
$|\mathcal{H}|$ | Replay buffer size | $2\times 10^{5}$
$K$ | Network update interval | 2
$\tau$ | Soft update factor | 0.001
$\gamma$ | Discount factor | 0.99
$\kappa_{\mathrm{f}}$ | - | 1000

Fig. 6 plots the PAP power consumption and air-time as a function of the
speed. The vertical flying power consumption increases with speed since the
PAP requires more power to overcome the downward drag force. When the PAP is
flying horizontally, the power consumption initially decreases and then
increases after 11 m/s: because the magnitude of power required to overcome
the rotor-induced drag force decreases with the PAP velocity; in the low-speed
regime, it dominates the power consumed to overcome the fuselage and rotor
profile drag forces. Correspondingly, the maximum PAP air-time, using
Algorithm 1, is obtained as 1616 seconds (s) when the PAP is flying at a speed
of 11 m/s. The figure also shows the importance of considering the Peukert
effect during the trajectory planning of the PAP; neglecting the Peukert
effect overestimates the air-time of the PAP that could force the PAP to
initiate landing procedure before completing the planned trajectory.

Figure 6: Variation of the PAP power consumption and air-time (endurance) as a
function of the flying velocity. H-fly represents a level forward flight at
the maximum height ($v_{m}\neq 0,$ $\epsilon_{m}=90^{\circ}$); v-fly
represents axial climb or descent ($v_{m}\neq 0,$ $\epsilon_{m}=0$).

(a) Trajectory.

(b) Speed.

(c) Trajectory.

(d) Speed.

(e) Trajectory.

(f) Speed.

Figure 7: Sample PAP trajectories obtained using the trained actor network.

(a) Fair energy efficiency.

(b) Fairness index.

(c) Energy-efficiency.

(d) PAP Airtime.

Figure 8: FEE, FI, EE, and PAP air-time improvements as the training
progresses. The training procedure is repeated for 16 different random seeds.
The shadow regions around the plots show a 95% confidence interval across
randomized repetitions.

Finally, we perform two types of RL training to showcase the effectiveness of
our approach in different scenarios. The first (offline) approach assumes the
same fixed user positions for both the training and testing part. This
analysis is done to evaluate the capability of the actor network to solve
problem where all users are uniformly spaced along the x and y directions, as
shown in Fig. 5. This fixed arrangement has very sparse GNs which makes it
difficult for the FI problem. The second approach considers random positions,
such as in a point process, where the x and y coordinates of each GN are
uniformly distributed for each training episode. In this approach the testing
is done on a set of random GN arrangements that the agent has not used for
training before, which also includes the fixed uniform positions. This is a
slightly easier problem when solving fairness problems such as FI due to the
likelihood of users to cluster and/or disperse due to the entropy of the
system.

###  IV-A Fixed Uniform User Positions (Offline RL)

The resulting trajectories following the training process of the actor network
for the first arrangement are given in Fig. 7. We notice that in the case of
suburban deployments, the TD3 DRL method behaves very similarly to the stop-
and-hover baseline deployment. This is due to the very high likelihood of
having a LoS with all users and gives less relevance to the position of the
PAP. However, opposite to the baseline case, the DRL method performs
occasional repositioning to maintain better FEE. Finally, the TD3-DRL
implementation keeps the PAP in constant movement with speeds around its most
energy efficient velocity. We note that the most energy efficient velocity
varies with the aerodynamics of the specific UAV and can thus be different
with UAVs from different manufacturers. Opposed to the suburban scenario, in
Fig. 7 we can see in the subplots c) and e) that the PAP maintains much more
dynamic movements for the urban and dense urban scenarios respectively. This
is a superior approach to the stop-and-hover one, due to the short bursts of
better LoS connectivity when the PAP travels above each GN. However, these
bursts need to be balanced over the longer period of service and thus the PAP
is always kept on the move. Finally, it is noticeable that in these two
scenarios, the PAP flies off to a position that is far from the center of the
area of service. This is significant for keeping the nodes equally serviced,
and thus have improved FEE, in the more challenging propagation environments.

Figs. 8(a)-8(d) show the respective improvement in the fair energy efficiency,
fairness index, energy efficiency, and PAP air time as the training
progresses. As seen in the figures, initially, the agent tries random
trajectories to explore the state-action space causing relatively shorter
episodes with low FEE, FI, and EE values. Later, this experience helps the ML
model to converge to a better policy that improves the optimization metric.
Moreover, the testing values (the values after 1000 episodes) using the
proposed algorithm outperform the baseline scenarios in all the considered
environments, as given in Table V.

TABLE V: Improvement with respect to the baselines.

|  | Proposed | Baseline1 | Improvement | Baseline2 | Improvement
---|---|---|---|---|---|---
Suburban | FEE | 2.039 | 1.085 | 87.93 % | 1.083 | 88.27%
FI | 0.896 | 0.975 | -8.1 % | 0.985 | -9.03%
EE | 2.276 | 1.113 | 104.49% | 1.098 | 107.28%
Urban | FEE | 1.4 | 0.376 | 272.34% | 0.508 | 175.59%
FI | 0.968 | 0.681 | 42.14 % | 0.971 | -0.31%
EE | 0.876 | 0.552 | 58.69 % | 0.523 | 67.49%
Dense urban | FEE | 0.853 | 0.204 | 318.13 % | 0.329 | 159.27%
FI | 0.902 | 0.617 | 46.19 % | 0.976 | -7.58%
EE | 0.946 | 0.331 | 185.8 % | 0.337 | 180.71%

The maximum gain over Baseline1 is achieved when the PAP is deployed in a
dense urban scenario. This is because the considered setup places a subset of
GNs in the NLoS regime of the PAP’s hovering point, thereby giving a low
baseline FI value. The proposed algorithm improves the FI value by moving the
PAP around the GNs, as shown in Fig. 7(e). The performance gain achieved by
Baseline2 is competitive in urban and dense urban scenarios, with regards to
Baseline1, because the corresponding trajectory improves the service fairness
among all the GNs. Interestingly, the FEE performances of Baseline1 and
Baseline 2 in a suburban scenario are comparable since all the GNs are in LoS
with the PAP throughout the respective placement policy, giving high FI
values. This leads us to the conclusion that the suburban environment is not
challenging enough for the problem of trajectory with regards to the scale of
our implementation. . In all the trajectories, the PAP climbs to the maximum
altitude after leaving the starting point and then follows a horizontal flight
during the remaining endurance: as the altitude increases, the throughput
between the PAP and a node increases due to an improved LoS probability
between them; furthermore, the PAP power consumption during a vertical flight
is much higher compared to a horizontal flight as shown in Fig. 6. Hence,
flying horizontally at the maximum altitude increases the PAP air time as well
as the number of bits transmitted to the GNs thereby improving the FEE value
of the system. Also, the speed plots of Fig. 7 show that the actor proposes to
fly the PAP at the optimal flying speed that maximizes the air-time of the
PAP.

(a) FEE Suburban

(b) FEE Urban

(c) FEE Dense urban

Figure 9: Training and testing FEE plots. Circles inside the box plots
represent the average test FEE values. The second and third box plots
represent Baselines 1 and 2, respectively.

###  IV-B Randomized Uniform User Positions (Online RL)

In this section, we describe the method adopted from [34] to generalize the
training so that the trained actor network performs well for any set of user
positions. The system is trained for a fixed number of episodes
$N_{\mathrm{train}}$, where the x and y coordinates of each GN are uniformly
distributed for each training episode. After every 10 training episodes, we
evaluate the actor network on a total of $N_{\mathrm{eval}}$ evaluation
episodes with disabled learning to assess the current performance of the actor
network. We repeat this training procedure for $N_{\mathrm{seed}}$ times, each
with a different random seed. The average FEE value after each evaluation
phase is used as a metric to select the best-performing parameter ($\theta$)
of the actor network ($\mu_{\theta}$). At the end of the training procedure,
we further evaluate the learned policy by assessing its performance in
$N_{\mathrm{test}}$ episodes, each with a different placement of GNs that the
agent has not seen during the entire training process. Moreover, this test
phase happens without exploration and learning.

Fig. 9 shows the training and testing performances for suburban, urban, and
dense urban scenarios with $N_{\mathrm{train}}=1000$, $N_{\mathrm{eval}}=16$,
and $N_{\mathrm{seed}}=8$. The shadow regions around the plots show a 95%
confidence interval across randomized repetitions. In all the scenarios, the
FEE value improves with the training as seen in Fig. 8(a). The mean and median
FEE values obtained after testing the learned policy over
$N_{\mathrm{test}}=512$ episodes outperform the mean baseline performances. As
observed previously, the maximum and minimum performance gains are observed in
dense urban and suburban scenarios, respectively. Additionally, the test
performances (with learning disabled) are comparable with the performances at
the end of the training phase. Thus the learned policy can be used to design
an energy-efficient 3D trajectory for a PAP deployed to serve any given
distribution of the GNs while guaranteeing per-user service fairness.

##  V Conclusion

In this paper, we considered a UAV in the role of a portable access point
(PAP) that aims to maximize the novel fairness-based energy efficiency metric,
fair energy efficiency (FEE). Optimizing the energy-efficiency of PAPs is
important but this should not come at the expense of service fairness. The
method we propose here strikes a good balance between both. Moreover, we
defined a pragmatic non-linear discharge behavior of the PAP battery, as the
Peukert effect. As the first work to investigate the Peukert effect in PAP 3D
trajectory optimization for wireless IoT services, we initially investigated
the impact of the non-linearity of the energy storage. As such, we deducted
that neglecting the Peukert effect overestimates the PAP air time which could
force the PAP to perform an early landing. Given the non-convex FEE
maximization problem with non-tractable constraints we proposed an adapted
implementation of a twin delayed deep deterministic policy gradient deep
reinforcement learning (TD3-DRL) framework. The optimal solutions provided by
TD3-DRL varied by the properties of the propagation environment. The
improvements of using the TD3-DRL in suburban scenarios are moderate with a
gain up to 80% in suburban over the baseline scenarios and around 200% and
300% in the urban-and dense urban scenarios respectively. Finally, we
generalize the network to any set of GN positions. Thus, we can summarize,
that our TD3-DRL implementation provides a robust solution for PAP trajectory
optimization in both strongly LoS and strongly NLoS environments.

## References

  * [1] A. Fotouhi, H. Qiang, M. Ding, M. Hassan, L. G. Giordano, A. Garcia-Rodriguez, and J. Yuan, “Survey on uav cellular communications: Practical aspects, standardization advancements, regulation, and security challenges,” _IEEE Communications surveys & tutorials_, vol. 21, no. 4, pp. 3417–3442, 2019.
  * [2] “Study on enhanced lte support for aerial vehicles (release 15),” _3GPP, Sophia Antipolis, France, Rep. 3GPP TR 36.777, Dec. 2017_ , 2017.
  * [3] A. Al-Hourani, S. Kandeepan, and S. Lardner, “Optimal lap altitude for maximum coverage,” _IEEE Wireless Communications Letters_ , vol. 3, no. 6, pp. 569–572, 2014.
  * [4] A. Filippone, _Flight performance of fixed and rotary wing aircraft_. Elsevier, 2006.
  * [5] K. Khan, M. A. Hossain, A. Obaydullah, and M. Wadud, “Pkl electrochemical cell and the peukert’s law,” _IJARIIE_ , vol. 4, no. 2, pp. 4219–4227, 2018.
  * [6] S. Shakoor, Z. Kaleem, M. I. Baig, O. Chughtai, T. Q. Duong, and L. D. Nguyen, “Role of uavs in public safety communications: Energy efficiency perspective,” _IEEE Access_ , vol. 7, pp. 140 665–140 679, 2019.
  * [7] I. Donevski and J. J. Nielsen, “Dynamic standalone drone-mounted small cells,” in _2020 European Conference on Networks and Communications (EuCNC)_. IEEE, 2020, pp. 342–347.
  * [8] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage,” _IEEE Communications Letters_ , vol. 20, no. 8, pp. 1647–1650, 2016.
  * [9] M. Alzenad, A. El-Keyi, F. Lagum, and H. Yanikomeroglu, “3-d placement of an unmanned aerial vehicle base station (uav-bs) for energy-efficient maximal coverage,” _IEEE Wireless Communications Letters_ , vol. 6, no. 4, pp. 434–437, 2017.
  * [10] M. Alzenad, A. El-Keyi, and H. Yanikomeroglu, “3-d placement of an unmanned aerial vehicle base station for maximum coverage of users with different qos requirements,” _IEEE Wireless Communications Letters_ , vol. 7, no. 1, pp. 38–41, 2017.
  * [11] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Mobile unmanned aerial vehicles (uavs) for energy-efficient internet of things communications,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 11, pp. 7574–7589, 2017.
  * [12] J. Cui, H. Shakhatreh, B. Hu, S. Chen, and C. Wang, “Power-efficient deployment of a uav for emergency indoor wireless coverage,” _IEEE Access_ , vol. 6, pp. 73 200–73 209, 2018.
  * [13] Q. Zhang, M. Mozaffari, W. Saad, M. Bennis, and M. Debbah, “Machine learning for predictive on-demand deployment of uavs for wireless communications,” in _2018 IEEE global communications conference (GLOBECOM)_. IEEE, 2018, pp. 1–6.
  * [14] S. Eom, H. Lee, J. Park, and I. Lee, “Uav-aided wireless communication designs with propulsion energy limitations,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 651–662, 2019.
  * [15] W. Shi, J. Li, H. Wu, C. Zhou, N. Cheng, and X. Shen, “Drone-cell trajectory planning and resource allocation for highly mobile networks: A hierarchical drl approach,” _IEEE Internet of Things Journal_ , vol. 8, no. 12, pp. 9800–9813, 2020.
  * [16] C. You and R. Zhang, “Hybrid offline-online design for uav-enabled data harvesting in probabilistic los channels,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 6, pp. 3753–3768, 2020.
  * [17] I. Valiulahi and C. Masouros, “Multi-uav deployment for throughput maximization in the presence of co-channel interference,” _IEEE Internet of Things Journal_ , vol. 8, no. 5, pp. 3605–3618, 2020.
  * [18] Q. Wu and R. Zhang, “Common throughput maximization in uav-enabled ofdma systems with delay consideration,” _IEEE Transactions on Communications_ , vol. 66, no. 12, pp. 6614–6627, 2018.
  * [19] H. Wu, F. Lyu, C. Zhou, J. Chen, L. Wang, and X. Shen, “Optimal uav caching and trajectory in aerial-assisted vehicular networks: A learning-based approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 12, pp. 2783–2797, 2020.
  * [20] Y. Zeng, J. Xu, and R. Zhang, “Energy minimization for wireless communication with rotary-wing uav,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 4, pp. 2329–2345, 2019.
  * [21] N. Babu, M. Virgili, C. B. Papadias, P. Popovski, and A. J. Forsyth, “Cost-and energy-efficient aerial communication networks with interleaved hovering and flying,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 9, pp. 9077–9087, 2021.
  * [22] Y. Zeng and R. Zhang, “Energy-efficient uav communication with trajectory optimization,” _IEEE Transactions on Wireless Communications_ , vol. 16, no. 6, pp. 3747–3760, 2017.
  * [23] N. Babu, C. B. Papadias, and P. Popovski, “Energy-efficient 3-d deployment of aerial access points in a uav communication system,” _IEEE Communications Letters_ , vol. 24, no. 12, pp. 2883–2887, 2020.
  * [24] I. Donevski, N. Babu, J. J. Nielsen, P. Popovski, and W. Saad, “Federated learning with a drone orchestrator: Path planning for minimized staleness,” _IEEE Open Journal of the Communications Society_ , vol. 2, pp. 1000–1014, 2021.
  * [25] R. Ding, F. Gao, and X. S. Shen, “3d uav trajectory design and frequency band allocation for energy-efficient and fair communication: A deep reinforcement learning approach,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 12, pp. 7796–7809, 2020.
  * [26] C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-efficient uav control for effective and fair communication coverage: A deep reinforcement learning approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 36, no. 9, pp. 2059–2070, 2018.
  * [27] A. Mazaherifar and S. Mostafavi, “Uav placement and trajectory design optimization: A survey,” _Wireless Personal Communications_ , vol. 124, no. 3, pp. 2191–2210, 2022.
  * [28] A. Al-Hourani, S. Kandeepan, and A. Jamalipour, “Modeling air-to-ground path loss for low altitude platforms in urban environments,” in _2014 IEEE global communications conference_. IEEE, 2014, pp. 2898–2904.
  * [29] https://files.gwl.eu/inc/_doc/attach/StoItem/5230/Datasheet_LG_MH1.pdf.
  * [30] B. R. Marks and G. P. Wright, “A general inner approximation algorithm for nonconvex mathematical programs,” _Operations research_ , vol. 26, no. 4, pp. 681–683, 1978.
  * [31] Z. Wang, H. Li, Z. Wu, and H. Wu, “A pretrained proximal policy optimization algorithm with reward shaping for aircraft guidance to a moving destination in three-dimensional continuous space,” _International Journal of Advanced Robotic Systems_ , vol. 18, no. 1, p. 1729881421989546, 2021.
  * [32] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International conference on machine learning_. PMLR, 2018, pp. 1587–1596.
  * [33] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [34] M. P. Mota, A. Valcarce, J.-M. Gorce, and J. Hoydis, “The emergence of wireless mac protocols with multi-agent reinforcement learning,” in _2021 IEEE Globecom Workshops (GC Wkshps)_. IEEE, 2021, pp. 1–6.


# A degree of image identification at sub-human scales could be possible with
more advanced clusters

Prateek Y J
Sponsored by Microsoft India
<EMAIL_ADDRESS>

###### Abstract

The purpose of the research is to determine if currently available self-
supervised learning techniques can accomplish human level comprehension of
visual images using the same degree and amount of sensory input that people
acquire from. Initial research on this topic solely considered data volume
scaling. Here, we scale both the volume of data and the quality of the image.
This scaling experiment is a self-supervised learning method that may be done
without any outside financing.We find that scaling up data volume and picture
resolution at the same time enables human-level item detection performance at
sub-human sizes.We run a scaling experiment with vision transformers trained
on up to 200000 images up to 256 ppi.

_K_ eywords Machine Learning $\cdot$ Image Recognition

##  1 Introduction

While initial investigations in this domain primarily focused on the scaling
of data volume, the present research takes a bold leap by not only scaling the
data volume but also enhancing the image quality. This ambitious scaling
experiment adopts a self-supervised learning approach that is both resource-
efficient and feasible even without external financial support. Remarkably,
our findings unveil the possibility of achieving human-level image
identification performance at scales below human capabilities, achieved
through the simultaneous scaling of data volume concentration and snapshot
resolution. In order to do this, we conduct a scaling experiment using vision
transformers and train them on a large data set with up to 200000 pictures,
each of which has a resolution of 256 ppi.

##  2 Evaluation Factors and Issues

I answer that query in relation to a particular skill, namely the capacity to
recognize visual images in the real world. It is difficult to directly assess
the data efficiency of deep learning models trained with self supervised
learning methods with regard to people for a variety of explanations.

###  2.1 Inconsistency in dimensions

Inconsistency in dimensions: They model frequently function with significantly
smaller impulses than the human cerebral cortex does, as seen, for instance,
when comparing the size of common images used in computer vision with the
quantity of detectors in the parafovea of an average person.

###  2.2 Inconsistency in Model Scale

When assessing the number of parameters in a model to the number of
neurotransmitters in the functioning model they are often significantly less
than the functioning nervous system, or indeed the visual regions within the
nervous system.

##  3 Control Conditions

We undertake control conditions to find out whether the present Self-Learning
algorithms can reach sub-human image cognizant scales for model scale, and
image dimensions in order to address these discrepancies. In this section we
take into account scaling both components simultaneously and come to a
decision. In the present trials, we additionally train our models on a roughly
two-fold bigger collection of images that resemble real beings.

###  3.1 Experimental data

A total of two-thousand images of human-like footage from four different data-
sets make up the whole collection of training data. The videos have two
distinct characteristics: In contrast to common pictures data-sets in machine
learning, which usually consist of significantly fewer photographs, most of
the content are (i) organic, independent head-cam videos taken from the
perspective of grown-up or toddler camera users as they go about their daily
lives, and (ii) these are time-wise extended, continuous videos with typical
run-times of tens of a few seconds to minutes. The following clip data-sets
make up the merged training set:
The individual picture data-sets that make up our integrated training data set
are broken out in Figure 1 along with their sizes in thousands. Image-Net,
which makes over 50% of our total training data, is by far the largest
contributor. About 31.5% of the training data come from the CelebA dataset,
while 5% come from the CIFAR-10 dataset and 13.5% come from the ADE20K
dataset. As a consequence, we anticipate that people who choose to repeat the
experiments in this study using solely publicly available data sources would
obtain outcomes that are strikingly comparable to those presented here.

We train models on the whole dataset and on constantly randomized random
portions of it in order to investigate the scaling of image identification
performance with the quantity of human-like visual input utilized during self-
supervised pretraining. We explicitly construct models on percentages of the
dataset that are 100 percent, 50 percent, 25 percent and 5 percent. These
selections encompass a 20-fold variance in data size, encompassing around two
hundred thousand to ten thousand captures of labelled images in every instance
(from the biggest through the tiniest). We repeat the subset selection a total
of four times for every data set size because it is stochastic.

####  3.1.1 Image-Net

The Image-Net Large Scale Visual Recognition Challenge (ILSVRC) 2012–2017
image categorization and localization data-set is the most frequently used
subset of Image-Net[Russakovsky et al., 2015]. There are 1,281,167 training
photos, 50,000 validation images, and 100,000 test images in this dataset,
which covers 1000 item classes. On Kaggle, this subset is accessible here.

Figure 1: Image Data-set proportions used for control-Conditions

####  3.1.2 CIFAR-10

Five learning groups and one evaluation batch, each containing 10,000
snapshots, make up the data-set. The remaining photos are distributed across
the training batches in random order, however certain training batches can
have a disproportionate number of images from a particular class.The training
blocks are made up of a total of 5000 images from every class. Access public
database here

####  3.1.3 CelebA

More than 200K celebrity photos with 40 feature annotations each make up the
large-scale face attributes collection known as Celeb Faces Attributes
collection[Zhang et al., 2019]. This collection of photos includes a wide
range of poses and cluttered backgrounds. With 10,177 distinct identities,
202,599 face photos, 5 landmark locations, and 40 binary attribute annotations
per image. Access public database here

####  3.1.4 ADE20K

ADE20K is made up of more than 27K images from the SUN and Locations
databases. The photos have been meticulously annotated with over 3K different
images. Numerous images also show individual parts and pieces of other
sections. Register and access the database here

###  3.2 Models and Checkpoints

In our tests, we only utilize vision transformer (ViT) modeling [Dosovitskiy
et al., 2021]. The models that we optimize include four common sizes: ViT-
Hybrid/S/L/B,. These models, which range in size from the smallest to the
largest model by a factor of twenty eight. We take into account three distinct
spatial resolutions for the picture resolutions: 476 pixels, 448 pixels and
226 pixels.

Our preferred Self Supervised learning method is masked auto-encoders. For the
objectives of this work, masked auto-encoders offer a number of benefits
against conventional self-supervised pictorial representational training
methods. They first require relatively little data augmentation, in contrast
to the majority of existing visual Self Supervised methods. This is helpful
for our objectives because other Self Supervised learning algorithms’
significant data augmentations reduce the human-likeness of the training data.
Second, we select a masking proportion of Eighty percent since masked auto-
encoders perform well with such large masking proportions. According to the
quantity of pictures utilized for self-supervised pre-training, estimated
reliability of validation on Image-Net. Performance grater than ninety percent
denotes performance at the level of a human. Four distinct models are
represented by various colors in the tale. The matches to the first equation
are shown by the solid lines. The expected precision under four fictitious
situations are shown by the lines of best fits,for additional details on these
instances, see Column one . Findings for the more lenient modifying control
are shown in the figure 3, while those for the stricter fine-tuning control
models without fine-tuning are shown in figure 2.
Figure 2 illustrates the findings of our Image Network scalability research.
We use an intuitive polynomial function in logarithmic order to represent the
impact of data size, model size, and picture quality on image recognition
precision:

$Precision=(\beta_{i}+\log_{i}\alpha_{i})(\log_{ppi}\alpha_{ppi}+\beta_{ppi})$
(1)

where i is the scaled data amount which is measured in thousands here and ppi
is the scaled image resolution per test.

Figure 2: Consequences of the validation test accuracy seen in the test with
no fine-tuning  Figure 3:  Consequences of the validation test accuracy seen
in the test with two percent fine-tuning

Table 1 presents three possible outcomes based on the following situations:
(i) a practical reference circumstance matching to our largest and best-
performing model thus far, the VisionTransformer-H model, that was constructed
using all of the approximately 200000 images from the public data.

A possible circumstance where we quadruple 1 each of i, ppi, with respect to
the benchmark situation; a possible scenario where we quadrupled 1 each of i,
ppi with regard to the benchmark case. The image recognition rate for every
one of these situations is shown in Table 1 as the combination of real and
predicted accuracy.

The anticipated precision now surpasses the ninety percent accuracy we have
defined as the lower limit on human-level precision on validation in the
estimated scenario and under the additional lenient finetuning condition. This
result is very encouraging because it only calls for a 522B variable Vision
Transformer model trained with around 200000 images of public data. The
increases in data amount and image resolution needed under this scenario are
also relatively small.

Table 1: Under a real-world reference situation, evaluation scaled data amount and picture resolution, and two forecasts based on adaptations with reference to 2 and 3 Estimation Rate | Data Amount(Thousands) | i (parameters) | Validation Accuracy (%) | Data Resolution(ppi)
---|---|---|---|---
Current Test | 200 | 0.3B | 40 | 256
4 x Test | 2400 | 9.6B | 65 | 1280
10 x Test | 12600 | 522B | 91 | 4096
Figure 4:  Consequences of the validation with human benchmark with no fine-
tuning  Figure 5:  Consequences of the validation with human benchmark with
two percent fine-tuning

##  4 Conclusion

Using highly general self-supervised machine learning techniques and deep
learning architectures without significant inductive biases, our results show
that human-level precision as well as resilience in visual image recognition
can be achieved through anthropomorphic visual perception at sub-human scales
of data amount and image quality.

## Acknowledgments

This was was supported in part by Microsoft with their Azure Credits porgram

## References

  * [Dosovitskiy et al., 2021] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021).  An image is worth 16x16 words: Transformers for image recognition at scale.
  * [Russakovsky et al., 2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015).  Imagenet large scale visual recognition challenge.
  * [Zhang et al., 2019] Zhang, H., Chen, W., Tian, J., Wang, Y., and Jin, Y. (2019).  Show, attend and translate: Unpaired multi-domain image-to-image translation with visual attention.