# Deep Reinforcement Learning for Dynamic Band Switch in Cellular-Connected
UAV

Gianluca Fontanesi1, Anding Zhu1 and Hamed Ahmadi12
1School of Electrical and Electronic Engineering, University College Dublin,
Ireland
2Department of Electronic Engineering, University of York, Heslington, York
YO10 5DD, United Kingdom

###### Abstract

The choice of the transmitting frequency to provide cellular-connected
Unmanned Aerial Vehicle (UAV) reliable connectivity and mobility support
introduce several challenges. Conventional sub-6 GHz networks are optimized
for ground Users (UEs). Operating at the millimeter Wave (mmWave) band would
provide high-capacity but highly intermittent links. To reach the destination
while minimizing a weighted function of traveling time and number of radio
failures, we propose in this paper a UAV joint trajectory and band switch
approach. By leveraging Double Deep Q-Learning we develop two different
approaches to learn a trajectory besides managing the band switch. A first
blind approach switches the band along the trajectory anytime the UAV-UE
throughput is below a predefined threshold. In addition, we propose a smart
approach for simultaneous learning-based path planning of UAV and band switch.
The two approaches are compared with an optimal band switch strategy in terms
of radio failure and band switches for different thresholds. Results reveal
that the smart approach is able in a high threshold regime to reduce the
number of radio failures and band switches while reaching the desired
destination.

###### Index Terms:

UAV, UE, trajectory, Deep Reinforcement Learning, cellular network, sub-6 GHz,
mmWave

3G

    Third Generation
4G

    Fourth Generation
5G

    Fifth Generation
6G

    Sixth Generation
3GPP

    3rd Generation Partnership Project
BB

    Base Band
BBU

    Base Band Unit
BER

    Bit Error Rate
BH

    Backhaul
BPP

    Binomial Point Process
BS

    Base Station
BW

    bandwidth
C-RAN

    Cloud Radio Access Networks
CAPEX

    Capital Expenditure
CDF

    Cumulative Distribution Function
CoMP

    Coordinated Multipoint
CPRI

    Common Public Radio Interface
CU

    Centralized Unit
D2D

    Device-to-Device
DAC

    Digital-to-Analog Converter
DAS

    Distributed Antenna Systems
DBA

    Dynamic Bandwidth Allocation
DDQN

    Double Deep Q-Learning
DL

    Downlink
DNN

    Deep Neural Network
DQN

    Deep Q-Network
DDQN

    Double DQN
DU

    Distributed Unit
FBMC

    Filterbank Multicarrier
FEC

    Forward Error Correction
FH

    Fronthaul
FL

    Federated Learning
FFR

    Fractional Frequency Reuse
FSO

    Free Space Optics
GS

    Ground Station
GSM

    Global System for Mobile Communications
HAP

    High Altitude Platform
HetNet

    Heterogeneous Network
HL

    Higher Layer
HARQ

    Hybrid-Automatic Repeat Request
KPI

    Key Performance Indicator
kg

    Kilogramm
IoT

    Internet of Things
LAN

    Local Area Network
LAP

    Low Altitude Platform
LL

    Lower Layer
LoS

    Line of Sight
LTE

    Long Term Evolution
LTE-A

    Long Term Evolution Advanced
MAC

    Medium Access Control
MAP

    Medium Altitude Platform
MC

    Monte Carlo
MDP

    Markov Decision Process
ML

    Medium Layer
MME

    Mobility Management Entity
mmWave

    millimeter Wave
MIMO

    Multiple Input Multiple Output
ML

    Machine Learning
MSE

    Mean Square Error
NFP

    Network Flying Platform
NFPs

    Network Flying Platforms
NN

    Neural Network
NLoS

    Non-Line of Sight
RL

    Reinforcement Learning
NR

    New Radio
OFDM

    Orthogonal Frequency Division Multiplexing
PAM

    Pulse Amplitude Modulation
PAPR

    Peak-to-Average Power Ratio
PDF

    Probability Density Function
PER

    Prioritized Experience Replay
PGW

    Packet Gateway
PHY

    physical layer
PP

    Poisson Process
PSO

    Particle Swarm Optimization
PTP

    Poin to Point
QAM

    Quadrature Amplitude Modulation
QoE

    Quality of Experience
QoS

    Quality of Service
QPSK

    Quadrature Phase Shift Keying
RAU

    Remote Access Unit
RAN

    Radio Access Network
RB

    Resource Block
RF

    Radio Frequency
RN

    Remote Node
RRH

    Remote Radio Head
RRU

    Remote Radio Unit
RRC

    Radio Resource Control
RRU

    Remote Radio Unit
RSS

    Received Signal Strength
RSRP

    Reference Signals Received Power
RU

    Remote Unit
SCBS

    Small Cell Base Station
SDN

    Software Defined Network
SINR

    Signal-to-Noise-plus-Interference Ratio
SIR

    Signal-to-Interference Ratio
SNR

    Signal-to-Noise Ratio
SON

    Self-organising Network
TETRA

    Trans-European Trunked Radio
TD

    Temporal Difference
TDD

    Time Division Duplex
TD-LTE

    Time Division LTE
TDM

    Time Division Multiplexing
TDMA

    Time Division Multiple Access
UE

    User Equipment
UL

    Uplink
UAV

    Unmanned Aerial Vehicle
UPA

    Uniform Planar Square Array

##  I Introduction

Integrating Unmanned Aerial Vehicles into cellular communication systems as
User Equipments is envisioned as an effective solution to support the UAV’s
mission specific rate-demanding data communication while improving the
robustness of the UAV navigation [1]. This vision of cellular connected UAVs
communication, however, poses new research challenges due to the significant
differences from conventional communication systems. UAVs-UE have typically
higher altitude, higher mobility and have more stringent constraints on the
power and operational time than the corresponding ground ones [2]. In
addition, the existing cellular network operating at sub-6 GHz is bandwidth
limited and perform poorly at high UAV heights, due to the interference
perceived from the down-tilted antennas at the ground Base Stations [3]. As a
consequence, during its trajectory, a UAV is very likely to experience radio
link failures. The UAVs’ mobility and flexibility offer a degree of freedom to
circumvent these issues. The UAV path design that aims to respect a quality-
of-connectivity constraint and minimize the travelling time goes under the
name of Communication-aware trajectory. Several works have optimized the UAV
trajectory under connectivity constraints using graph based [4] or dynamic
programming based solutions [5]. The above traditional optimization solutions
are time consuming and computationally complex. For this reason, Reinforcement
Learning (RL) approaches have been recently investigated. Compared to a
traditional optimization approach, RL is able of making decisions interacting
iteratively with the environment. A double Q-Learning approach is proposed in
[6] to solve a joint trajectory and outage time constraint problem. A Temporal
Difference (TD) learning method is utilized in [7] to design the UAV-UE
trajectory while minimizing the mission completion time and the disconnection
duration.

Besides, exploiting their mobility, UAVs can establish short Line of Sight
(LoS) communication links, that represents an ideal situation to transmit at
millimeter Wave (mmWave) band. A mmWave link offers a wide spectrum and
enables the use of directional beamforming, providing high data rate [8]. The
work in [8] addresses the cellular connected UAV communication-aware
trajectory problem learning simultaneously the mmWave beam and trajectory via
Deep Q-Network (DQN) to improve the Uplink (UL) performance. However, due to
the severe attenuation and sensitivity to blockages, mmWave links are highly
intermittent, leading to frequent radio failures at low UAV altitudes [9].

A promising solution to improve the system robustness is to support two
different frequency ranges, leading to integrated dual mode sub-6 GHz/mmWave
systems [10]. Dual connectivity would allow to exploit the complementary
advantages of both the frequency bands and reduce consistently outage status
through band switch algorithms. Some literature has recently studied the band
switch problem for conventional ground UEs in a 2D environment. Works in
[11]-[12] propose Machine Learning (ML) classifiers that select the best band
based on previous rate measurements within a temporal window. Within the UAV
communication context, in [13] UAVs are equipped with dual band mmWave and
sub-6 GHz communication modules to act as relays to minimize the total service
time, but the switch algorithm between the bands is not investigated.

In this paper, differently from previous works, we aim to solve the
communication aware 3D trajectory problem of UAV-UE proposing a learning
strategy to both manage the trajectory and the band switch policy. We firstly
formulate a trajectory problem in order to minimize the travelling time and
radio failures of the UAV. Then we propose a 3D DQN based algorithm to
dynamically solve it adjusting the position and the operating frequency of the
UAV-UE. Finally, we present also a blind switch strategy and introduce a
baseline optimal band switch policy. We compare the performance of these three
approaches in term of number of radio failures and band switches in the result
section. We observe that the smart approach significantly improves the
performance of the ground BS link for stringent thresholds.

##  II System Model

###  II-A Network Topology

Consider the downlink of a dual band wireless cellular network, that operates
in two frequency bands with frequency $f_{c}$ and bandwidth $B_{c}$,
$c\in\\{1,2\\}$. A set of ground Distributed Units is deployed in an area
$\mathcal{X}\in\mathbb{R}^{3}$, connected to a single Centralized Unit (CU),
and serve a set of $N$ UAV-UEs. Both DUs and UEs are equipped with interfaces
which allow them to transmit at both frequency bands. We assume the
transmission occurs in a single frequency at a time, generating inter-cell
interference to neighboring cells working at the same frequency band. Each DU
is assumed to have $n_{b}$ Resource Blocks and transmits with same power
$P_{DU}^{c}$. Without loss of generality, we focus on a single link to a UAV-
UE (hereafter addressed as UAV).

###  II-B Ground-to-Air Channel Model

In this section we present the path loss, antenna and fading models,
considering $f_{1}$ a lower sub-6 GHz band and $f_{2}$ the mmWave band.

####  II-B1 Path Loss and Antenna Model

The ground-to-air channel model is subject to LoS/ Non-Line of Sight (NLoS)
variations based on the building distribution in $\mathcal{X}$ and UAV height
that lead to blockages. For modeling the path loss $L^{1}(d)$ at sub-6 GHz we
consider the urban Macro (UMa) path loss specified in 3rd Generation
Partnership Project (3GPP) [14]. The path loss $L^{2}(d)$ at mmWave follows a
path loss model [9]:

$L^{2}(d)=\begin{cases}l^{L}(d)=X_{L}d^{-\alpha_{L}};\\\
l^{NL}(d)=X_{NL}d^{-\alpha_{NL}};\end{cases}$ (1)

where $d$ is the ground BS-UAV distance, parameters $\alpha_{L}$,
$\alpha_{NL}$ and $X_{L}$, $X_{NL}$ represent, respectively, the path loss
exponent for LoS/NLoS and the path loss at 1 meter distance. Each DU has three
sectors separated by $None$ as characterized by 3GPP specification [14]. Each
sector is equipped with a vertical $N_{1}$-element uniform linear array (ULA)
at $f_{1}$ and a Uniform Planar Array (UPA) $N_{2}\times N_{2}$ at $f_{2}$
tilted with angle $\phi_{1}$ and $\phi_{2}$. We denote the total ground DU
directional radiation pattern as $G(\Theta,\phi)$, where $\Theta$, $\phi$ are
elevation and azimuth angles. We assume the UAV is equipped with a
conventional omni-directional antenna of unitary gain in any direction to
maintain low complexity and cost at the UAV.

####  II-B2 Fading Model

We model the small scale fading fading power $f_{0,i}^{2}$ with $i$
$\\{LoS,NLoS\\}$ as a Nakagami-m fading model, that covers a wide range of
fading environments, including both sub-6 GHz ($m=1$) and mmWave ($m=3$).
Accordingly, $f_{0}^{2}$ follows a Gamma distribution with
$\operatorname{\mathbb{E}}[f_{0}^{2}]=1$.

###  II-C UAV Mobility Model

The UAV moves at constant speed $V=V_{max}$ along a 3D trajectory of duration
$T$ that can be divided into $K$ discrete segments with interval
$\delta_{k}=T/K$, $k=\\{1,...,K\\}$. $\delta_{k}$ is chosen arbitrarily small
so that within each step the large scale signal power received by the UAV
remains approximately unchanged. Each segment is thus described by its
discrete coordinates $\mathbf{q_{k}}=[x_{k},y_{k},h_{k}]$. The trajectory of
the UAV starts from a random position $\mathbf{q_{I}}\in\mathcal{X}$ in the
given area of interest $\mathcal{X}$, delimited by borders
$[x_{min},x_{max}]\times[y_{min},y_{max}]$ while the altitude of the UAV must
satisfy $h_{k}\in[h_{min},h_{max}]$, where $h_{min}$ represents the lower
bound of UAVs’ altitude to avoid collisions and $h_{max}$ represents the upper
bound. The trajectory ends when the UAV reaches a final destination
$\mathbf{q_{F}}$.

###  II-D Communication Model

The achievable rate $R_{A}^{c}$ at $\mathbf{q_{k}}$ in band $c$ can be
expressed as:
$R_{A}^{c}(\mathbf{q_{k}})=B_{c}\operatorname{\mathbb{E}}(log_{2}(1+SINR^{c}(\mathbf{q_{k}}))),$
where $B_{c}$ is the bandwidth assigned to the link and $SINR^{c}$ the Signal-
to-Noise-plus-Interference Ratio (SINR). At each trajectory step
$\mathbf{q_{k}}$, the UAV associates with the m-th DU at position
$\mathbf{b}_{m}=[x_{m},y_{m},h_{m}]$ providing the highest average Reference
Signals Received Power (RSRP) at the current frequency. Assuming that the
associated cell remains unchanged within each trajectory step, the SINR at the
UAV can be denoted as:

$SINR^{c}(\mathbf{q_{k}},m)=\frac{\gamma_{m}^{c}(k)}{\sigma^{2}+\sum_{j\neq
m}\gamma_{j}^{c}(k)},$ (2)

where $\gamma_{m}^{c}(k)$ is the received power
$\gamma_{m}^{c}(k)=P_{DU}L^{c}(\left\lVert\mathbf{b}_{m}-\mathbf{q}_{k}\right\rVert_{2})f_{0}^{2}(\mathbf{q}_{k})G(\Theta,\phi)$
and terms in the denominator are, respectively, the spectral noise and the
interference. Note that a subscript to emphasize the $\\{LoS,NLoS\\}$
dependence of $L()$ and $f$ has been omitted to lighten the notation.

To maintain a reliable connection, DU-UAV link must satisfy a rate threshold
$R_{TH}$. We define a radio failure on the link if
$R_{A}^{c}(\mathbf{q_{k}})<R_{TH}$ occurs at step $k$. In addition, we
introduce a failure indicator as:

$F(\mathbf{q_{k}})=\begin{cases}1,&\mbox{if
}R_{A}^{c}(\mathbf{q_{k}})<R_{TH}\\\ 0,&\mbox{otherwise}.\end{cases}$ (3)

To avoid a radio failure, at each discrete step $k$ the UAV may require the CU
a band switch to change the transmission frequency. Prior to formulating the
problem, in what follows we introduce the band switch procedure and
signalling.

###  II-E Band switch procedure

The band switch policy industry standard for a conventional ground UE is
composed of several iterative steps which can be summarized as follows : 1) if
the serving rate $R_{A}^{c}$ at current frequency drops below threshold
$R_{TH}$, the UE initiates a band switch procedure with the serving cell, 2)
the UE utilizes a time $\Delta_{t}$ to measure if a cell at target frequency
offers a higher RSRP than the serving cell, 4) if yes, the UE triggers a band
switch procedure and reports the measurement back to the DU, 5) the CU makes a
decision on the suitable DU at target frequency for band switch, based on the
UE measurements. Applying the above policy to UAV communications would
introduce several challenges. The mobility of the UAV might cause deep RSRP
fluctuations between two consequent trajectory steps $\mathbf{q_{k}}$ and
$\mathbf{q_{k+1}}$. This might lead to subsequent band switch procedures and
potential ping-pong effect. Note that, due to the measurement gap, in case of
several band switches the effective throughput at the UE suffers a significant
reduction [12]. Thus, we modify the above band switch procedure to adapt it
for a UAV moving between cells belonging to different DUs controlled by a
central CU. Here, motivated by the above mentioned challenges, we formulate a
real-time trajectory and band switch procedure exploiting the capabilities of
a RL based policy. In the next sections we formulate the problem and propose
two DQN algorithms for joint UAV-UE trajectory and band switch.

###  II-F Problem formulation

The goal of the UAV is to reach the destination in the minimum amount of steps
while minimizing the number of radio failures. At the same time, to avoid a
radio failure and meet the quality of service requirement, at the beginning of
each discrete step $k$, the operating frequency can be switched between
$f_{1}$ and $f_{2}$. Denoting as $K$ the trajectory steps, $N_{s}$ the number
of band switches, the above problem can be mathematically formulated as:

$\displaystyle\min_{N_{s},K,\mathbf{q}_{k}}\quad$
$\displaystyle\kappa_{1}\sum_{k=1}^{K}F(\mathbf{q_{k}})+\kappa_{2}K+\kappa_{3}N_{s}$
(4a) s.t. $\displaystyle\mathbf{q}_{0}=q_{I},$ (4b)
$\displaystyle\mathbf{q}_{K}=q_{F},$ (4c)

where $\kappa_{1},\kappa_{2},\kappa_{3}$ weight the summation of the
arguments, (4b), (4c) are the starting and final positions. Problem (4) cannot
be readily handled by conventional optimization methods due to the non trivial
design of $R_{A}^{c}(\mathbf{q_{k}})$, function for each position of the
fading, and the antenna model described in Section II.

To tackle this challenging problem, we propose a novel approach by leveraging
the DQN technique. DQN based frameworks are able to solve the joint trajectory
and band switching problem by exploring different actions, and without the
need of prior knowledge or predefined dataset to train the network.

##  III Band Switching as a Deep Reinforcement Learning Problem

In the standard RL setting an agent acts in an environment over discrete time
steps. Given that problem (4) has been formulated in a discrete form, we can
directly map it into a Markov Decision Process (MDP) 5-tuple
$\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$, with
space state $\mathcal{S}$, action space $\mathcal{A}$, state transition
probability $\mathcal{P}(s^{\prime}|s,a)$, reward function
$\mathcal{R}(s,a,s^{\prime})$, and discount factor $\gamma\in[0,1)$. At each
time step $k$, the agent receives an observation $o_{k}$ and selects an action
$a_{k}$, receives reward $r_{k+1}$ from the environment and goes from state
$s_{k}$ to a new state $s_{k+1}$ where the cycle restart. We assume an
episodic setup, where an episode starts with time step $0$ and concludes with
time step $K$. Here the states consist in the 3D location of the UAV
$\mathbf{q_{k}}$ and band switch indicator $w_{k}\in\\{0,1\\}$, such as
$s_{k}=\\{\mathbf{q_{k}},w_{k}\\}$ $\in\mathcal{S}$. Both the location and the
band switch indicator are the input of the neural network. The agent, placed
at the CU to overcome the limited onboard computing capacity, selects an
action in a set of $N$ actions $\\{a_{1},...a_{N}\\}$ in the action space
$\mathcal{A}$. The action can be represented by parameters
$a_{k}=\\{\eta_{k},w_{k}\\}$ where $\eta_{k}$ corresponds to the movements of
the UAV in altitude (up or down) or in the horizontal direction (left-right-
forward-back). In addition, if $w_{k}$ is positive, the CU triggers a band
switch in the network operating frequency without a measurement gap. Based on
the selected action the UAV moves in the desired direction for a distance
$\delta_{k}$ at speed $V_{max}$ reaching the position $\mathbf{q_{k+1}}$. Our
goal is to predict the UAV direction along with the transmitting frequency to
avoid radio failures and reach the destination using the current rate
measurement. With this aim we define as reward function as:
$r_{k}=\lambda_{1}\mathcal{F}_{1}+\lambda_{2}\mathcal{F}_{2}+\lambda_{3}\mathcal{F}_{3}.$
$\mathcal{F}_{1}$ is a negative cost and forces the UAV to reach the
destination in the minimum amount of steps. $\mathcal{F}_{2}$ denotes the cost
for entering in low normalized rate regions
($R_{TH}/R_{A}^{c}(\mathbf{q_{k}})$), that lead to radio failures.
$\mathcal{F}_{3}$ accounts for the band switches occurred until time step $k$.
Note that $\mathcal{F}_{3}$ is a cost since a high number of band switches is
not desirable. Thus, $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ balance the
impact of the three factors on the reward function and need to be properly
adjusted.

In order to achieve the desired goal, we need to find a policy that maximizes
the long-term average of both connectivity and path guidance. We recall that a
policy $\pi(s_{k},a_{k})$ is defined as the probability of taking action
$a_{k}=a$ in state $s_{k}=s$, through interacting with the environment. Based
on this, the problem is equivalent to find a policy $\pi(s_{k},a_{k})$ that
maximizes a cumulative discounted reward over the long run:

$G_{k}=\sum_{i=0}^{K-1}\gamma^{i}r_{k+i+1}.$ (5)

The discount factor $\gamma$ in (5) regulates the relative importance of
future and immediate rewards.

After being formulated as an MDP, Problem (4) can now be solved via different
RL algorithms.

###  III-A DDQN for UAV-UE Trajectory and Smart Band Switch

In the convention RL approach, for a given policy $\pi$, a state-action
function $Q(s,a)$ is introduced to represent the expected return after taking
action $a_{k}$ in state $s_{k}$. In order to deal with large state/action
space dimensions DQN was introduced in [15] to approximate the optimal
Q-function using a Deep Neural Network (DNN) such that $Q(s,a;\theta)\approx
Q(s,a)$ where $\theta$ are the network’ weights. The transitions
$(s,a,r,s^{\prime})$, with $s^{\prime}=s_{k+1}$ are stored into a memory of
finite capacity size $H$ and randomly picked when performing a Q-value update.
At each iteration, DQN is trained to minimize the loss:

$L(\theta)=E[(y_{k}-Q(s_{k},a_{k}|\theta_{k})^{2})]$ (6)

where $y_{k}$ is the target function given by
$y_{k}=r_{k}+\gamma\max_{a}Q(s_{k+1},a|\theta^{-})$, where $\theta^{-}$ is
introduced following the target network mechanism. We denote the primary DNN
network weight matrix and target DNN weight matrix as $\theta$ and
$\theta^{-}$. We consider a fully connected DNN for both networks and the DNN
parameters $\theta^{-}$ are updated with the parameters $\theta$ every fixed
number of steps $C$ to improve the convergence of the algorithm. Among the
improvements and extensions of the baseline DQN algorithm, we implement Double
DQN (DDQN) which reduces the overstimation bias of DQN with a simple
modification of the update rule. In particular, the target function becomes:

$y_{k}=r_{k}+\gamma\max_{a}Q(s_{k+1},\operatorname*{arg\,max}Q(s_{k+1},a_{k+1},\theta_{k}),\theta_{k}^{-}).$
(7)

In Algorithm (1) we introduce the complete pseudocode.

1

2Initialize: maximum number of episodes, the replay memory $H$ with capacity
$N$, mini batch size $B$ ;

3 the DQN $Q$ network with coefficients $\theta$, the target network
$\tilde{Q}$ with coefficients $\theta^{-}=\theta$;

4 Algorithm hyperparameters ($\epsilon$,
$\lambda_{1}$,$\lambda_{2}$,$\lambda_{3}$,$\gamma$), Rate Threshold $R_{TH}$;

5 for  _episode = 1,…,Max episode_ do

6 Initialize $q_{0}=\\{\mathbf{q_{I}}\\}\in\mathcal{S}$, $w=0$, set step
$k\xleftarrow[]{}0$;

7 for  _each step of episode_ do

8 if  _$rand() <\epsilon$_ then

9 select action $a_{k}$ randomly ;

10 else

11 choose action
$a_{k}=\operatorname*{arg\,max}Q(\mathbf{s_{k}},a_{k},\theta)$

12 end if

13

14 end if

15 Agent execute action $a_{k}$, observe $\\{\mathbf{q_{k+1}},w_{k}\\}$;

16 if  _$w_{k}==1$_ then

17 $f_{c}=!f_{c}$ (Band Switch)

18 end if

19 if  _Destination reached_ then

20 Terminate Episode ;

21

22 end if

23 Store transition $\\{\mathbf{s}_{k}$, $a_{k}$, $r_{k}$, $\mathbf{s}_{k+1}$}
in $H$;

24 Compute reward $r_{k}$;

25 Randomly sample mini batch from replay memory $H$;

26 Perform a gradient descent using (6) and (7);

27 Update $\tilde{Q}=Q$ every $C$ steps;

28

29 end for

30

31 end for

Algorithm 1 DDQN Algorithm for Band Switch Trajectory

####  III-A1 Analysis of the Smart Switch Algorithm

The proposed algorithm aims to solve the challenges of the state of the art
band switch policy presented in Section II-E. After the initialization
procedures, the agent performs an initial exploration of the state space
through an $\epsilon$-greedy policy (lines ($7-12$) in Algorithm (1)). The UAV
moves in a custom environment consisting of the area under consideration
$\mathcal{X}$ covered by the dual band network, the possible UAV locations and
discrete action space. At each episode, the environment resets the source UAV
location, computes new observations and rewards based on the desired action
until it reaches destination. Since the band switch indicator is included as
input state of the DNN, goal of the agent is to learn by experience in which
states bandswitching is beneficial, without the need for measurement of gaps.

Next, we introduce a Blind DQN Algorithm for the joint UAV-UE Trajectory and
Band Switch and an Optimal Algorithm.

###  III-B DDQN for joint UAV-UE Trajectory and Blind Band Switch

To solve the optimization problem (4) we propose also a Blind algorithm, that
focuses on learning the best trajectory to reach the destination and switches
the transmitting frequency any time a radio failure occurs. Differently from
the Algorithm presented in Section III-A, the states consist of the 3D
location of the UAV and the action in the direction only. In addition, in the
reward function, $F_{3}=0$. The Algorithm pseudocode is omitted here since it
is similar to the one presented in Algorithm 1. This band switch algorithm
without information about the instantaneous rate at both frequencies, cannot
guarantee that the throughput after the band switch will be higher than the
original one.

###  III-C Optimal DDQN for UAV-UE Trajectory

Finally, as benchmark, we introduce an Optimal Algorithm. At each trajectory
step $\mathbf{q_{k}}$ the agent knows the instantaneous achievable rate of the
two different bands. Therefore, the CU at each step coordinates the DUs to
transmit at the best frequency, minimizing the radio failures due to a wrong
choice about the band.

##  IV Simulation Results

In this section we evaluate the performance of the proposed DQN algorithms and
we benchmark them with the Optimal Policy.

Parameter |  Description |  Value
---|---|---
$h_{max}$/$h_{min}$ |  UAV Max/Min Height |  120/60 m
$N_{1}$/$N_{2}\times N_{2}$ |  Antenna Element $f_{1}$/$f_{2}$ |  8/64
$\phi_{1}$/ $\phi_{2}$ |  Antenna Tilt $f_{1}$/$f_{2}$ |  -10/10 $\deg$
$\sigma^{2}$ |  Noise Power $f_{1}$/$f_{2}$ |  -204/-120 db/Hz
$\delta_{k}$ |  time step length |  0.5 s
$\epsilon$ |  $\epsilon$-greedy variable |  0.4
$K$ |  UAV Max Moves |  200
$\gamma$ |  discount factor |  1
$H$ |  Replay Memory Size |  100000
$\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ |  Rewards weights |  0.1, 0.8, 0.1

TABLE I: Simulation parameters

We consider a deployment of 5 dual band DUs in an area of side $2$ km where
buildings are modeled as for the ITU-R urban statistical model [16] with
maximum height of $50$m. The two frequency bands considered are $f_{1}=2$ GHz
and $B_{1}=180$ kHz, and $f_{2}=28$ GHz and $B_{2}$ = 1800 kHz. We use one RB
at sub-6 GHz while ten at mmWave [12]. We choose to transmit at $0.1$ W at
mmWave and at $1$ W at sub-6 GHz. In the proposed DQN algorithms, the DNN
consists of input layer, four hidden layers, one output layer, all fully
connected feedforward, activated using Rectified Linear Units (ReLU) and
trained with Adam optimizer. Each training run was conducted using a batch
size of $32$ for a minimum of $3500$ iterations. Table I presents all
simulation parameters.

###  IV-A Simulation Environment

In Figure 1(a), we show the rate trend along with the same trajectory episode
at sub-6 GHz and mmWave.

(a)

(b)

Figure 1: Evolution of the UAV-UE rate at sub-6 GHz and mmWave along one
single trajectory episode (a). Samples are collected at each step k, according
to the framework described in Section II-D. In (b) we show the Radio Failures
for a 3D trajectory at sub-6 GHz and mmWave

The rate has sudden drops both at sub6GHz and mmWave, due to the presence of
fading and obstacles. In particular, at mmWave, the rate can be very high or
very low, due to the higher bandwidth and, at the same time, high sensitivity
to blockages. In Figure 1(b) we show the total percentage of detachments
completing the UAV missions at one frequency only, sub-6 GHz or mmWave band,
and without any band switch mechanism. Intuitively, the number of failures
increases as we increase the rate threshold. Both the above results motivate
the band switching approach presented in this paper.

###  IV-B Band Switch Policies

The UAV-UE starts its trajectory at sub-6 GHz and can switch to the 28 GHz
band. We consider a low threshold of $150\text{Kbps}$ at sub-6 GHz and
$3\text{Mbps}$ at mmWave (Treshold 1), a threshold of $300\text{Kbps}$ at
sub-6 GHz and $3\text{Mbps}$ at mmWave (Treshold 2) and a high Threshold 3 of
$400\text{Kbps}$ at sub-6 GHz and $4\text{Mbps}$ at mmWave. In Figure 2(a) we
compare the blind, the smart and optimal switch algorithms for increasing
thresholds in term of average percentage of radio failures.

(a) Radio Failures for the Blind and Smart approaches, compared with an
Optimal policy

(b) CDF of number of switches for the Blind and Smart Algorithm with
thresholds 1 and 3

Figure 2: Performance of the band switch algorithms

We observe that at low thresholds, the smart approach performs similarly to
the blind approach. By increasing the threshold, and as consequence, the
probability of a radio failure, we observe a higher increase in the number of
radio failures for the blind approach. While the blind approach switches the
band anytime a radio failure occurs, the smart one offers the agent a generic
framework that, interacting with the environment, learns to minimize the
number of radio failures switching the band only when the target frequency
provides higher throughput. As confirmation of this, in Figure 2(b) we plot
the Cumulative Distribution Function (CDF) of the number of band switches of
the blind and smart approach for Threshold 1 and 3. We observe that in a high
throughput regime, the smart approach outperforms the blind approach and can
significantly reduce the number of switches.

##  V Conclusion

In this paper, we propose a DQN based approach to solve a communication aware
trajectory problem for cellular UAV. In order to minimize the number of radio
failures while reaching its target destination, we consider the UAV to exploit
band switch policies without prior knowledge of the environment. We define and
compare two joint trajectory and band switch DQN algorithms, that
differentiate based on how the agent decide to switch the transmitting
frequency. We show that a smart switch approach, where the band switch
indicator is included as input state of the DNN, is able to outperform a blind
approach in a high threshold regime.

## Acknowledgment

This work was supported by the Irish Research Council under Grant
GOIPG$/2017/1741$ and in part by the Science Foundation Ireland under Grant
Numbers $13/$RC$/2077$ and $17$/NSFC/$4850$.

## References

  * [1] Y. Zeng, J. Lyu, and R. Zhang, “Cellular-Connected UAV: Potential, Challenges, and Promising Technologies,” _IEEE Wireless Commun._ , vol. 26, no. 1, pp. 120–127, 2018.
  * [2] M. Mozaffari _et al._ , “A tutorial on UAVs for wireless networks: Applications, Challenges, and Open Problems,” _IEEE commun. surveys & tutorials_, vol. 21, no. 3, pp. 2334–2360, 2019.
  * [3] G. Geraci _et al._ , “Understanding UAV cellular communications: From existing networks to massive MIMO,” _IEEE Access_ , vol. 6, pp. 67 853–67 865, 2018.
  * [4] S. Zhang and R. Zhang, “Trajectory Design for Cellular-Connected UAV under Outage Duration Constraint,” in _Proc. IEEE International Conference on Communications (ICC)_ , May 2019.
  * [5] E. Bulut and I. Guevenc, “Trajectory Optimization for Cellular-Connected UAVs with Disconnectivity Constraint,” in _Proc. IEEE International Conference on Communications (ICC)_ , May 2018.
  * [6] B. Khamidehi and E. S. Sousa, “A Double Q-Learning Approach for Navigation of Aerial Vehicles with Connectivity Constraint,” in _Proc. IEEE International Conference on Communications (ICC)_ , Jun. 2020\.
  * [7] Y. Zeng and X. Xu, “Path Design for Cellular-Connected UAV with Reinforcement Learning,” in _Proc. IEEE Global Communications Conference (GLOBECOM)_ , Dec. 2019.
  * [8] P. Susarla _et al._ , “Learning-Based Trajectory Optimization for 5G mmwave Uplink UAVs,” in _Proc. IEEE International Conference on Communications (ICC)_ , Jun. 2020.
  * [9] G. Fontanesi, A. Zhu, and H. Ahmadi, “Outage Analysis for Millimeter-Wave Fronthaul Link of UAV-Aided Wireless Networks,” _IEEE Access_ , vol. 8, pp. 111 693–111 706, June 2020.
  * [10] O. Semiari _et al._ , “Integrated Millimeter Wave and Sub-6 GHz wireless Networks: A Roadmap for Joint Mobile Broadband and Ultra-Reliable Low-Latency Communications,” _IEEE Wireless Commun._ , vol. 26, no. 2, pp. 109–115, April 2019.
  * [11] D. Burghal, R. Wang, and A. F. Molisch, “Deep Learning and Gaussian Process based Band Assignment in Dual Band Systems,” _arXiv preprint arXiv:1902.10890_ , 2019.
  * [12] F. B. Mismar _et al._ , “Deep Learning Predictive Band Switching in Wireless Networks,” _IEEE Trans. Wireless Commun._ , Sept. 2020.
  * [13] H. Ghazzai _et al._ , “Trajectory Optimization for Cooperative Dual-Band UAV Swarms,” in _Proc. IEEE Global Communications Conference (GLOBECOM)_ , Dec. 2018.
  * [14] “Technical specification group radio access network; study on enhanced LTE support for aerial vehicles (release 15),” 3GPP, TR 36.777, Dec 2017.
  * [15] V. Mnih _et al._ , “Human-Level Control Through Deep Reinforcement Learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [16] _Propagation Data and Prediction Methods Required for the Design of Terrestrial Broadband Radio Access Systems Operating in a Frequency Range From 3 to 60 GHz_ , ITU-R Recommendation P.1410-5, 2012\.

  *[UAV]: Unmanned Aerial Vehicle
  *[UAVs]: Unmanned Aerial Vehicle
  *[UE]: User Equipment
  *[RL]: Reinforcement Learning
  *[TD]: Temporal Difference
  *[LoS]: Line of Sight
  *[mmWave]: millimeter Wave
  *[DQN]: Deep Q-Network
  *[UL]: Uplink
  *[UEs]: User Equipment
  *[ML]: Machine Learning
  *[BS]: Base Station
  *[CU]: Centralized Unit
  *[DUs]: Distributed Unit
  *[DU]: Distributed Unit
  *[NLoS]: Non-Line of Sight
  *[3GPP]: 3rd Generation Partnership Project
  *[SINR]: Signal-to-Noise-plus-Interference Ratio
  *[RSRP]: Reference Signals Received Power
  *[MDP]: Markov Decision Process
  *[DNN]: Deep Neural Network
  *[DDQN]: Double DQN
  *[RB]: Resource Block
  *[CDF]: Cumulative Distribution Function