Unmanned Vehicles in 6G Networks: A Unifying Treatment of Problems, Formulations, and Tools

Winston Hurst, Spilios Evmorfos, Athina Petropulu, Yasamin Mostofi

Winston Hurst and Yasamin Mostofi are with the Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA 93117 USA (email: {winstonhurst, ymostofi}@ece.ucsb.edu).

Spilios Evmorfos and Athina P. Petropulu are with the Department of
Electrical and Computer Engineering, Rutgers, The State University of New
Jersey, Piscataway, NJ 08854 USA (email<EMAIL_ADDRESS>
This work was supported in part by NSF RI award 2008449 and ARO under Grants W911NF2110071 and W911NF2320103

Unmanned Vehicles (UVs) functioning as autonomous agents are anticipated to play a crucial role in the 6th Generation of wireless networks. Their seamless integration, cost-effectiveness, and the additional controllability through motion planning make them an attractive deployment option for a wide range of applications. However, despite their potential, the convergence of UVs and wireless systems brings forth numerous challenges that require attention from both academia and industry.  This paper then aims to offer a comprehensive overview encompassing the transformative possibilities as well as the significant challenges associated with UV-assisted next-generation wireless communications. Considering the diverse landscape of possible application scenarios, problem formulations, and mathematical tools related to UV-assisted wireless systems, the underlying core theme of this paper is the unification of the problem space, providing a structured framework to understand the use cases, problem formulations, and necessary mathematical tools.  Overall, the paper sets forth a clear understanding of how unmanned vehicles can be integrated in the 6G ecosystem, paving the way towards harnessing the full potentials at this intersection.

Unmanned Vehicles, 6G, wireless communications, autonomy

§ INTRODUCTION

6G cellular systems are envisioned to make significant progress in the advancement of wireless communication technology. This has caused much excitement and anticipation in the scientific and high-tech communities, as 6G wireless networks promise unparalleled capabilities that surpass those of 5G systems [1, 2, 3]. In addition to improving reliability, latency, and capacity by orders of magnitude, 6G will introduce new dimensions to system services, including those related to sensing, artificial intelligence (AI), and positioning, all while responding to societal challenges such as climate change and inequity through an emphasis on sustainability and closing the digital divide [4].

List of Key Abbreviations.

3GPP     Third Generation Partnership Project

5G     Fifth generation

6G     Sixth generation

AV     Autonomous Vehicle

BCD     Block Coordinate Descent

BS     Base Station

DRL     Deep Reinforcement Learning

HAPS     High Altitude Platform Station

ITU     International Telecoms Union

LEO     Low Earth orbit

LoS     Line-of-sight

MIMO     Multiple-input, multiple output

mmWave     millimeter Wave

MIP     Mixed-integer Program

MILP     Mixed-integer Linear Program

MINLP     Mixed-integer Nonlinear Program

NLoS     Non-LoS

NOMA     Non-Orthogonal Multiple Access

OFDMA     Orthogonal Frequency Division Multiple Access

OMA     Orthogonal Multiple Access

PD-NOMA     Power-Domain NOMA

PRB     Physical Resource Block

QoS     Quality of Service

RL     Reinforcement Learning

RSMA     Rate-Splitting Multiple Access

SCA     Successive Convex Approximation

SDMA     Space division Multiple Access

SOCP     Second-order Cone Program

TBS     Terrestrial Base Station

TDMA     Time Division Multiple Access

THz     Terahertz

UAV     Unmanned Aerial Vehicle

UV     Unmanned Vehicle

UV-BS     UV acting as a BS

There are a number of inter-related challenges to overcome before realizing the full 6G vision. Some of these are posed directly by the stated network capabilities. How can ubiquitous coverage be achieved in areas where establishing network infrastructure is difficult, such as in maritime environments or in rural areas [5]? Others arise as we consider use cases. For instance, how can throughput, reliability, and latency improvement be guaranteed in dense Internet of Everything (IoE) networks [2], or for extended reality applications [6]. Still more challenges are found as we consider issues associated with specific technologies, such as accounting for the acute attenuation and penetration loss of THz communication [7].

Amidst these challenges, one transformative player emerges in the 6G landscape: unmanned vehicles. Autonomous, unmanned vehicles are envisioned to have a crucial impact in shaping the future of 6G cellular systems, enabling them to overcome the limitations of conventional communication paradigms. For instance, they are anticipated to facilitate novel use cases, enhance network efficiency, improve system intelligence and resilience, and further support the high demands of truly immersive applications.

In this paper, we lay out a critical exploration of the dynamic intersection of unmanned vehicles and 6G cellular systems, aiming to provide a comprehensive and methodical survey that encompasses the underlying challenges, the associated problem formulations, and mathematical tools essential for their integration. At the core of this research endeavor lies a central theme — the unification of the problem space. When considering the integration of unmanned vehicles with 6G systems, researchers encounter many different scenarios, several different problem formulations, and an array of mathematical tools. This makes it considerably challenging to comprehend the full spectrum of possibilities in this area and harness them effectively. To address this, this paper takes a fresh perspective by offering a unifying approach. More specifically, it provides a structured framework, with an emphasis on unifying the scenarios, underlying problem formulations, and necessary mathematical tools, in order to enable successful navigation of this complex domain. By doing so, we aim to equip researchers, engineers, and stakeholders with a clear understanding of how unmanned vehicles can be integrated in the 6G ecosystem, which can facilitate the development of innovative solutions that effectively utilize the opportunities presented at this intersection.

We next provide a brief summary of the envisioned capabilities as well as supported applications that 6G can provide, followed by an overview of various types of unmanned vehicles relevant to the 6G ecosystem. This then paves the way for our methodical characterization of the roles UVs can play in the 6G landscape. We end the section with a summary of our contributions.

§.§ 6G: Use Cases and Enabling Technologies

Recent years have seen much discussion and speculation about beyond-5G communication networks, their capabilities, enabling technologies, and supported applications. In June 2023, the International Telecoms Union (ITU) published the IMT-2030 Framework [8], which lays out an official road map for 6G standardization and brings a level of concreteness to the discussion. Core use cases, performance capabilities that support these, and technologies that, in turn, enable the cited performance, are briefly introduced below.

Key performance indicators (KPI) for 5G [9] and 6G [4].

§.§.§ Use Cases and Applications

The ITU envisions six core usage scenarios for 6G networks, three of which are extensions of 5G capabilities. Immersive communication expands upon enhanced mobile broadband (eMBB), with the capacity to support AR and VR experiences [6]. Massive communication enhances massive machine-type communication (mMTC) by handling very high traffic loads generated by huge numbers of heterogeneous sensors and devices [10], with envisioned application in smart cities [11] and agriculture [12], to name a few. Moving from 5G to 6G will raise Ultra Reliable and Low-Latency Communication (URLLC) to Hyper Reliable and Low-Latency Communication (HRLLC), with performance sufficiently robust to support autonomous vehicles in industry 4.0 [3] and remotely controlled medical robotics [13], among other applications.

Three other core use cases are new in 6G. Integrated sensing and communication (ISAC) will allow communication signals to also provide information about the physical environment, a feature useful for, e.g.,  autonomous vehicles navigating through dynamic urban areas [14]. Ubiquitous connectivity will close the digital divide, providing cellular and internet access to under-served areas. Finally, the 6G network will empower integrated artificial intelligence and communication, facilitating intelligent, data-driven decision-making for many applications, from smart cities [15] to self-driving cars [16].

§.§.§ Enabling the 6G Vision

Realizing the 6G vision will depend on underlying technical improvements to the network. Figure <ref> illustrates several key performance indicators (KPI) and calls out how these improve over current 5G metrics. To make this enhancement possible, a number of important emerging technologies will be employed. We next highlight a few of these.

A high-level view of the various roles UVs play in the 6G ecosystem.

The use of high frequency bands creates greater capacity by permitting larger bandwidths for communication. In 6G, the rollout of mmWave that began in 5G will mature, and sub-THz and THz communication links will come online. However, these spectra bring challenges in a number of dimensions, including hardware design, channel modeling, and rapid signal attenuation [7].

Massive multiple-input, multiple-output (mMIMO) plays a key role in 5G, and multi-antenna techniques will continue to be key enablers in 6G through extreme MIMO (eMIMO) [4]. More specifically, higher communication frequencies have smaller wavelengths, and antenna sizes shrink proportionally. This allows very high numbers of them to fit on a single array. Beamsteering can then be used to overcome the fast attenuation of the signal or to provide access to multiple users simultaneously by spatial multiplexing [17].

Reconfigurable intelligent surfaces (RIS) or meta-surfaces (MS) [18, 19, 20] will provide similar beamforming benefits, but in a passive way. These artifacts can be deployed in the environment to dynamically alter the channel quality between nodes by intelligent manipulation of impinging EM fields. By steering signals around large obstacles in the environment, they can ameliorate issues that arise when no line of sight (LoS) path exists between a transmitter and receiver [21].

The high connection density requirements specified in IMT-2030 (see Figure <ref>) will necessitate new multi-access schemes.  Emerging candidates, such as Power-Domain Non-Orthogonal Multiple Access (PD-NOMA) [22] and Rate-Splitting Multiple Access (RSMA) [23], are promising methods which can leverage more capacity out of limited communication resources (time, frequency, and space) through intelligent signal processing and interference management.

Artificial intelligence (AI) can enhance almost all aspects of the 6G ecosystem, whether for traffic prediction, channel learning, or resource management. These methods are desirable because they can handle the increased complexity of the ecosystem while also allowing for flexible operation to match the dynamic communication environment [24].

Finally, the realization of the 6G vision depends on diverse network topographies, as seen in Figure <ref>. For example, Space-Air-Ground Integrated Networks (SAGIN) and Non-Terrestrial-Networks will be crucial in providing connectivity in scenarios where terrestrial-based cell systems fall short. In this context, unmanned vehicles (UVs), especially UAVs and HAPS, will play a key role in realizing the 6G vision, which is the main motivation for this survey paper. For instance, by leveraging mobility and strategic positioning, they can facilitate dynamic service provisioning, adapting to changing network conditions and traffic patterns, while also improving link quality through direct line-of-sight channels. We next discuss different types of UVs in more detail, before providing a more thorough discussion of their place in 6G and beyond.

6G: Enabling technologies and enabled applications.

§.§ Unmanned Vehicles

While the history of unmanned mobile infrastructure in communication systems traces back to the 1940s [25], the utilization of UVs in wireless systems has predominantly been confined to specialized military applications. The advent of the next generation of wireless networking (encompassing Beyond-5G and 6G systems [26]), necessitates adaptable and mobile infrastructure capable of meeting new KPIs. UVs are envisioned to play an integral role in these advanced wireless systems, unlocking the potential of wireless autonomy by seamlessly integrating components capable of efficient sensing, decision-making, and communication.

Fundamentally, UVs can operate on land, water or in the air. Each medium imposes distinct design considerations. Given their versatility in deployment across various settings, we put a great emphasis on Unmanned Aerial Vehicles (UAVs) in this paper. However, we also address the relevance of Unmanned Ground Vehicles (UGVs) as consumers of 6G services. Notwithstanding their importance, our discussions omit underwater Unmanned Vehicles due to the inherently unique characteristics of communication methods needed in underwater conditions.

6G UAVs, categorized by operational characteristics [27, 28], exhibit altitude-based distinctions. Within this framework, three primary groups emerge:
High-level Aerial Platforms (HAPS) operate at altitudes ranging from 15km to 25km above ground level, facilitating communication over a radius of 30km and maintaining extended flight duration.
Mid-level Aerial Platforms (MAPS) navigate altitudes spanning 5km to 15km, covering a communication area with a radius of 5km. These platforms, smaller than HAPS, serve as relays connecting higher-altitude HAPS with lower-altitude ground users or UVs.
Low-level Aerial Platforms (LAPS) encompass UVs operating at altitudes below 5km above ground level. Generally smaller than counterparts in the other categories, LAPS provide a narrower area of coverage.

UAVs are also categorized in terms of size into distinct subgroups. Micro Aerial Vehicles (MAV) weigh between 50g and 2kg, offering a typical flight range under 10km. Small unmanned Aerial Vehicles (sUAV), within the weight range of 2kg to 50kg, typically feature a 2m wingspan and possess battery capacity supporting a flight range of approximately 15km. Conventional UAVs, spanning weights from 5kg to 150kg, are equipped with battery life sufficient for flight distances ranging from 10km to around 250km. Lastly, Personalized Aerial Vehicles (PAV), designed for large-scale transportation and cargo delivery, exhibit weights from 140kg up to 1500kg and often boast a flight range that exceeds 250km

Regarding the wing type, UAVs can be equipped with fixed, flapping or rotary wings of varying sizes, depending on the requirements of the flight mission.

The employment of UVs in 6G networks comes with the promise of seamlessly integrating Terrestrial and Non-Terrestrial Networks in the pursuit of connecting both humans and machines [29]. The mobility of UVs enables increased throughput and decreased latency, since the introduced controllable degrees of freedom can be leveraged in order to adjust to fluctuating conditions of network traffic and communication channels. Furthermore, they can assist in achieving ubiquitous and uninterrupted connectivity by dynamically adjusting link quality. Finally, the dynamic motion of UVs promises extension of the network coverage that is being realized by the current immovable infrastructure (static base stations and relay antennas), a direction consistent with the vision of closing the digital divide [30]. In the next section, we provide a framework for understanding the role of UVs in next-generation communication networks.

§.§ Unmanned Vehicles in the 6G Ecosystem

The use of UVs can both enable and be enabled by 6G networks. Throughout this paper, we refer to the UV as a provider when its purpose is to aid in the realization of the promises of 6G, and we call it a consumer when it depends on the network's capabilities to complete a mission. This duality serves as an organizing principle through the rest of this paper.

§.§.§ UVs as Providers

Using UVs as providers can address many important challenges. As part of space-air-ground integrated networks (SAIGN), HAPSs can help close the digital divide by obviating expensive build-out of terrestrial infrastructure in rural areas, or provide connectivity to maritime nodes, where infrastructure cannot be built [31]. Furthermore, the elevated nature of aerial UVs makes them well suited to close coverage holes (dead zones), as they frequently experience strong LoS communication channels with ground users. Moreover, their mobility opens the door to real-time network capacity provisioning, either to handle hotspots due to infrequent but acute high-traffic events, to provide temporary solutions when permanent terrestrial infrastructure is offline, or to provision dynamic coverage to a set of mobile users. A sample of these provider roles is shown in Figure <ref>.

Sample envisioned applications of UVs as providers in next-generation communication systems. (Top left) As part of a Space-Air-Ground Integrated Network (SAGIN), a HAPS provides connectivity to isolated and under-served areas. (Top right) A UV uses integrated sensing and communication (ISAC) to simultaneously extend connectivity to and further sense end users. (Bottom left) UVs aid a base station by acting as relays. Their elevated position allows them to serve users in dead zones, and by working together, they can extend the area covered by the terrestrial base station (TBS). (Bottom right) A team of UVs provides dynamic network provisioning for a crowd during a large event.

Commercial enterprises have begun making this vision a reality. For example, in 2022, AT&T successfully demonstrated the use of a UAV as a flying 5G base station (BS) [32], while Aalto's Zephyr HAPS promises direct-to-device 5G connectivity with $5-10\,$ms of latency as soon as 2024 [33]. As recognition of the advantages offered by UVs grows, their use will become more commonplace.

Across the variety of settings in which UVs may be placed as providers, we highlight two general scenarios. First, as UV base stations (UV-BSs), they must optimize their position and allocation of limited communication resources among multiple users, while mitigating interference with users connecting via other access points. This scenario covers many problems, including the deployment of HAPS and UAVs to provision additional network capacity or take the place of damaged, disabled, or non-existing  terrestrial base station (TBS). Second, UVs may be deployed for cooperative communication for relaying using multihop routing or distributed beamforming. This captures problems related to extending the network or covering dead zones.

§.§.§ UVs as Consumers

On the other hand, UVs as consumers will enjoy robust support in next generation networks, enabling many other exciting applications. 3GPP Release 17, published in 2022, includes specific guidance on incorporating UAVs into the 5G environment [34], with additional studies coming in 2024 as part of Release 18 [35]. Furthermore, the FAA's Integration Pilot Program, developed to catalyze the incorporation of UAVs into U.S. airspace, published its finding in 2020 [36], and has since been replaced by the BEYOND program, which continues to collect flight data and develop standards, with an eye towards beyond-visual-line-of-sight operation [37].

With standards in development, a number of enterprises have already begun deploying both ground and aerial UVs for a broad range of applications. Well known commercial examples include Alphabet's Wing [38] and Amazon's delivery drone program [39], which employ cargo UAVs for last-mile delivery, while Nuro [40] provides a similar service with autonomous ground vehicles. Autonomous vehicles have also found applications in warehouse management, with Amazon's Proteus and Sequoia platforms demonstrating the possibilities in this area [41]. Future industrial applications include precision agriculture and facility surveillance [42]. Furthermore, Tesla [43], Waymo [44], and Uber [45] all have self-driving cars in various stages of development, and the vision of fully autonomous automobiles has taken root across the industry. In all these cases, communication can facilitate mission completion by allowing for command and control, information sharing, and access to edge computing services.

For optimal performance, UVs must jointly optimize their trajectories and communication as they complete their missions, with a few key problem scenarios deserving attention. Most fundamentally, UV users must maintain connectivity for command and control, or, if connectivity is lost, they must reconnect as soon as possible. They may additionally need to schedule wireless offloading of data collected from, e.g., onboard cameras, IoT devices, or wireless sensor networks, while accounting for the quality of the underlying communication channel. The data collection itself may involve deciding on the best way to visit a number of sites while taking into account the timeliness of data transfer back to a remote base station. Examples of these problems are illustrated in Figure <ref>.

Sample common tasks for UV operation in 6G networks. (Top left) A UV navigates to a destination while maintaining connectivity. (Top right)  A UV collects data from a dynamic WSN. (Bottom left) A UV must jointly plan its trajectory and transmission power/communication rate while offloading data to a remote station. (Bottom right) A UV, initially disconnected from the network, seeks to reconnect as quickly as possible.

§.§ Our contributions

Using the mobility of unmanned vehicles to enable new forms of connectivity is not a new area, and dates back to work as early as 2008.  For instance, in [46] the term communication-aware motion planning was coined to refer to path planning that considers its impact on communication parameters, while [47] showed how a UV can actively plan its path to enforce new forms of connectivity. This motivated subsequent work since then that showcased the potentials of unmanned vehicles to enable new forms of connectivity [48].  While these explorations were for a general system, the possibilities and challenges offered by the 6G vision created great new interest in this area since 2019.

Along this line, the past few years have produced a number of valuable surveys and tutorials on UVs in communication systems [49, 50, 51, 52, 53, 54, 55, 56].  Some of these focus on particular topics, such as machine learning [52], IoT networks [49], or path planning [56], while others provide a high-level perspective of the potentials and challenges in this space. Moreover, most such work are written from a 5G perspective.  In this context, this paper takes a unique perspective by focusing on the optimization problems that arise when planning the operation of UVs in 6G communication networks. Its primary contribution lies in the unification of the problem space, providing a structured framework to understand the use cases, underlying problem formulations, and the needed mathematical tools.  Overall, the paper aims to furnish the readers with a systematic understanding of the potentials and challenges offered by the integration of UVs into the 6G communication systems.

The rest of the paper is organized as follows. In Section <ref>, we introduce the two key types of optimization variables — those related to communication resource allocation and UV mobility — and discuss key modelings considerations for each in the context of 6G. Section <ref> discusses the aspects of these systems that make optimization mathematically challenging and provides a set of key tools that can be used to navigate this space. With this groundwork laid, Section <ref> then takes a unifying perspective of the problem formulation space, methodically categorizing the large array of potential scenarios into a set of fundamental problems, all the while bringing key insights into relevant sub-problems.  Finally, Section <ref> gives other important considerations for the optimization of UV operation, which is followed by concluding remarks in Section <ref>. We note that Sections <ref> and <ref> are written so that readers familiar with their content may choose to move directly to Section <ref>.

§ KEY CONCEPTS AND MODELING

The previous section highlights the many roles UVs will play in next-generation communication systems, but the realization of UVs' full potential depends on intelligent, optimized deployment of these assets in the network. In Section <ref>, we provide a unifying and structured treatment of the corresponding problem space. This section then identifies core components of these systems and presents relevant models needed for tackling the problem space. Throughout, we identify common types of constraints and performance metrics, and we end the section with a high-level problem formulation.

§.§ Common Core Components

To identify common system components, consider $N_b$, a bound on the number of bits that can successfully be sent between a transmitter and a receiver over a given time period, $\Delta t$. From Shannon's channel capacity for additive white Gaussian noise (AWGN) [57], this can be expressed as

\begin{equation}\label{eq:shannon_bound}
N_b =  \Delta t\, B\, \text{log}_{2}\big(1+\text{SINR}\big),\;\text{SINR} = {P |H|^2}/{(\sigma^2 + I)}
\end{equation}

where $B$ is the channel bandwidth, SINR is the signal-to-interference-plus-noise ratio, $P$ is the transmit power, $|H|$ is the channel gain,  $\sigma^2$ is the noise power at the receiver, and $I$ is interference power that arises from other communication links using the same physical resource block (PRB).

This bound indicates the two avenues through which UV operation impacts communication. First, $N_b$ depends on the allocation of wireless resources: the time spent communicating over the channel ($\Delta t$), the transmit power ($P$), and the bandwidth of the channel ($B$). In general, these resources may need to be allocated among multiple users simultaneously, and the interference, $I$, may depend on allocation to other links in the network. Second, $N_b$ is a function of the communication channel, $H$, which, while generally considered exogenous, is controllable to a degree through the mobility of the UV.

Based on the discussion above, we identify three general components of these systems which merit attention. First, the communication channel affects the quality of communication over a link. Second, the allocation of communication resource across over multiple users, together with the channel, determines the communication performance. Third, mobility gives the UV some control over the channel while also allowing for the completion of other mission-specific objectives. We next discuss each of these high-level components, with an emphasis on considerations for 6G and beyond.

§.§ Communication Channel Models

The channel plays a central role in the performance of the system, and thus appropriate modeling is very important. However, channels display complex spatio-temporal variation, making them challenging to model and predict. We next discuss several key modeling concepts and then move to considerations for 6G and beyond.

§.§.§ Statistical Channel Models

The spatial dynamics of channels are generally modeled as the superposition of three components. Path loss describes the large-scale attenuation of the signal power as it propagates through space. Shadowing captures the impact of large-scale blockages which create Non-Los (NLoS) communication links. Finally, multi-path or small-scale fading models the small-scale variation due to multiple copies of the signal arriving at a receiver via different paths. When the multipath delay spread is relatively small, a general model has the form

\begin{equation}
|H|^2 = H_{\text{PL},0}d^{-\alpha_\text{PL}}\; \alpha_{\text{SH}}^2 \; \gamma_{\text{MP}}
\end{equation}

where $H_{\text{pl},0}$ is the path loss intercept (i.e., the path loss at a reference distance of $1\,$m), $\alpha_\text{PL}$ is the path loss exponent, $d$ is the distance between the transmitter and the receiver, $\alpha_\text{SH}^2$ is the shadowing power, and $\gamma_{mp}$ is the multipath effect. While the path loss intercept and exponent are modeled as scalar values,  shadowing, $\alpha_{\text{SH}}^2$, is generally modeled as a spatially-correlated log-normal random variable, and multipath effects can be modeled with Nakagami, Rayleigh, Rician, or log-normal distributions [58, 59]. The values of the path loss intercept and path loss exponent parameters, as well as the parameters for the shadowing and multipath distributions, depend on the carrier frequency and environmental factors. For the case of wideband channels where delay spread needs to be considered, the same modeling principles can be extended over the delay dimension, resulting in the overall power-delay profile of the channel.

§.§.§ Channels for UAVs and HAPSs

For UVs deployed at high altitudes, the shadowing and multipath components may be omitted to model scenarios of unimpaired propagation between nodes, both for air-to-air links between UVs and for air-to-ground links when the height of a HAPS or UAV results in a dominant line-of-sight (LoS) link [60].

For air-to-ground channels, shadowing is often accounted for by using different path loss intercepts and exponents depending on whether a dominant LoS path exists between the transmitter and receiver:

\begin{equation}\label{eq:LOS_NLOS}
|H|^2 =\begin{cases}
H_{\text{PL, L},0}||q_i - q_j||_2^{-\alpha_\text{PL, L}}&\hspace{-0.1in}\text{w/ prob.}\, p_{\text{LoS}}\\
H_{\text{PL, N},0}||q_i - q_j||_2^{-\alpha_\text{PL, N}}&\hspace{-0.1in}\text{w/ prob.}\, (1 - p_{\text{LoS}})
\end{cases}
\end{equation}

where $H_{\text{PL, L},0},\,\alpha_\text{PL, L}$ and $H_{\text{PL, N},0},\,\alpha_\text{PL, N}$ are the path loss intercept and exponent for the LoS and NLoS conditions, respectively, and $p_\text{LoS}$ is the probability of the existence of the LoS path.

Three common models for calculating $p_\text{LoS}$ appear in the literature. A method used by the ITU relies on three parameters to statistically characterize the probability of a LoS channel existing between a UAV and a ground node [61]. The parameters of this model are $\beta$, the number of buildings per square kilometer, $\alpha$, the fraction of area covered by buildings, and $\kappa$, the parameter for a Rayleigh distribution describing the building heights. The formula is given by

\begin{equation*}
p_{\text{LoS}}=\prod_{n=0}^{\max\left(0,o-1\right)}\left[1-\exp\left(-\frac{\left(h_{\text{gr}}+\frac{\Delta h\left(n+0.5\right)}{o}\right)^{2}}{2\kappa^{2}}\right)\right],
\end{equation*}

where $h_{\text{gr}}$ is the height of the ground node, $o=\lfloor \sqrt{\alpha \beta} \rfloor$ with $\lfloor\cdot \rfloor$ the floor operator, and $\Delta h$ is the difference in height between the UAV and the ground node.

Alternatively, 3GPP has produced recommendations TR36.777 [62] and TR36.873 [63], which calculate $P_\text{LoS}$ as:

\begin{equation*}
\begin{cases}
1,  \hspace{0.1in} \text{if}\,d_{\text{horiz}}\leq d_{1}\, \text{or}\, h_u \geq 100\\
\frac{d_{1}}{d_{\text{horiz}}} +
\left(1-\frac{d_{1}}{d_{\text{horiz}}}\right)\text{exp}\left(\frac{-d_{\text{horiz}}}{d_{2}}\right), \hspace{0.1in} \text{if}\; d_{\text{horiz}} > d_{1}
\end{cases},
\end{equation*}

where $h_u$ is the height of the UAV, $d_{\text{horiz}}$ is the horizontal distance between the UAV and the ground node, $d_1=\text{max}\{460\text{log}_{10}h_u -700, 18\}$, and $d_2=4300\text{log}_{10}(h_u)-3800$. Importantly, this model holds for $22.5\,\text{m}<h_u\leq 300\,\text{m}$.

Finally, work in [64, 65], based on the ITU model, finds that in many scenarios, the probability of a LoS channel is modeled well as

\begin{equation}\label{eq:plos_alhourani}
p_\text{LoS}=\frac {1}{1+b_1 \exp (-b_2 (\theta-a))},
\end{equation}

where $b_1$ and $b_2$ are environment parameters, and $\theta$ is the UAV's angle of elevation with respect to the ground node.

Additional details on air-to-ground communication channels may be found in [66, 67].

§.§.§ Communication Channels at mmWave and Beyond
While the modeling discussed thus far suffices for a wide range of frequencies, modeling 6G frequencies (mmWave and THz) requires additional considerations. Modeling these channels continues to be an area of active research, with details found in [68, 69].

Acute Signal Attenuation:
mmWave and THz communication signals attenuate much more quickly as they travel through space, due to the physics of electromagnetic propagation as well as increased impacts of atmospheric absorption, necessitating the use of beamforming for energy-efficient communication. Furthermore, signals at these frequencies do not propagate well through most solid material due to acute penetration loss. Consequently, the existence of a LoS path between the transmitter and receiver becomes more important, though technologies such as RIS may mitigate this challenge by steering around the blockage.

Sparse Multipath In part due to acute signal attenuation, 6G frequencies tend to be multipath sparse, so the channels tends to be dominated by path loss and shadowing.

§.§.§ Temporal Channel Variation

Importantly, communication channels may vary across times as well as space, due to changes in the environment. To account for this, a block-fading model assumes the channel is temporally stationary over a coherence time of length $\Delta_C$ (see, e.g., [70]). Alternatively, the shadowing power over both space and time may be modeled with a Gaussian process with an autocorrelation function

\begin{equation}\label{eq:spatio_temproal_AR}
R\big(\alpha_\text{SH}(q,t), \alpha_\text{SH}(q',t')\big)=\alpha_\text{SH}^2 e^{-\frac{||q-q'||_2}{d_c}-\frac{|t-t'|}{t_c}},
\end{equation}

where $d_c$ and $t_c$ give the rate of decorrelation in space and time, respectively [71].

§.§.§ Channel Prediction

For fixed transmitters and receivers, there are many established techniques to estimate the channel gain after receiving a packet. However, for trajectory planning purposes, it is often important to know the channel gain at points not yet visited by the UV. The UV may employ a statistical modeling framework for channel prediction across the workspace. For example, given a few prior channel measurements, [59] proposes a Gaussian-process based predictor which gives the probability distribution over the channel gain values at any unvisited location in the workspace. More recently, [72] uses a convolutional deep neural network to predict channel values across the workspace with only a few prior measurements and without knowledge of the transmitter location. In [73], the authors demonstrate how the mobility of a UV can be used to facilitate full ray makeup channel prediction, enabling the prediction of the magnitude, delay and angle of arrival of all the incoming paths at unvisited locations.

[74] extends the Gaussian process model across space and time for predicting spatio-temporal variations. The work of [75] proposes a distributed stochastic optimization method in order to predict the SNR at future time steps for mobile relay motion control, assuming that the log-magnitude of the channel evolves as a Gaussian process that is correlated across time and space along the lines of (<ref>). The work by [76] proposes a correlated bandits approach in order to predict the spatio-temporally correlated channels.

§.§.§ Ray Tracing
Ray tracing models communication channels by tracing the path of electromagnetic radiation from the transmitter to the receiver and explicitly accounting for phenomena such as reflection, diffraction, refraction, and scattering by splitting, attenuating, and redirecting the rays. This approach requires detailed information about the environment, and for all but the simplest cases needs computationally-intensive simulation. As a result, these methods have generally been used in offline settings, with desktop applications such as Remcom's Wireless InSite [77] capable of producing high-fidelity simulated channels. However, recent developments in mobile hardware, such as Qualcomm's Snapdragon 8 Gen 2 [78], have raised the prospect of real-time ray tracing on small, energy-limited devices. This, in conjunction with real-time data from UV-mounted vision systems, may eventually permit this approach for channel modeling on the fly.

§.§.§ Realistic Channel Environments

Regardless of which channel model is used, good modeling and validation require grounding in real-world measurement campaigns. Standards and recommendations written by bodies such as the ITU and 3GPP are good resources for finding realistic parameters for statistical channel modeling [61, 62, 63], as are academic works which take on the challenge of measurement campaigns, several of which are summarized in [67].

Alternatively, realistic channel data may be synthesized. For example, [79] generates statistically realistic channel spatial variations using a Gaussian process model for shadowing (in dB domain), with an exponential correlation function, as discussed earlier, while also accommodating multipath effects if needed, based on the work in [80], and [81] provides sophisticated channel simulation for 5G and 6G mmWave and sub-THz channels in environments studied by 3GPP. Similarly, the work of [82] (as outlined in Theorem 2) shows how to synthesize channels that have the spatio-temporal characteristics of (<ref>) based on a stable order-1 vector autoregression, with a corresponding software package found in [83]. As noted earlier, when sufficient environmental information is available, ray tracing software may also be used.

§.§.§ Model-Free Channels
While we have focused on analytical or simulated models, data-driven, model-free approaches offer an interesting avenue to capture channel characteristics, particularly in the context of Reinforcement Learning [84]. A key challenge here lies in undertaking measurement campaigns large enough to capture channel variations across both space and time.

§.§ Communication: Interference Management, and Metrics

For a single communication link, the allocation of limited power and bandwidth over a finite time, together with the channel, determine the performance of the communication system, as indicated by the bound in (<ref>). When multiple links are considered, the picture becomes more complex due to the possibility of interference and the need to aggregate the performance over each link. We next discuss interference management via multiple access schemes before reviewing important metrics of the communication system's performance.

§.§.§ Multiple Access Schemes

Multi-access schemes give structure to the allocation process by defining rules which help mitigate interference between users connecting to the same BS. Traditionally, orthogonal multiple access (OMA) schemes have been used to ensure that no two users send or receive data on the same PRB by splitting either time (time-division multiplexing (TDM)), or bandwidth (orthogonal frequency division multiplexing (OFDM)) into equal, non-overlapping partitions of a predetermined size and assigning a single user to each.

To achieve the objectives of 6G, physical resources will need to be used more efficiently, leading to increased interest in non-orthogonal multiple access schemes, which allow multiple channels to occupy the same PRB. For example, massive MIMO, a key enabler of 5G, allows for space-division multiple access (SDMA) by creating strong antenna gains in multiple directions simultaneously [17]. In 6G, ultra-massive MIMO [6] or extreme MIMO [4] will continue to develop this approach. Alternatively, PRBs can be reused by employing intelligent signal processing and interference management with techniques such as power-domain NOMA (PD-NOMA) [22] and rate-splitting multiple access (RSMA) [23]. Table <ref> provides an overview of these schemes with associated resource constraints and inter-cell interference expressions.

Important multi-access communication schemes with related constraints and interference values. Here, $P_u$ is the power allocated to user $u$, and $P_\text{MAX}$ is the maximum available transmit power. $s_{uc}$ and $f_{uc}$ are binary variables indicating whether user $u$ is using time slot (TDM) or sub-channel (OFDM) $c$, respectively. $\vec{w}_u$ is the beamforming vector for user $u$, and $\vec{h}_u^H$ is the hermitian transpose of the channel vector $\vec{h}_u=[H_{u,1},..,H_{u,j},...,H_{u,N_t}]$ where $H_{u,j}$ is the channel between the $j^\text{th}$ antenna and the user, and $N_t$ is the number of transmit antennas. For PD-NOMA, the users are sorted by the amount of power allocated to each and indexed in ascending order. For RSMA, $P_\text{cmn}$ is the power allocated to the common component and $P_u$ refers to the private component for each user.

Scheme     Channels Defined by    BS DL Power Constraint     Channel Constraints     DL Intra-cell Interference

TDM     Orthogonal time allocation    $P_u \leq P_{\text{MAX}}$     $\sum_{u\in \mathcal{U}} s_{u,c}\leq 1$     None

OFDM     Orthogonal frequency blocks     $\sum_{u\in \mathcal{U}} P_u \leq P_{\text{MAX}}$     $\sum_{u\in U} f_{uc}\leq 1$     None

SDMA     Quasi-orthogonal spatial beams     $\sum_{u\in \mathcal{U}} ||\vec{w}_u||_2^2 \leq P_{\text{MAX}}$     Limited by number of antennas     $I_u = \sum_{u'\in \mathcal{U}, u'\neq u} |\vec{h}_{u}^H \vec{w}_{u'}^2|^2$

PD-NOMA     Channel reuse     $\sum_{u\in \mathcal{U}} P_u \leq P_{\text{MAX}}$     –     $I_{u_i} = \sum_{j=i+1}^n P_{u_j} |H_{u_i}|^2$

RSMA     Partial channel reuse     $P_\text{cmn} + \sum_{u\in\mathcal
{U}} P_u \leq P_\text{MAX}$     –     See [23]

§.§.§ Communication Metrics

In IMT-2030, the ITU has set out a variety of metrics, both new and long-standing, to measure the performance of 6G systems, as we review next.

Fundamentally, the SINR of individual communication links determines the performance of the network. The SINR in turn determines the maximum achievable spectral efficiency, $r^{\text{max}} = \text{log}_{2}(1+\text{SINR}),\,$ and by extension the maximum achievable communication rate. The performance may also be measured in terms of energy efficiency, i.e., bps/W.

On the other hand, by modeling data generation at an end user as an $M/G/1$ queue, the latency across a single link is

\begin{equation*}
l = \frac{\lambda}{2 (B\,r^\text{max}\nu)((B\,r^\text{max}/\nu) - \lambda)} + \frac{\nu}{Br^\text{max}},
\end{equation*}

where $\nu$ is the packet size [85] and $\lambda$ is the data generation rate for user $u$ in bps.

Another key communication metric is the bit-error rate (BER), which is the ratio of the number of bit errors to the total number of transmitted bits. The BER may be calculated in various ways depending on factors such as underlying modulation, coding scheme, and channel characteristics [58]. Assuming an M-QAM communication constellation with spectral efficiency $r$, the reliability in terms of the bit error rate is well approximated by

\begin{equation*}
p_\text{b} \approx 0.2\text{exp} \left(-1.5 \text{SNR}/\left(2^r -1\right)\right).
\end{equation*}

for certain SNR regimes [58].

Other important metrics include coverage, outage probability, and the number of handoffs experienced by a user, i.e., the number of times the user's BS association changes, while secrecy rate provides a way of evaluating the security and privacy of the system [86].

§.§ UV Mobility: Kinematics and Energy Models

So far, we have discussed key communication concepts and models. We now address another important facet, the positioning and movement of UVs. Mobility supports UVs both as providers and consumers in the network by giving some degree of control over the communication channel, while also allowing for the completion of mission-specific tasks such as data harvesting or automated delivery. We next briefly present key considerations for modeling the mobility of UVs including kinematic and energy consumption models.

§.§.§ Dynamics and Kinematic Constraints

A basic model treats the UV as a single point, $q$, constrained to lie in some region $\mathcal{Q}\in\mathbb{R}^3$, which may account for obstacles or no-fly zones. In the multi-UV case, a minimum distance constraint between UVs may enforce collision avoidance.

A basic second-order dynamic system provides a basic model of mobility, expressed as

\begin{equation}
\dot{q} = v,\;\;\dot{v} = u,
\end{equation}

where the dimensions of velocity, $v$, and control input (acceleration), $u$, match the dimension of $q$. When planning a 2D trajectory, either as a ground vehicle or when flying at a fixed altitude, the UV's velocity, $v$, is also constrained to be in some range, i.e., $v_\text{MIN} \leq \|v\| \leq v_\text{MAX}$, with the minimum and maximum values dependent on the specifics of the UV. For ground vehicles and rotary-wing UAVs, $v_\text{MIN} = 0$, while fixed-wing UAVs will have some non-zero minimum flight speed. For 3D trajectories, the velocity constraint may be further decomposed into two separate vertical and horizontal maximum velocities, as for many UAVs, the mechanics of moving vertically differ significantly from those of moving horizontally [87, 88]. Acceleration should be similarly bounded to capture realistic kinematic constraints.

This basic model has the advantage of simplicity and serves when planning for rotary-wing  UAVs or, more generally, when the UV has a small turn radius compared to the scale on which the planning occurs. At other times, however, more detailed kinematics are important. For many ground vehicles, a basic car model offers a simple yet descriptive representation [89]. This differential model includes not only the position of the vehicle, $q$, but the orientation $\theta_o$ as well. The system takes as input a velocity, $v$ and steering angle, $\phi$, and the dynamics are

\begin{equation}
\dot{q} = v[\text{cos}\theta_o,\,\text{sin}\theta_o],\,\dot{\theta_o} = \phi_o,
\end{equation}

with $\dot{q} = [\dot{q}_{x}, \dot{q}_{y}]$ denoting the velocity in the x- and y-coordinates. Generally, the steering angle is constrained, i.e., $|\phi_o|\leq \phi_{o,\text{MAX}}$, and the minimum turning radius is given by $L/\text{tan}\phi_{o,\text{MAX}}$, with $L$ as the distance between the two axles. If $v$ is allowed to be negative, the vehicle may go in reverse, but when the velocity is constrained so that $v\in\{0,1\}$, we have the well-studied Dubins model [90].

The kinematics of fixed-wing aircraft are quite complex, but a simple extension of the model above gives an elementary yet useful set of dynamics. In particular, the dynamics at a fixed flying altitude and constant velocity can be modeled directly as a Dubins model, while any changes in altitude can be captured by the independent linear system $\dot{q}_z = v_z$, for a vertical velocity $v_z$ [89].

The above models give a more accurate sense of a UVs movement capabilities than the simple point model first presented. However, some scenarios, such as when studying air-frame occlusion or beam-misalignment, require rigorous understanding of UV kinematics [91, 67]. In these cases, appropriate models must be found using established techniques from robotics.

§.§.§ UV Energy Models

UVs generally have access to limited energy, so understanding the energy requirements for a proposed trajectory is critical to successful operation. As with kinematics, exact expressions for energy consumption are tied to the specifics of the UV under consideration, but some general models capture important features for different types of UVs. For example, [92] develops an empirical model for a pioneer ground robot given by $P(v) = \kappa_1 v + \kappa_2$, where $\kappa_1$ and $\kappa_2$ are determined by the load and mechanics of the robot. Alternatively, [93] gives the instantaneous power consumption for a rotary-wing UAV as a function of velocity with the following formula:

\begin{align}\label{eq:rotary_UAV_energy_model}
P(v)=&\underbrace {P_{0} \left ({1 + \frac {3v^{2}}{U_{\mathrm {tip}}^{2}} }\right)}_{\text {blade profile}}+ \underbrace {P_{i} \left ({\sqrt {1 + \frac {v^{4}}{4 v_{0}^{4}}}-\frac {v^{2}}{2v_{0}^{2}}}\right)^{1/2}}_{\text {induced}} \\&+\,\underbrace {\frac {1}{2} d_{0} \rho s_r A v^{3}}_{\text {parasite}},\notag
\end{align}

where, $v$ is the velocity, $P_0$ and $P_i$ are constants, $U_\text{tip}$ is the speed of the tip of the rotor blade, $v_0$ is the mean rotor induced velocity in hover, $d_0$ is the fuselage drag ratio, $s_r$ is the rotor solidity, $\rho$ is the air density, and $A$ is the rotor disk area. Importantly, this model is not convex in $v$, but a convex approximation fits well outside the low-velocity regime.

Finally, for a fixed-wing UV flying at a fixed altitude, [94] derives an energy model which accounts for drag, list, and thrust. For the velocity vector $\mathbf{v}\in \mathbb{R}^2$ and its time derivative, $\dot{\mathbf{v}}$, the expression is given as:

\begin{align}\label{eq:fw_UAV_energy_model}
P(\mathbf{v}, \dot{\mathbf{v}})= c_1 \|\mathbf{v}\|^3+\frac{c_2}{\|\mathbf{v}\|}(1+\frac{\|\dot{\mathbf{v}}\|^2 - \frac{\dot{\mathbf{v}}^T \mathbf{v}}{\|\mathbf{v}\|^2}}{g^2}) + \frac{m\|\dot{\mathbf{v}}\|}{4},
\end{align}

where $m$ is the mass of the UV, $g$ is acceleration due to gravity, and $c1$ and $c2$ are constants determined by other parameters of the UV air frame. The $1/\|\mathbf{v}\|$ and $\|\mathbf{v}\|^3$ terms capture high power consumption at both very low and very high velocities.

§.§ A High Level Problem Formulation

To conclude, we present a high-level problem formulation consisting of an objective — which may account for the communication performance/cost, motion costs, and other mission objectives — and constraints which can be motivated by communication, motion, or mission-specific considerations. The decision variables can be broadly categorized as either communication- or trajectory-related, with the possible addition of mission-related variables representing concepts such as sensing. Let $\mathcal{R}$, $\mathcal{T}$, and $\mathcal{M}$ represent collections of variables that determine the allocation of communication resources, the UV's positioning, and other mission-specific optimization variables, respectively. Then a high-level formulation is shown in (<ref>),

ℛ, 𝒯,
f_GLB( f_com(ℛ, 𝒯)_Communication, f_𝒦(ℛ, 𝒯)_Motion,  f_m(ℛ, 𝒯, ℳ)_Mission )

g_com(ℛ, 𝒯) ≤0

h_com(ℛ, 𝒯) = 0
Communication constraints

g_𝒦(𝒯, ℳ) ≤0

h_𝒦(𝒯, ℳ) = 0
Motion constraints

g_m(ℛ, 𝒯, ℳ) ≤0

h_m(ℛ, 𝒯, ℳ) = 0
Mission constraints  ,

The objective may include communication, motion, and mission terms, combined as a weighted sum or in ratios. Not all terms will be present in all formulations, and furthermore, performance in any one of these areas may also be captured as constraints.

Communication constraints come in two types. First, there are communication resource constraints, which ensure that time, bandwidth, or spatial degrees of freedom are not over-allocated. Second, there are constraints tied to communication related metrics, including constraints on the minimal latency, SNR, or outage probability. Importantly, when a stochastic channel model is used, the constraints must be defined either in expectation or in some other probabilistic sense (e.g., the constraint is met with some probability).

Kinematic constraints can also be subdivided into a few categories. First, there are kinematic constraints related to what movement is physically possible by the UV. Second, there may be safety constraints, such as a minimal flying height for a UV or a minimum distance between any two UVs. Finally, there may be constraints related to energy consumption for motion.

Finally, the mission-specific problem may introduce additional constraints or objectives. For example, if the UV is tasked with visiting a number of sites, we may additionally have a constraint that captures this requirement, or the objective may be to do so in the minimum amount of time, subject to, e.g., energy and power constraints. These constraints may also ensure that mission objectives, such as sensing quality, meet certain thresholds.

§ AN OPTIMIZATION TOOLKIT

In the previous section, we highlighted key modeling considerations and developed a high-level problem formulation which captured the many different optimization problems that arise when considering the operation of UVs in next-generation communication networks. However, it is difficult to develop a similarly generic solution method due to the wide variety of specific problem instances represented by the generic formulation. Rather, this section presents a number of important tools which find broad application, and these are classified into two categories. The first category comprises analytical methods, which generally produce interpretable solutions with well-defined optimality properties. The second category encompasses learning-based approaches, particularly in the context of stochastic dynamic programming with stochastic function approximation. These powerful methods are capable of solving otherwise-intractable problems at the expense of interpretability of the solution.

§.§ Analytical Methods

Many optimization tools have been developed in fields such as computer science, operations research, robotics, and control, and set forth well-developed frameworks with clear understanding of optimality guarantees and computational complexity. Below, we discuss facets of UV operation that add complexity to problem formulations and highlight useful analytical tools for tackling them.

Handling Non-Convexity with Convex Approximation:
In many instances, meaningful measurement of performance requires consideration of nonlinear objectives or constraints (see Section <ref>). For example, spatially non-convex channel gains or non-convex energy models, such as that of Eq. (<ref>), create non-convexity in trajectory design problems.

For non-convex objective functions, successive convex approximation (SCA) provides an iterative method that guarantees a locally-optimal solution under permissive technical assumptions [95]. Specifically, for an objective function $f(x)$, the method finds a sequence of intermediate solutions $\{x_k\}$ by solving a series of simpler problems, $x_{k+1} = \argmin_{x\in\mathcal{X}} \tilde{f}(x|x_{k})$, where $\mathcal{X}$ is the solution space and $\tilde{f}(x|x_{k})$ is a strongly convex local approximator of $f$, possibly parameterized by the current iterate $x_k$. Newton's method is a common example of SCA, in which $\tilde{f}(x|x_{k})$ is the second-order Taylor-series expansion around the current iterate, $x_k$.

In some instances, coupling between optimization variables makes the problem difficult to solve. In such cases, block coordinate descent (BCD) simplifies the problem by partitioning the variables into non-overlapping subsets (blocks) and iteratively optimizing over a single block at a time. When the problem is convex in the block variables, standard convex optimization methods can be used. Otherwise, methods like SCA can be used to update a single block. Various optimality results hold under certain assumptions (see [96] and references therein). For UV problems, treating positioning and communication optimization variables as distinct blocks is often a useful way to decompose the problem.

Handling Combinatorics with Mixed Integer Programs:
Modeling concepts such as user-BS association or visit sequence among a set of WSN cluster heads introduces binary variables, giving these problems a combinatorial aspect. Mixed integer programs (MIPs), a broad class of mathematical programs characterized by the presence of both continuous- and integer-valued variables, can model these scenarios. While the general MIP is NP-hard, for special cases, extensive literature offers established methods for finding high-quality solutions.

Within the umbrella of MIPs, there are several important subclasses of problems characterized by the nature of the objective and constraint functions. In mixed-integer linear programs (MILPs), the objective and all constraint functions, $f_i$, are linear, whereas in a mixed-integer nonlinear program (MINLP)[97], the objective and/or constraints may be nonlinear. Within the class of MINLPs, we further have a mixed-integer quadratic program (MIQP) [98] if the objective is quadratic and the constraints are linear, and a mixed-integer second-order cone program (MISOCP) [99] if the objective is linear and all constraints are expressed as second-order cone constraints.

Branch-and-cut or branch-and-bound methods are common candidates for finding solutions of MIPs. These methods iteratively tighten upper or lower bounds on the optimal solution via relaxations of the integer variables or the nonlinear constraints/objective. In the case of MILPs, the resulting relaxations are linear programs (LPs), which can be solved very quickly, so that optimal solutions may be found quite efficiently. Similarly, the convexity of the relaxations of MIQPs, MIQCPs and MISOCPs allow reasonably-sized problem instances to be solved optimally in a modest amount of time. A number of commercially available software packages are capable of solving these types of problems, including IBMs CPLEX [100], MOESK [101], and GUROBI [102]. For general MIPs, however, finding optimal solutions is quite challenging, and finding high-quality approximate solutions is often acceptable.

Finding Visit Orders Using Vehicle Routing Problems: The vehicle routing problem (VRP) [103] refers to a general set of problems in which one or more vehicles must visit a number of locations in the shortest time possible. These problems may be posed as MIPs, with integer variables giving the sequence of site visits, and they include the classical traveling salesperson problem (TSP). Time window, energy, and vehicle-capacity constraints may be integrated into the problem as well.

The core modeling concept of the VRP is a weighted graph, with nodes representing locations to visit and edge weights giving the cost to move between sites. Works such as [104, 105, 106] provide an overview of the application of these types of problems for UV trajectory design. This framework can be made communication aware by incorporating communication aware metrics into the values of the edge weights.

Solving Shortest-Path Problems with Graph-Theoretic Algorithms:
Many problems of interest can be cast as shortest-path problems, particularly for scenarios where the UV is a consumer of network connectivity. For example, a cargo UAV may need to find the shortest path to its destination while respecting a QoS constraint or minimizing handoffs across BS. In these cases, the space may be discretized and modeled abstractly as a graph, and $\text{A}^*$ [107] or similar graph-based algorithms can produce optimal solutions.

Randomized methods from the robotics community provide interesting alternatives. Rapidly-expanding random trees (RRT) find paths between given start and end positions by building out trees of partial solutions constructed from random samples of the UVs configuration space. These methods can quickly find feasible solutions, then iteratively improve, making them strong candidates for real-time applications, where available time for computation is unknown and short. Furthermore,  they naturally handle obstacle avoidance as well as kinematic constraints, and a variant, $\text{RRT}^*$, provides asymptotic optimality guarantees [108].

Sequential Decision-Making with Dynamic Programming and Optimal Control:
Both the VRP and shortest-path problems are examples of more general sequential decision-making problems, for which dynamic programming (DP) and optimal control provide highly developed frameworks in discrete- and continuous-time settings, respectively. While some DP problems can be solved via analyzable and provably-optimal methods such as value iteration, many problems necessitate a learning-based approach. We discuss these methods further in the next subsection, but before doing so, we highlight the use of analytical optimal control.

When the dynamics of the problem are described with ordinary differential equations (ODEs), e.g., when using a more developed kinematics model, optimal control [109] offers an established framework for planning the UV's operation. Two key entities must be well-defined before posing the optimal control problem. First, an ODE in the form  $ \dot{x}(t) = f(x(t), u(t))$ describe the evolution of the trajectory, $x(t)$, over time, with different control inputs, $u(t)$, producing different trajectories. Second, a cost functional

\begin{equation*}
J\left(u(t)\right) = \int_{t=t_0}^{t_f} L(x(t), u(t))\, dt\, +  K(x(t_f), t_f)
\end{equation*}

provides a way to compare the quality of different inputs, where $L$ and $K$ are functions capturing the instantaneous and terminal costs/rewards, respectively.

As discussed in [110], optimal control can be conceptualized as DP in continuous time. Analogous to the Bellman equation, the Hamilton-Jacobi-Bellman equation characterizes the optimal control policy, and when the cost functional has special structure, more specific results may be applied. For example, if $f(x(t), u(t))$ describes a linear time-invariant system and $L(x,u)$ can be expressed in the quadratic form $x^T Q x + u^T R u + 2x^T Nu$ where $Q$ and $R$ are symmetric positive-definite matrices, and if there are no terminal costs, i.e., $K(x,u) = 0$, we have a linear-quadratic regulator (LQR) problem, for which solutions may be found by solving an algebraic Riccati equation [111].

§.§ Learning-based Methods

Trajectory design for UVs often requires optimizing system parameters, addressing long-term operation, and managing stochastic uncertainties. Stochasticity is inherent in this domain, particularly due to difficult-to-predict spatial and temporal channel variations brought about by shadowing and multipath effects [112]. Joint mobility of end-users and UVs further induces additional uncertainty. By taking a data-driven perspective and including stochastic function approximation, learning-based methods provide for the optimization of UV-assisted communication systems while ensuring robustness against uncertainty [113].

In this context, the Reinforcement Learning (RL) paradigm  [114, 115] considers an agent that seeks to learn a policy for choosing actions by interacting with the environment. The agent's goal is to maximize a notion of cumulative reward over time, more specifically, the expected sum of rewards that the agent collects over a horizon of interactions.

Mathematically, the RL problem can be formulated as a Markov Decision Process (MDP). An MDP is defined by a tuple $ (\mathcal{S}, \mathcal{A}, \mathcal{P}, R, \gamma) $, where:

* $ \mathcal{S} $ is the set of states representing the environment's observable conditions.

* $ \mathcal{A} $ is the set of actions that the agent can take to interact with the environment.

* $ \mathcal{P}(\mathbf{s}_{k+1}|\mathbf{s}_k, \mathbf{a}_k) $ is the transition function, specifying the probability of transitioning to state $ \mathbf{s}_{k+1} $ given the current state $ \mathbf{s}_k $ and action $ \mathbf{a}_k $.

* $ R(\mathbf{s}_k, \mathbf{a}_k) $ is the reward function that provides a scalar feedback signal to the agent after taking action $ \mathbf{a}_k $ in state $ \mathbf{s}_k $. An instantiation of the reward function at time step $k$ can be denoted as $r_{k}$.

* $ \gamma \in [0,1] $ is the discount factor, representing the agent's preference for immediate rewards over future rewards.

The agent's goal is to learn an optimal policy $ \pi^* $ that maps states to actions, such that the expected cumulative reward, also known as the return, is maximized:

\begin{equation}
\pi^{*} = \argmax_{\pi} \mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty} \gamma^{k} R\left(\mathbf{s}_{k}, \mathbf{a}_{k} \equiv \pi(\mathbf{s}_{k})\right)\right].
\end{equation}

The state-action value function $Q^{\pi}$ is defined as the expected discounted sum of rewards starting from a state-action pair and following the policy $\pi$ thereafter, i.e.,

\begin{equation}
Q^{\pi}(\mathbf{s},\mathbf{a}) = \mathbb{E}_{\pi} \Bigl[\sum_{k=0}^{\infty} \gamma^{k} R(\mathbf{s}_k, \pi(\mathbf{s}_k))| (\mathbf{s}_0, \mathbf{a}_0) = (\mathbf{s},\mathbf{a})\Bigr].
\end{equation}

The optimal value function $Q^{*}(\mathbf{s},\mathbf{a})$ is the fixed point of the Bellman backup operator [116]:

\begin{equation}
Q^{*}(\mathbf{s},\mathbf{a}) = \mathbb{E}_{\mathbf{s}' \sim \mathcal{P}} \Bigl[R(\mathbf{s},\mathbf{a}) + \gamma \max_{a'} Q(\mathbf{s}', \mathbf{a}')\Bigr]   .
\label{bellman}
\end{equation}

In Deep RL (DRL) [117], the optimal state-action value function is often parameterized as a neural network with parameters $\mathbf{w}$, denoted as $Q_{\mathbf{w}}(\mathbf{s}, \mathbf{a})$, also known as the critic or value network. Approximating the value function involves using batches of transitions, typically stored in an Experience Replay $D_\text{ER}$, and applying the following gradient descent update rule on the critic:

\begin{equation}%\label{update rule q}
\begin{split}
\mathbf{w} \rightarrow \mathbf{w}
+  \eta \mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}',r) \sim D_\text{ER}}\Bigl[ \Bigl (&Q^{*}_{\mathbf{w}}(\mathbf{s},\mathbf{a}) - Q_{\mathbf{w}}(\mathbf{s},\mathbf{a})\Bigr)\\ &* \nabla_{\mathbf{w}}Q_{\mathbf{w}}(\mathbf{s},\mathbf{a})\Bigr] ,
\end{split}
\end{equation}

where $*$ is the standard multiplication operator and the scalar parameter $\eta$ denotes the learning rate.

In the context of  MDPs with discrete action spaces, achieving an accurate estimate of the value function is deemed sufficient for optimal control [118]. This  arises from the fact that, at each state $\mathbf{s}_{t}$, the agent can straightforwardly select the action $\mathbf{a}_{t}$ that maximizes the value estimator.

Discrete action spaces are prevalent in UV problems, often resulting from the discretization of the UV's operational space, thus inducing a discrete action space MDP. Moreover, in various scenarios, the action space is inherently discrete. For instance, in situations where decisions like the UV-BS or the UV-end user associations are made at every time step, the action space inherently takes on a discrete nature.

However, when dealing with continuous action spaces, the maximization with respect to the action requires a different approach. In such a scenario, action selection is achieved by parametrizing the policy with a neural network with parameter vector $\phi$, $\pi_{\phi}(\mathbf{a}_{k} | \mathbf{s}_{k})$. The parameter vector $\phi$ is adjusted with stochastic gradient descent to choose the action at each state that maximizes the estimation of the critic

\begin{equation}
\nabla_{\phi} J(\phi) \approx \mathbb{E}_{\mathbf{s}_{k} \sim \mathcal{P}} \left[\nabla_{\mathbf{w}} Q_{\mathbf{w}}(\mathbf{s}_{k}, \pi_{\phi}(\mathbf{s_{k}}) )\nabla_{\phi} \pi_{\phi} (\mathbf{s}_{k}) \right].
\end{equation}

In UV-assisted wireless communications, decision variables like energy expenditure, motion space, and bandwidth allocation inherently take values from a continuous set. Discretizing the solution set is conventionally non-scalable due to the curse of dimensionality [120]. Thus, a more effective strategy is to use continuous action space DRL approaches for addressing these complex problems.

Various versions of the policy gradient algorithm adapt gradient estimation for different purposes. Proximal Policy Optimization (PPO) [121] employs a clipped surrogate objective to maintain stability during policy updates, preventing large changes when value estimates are noisy. Similarly, Trust Region Policy Optimization (TRPO) [122] confines policy updates within a trust region of the parameter space to ensure smooth transitions in learning steps.

Certain methods avoid using a function approximator to parameterize the value function and concentrate solely on parametrizing the policy. These approaches replace the parametrization of the critic with Monte Carlo estimates of the value function, fitting within the Baseline Policy Gradient (BPG) framework. Here, the policy is often presented as a learnable distribution over the action space, and the baseline policy gradient is usually formulated as follows:

\begin{equation}
\nabla_{\phi} J(\phi) \approx \mathbb{E}_{\mathbf{s}_{k} \sim \mathcal{P}} \left[\hat{R}_{k}\nabla_{\phi} \log \pi_{\phi} (\mathbf{s}_{k}) \right],
\end{equation}

where $\hat{R}_{k}$ is the Monte Carlo estimate of the expected sum of rewards corresponding to the state-action pair $(\mathbf{s}_{k}, \mathbf{a}_{k})$. Since Monte Carlo estimates usually exhibit high variance, the baseline policy gradient methods are considered unstable.

§.§.§ On-policy vs Off-policy

A fundamental distinction exists between On-policy and Off-policy DRL methods. On-policy methods optimize the current policy based on its generated data, discarding experiences from previous policies, which can slow learning. In contrast, Off-policy methods decouple data collection from optimization, using Experience Replay [117] to store past experiences for more sample-efficient updates. While On-policy methods like TRPO [122] and PPO [123] prioritize stability, Off-policy methods (e.g., Deep Q-Learning (DQL) [117] and Soft Actor-Critic (SAC)[124]) often demonstrate greater sample efficiency and flexibility in learning from diverse experiences.

§.§.§ Model-based vs Model-free

The dynamics of the MDP are encapsulated in the probability distribution over subsequent states and rewards given the current state-action pair $\mathcal{P}(\mathbf{s}_{k+1}, r_{k} | \mathbf{s}_{k}, \mathbf{a}_{k})$. The presence or absence of assumptions concerning these dynamics delineates the distinction between Model-based and Model-free  DRL approaches.

In Model-based DRL, the agent has an internal representation of transition dynamics, either assumed beforehand or estimated through data-driven techniques [125, 126]. With this insight, the agent can simulate trajectories to determine the optimal policy. In Model-free approaches, there is no internal representation of the underlying dynamics. Instead, the agent directly learns the value function and optimal policy through an iterative trial and error process.

Exploring the intersection of UV-assisted wireless communications and DRL reveals the distinction between Model-based and Model-free approaches, especially regarding the underlying channel statistics. In other words, to comprehend the MDP dynamics, the UV agent requires specific assumptions about the statistical evolution of communication channels over time and space [127, 128].

§.§.§ DRL Algorithms

DRL is built on a core theoretical framework, but there are many specific algorithms which handle the various cases discussed above.
Table <ref> summarizes the characteristics of various Model-free DRL algorithms. Applications of these methods to optimize UV operation in communication systems are found in Section <ref>.

The characteristics of various Model-free DRL approaches.

Algorithms     On-Policy     Off-Policy     Critic

DQL [117]          $\checkmark$     $\checkmark$

Dueling DQL [129]          $\checkmark$     $\checkmark$

SAC [124]         $\checkmark$     $\checkmark$     $\checkmark$

TD3 [130]          $\checkmark$     $\checkmark$     $\checkmark$

PPO [121]    $\checkmark$           $\checkmark$     $\checkmark$

TRPO [122]    $\checkmark$           $\checkmark$     $\checkmark$

DDPG[131]          $\checkmark$     $\checkmark$     $\checkmark$

BPG[115, 114]    $\checkmark$                $\checkmark$

Zeroth-order DPG[132]    $\checkmark$                $\checkmark$

§ REPRESENTATIVE PROBLEM FORMULATIONS

As made apparent in previous sections, the set of possible problem formulations that arise when optimizing the operation of UVs in next-generation networks is incredibly large and diverse. This diversity stems not only from the myriad roles and applications of UVs in the network, as discussed in Section <ref>, but also from the range of modeling choices, technical assumptions, and optimization tools that may apply to a single problem, as discussed in Sections <ref> and <ref>.

Despite this variety, work in this area naturally clusters into a set of core problems, which we illustrate next with a series of representative problem formulations. These are presented with a level of abstraction so as to be agnostic to, e.g., UV kinematics or channel modeling, while highlighting the fundamental question posed in each. We first present analytic optimization formulations (see Section <ref>), then present a selection of these in the context of reinforcement learning. Throughout, we discuss the importance of each problem and give examples from the literature.

Sample papers addressing the optimization of UVs in the role of providers.

UV Mission    Ref.     Objective     Tools     Notes

16UV as BS     [133]     Worst-case SINR     MIP, SCA     OFDMA, sub-channel assignment, HAPS and TBSs.
[134]      Sum-rate     SCA     PD-NOMA, maritime environment.
[135]     Number of users     MINLP     OMA, 3D placement.
[136]      Sum-rate     SCA, Semi-definite relaxation     MIMO, full-duplex communication.
[137]      User-avg. latency     Optimal Transport     OFDMA, non-terrestrial network (NTN).
[138]      Worst-case Throughput      BCD/SCA     TDMA, trajectory planning.
[139]      Total communication rate      RL     SIMO, Q-learning for motion control in order to maximize communication rate.
[140]      Throughput      DRL     MIMO, DPG for weighted throughput maximization subject to fairness.

[141]      Energy consumption      DRL     Motion Control of UV base stations in order to minimize energy expenditure subject to a constraint of perpetual connectivity.

[142]      Task Completion       TD-learning     TD-learning with tile coding for motion control of UV base stations.

[143]      Sum-rate      DRL     UV continuous motion control for collective sum-rate maximization.

[144]      Throughput      DRL     MIMO, SARSA and offline RL for throughput maximization under fading.

3ISAC     [145]     Generic sensing and communication     Calculus of variations     Sensing performance measured with sensing SNR.
[146]     Sum-rate     MIP, BCD, SCA     MIMO, SDMA for sensing and communication.

SINR     Distributed stochastic optimization     Distributed beamforming. A queue of previous positions and channels is used to solve a stochastic surrogate for channel prediction.
[148]     Bit-error probability, motion energy     KKT conditions     Multihop robotic relay, uncertain channel environment with obstacles, Gaussian-process based channel prediction.
[149]     Abstract task completion     SOCP, control barrier functions, $\text{RRT}^*$     Multihop relay. Extensive real-world experimentation.

Motion and communication energy     Multiple-choice knapsack problems      Distributed robotic beamforming, Accounts for uncertainty in shadowing and multipath fading via stochastic channel prediction.

[153]     SINR     DRL     Disrtibuted beamforming relay. Fourier features as preprocessing to estimate the value function depending on spatio-temporally correlated channels.
[154]     Secrecy rate     Stochastic programming     Distributed beamforming relay. Distributed stochastic programming for secrecy rate maximization under cooperation.

[155]     Throughput     DRL     Single UV relay. Motion control for average system throughput maximization.

[156]     Transmit power     BCD     BCD to optimize UV positioning in multihop relaying.
[157]     Maintaining connectivity     Particle Swarm Optimization     Particle Swarm Optimization for UV placement for multihop relaying.
[158]      Min communication rate     BCD, Gibbs sampling     3D UV placement and power and bandwidth allocation for multihop relaying.

[159]     Energy     Multi-agent DRL     Single UV relay. Multi-agent RL surpasses hybrid precoding for energy minimization.

§.§ Unmanned Vehicles as Providers

In this subsection, we present three broad categories of problems in which UVs act as providers. In the first, UVs provide connectivity to the core network for multiple users, possibly in environments which include terrestrial BS (TBS) as well. In the second, the UV uses ISAC to perform a sensing task while simultaneously communicating. In the third, UVs work cooperatively to extend connectivity by relaying either with multihop communication or distributed beamforming.

§.§.§ Providing Multiple Access with Resource Allocation

When UVs act as base stations BSs (UV-BS), either as a cost-effective solution to closing the digital divide or as dynamic assets to relieve temporary high-traffic loads in urban areas, a core problem is resource allocation: What is the best way to distribute a UV-BS's finite wireless resources (power, time, bandwidth, and spatial degrees of freedom) across multiple concurrent users?

A UV acting as a base station must support concurrent users while mitigating inter-cell interference.

Resource allocation over multiple simultaneous links for UV-BS differs from existing results on multi-access for TBS in two important ways. First, the mobility of UV-BSs creates an additional degree of freedom for planning. Second,the elevated position of aerial UV-BSs results in a high probability of a strong LoS channels to unintended receivers, so that mitigating inter-cell interference takes on greater importance. A high-level depiction of the problem setting is shown in Figure <ref>.

To pose a general problem, let $\mathcal{U}$ denote the set of users, $\mathcal{C}_i, i \in \mathcal{U}$ indicate a per-user communication metric (Section <ref>), and $f_\text{comm}$ here be an abstract aggregation operator which takes as input each user's link-level performance and produces a single, system-level metric. Let $\mathcal{B}_{\text{UV}}$ denote the set of UV-BS, $\mathcal{B}_{\text{TBS}}$ the set of TBS, with $\mathcal{B}=\mathcal{B}_{\text{UV}} \cup \mathcal{B}_{\text{TBS}}$. The problem consists of finding the binary BS-to-user associations, $\{a_{bi}\}_{(bi)\in\mathcal{B}\times \mathcal{U}}$, the locations of the UV BS, $\{q_b\}_{b\in \mathcal{B}_{\text{UV}}}$, and the resources allocated from each BS to each user, $\{\mathcal{R}_{bi}\}_{(bi)\in\mathcal{B}\times \mathcal{U}}$, which maximize the aggregate metric of system performance subject to positioning, communication resource, and QoS constraints. The problem is given below:

\beginmaxi![1]
f_com({ 𝒞_i ({a_bi},  {ℛ_bi},  ℋ_i ({q_b})) }_i ∈𝒰)

g_com({ℛ_bi}) ≤0, ∀b∈ℬ
𝒞̃_i({a_bi},  {ℛ_bi}, ℋ_i({q_b}) ) ≥𝒞̃_i,req, ∀i ∈𝒰
∑_b ∈ℬ a_bi = 1, ∀i ∈𝒰
a_bi ∈{0, 1}, ∀  b∈ℬ, i∈𝒰
q_b ∈𝒬_b,  ∀b∈ℬ
\endmaxi!
where, for brevity, we have written the association and resource allocation variables as $\{z_{bi}\}$ rather than $\{\{z_{bi}\}\}_{(bi)\in\mathcal{B}\times \mathcal{U}}$ for $z\in \{a, \mathcal{R}\}$, and $\mathcal{H}_i(\{q_b\})$ represents the set of abstract channels from all BS to user $i$ (see Section <ref>). The link-level metric $\mathcal{C}_i$ is a function of BS-to-user associations as well as resource allocation and the communication channels as determined by the BS positions, and (<ref>) is a QoS constraint which requires that a link-level metric $\tilde{\mathcal{C}}_i$ exceeds a value $\tilde{\mathcal{C}}_{i,\text{req}}$ for each user. Constraint (<ref>) ensures communication resources are not over-allocated, and (<ref>) captures positioning constraints such as obstacles, no-fly zones or fixed flying altitudes.

There are several important challenges raised in this generic problem statement. The binary user association variables introduce a combinatorial element to the problem, which is naturally expressed as a MIP. Non-convexity enters through inter-cell interference, highlighted by the dependence of $\mathcal{C}_u$ and $\tilde{\mathcal{C}}_u$ on all resource allocation variables, as well as through non-convex spatial variations in the communication channels, which enter the problem through $q_b$.

Many problems may be formulated through specific choices of the link-level metric, aggregation operator, resources allocation constraints, and positioning constraints. We next review a few interesting examples.

Providing Multiple Access with Given Position: In some cases, such as when dealing with quasi-stationary HAPSs, the positioning of the UV is assumed fixed, so that optimization only occurs over the communication resource variables. For example, in [133], a system consisting of multiple TBSs and a single HAPS serve a set of ground users using OFDMA. The authors formulate a problem which seeks to maximize the minimum SNR over all users by optimally selecting a sub-carrier frequency channel and power allocation for each user. An iterative procedure based on successive convex approximation (SCA) repeatedly solves a MIP to find solutions to the problem which enhance the worst-case SNR among the users.

A multi-UV variation appears in [134], in which a team of UAV-BSs, TBSs, and a satellite jointly provide communication to several maritime users using non-orthogonal multi-access (NOMA). The transmit powers of the UAV-BSs and TBSs are jointly optimized to maximize the sum-rate under the constraint that interference does not significantly degrade communication for the users connected via satellite. To solve the problem, an iterative SCA method is proposed, and results show that the NOMA multi-access scheme greatly improve the sum-rate compared to OMA schemes in this maritime scenario.

Multiple Access with Optimized Positioning: In many cases, a UV-BS deployed to service an area will operate from or near a fixed point. In [135], the authors examine the 3D-placement problem for a UAV which is deployed to improve the network capacity. The UAV services all users for which the channel between the user and the UAV achieves some threshold value, and the 3D placement of the UAV must be chosen to maximize the total number of users serviced by the UAV. The problem is formulated as a mixed-integer nonlinear program (MINLP) which is solved using an interior-point method and a bisection search.

In addition to UAV placement, [136] optimizes beamforming weights and uplink-user transmit power in a scenario where a full-duplex UAV-BS simultaneously handles uplink and downlink traffic from distinct sets of users with LoS/NLoS mmWave channels. The challenging non-convex problem is solved by first optimizing the UAV's location using SCA, then finding the UAV-BS beamforming weights and uplink user transmit powers by relaxing a rank-1 matrix constraint. It is shown that omitting optimization of the UAV-BS's location reduces sum-rate by nearly $1\,$b/s/Hz, and that greater performance loss occurs when using only half-duplex.

The work in [137] develops a fully aerial network, with designated UAV-BSs providing connectivity to UAV end users using OFDMA. The authors pose the problem of partitioning the coverage region among the UAV-BS to achieve the minimal average communication latency, and the problem is solved using optimal transport [160].

Multiple Access with Trajectory Optimization: When optimizing over time, additional kinematic and energy constraints may enter the problem. We express these generically as

\begin{equation}\label{constraint:generic_kinematics}
\begin{rcases}
\hspace{-0.2in}g_{\mathcal{K}}(\{q[k]\}_{k=1}^K)& \leq 0\\
\hspace{-0.2in}h_{\mathcal{K}}(\{q[k]\}_{k=1}^K)& = 0
\end{rcases}\,\text{Motion constraints}
\end{equation}

where $k$ and $K$ are the discrete-time index and horizon, respectively. For specific examples of such constraints, see Section <ref>. For brevity, in the sequel we denote $\{q[k]\}_{k=1}^K$ with $\{q[k]\}$, unless clarity demands otherwise.

The authors of [138] consider multiple UAV-BSs moving from given initial locations to given final locations while providing connectivity to a set of end users using TDMA. A problem is formulated which maximizes the minimal time-averaged throughput among end users by jointly optimizing trajectories, transmit powers, and user associations. To solve the problem, block coordinate descent (BCD) is used in conjunction with SCA to find solutions. Numerical examples show that utilizing mobility significantly increases the minimal throughput compared to the placement of static BSs.

Thus far, we have looked at resource allocation and positioning for UV-BSs supporting multiple users. These problems are characterized by binary user-BS association variables and a need to mitigate inter-cell interference. Several examples have illustrated the performance gains achievable by deploying UVs and exploiting their mobility, and SCA has proved to be a useful tool for solving these problems. We next consider integrated sensing and communication (ISAC), in which shared time, frequency, and power resources are allocated not only among different communication users but also more broadly between sensing and communication tasks.

§.§.§ Integrated Sensing and Communication (ISAC)

Recently, there has been growing interest in integrating communication and sensing, particularly as communication systems begin to employ higher frequencies which are more amenable to use in radar-like ways. Fully integrating these two applications requires combining hardware, signal processing pipelines, and waveforms, as well as understanding how to allocate limited resources between these two tasks. A thorough survey of the area may be found in [161, 162, 14].

ISAC may use either orthogonal sensing and communication signals, or a single signal may be used for both applications.

Many advantages of UVs for communication also extend to sensing-related tasks. Mobility allows for repositioning to find better sensing locations, while the high probability of LoS paths for UAVs/HAPSs avoids occlusion in the sensing domain. [163] and [164] provide more discussion of ISAC specifically in the context of UAVs.

To pose a general problem, we consider two sets of resource-allocation variables, $\mathcal{R}_{C}$ and $\mathcal{R}_{S}$, representing the resources allocated to communication and sensing, respectively. While current sensing standards treat these resources as orthogonal, as in 802.11bf which uses time-division to isolate sensing and communication signals [165], finding dual-function waveforms that achieve sensing and communication simultaneously is an area of active research. For a single UV, a general problem formulation is given below:
\beginmaxi!|s|[2]
{q[k]}, {ℛ_C[k]},
f_GLB(f_com,  f_sense)

(<ref>) (Motion constraints)
g_com(ℛ_C[k], ℛ_S[k]) ≤0, ∀K.
\endmaxi!

The objective in (<ref>) captures the trade-off between communication performance, $f_{\text{co}}$, and sensing performance, $f_{\text{sense}}$, and both are functions of all optimization variables. Either the sensing or communication performance could alternatively be treated as a constraint, and Constraint (<ref>) captures the fact that resources are shared across sensing and communication applications. We next provide two interesting examples.

In [145], the authors study the optimal trajectories for a UAV tasked with sensing and communicating with a mobile ground node. They show that the sensing and communication quality at different points can be viewed as an artificial potential field, so that maximizing an abstract communication and sensing service can be found by determining the shape of a catenary [166]. Assuming path loss-only communication models, analysis shows the global optimality, existence, and uniqueness of the given solution.

The sensing performance may be taken as a constraint rather than part of the objective. For example, the work in [146] investigates a multi-antenna UAV which communicates with multiple users using TDMA while also periodically sensing a number of targets. To optimize the total communication rate, communication user and sensing targets are scheduled along with corresponding beamforming weights, subject to the constraint that when sensing is scheduled, the sensing SNR (distinct from the communication SNR) must exceed a given threshold. The problem is solved using a general BCD approach, and results show a trade-off between achievable communication performance and sensing frequency.

UV-enabled distributed relay beamforming can extend existing terrestrial networks.

As with communication, UVs can use mobility to improve sensing in ISAC scenarios. However, they must balance resource allocation and trajectory planning between communication and sensing objectives, which may either be competing or complementary. Furthermore, ISAC is an emerging paradigm, and as the technology matures, the place of UVs in enabling these applications will continue to develop.

§.§.§ Relaying and Cooperative Communication

In many scenarios, UVs can work cooperatively to enhance communication in the network. A key example is the use of UVs as relays, either by creating a chain of communication links to extend connectivity via multihop relay or coordinating transmission to achieve distributed beamforming.

A generic setting is the following: $M$ sources aim to communicate with $M$ destination nodes, but direct communication is not possible due to obstruction of a LoS path or significant distance. Facilitating this communication are $N$ UV relays which will move to locations $q_{n},\,n\in\{1,...,N\}$. The communication performance for each source-destination pair (SDP) is given by $C_m,\,m\in\{1,...,M\}$, which represents metrics such as data rate or bit error rate (BER). For each SDP, let $S_m$ be an abstract object representing the routing path from source to destination (e.g., a sequence of UVs), and let $\mathcal{R}_{n,m}$  denote the communication resources, allocated by UV $n$ to SDP $m$, with $\mathcal{R}_m=\{\mathcal{R}_{n,m}| \, n=1,..., N\}$. Then, a high-level problem can be formulated as below:

\beginmaxi!|s|[2]
{q_i}_i=1^N, {ℛ_m}_m=1^M,
∑_m=1^M C_m(ℛ_m, S_m, {q_i})

g_com({ℛ_m}) ≤0
g_r({S_m}) ≤0 .
\endmaxi!
Constraint (<ref>) ensures that the placement of the UVs is valid, e.g., enforcing minimum distances between UVs, while Constraint (<ref>) ensures that communication resources are not over allocated. Constraint (<ref>) requires that communication routing is valid, e.g., there are no cycles. For clarity of exposition, we have written the SDP-level communication performance, $C_m$, as a function of only the resources allocated to SDP $m$, but more generally, e.g., when communication for the different SDPs does not occur over orthogonal channels, $C_m$ may be a function of resources allocated to other SDPs as well. We next look at examples of these types of problems from the literature, focusing first on multihop routing, then proceeding to distributed beamforming approaches.

Multihop mobile relaying:
Multihop relaying [167] (see Fig. <ref> (top)), departs from the conventional point-to-point communication model by introducing a chain of data transmission through intermediary nodes. Such approaches can extend coverage and hedge against NLoS conditions. UVs can effectively serve as relays in multihop scenarios, as their inherent mobility and adaptability prove beneficial in responding to evolving communication conditions, such as variations in communication channels and the mobility of end-users.

As an example of such a scenario, [148] considers multiple UVs moving from initial positions to find locations which minimize the end-to-end bit error rate (BER) for a multihop relay scenario in the presences of obstacles and stochastic communication channels. For certain conditions, the problem is shown to be concave, and the KKT conditions can be used to produce solutions. [149] considers an ad hoc network of robots which must support multiple source-destination communication flows while performing an abstract task. For tractability, the problem is decomposed, with communication routing found using a second-order cone program (SOCP) [168], local control laws derived using barrier functions [169], and $\text{RRT}^*$ [108] providing high-level trajectories.

In [156], the authors consider a problem such as the one described above. They propose a Block Coordinate Descent (BCD) [170] method to find the positions of the UVs which minimize the total transmit power under a coverage constraint. The work of [157] examines a similar UV multihop scenario and proposes a particle swarm optimization approach to design the UV positions for maintaining connectivity. Finally, the authors of [158] examine a scenario with multiple source-destination pairs where communication is facilitated by multiple UV relays in a multihop setting. They propose an alternating optimization scheme that combines BCD with Gibbs-sampling [171] to jointly optimize the UV positions as well as the bandwidth and power allocation, with the objective of maximizing the minimum communication rate among source-destination pairs.

Distributed relay beamforming: Distributed relay beamforming [172], akin to multihop relaying, involves source-destination pairs seeking information exchange. Relays between these pairs receive signals simultaneously and transmit them collectively to destinations in synchronous fashion. Despite its classification as a multihop relaying case with two hops, it is often studied independently due to nuanced domain-specific considerations. These nuances involve cooperative relay decision-making to form a unified beam at the destination.

In recent years, beamforming in the context of UVs has gained significant attention. For example, [151, 150]  focus on minimizing both motion and communication energy while jointly optimizing the positions of multiple UVs and their corresponding beam weights for distributed robotic beamforming. The paper considers an uncertain channel environment and shows how to incorporate channel prediction using a Gaussian-process-based prediction model. Near-optimal solutions are then found through a series of multiple-choice knapsack problems [173], which can be solved efficiently despite being NP-hard in the general case.

The study in [174] explores scenarios where multiple relays engage in distributed beamforming for a single source-destination pair, incorporating quantized feedback. The authors recommend using the Generalized Loyd Algorithm [175] to design the quantizer for specified beamforming weights. In a related domain, [176] delves into employing multiple relays for distributed relay beamforming in a two-transceiver system. Two scenarios are considered: optimizing relay beamforming weights to minimize total transmit power while meeting receiver SNR constraints, and maximizing the lowest SNR between two receivers under a total transmit power constraint. Proven unique solutions and proposed iterative schemes showcase linear complexity per iteration.
In cooperative jamming, [177] focuses on UV motion control and noise nulling to maximize collective secrecy rates. [154] optimizes relay positioning for acceptable receiver-side QoS and maximizes ergodic secrecy rates with an eavesdropper.

The works [147, 75] consider scenarios with one source-destination pair and multiple UV relays, aiming to select UV positions and beamforming weights to maximize a utility metric. Assuming underlying channels evolve as Gaussian processes with correlations across time and space, the authors propose a distributed stochastic optimization method. This approach predicts channels for candidate UV positions based on historical data, achieving closed-form expressions for optimal beamforming weights once positions are determined.

Takeaways: The use of UVs as mobile relay nodes in relay networks introduces the element of dynamic positioning, an additional controllable parameter for optimizing various aspects of system performance. When multiple UVs provide this relay service together, their operation must be optimized jointly. The cooperative paradigm, whether the team of UVs acts as a single relay chain or employs distributed beamforming, greatly influences optimal positioning and resource allocation.

Maintaining connectivity: A fundamental problem for UVs is how to efficiently move between two locations while maintaining connectivity.

§.§ Unmanned Vehicles as Consumers

Supporting connectivity for UVs is an expected feature of 6G networks, with related standards actively under development [62]. In this section, we present several fundamental problem formulations in which UVs consume the services provided by 6G systems, beginning with the problem of maintaining connectivity, then moving towards more complex problems of wireless data-offload scheduling with data causality constraints in the context of IoT and WSN data collection.

Sample papers addressing the optimization of UVs in the role of consumers.

UV Mission     Ref.     Objective     Tools     Notes

Connectivity     [178]     Motion and communication energy     $\text{RRT}^*$     Obstacle avoidance, uncertain channel environment.
[179]     Mission time     Graph theory, Convex optimization      UAV associates with nearest BS.
[180]     Handovers     Dynamic programming     Cargo UAV application.

2Reconnecting to
the Network     [181]     Distance to connectivity      Game theory, Graph theory, Dynamic programming     Probability of success modeled as independent across discretized space. Shown to be NP-hard.
[182]     Distance to connectivity     Ornstein-Uhlenbeck process     Accounts for log-normal shadowing and generic multipath fading.

Scheduling      [183]     Motion and communication energy      KKT conditions     Velocity and transmission rate optimized over a given trajectory.

Motion and communication energy     Optimal control     Abstract Bolza optimal control problem.

20Site Visiting      [186]     Sum-rate and completion time     DRL     Deep Q-learning for motion control to optimize a convex combination of sum-rate and task completion time.
[187]     Avg. Wait Time     Polling systems, MISOCP, stochastic planning     Data collection and relay. Optimal robotic routing locations/policies. Long-run analysis of stationary regime
[188]     Avg. Wait Time     Polling systems, DRL     Data collection and relay. Dynamic visit sequence. Handles long-run average cost criterion.
[189]     Data collection and wireless energy transfer     Multi-agent DRL    Split the UVs into two groups (data harvesting and energy transfer) and employ multi-agent DRL since the policies are correlated.
[190]     Data rate     DRL for partial observability     Buffer overflow can cause unsuccessful transmission that induces partial observability.
[191]     Motion and communication energy     MILP     Sensing mission, collects set amount of data from each site.
[192]     Motion and communication energy     Monte Carlo tree search     Trajectory must pass within a certain distance of each site.
[193]     Device energy     Sequential  quadratic programming     Energy reduction for IoT device communication
[194]     Number of devices served     MINLP, SCA, Branch, reduce, & bound     Shown to be NP-hard.
[195]     UV energy     Vehicle routing problem     Capacitated vehicle routing problem with time windows.
[196]     Motion and communication energy     MILP     Sites visited on Hamiltonian path.
[197]     Peak/Avg. AoI     Ant colony optimization     Sensor-node-to-UAV wireless communication, UV moves to BS before offloading (data muling).
[198]     Avg. AoI     BCD, SCA, Convex optimization     Considers communication over both WSN-to-UAV and UAV-to-BS links.

§.§.§ Maintaining Connectivity while in Motion

Consider a UV which must move between two points while maintaining connectivity with a remote server in order to, e.g., receive control and non-payload communication (CNPC) [199] or upload sensed data in real time. These problems can be frequently posed as shortest-path problems, with distance metrics that account for transmission and motion power. As a baseline example, consider a UV moving from a given initial position, $q_0$, to a final position, $q_f$, while maintaining the SNR between the UV and a remote BS above a minimum threshold, $\text{SNR}_\text{th}$, as illustrated in Figure <ref>. This baseline formulated is

\beginmini!|s|[2]
{ℛ[k]}_k=1^K, K
λ_1 ℰ_𝒦({q[k]}) + λ_2 ℰ_com({ℛ[k]}) + λ_3 Δt K

(<ref>)   Motion constraints
g_com(ℛ[k])≤0 , ∀k
SNR(q[k], ℛ[k])≥SNR_th  ∀k
q[0] = q_0
q[K] = q_f
\endmini!
where, $\Delta t$ is the length of each time step, $\mathcal{E}_{\mathcal{K}}(\{q[k]\})$ is the motion energy required to follow trajectory $\{q[k]\}$, as determined by the UV's energy model (see Section <ref>), $\mathcal{E}_{\text{com}}$ is the communication energy, $\Delta t K$ is the mission completion time, and $\lambda_i,\, i\in\{1,2,3\}$ are scalar weights. Thus, the objective consists of a weighted sum of motion energy, communication energy, and mission completion time, with the latter particularly important in, e.g., drone delivery problems. We can formulate many related problems by moving performance metrics between constraints and objectives. For example, the SNR constraint can be relaxed by including a penalty term in the objective for the distance or time traveled with $\text{SNR}<\text{SNR}_{\text{th}}$, and the mission completion time can be recast as a constraint on the maximum allowed time.

A version of this problem is studied in [178], in which a UV moves through a complex environment with obstacles while maintaining connectivity to the core network via any one of several TBSs. Both fixed and variable transmit power problems are considered, and due to the additive nature of the objective function, the problem can be solved using a modified version of $\text{RRT}^*$. The resulting trajectories conserve communication energy by avoiding areas of poor channel quality. In a similar setting, [179] studies the problem of a UAV flying between an initial and final location in minimal time while maintaining connectivity with a set of TBSs. The authors find high-quality approximate solutions using tools from graph theory and convex optimization.

When the UV may choose from among multiple BS for connectivity, it should avoid excessive handoffs, which are expensive from the standpoint of communication resources. This is accounted for in [180], which studies the problem of a cargo UAV flying at a fixed altitude between two points which seeks to minimize a linear combination of energy consumption and number of handoffs while constrained to be disconnected no more than a given percentage of the path. The path planning problem is solved using a dynamic programming approach. As the flight altitude increases, the energy consumption increases and handoffs decrease, while stricter disconnectivity constraints also increase energy consumption.

Maintaining connectivity can be viewed as a shortest-path problem with an appropriate choice of distance metric. Consequently, graph theory and DP provide good approaches to solving these problems. The resulting paths improve communication by seeking out areas of high channel quality, possibly at the expense of additional motion energy.

§.§.§ Minimizing Distance to Connectivity with Stochastic Channel Models

Finding a connected spot: For a UV that is initially disconnected, planning a trajectory that minimizes the time to restore connectivity is challenging, particularly when the channel is not fully known.

When a UV loses connectivity, either while fulfilling a mission objective or due to the difficult-to-predict variations in the communication channel, it may need to quickly return to an area where communication is possible. Here, a problem naturally arises: how can the UV minimize the time it takes to find a connected point?

Consider a UV initially at a position $q_0$ where the channel quality does not permit communication with a distant BS. Without loss of generality, assume the UV moves at a constant velocity, $v$, and transmits with a fixed transmit power, $P$. The channel is not known, and at each point, the condition (<ref>) is met with some probability tied to the spatially-correlated statistics of the underlying channel. Further suppose there exists a known, non-empty set of points, $Q_\text{f}$, where, with probability 1, the channel is sufficiently strong to allow for communication when the UV is at any of those points. The problem of minimizing the distance traveled to connectivity can be posed as follows:

\beginmini!|s|[2]
{q[k]}_k=1^K, K
𝔼[∑_k=0^k_fpd-1  q[k] - q[k+1]_2 ]

Constraints (<ref>), (<ref>)q[K] ∈Q_f
\endmini!
where $k_\text{fpd}$ is defined as $\min \{k\,|\,\text{(\ref{constraint:user.MC.SNR}) holds}\}$, and, as $k_\text{fpd}$ is a random variable due to imperfect channel information, the expectation is taken with respect to $k_\text{fpd}$. Importantly, Constraint (<ref>) ensures the expectation is always well-defined.

The problem of minimizing the expected distance to connectivity in a general sense is studied in [181]. A graph-theoretical formulation is put forth, where at each node, the UV has some probability of success, and the UV must plan a path through the graph which minimizes the expected distance traveled until success occurs. A Markov decision process describes the problem, which is proven to be NP-hard. To find high-quality approximate solutions, the problem is recast as a potential game with the nodes acting as the players, and log-linear learning is used to find near-optimal solutions. Results show that first seeking out nearby locations with even modest probabilities of connectivity can reduce average distance traveled compared to moving directly to the nearest connected point.

When the UV's trajectory is given beforehand, [182] provides rigorous analysis of the first-passage distance (FPD), the expected distance the UV will travel before reconnecting. An Ornstein-Uhlenbeck [200] process is shown to model log-normal channel shadowing along a straight line path, which permits an analytical expression for the FPD. If small-scale fading is not negligible, a recursive formulation is developed that allows the FPD to be calculated over a discretization of the path.

Quickly finding connected points is critical for both the safety and performance of the UV. These problems have a similar shortest-path element as those in (<ref>), but when accounting for shadowing and path loss statistically, they become stochastic shortest path problems [110], which are much more difficult to solve.

§.§.§ Scheduling Wireless Data Uploading

Scheduling of wireless data offloading from a UV to a BS with complementary trajectory design can be a challenging problem, but it can also significantly reduce the energy needed for data transfer.

In a number of scenarios, UVs must offload data to a remote station while traveling. In these situations, the UV should jointly optimize its trajectory and the corresponding adaptive transmission power/rate over the whole trajectory, for efficient data transfer. The general problem scenario is shown in Figure <ref>.

A core problem involves a UV moving from $q_o$ to $q_f$, while offloading $D_0$ bits of data along the way. The problem can be formulated starting with Problem (<ref>) and replacing the SNR constraint (<ref>) with a data offloading constraint:

D[0] = D_0,

D[k] = max(D[k-1] - Δt B r(P[k], q[k]),0),

D[K] = 0,

where $B$ is the bandwidth of the communication channel, $r(P,\,q)$ gives the data rate between the UV and the BS when the UV transmits with power $P$ from position $q$. Alternatively, an additional term capturing the timeliness of data offloading can be added to the objective.

Scheduling when to upload data adds complexity, particularly for optimization of the transmit power. In problem (<ref>), it is easy to see that transmit power should be chosen to satisfy Constraint (<ref>) with equality, so that the choices of transmit power, $P[k]$, are not directly coupled over time. However, the introduction of Constraint (<ref>) results in tighter coupling of transmit power across time. We next present interesting studies related to this problem.

For a UV moving along a predefined trajectory, [183] considers jointly planning data transmission (i.e., communication rate/power) and velocity in order to minimize total energy costs. It is shown that the UV should move quickly through areas of poor communication channel quality and more slowly through regions of high channel quality (via an optimized velocity profile), so that it spends more time transmitting in more favorable areas. Using the channel prediction framework of [59], an adaptive, online strategy is proposed. The proposed method significantly reduces energy consumption compared to the baseline where the UV moves with constant speed and transmits with constant spectral efficiency.

A continuous time, energy-minimizing version of the problem appears in [184], where it is reformulated by moving Constraints (<ref>) and (<ref>) into the objective as additional penalty terms. This makes it in the form of an abstract Bolza optimal control problem [201]. This work is extended to a multi-UV case in [185], which proposes a distributed version of the optimal control solution. Results show that the UVs seek out areas of better channel quality and perform most of the wireless data uploading there.

Jointly adapting communication rate and UV speed to channel conditions can greatly improve communication energy efficiency. This may motivate planning trajectories that seek out areas of better channel quality, though the associated motion energy expenditure must also be weighed as well.

§.§.§ Site-Visiting Problems

As a final set of problems, we consider scenarios where the UV visits several sites to collect data either with onboard sensors, such as cameras, or through communication channels, e.g., in the case of data collection from wireless sensor nodes or IoT devices. Additionally, the problem may involve transferring the collected data to a remote BS, either over a wireless communication link, in which case the UV's role resembles that of a relay, or by physically moving the data to an access point, sometimes referred to as data muling [202]. A general problem scenario is illustrated in Figure <ref>.

Site visiting problems arise in a number of situations, including surveillance, surveying, and IoT/WSN systems. In addition to collecting data from the sites, the UV may also need to transfer the data back to a remote BS in a timely manner.

Usually, site visiting constraints ensure that all sites are visited, and the visit order must be determined, which introduces a combinatorial aspect to these problems. Site visiting constraints can take many forms. At the simplest, they may constrain the trajectory to pass through a (possibly singleton) set $\mathcal{Q}_\omega$, e.g., there exists a point in the trajectory, $q[k]$, where $q[k]\in \mathcal{Q}_\omega$ for all $\omega$ in the set of sites $\Omega$. When sites correspond to the locations of IoT devices or sensor nodes, the visiting constraint may ensure that a set amount of data is uploaded to the UV.

Site visiting problems also bring with them a distinct set of metrics, which may enter the system either as constraints or objectives. When the sites correspond to communication nodes, the energy required to upload the data to the UV is an important measure of system performance, as these nodes are frequently quite energy-limited. Additionally, metrics such as average delay and Age of Information (AoI) [203] capture the timeliness of data transfer to the BS.

A general site visiting problem can be expressed as:

{q[k]}, ℛ_UV,
f_GLB( ℰ_UV,  {ℰ_ω}_ω∈Ω, f_T({q[k]}, ℛ) )

Kinematic constraints
g_c({q[k]}, ℛ) ≤0 Comm. res. constraints

g_D(ℛ, {q[k]}) ≤0

h_D(ℛ, {q[k]}) = 0
Site Visiting

where $\mathcal{E}_\text{UV}$ is the total energy consumed by the UV, $\mathcal{E}_{\omega}$ is the energy expended by site $\omega$ to upload data to the UV, $\mathcal{R}_{UV}$ and $\mathcal{R}_\Omega$ are the communication resource variables for the UV and sites, respectively, with $\mathcal{R} = \mathcal{R}_{UV} \cup \mathcal{R}_\Omega$, and $f_T(\{q[k]\}, \mathcal{R}_{UV}, \mathcal{R}_\Omega)$ is a measure of the timeliness of data collection, such as mission completion time or AoI.

Data Collection for IoT/WSN: A number of works focus on the problem of data collection from IoT/WSN devices without explicitly modeling the data transfer from UV to BS. In these cases, the energy consumed by the devices, $\mathcal{E}_\omega$ is often given greater attention. For example, [193] studies the problem of deploying multiple UAVs to collect data from IoT devices in a way that minimizes energy expenditure by the devices. Using BCD and sequential quadratic programming, the authors find device-to-UAV associations, UAV position, and device transmit power schedules that greatly reduces the total energy consumed by the IoT devices compared to the scenario where the UAVs are statically placed in the environment.

In [194], the authors consider the challenging problem of multiple UAVs collecting data from IoT devices under time constraints, that is, when the devices are only awake for a subset of the time. They formulate a mixed-integer nonlinear program (MINLP) to find a trajectory and an allocation of communication resource which maximize the total number of IoT nodes serviced over a finite time frame. A similar time-constrained, multi-UAV problem is considered in [195], which instead focuses on the problem of minimizing the total UAV energy consumption while ensuring that a set amount of data is collected from each node. The problem is solved using results from the literature on capacitated vehicle routing problems with time windows (CVRPTW).

Sensing and Data Transfer: A different set of problems focus on the UV-to-BS offloading aspect of these problems. These problems are similar to the data offloading problem in Section <ref>, but with modified data dynamics. Specifically, after visiting each site, an additional amount of data enters the UVs onboard memory, so that Eq. (<ref>) is replaced by

\begin{equation*}
D[k] = \text{max}\big(D[k-1] + \sum_{\omega\in\Omega} I_{\omega}[k] D_\omega - \Delta t\, B \,r(P[k], q[k]),\,0\big),
\end{equation*}

where $I_\omega[k]$ indicates if site $\omega$ is visited at time step $k$ and $D_\omega$ is the data associated with site $\omega$.

In [191], a UV visits a number of sites where it collects data and transmits them to a remote station. Initially, the problem is considered in the absence of the sites, and the UV simply decides whether it is more energy efficient to transmit from the current location or to first expend energy to move to another location and then transmit. Then, the problem of minimizing energy consumption by co-optimizing the UV's motion and communication strategies for retrieving and relaying data from the site back to a BS is formulated as a MILP.

A related problem is studied in [196], in which multiple UVs service a number of sites to keep a generic Quantity of Interest (QoI) from growing unbounded while maintaining communication with a remote BS. Two cases of connectivity are considered. In the first, the UVs must establish a communication link to a remote BS after each site visit. In the second, the UVs must only connect once during their cyclic trajectories. The problem of minimizing the total energy consumption while guaranteeing boundedness of the quantity as well as respecting the connectivity constraints is formulated as a MILP.

Data Collection and Relay: A final set of problems considers uplink communication both from the sites to the UV and from the UV to the BS. These types of problems are the most general and frequently feature metrics related to the timeliness of end-to-end data transfer. For example, [187] studies the problem of minimizing the average wait time in a data collection and relay scenario. The authors consider a single UV relaying data between a set of spatially diverse and disparate source-destination pairs, where heterogeneous data accumulation rates at the sources necessitate different visiting frequencies. They then find the optimal robotic routing policies (e.g., optimal relay locations as well as the sequence with which the pairs are serviced). More specifically, the problem of minimizing the average time a bit of data waits at the source before being relayed to the destination is approached from a polling systems perspective [204], and several fundamental properties of the system are proved using results from queuing theory, all the while showing how stochastic robotic routing can be handled.

Another metric of the timeliness of data transfer is Age of Information (AoI) [203], which measures the recency of the most up-to-date data at the BS. In [197], the authors look at the problem of minimizing the AoI in a scenario where multiple UAVs are dispatched to collect data from clusters of devices and then return to a data center to offload the data. The AoI accounts for the travel time as well as the device-to-UV and UV-to-BS communication times. The problem is approached by first partitioning the sites among the UAVs with a variant of K-means [205], then using ant colony optimization [206] to find the visit order for each UAV. For a single UAV, end-to-end AoI is also addressed in [198], but the data transfer to the BS is constrained to happen directly after receiving the data from the device, that is, the UAV operates as a half-duplex relay. The authors use BCD along with SCA and convex optimization methods to produce trajectories and transmit power profiles which minimize the average AoI while respecting communication energy constraints for the UAV and devices.

The combinatorial element of site visiting and wireless offloading with causality constraints together produce very complex problems with elements of both continuous and discrete optimization coupled strongly across time. Furthermore, when the UVs collect data via wireless uploading from devices, additional considerations come into play, including the energy consumed by the devices and possibly timing windows. For end-to-end data transfer with data generating dynamically at each site, the system may be viewed as a network of queues [207], where data accumulating in queues at the sensor nodes are uploaded to the queue on the UV which in turn uploads the data to a BS. The system also bears resemblance to a polling system, in which a single server serves multiple queues, with some idle time incurred when switching between the queues.

§.§ Stochastic Dynamic Programming Formulations

Many of the analytical formulations of the previous section (along with the accompanying solution methods) account both for the inherent stochasticity of UV-assisted wireless systems and for the optimization of system parameters over an extended period of system operation. Nonetheless, the recent paradigm of DRL introduces a compelling framework for advancing these formulations. In a dedicated subsection, we delve into the evolving landscape of stochastic dynamic programming formulations within UV-assisted networking. Furthermore, we explore the application of learning-based methods (and more prominently DRL) to effectively address such settings.

§.§.§ Unmanned Vehicles as Providers

The current subsection provides an overview of UV problem formulations that adhere to the general framework of stochastic dynamic programming and consider that the UV acts as a provider of system services.
In such role, the UV can be a central node that provides connectivity with the core network, for example it could be an aerial base station (BS). Alternatively, the UV can be acting as an intermediate/relay node that extends coverage and connectivity.

UVs as Base Stations: Assume a UV acts as an aerial BS and serves $\mathcal{U}$ ground users, that are located at specific positions. The elevation level of the UV stays constant during the whole time horizon of operation. The UV starts at position $q_{0}$ and is supposed to finish its flying trajectory at position $q_{K}$ at time $K$. The channel at time $k$ between the UV and the ground user $i$, $\mathcal{H}_{i}$,
depends on the position of the UV $q[k]$, $\mathcal{H}_{i}(q[k])$. The transmission power of the UV at time step $k$ is denoted as $P[k]$. The communication rate for user $i$ at time $k$ is $\log_{2} ( 1 +  \text{SNR}(\mathcal{H}_{i}(q[k]))$. The set of all valid trajectories in the UV operation space that amount to overall time $T$ is denoted as $\mathcal{Q}$.

\beginmaxi!|s|[2]
𝔼_𝒬 [ ∑_k=0^K ∑_i ∈U  log_2( 1 + SNR(ℋ_i(q[k]) ) ]

q[k] ∈[q_min, q_max]^2  ∀k ∈[0, K]
log_2(1 + SNR(ℋ_i(q[k])) ≥ζ_min
∑_k=0^K P[k] ≤P_total
\endmaxi!

The goal is to design the UV trajectory  over time such that the sum of total communication rate is maximized in expectation. The constraints correspond to the fact that the initial and final positions of the UV are predetermined, the UV area of motion is confined within a $2D$ rectangular space, and there should be fairness among end users in the sense that the communication rate of any individual user is not below a certain threshold at all time steps. Finally, the overall power budget for transmission is below a certain predefined threshold.

The problem formulation calls for approaches where optimization unfolds through time. DRL is an organically suitable paradigm for tackling the problem. In order to employ DRL for settings that fall under the above formulation, the Markov decision process (MDP, see Section <ref>)must be defined:

* State: $\mathbf{s}[k] = f_{s}(q[k])$, the state is a vector that is a function of the UV position at time step $k$. In the simplest case, the state can be the position itself, or a window of previous positions.

* Action: $\mathbf{a}[t] = f_{a}(\delta q[k])$. The action is a function of the displacement of the UV for time step $k$. The displacement can be a vector that represents the difference between the position at the current step and the position in the subsequent step. The range of the displacement should adhere to the imposed kinematic constraints.

* Reward: $r[k] =  \sum_{i \in U}  \log_{2}\left( 1 + \text{SNR}(\mathcal{H}_{i}(q[k + 1]) \right)$. The reward is the overall communication rate achieved at time step $k+1$. One should note that, by the definition of the MDP, the reward is calculated after performing the action at the current state. In order to facilitate the fairness constraint, the reward can be augmented by the difference between the minimum communication rate among end users at each time step and the minimum acceptable communication rate $\zeta_{min}$ and the transmission power at every time step.

A significant body of work exists that falls under the aforementioned formulation. The work in [139] examines a scenario where a UV acts as a BS and the goal is to optimize the total communication rate over time. A Q-learning [208] approach is considered and the space of UV motion is discretized into a fine grid. In [140], the authors focus on a similar scenario with multiple UV-BSs, where the goal is to maximize a weighted throughput objective that relates to fairness among end users. A Multi-agent Deep Deterministic Policy Gradient approach is proposed that disentangles the contributions of the individual UVs. The authors in [144] examine a setting where a UV acts as a BS serving end users in the context of Open Radio Access Networks [209] under MIMO fading channels. The UV motion is discretized, and two RL approaches are proposed to maximize the network's throughput over time. The first approach is a variant of SARSA [210] and the second approach is predicated on offline Q-learning [211].

The work of [141] examines the trajectory optimization of multiple UVs that serve a ground cellular network. The goal is to design the trajectories of the UV swarm and the task completion time in order to minimize energy expenditure, subject to a constraint that pertains to the continuous connectivity of the UVs with the ground cellular network. The authors consider two scenarios: one where the radio channel maps are known to the swarm and one where the channel maps are initially unknown. For the scenario where the channel maps are known, they employ a convex optimization approach along with a dynamic-weight shortest path algorithm. For the second scenario, the aforementioned approach is non-applicable, therefore the authors propose a deep Q-learning approach with a dueling architecture in order to optimize the UV trajectories over time. The work of [142] examines a multi-UV cellular network, where the UV trajectories are optimized with a TD-learning [212] approach in order to minimize the overall task completion time. The state-action value function is linear and tile-coding [213] is implemented in order to ameliorate the effect of the large state-action space. A similar scenario is considered in [143], with the difference being that only one UV is considered. A Deterministic Policy Gradient algorithm is developed to optimize the UV motion with the goal of maximizing the uplink sum-rate.

Cooperative Communication: Besides the UV operating as a BS, there has been increasing interest in applications where the UVs are deployed as intermediate nodes in the cascade of the network [214]. The inherent mobility of UVs can be leveraged in order to increase coverage and improve the QoS on the receiver side.

Let us consider $I$ UVs acting as relays that facilitate the communication between a source $S$ and a destination $D$. The locations of source and destination are fixed throughout the operation, therefore the dependence on their location is dropped from all expressions. The source transmits symbol $s[k] \in \mathbb{C}$ with power $P$ and the signal received at UAV/relay $i$ which is at position $q_{i}[k]$, $x_{i}[q_{i}[k], k]$ depends on the position of the relay and the channel from the source to the relay/UV. The channel from the source to the relay, $\mathcal{H}_{Si}$ also depends on the relay position and possible on the time (assuming spatio-temporally correlated channels [82]). The signal received by relay $i$, $x_{i}[k]$ can be generally expressed as:

\begin{equation}
x_{i}[k] = x_{i}\left(q_{i}[k], \mathcal{H}_{Si}[k]\right) .
\end{equation}

When considering narrowband flat fading channels, the effect of the channel on the transmitted signal is multiplicative [127, 82]. On the other hand, wideband channels of 6G systems are frequency selective [215, 216] (refer to Section <ref>).