#  Prescribing Optimal Health-Aware Operation for Urban Air Mobility with Deep
Reinforcement Learning

Mina Montazeri  Chetan S. Kulkarni  Olga Fink  Laboratory of Intelligent
Maintenance and Operation Systems, EPFL, Switzerland.  Urban Energy Systems
Laboratory, Swiss Federal Laboratories for Materials Science and Technology,
Dübendorf, Switzerland.  KBR, Inc., NASA Ames Research Center, Mountain View,
CA 94035, USA

###### Abstract

Urban Air Mobility (UAM) aims to expand existing transportation networks in
metropolitan areas by offering short flights either to transport passengers or
cargo. Electric vertical takeoff and landing aircraft powered by lithium-ion
battery packs are considered promising for such applications. Efficient
mission planning is crucial, maximizing the number of flights per battery
charge while ensuring completion even under unforeseen events. As batteries
degrade, precise mission planning becomes challenging due to uncertainties in
the end-of-discharge prediction. This often leads to adding safety margins,
reducing the number or duration of potential flights on one battery charge.
While predicting the end of discharge can support decision-making, it remains
insufficient in case of unforeseen events, such as adverse weather conditions.
This necessitates health-aware real-time control to address any unexpected
events and extend the time until the end of charge while taking the current
degradation state into account. This paper addresses the joint problem of
mission planning and health-aware real-time control of operational parameters
to prescriptively control the duration of one discharge cycle of the battery
pack. We propose an algorithm that proactively prescribes operational
parameters to extend the discharge cycle based on the battery’s current health
status while optimizing the mission. The proposed deep reinforcement learning
algorithm facilitates operational parameter optimization and path planning
while accounting for the degradation state, even in the presence of
uncertainties. Evaluation of simulated flights of a National Aeronautics and
Space Administration (NASA) conceptual multirotor aircraft model, collected
from Hardware-in-the-loop experiments, demonstrates the algorithm’s near-
optimal performance across various operational scenarios, allowing adaptation
to changed environmental conditions. The proposed health-aware prescriptive
algorithm enables a more flexible and efficient operation not only in single
aircraft but also in fleet operations, increasing the overall system
throughput.

###### keywords:

Mission planning, Prescriptive health-aware operation, Deep reinforcement
learning, Lithium-ion battery, Real-time control.

##  1 Introduction

Urban Air Mobility (UAM) represents a paradigm shift in transportation, aiming
to revolutionize urban mobility in densely populated metropolitan areas. With
the goal of addressing challenges such as traffic congestion, limited
accessibility, and environmental concerns, UAM offers a promising solution
through short-distance flights for both passengers and cargo [1, 2]. Among the
diverse range of aircraft types being explored for UAM applications, electric
vertical takeoff, and landing (eVTOL) aircraft, propelled by advanced lithium-
ion battery packs, have emerged as a particularly promising option [3, 4].
These cutting-edge aircraft possess remarkable potential in terms of their
ability to provide highly efficient, environmentally friendly, and
exceptionally quiet air transportation within urban environments [5].

In the context of UAM, efficient mission planning plays a crucial role in
optimizing the utilization of eVTOL aircraft and ensuring their viability as a
mode of transportation. The primary objective is to plan missions to maximize
the number of flights that can be performed on a single battery charge while
ensuring the successful completion of each flight or mission, even in the face
of unforeseen events or disruptions such as adverse weather conditions [6, 7].
Various methods have been explored for path planning, including guidance
algorithms using Markov Decision Processes (MDP) [8, 9], Reinforcement
Learning (RL) [10, 11, 12], Particle Swarm Optimization (PSO) [13, 14, 15],
visibility graph algorithms [16], Dynamically Directed Graph Algorithm (DDGA)
[3], evolutionary algorithm [17], and chimp optimization algorithm [18].
However, a common limitation of these approaches is the neglect of important
factors such as battery aging and the absence of joint mission planning with
real-time control of operational parameters, which are critical in real-world
UAM systems [19]. As batteries degrade over time, accurately predicting the
end-of-discharge becomes increasingly challenging, leading to increased
uncertainty in estimating the available energy for flight operations [20, 21].
These uncertainties pose significant challenges to mission planning,
necessitating innovative approaches to effectively address them.

While previous research has investigated the ability to predict the end-of-
discharge of batteries using both model-based [22, 23] and data-driven
approaches [24, 25, 20], the efficiency of this prediction in determining
optimal aircraft trajectories remains unexplored. Finding optimal trajectories
becomes more complex in the face of unforeseen events or disturbances, such as
adverse weather conditions, where the decision algorithm must discern whether
its low performance is due to the current battery health state or weather
conditions. As such, there is a pressing need to develop advanced algorithms
and methodologies that can effectively address the joint challenges of mission
planning and health-aware real-time control of operational parameters in the
context of battery aging and adverse weather conditions.

This paper proposes a deep reinforcement learning algorithm designed to
address the intricate challenge of simultaneous mission planning and health-
aware real-time control of operational parameters for eVTOL aircraft. The
algorithm takes into account crucial factors such as battery aging and
unforeseen events. Effective decision-making relies on a comprehensive
understanding of how battery aging and unexpected events impact discharge
trajectories, with variations specific to each mission. The proposed algorithm
leverages the recently introduced Dynaformer algorithm, a transformer-based
architecture [20], to forecast the voltage profile of the entire flight. This
prediction relies on a brief observation time period (20s) during which both
the current and voltage profiles are collected. Following this observation
phase, the algorithm implicitly infers battery aging information through the
encoder part of the Dynaformer and conditions the prediction of the discharge
trajectory based on the currently planned flight profile. The decision-making
process leverages deep Q-Learning, a form of deep reinforcement learning that
enables the aircraft to make optimal decisions, in scenarios where the action
space is continuous. Specifically, the algorithm empowers the aircraft to make
strategic decisions, such as selecting destinations and their order, as well
as tactical decisions regarding the altitude for each destination, all while
considering the information on the health condition.

To comprehensively evaluate the efficiency of our proposed algorithm, we
conducted extensive simulations using a National Aeronautics and Space
Administration (NASA) conceptual multirotor aircraft model. These simulations
are executed within a hardware-in-the-loop experimental framework,
encompassing realistic flight scenarios and diverse environmental conditions,
including various wind conditions [26, 27, 28]. Our assessment covered a range
of mission scenarios to gauge the performance of our proposed health-aware
prescriptive algorithm. These mission scenarios represent three distinct
flight scenarios. The first mission scenario comprises a single flight, while
the second scenario involves the aircraft reaching multiple destinations, each
assigned with varying levels of priority. The objective of this mission is to
reach as many high-priority destinations as possible. In the third scenario,
the mission requires the aircraft to reach all predefined destinations with
the minimum number of charging cycles.

Our research outcomes have significant implications for the operation of eVTOL
aircraft within UAM systems. Our proposed algorithm not only enhances the
operation of individual aircraft for diverse mission scenarios but also
contributes to the overall efficiency and sustainability of fleet operations.

In each of the three scenarios, the algorithm successfully achieved its
objective by determining optimal paths and altitudes for each flight, taking
into account the prevailing battery health status, wind conditions, and
scenario priorities. Therefore, optimizing operational parameters and path
planning, while considering battery degradation and uncertainties, is crucial
for ensuring the long-term viability and success of UAM as a reliable and
scalable transportation solution.

##  2 Methodology

###  2.1 Methodological Framework

The goal of this work is to develop a decision-making model capable of
simultaneously handling mission planning and health-aware real-time control of
operational parameters in various mission scenarios. The key innovation lies
in the incorporation of the battery’s health status and the ability to respond
to unforeseen events, such as adverse weather conditions, within our decision-
making algorithm. In this case, to make accurate decisions, it is crucial to
understand how the battery state and unexpected events affect the discharge
curve in a specific mission. To accomplish this, we employ the health-aware
discharge prediction module, which consists of a Dynaformer to forecast the
voltage profile of the entire flight. This forecasting relies on a brief
observation time period (20s) during which both the current and voltage
profiles are collected, as detailed in [20]. Subsequently, leveraging the
discharge prediction, we introduce a decision-making agent employing deep
reinforcement learning (DRL). This empowers the aircraft controller to acquire
optimal decisions even in the absence of precise knowledge about flight
circumstances, such as adverse weather conditions. Consequently, the aircraft
can determine destinations (mission planning), their sequence (mission
planning), and the altitude for each destination (control of operational
parameters) using DRL while considering the current system health state for
decision-making. To ensure optimal decision-making, it is crucial to
appropriately define the state and reward. In this work, the state vector
encompasses key information, including the predicted voltage discharge curve,
implicitly inferred battery health representation, next destination locations,
and the count of destinations reached. This comprehensive state representation
provides the necessary environmental information for making optimal decisions.
Regarding the reward, the aircraft receives positive reinforcement upon
successfully reaching each destination. However, substantial penalties are
imposed if the battery voltage falls below the end-of-discharge level at any
point during the mission, before reaching the destination, or if the aircraft
arrives at the charging station. This penalty structure is designed to
incentivize the aircraft to efficiently reach multiple destinations within a
single charging cycle, provided the battery voltage remains above the end-of-
discharge threshold.

To assess the effectiveness of our proposed algorithm, we conducted
simulations employing a NASA conceptual multirotor aircraft utilizing the Li-
ion battery pack [26, 27, 28]. These simulations were executed within a
hardware-in-the-loop experimental framework, encompassing realistic flight
scenarios with various battery health states and varying operational wind
conditions. Since the health-aware discharge prediction module is available at
the level of a single cell, the simulator data is then scaled down to a single
18650 cell. All subsequent calculations are performed using the scaled values.

An overview of the proposed health-aware optimal operation framework is
illustrated in Figure 1. In the following, the main building blocks of this
figure are explained in detail.

Figure 1:  Overview of the proposed health-aware optimal operation method for
urban air mobility.

###  2.2 Health-Aware Discharge Prediction Module

To make an optimal decision for joint mission planning and real-time control
of operation parameters, we first need to predict the voltage discharge curve
of the battery under the current battery health conditions. To accomplish
this, we develop the health-aware discharge prediction module (red module in
Figure 1) that predicts the voltage discharge curve by considering the impact
of battery health on the voltage discharge trajectories. To design this
module, we employ a Dynaformer that forecasts the voltage profile of the
entire flight, relying on a brief observation time period (20s) during which
both the current and voltage profiles are collected [20].

In the first step, a substantial and diverse dataset is generated,
incorporating synthetic voltage curves derived from distinct input current
profiles and various health conditions, covering a broad spectrum of
combinations of health parameters. This involves the utilization of the
recently introduced open-source NASA simulator of the electrochemical battery
model, implementing the single-cell battery model outlined in [29, 30]. The
foundational element required for constructing the dataset is a set of
parameters representing the battery’s degradation level. This study
concentrates on two such parameters: $q_{max}$, capturing the total amount of
available active Li-ion, and $R_{0}$, accounting for the increase in internal
resistance. To create this comprehensive dataset, a significant number of
$(q_{max},R_{0})$ pairs and current profiles are sampled, ensuring that the
resulting voltage curves generated by the simulator display maximum diversity.
For each flight the values of $q_{max}$ and $R_{0}$ were randomly drawn from
uniform distributions with supports between 5000 C and 8000 C and 0.017
$\Omega$ and 0.45 $\Omega$, respectively. All other parameters and initial
conditions are maintained at typical values assumed for Li-ion 18650 batteries
(see table 1 in [29]).

Dynaformer employs a transformer neural network architecture, as depicted in
Figure 1. In line with observations from [20], the encoder component of
Dynaformer implicitly learns the health state of the battery from
observations, even in the absence of ground truth information regarding
degradation conditions, which is typically unavailable in real-world
applications. Specifically, the encoder infers the health state with only the
initial segment of current/voltage profiles. In this implementation, the first
20 seconds of both the current and voltage profiles are used as inputs to the
encoder for health inference. Subsequently, based on the encoder’s output, the
decoder predicts the full voltage discharge trajectory extending until the end
of discharge, conditioned on any given input current load profile. Overall,
the Dynaformer module plays a crucial role in our framework for health-aware
discharge prediction, facilitating effective decision-making for the
operational flight’s parameters with a focus on health awareness.

###  2.3 Deep Reinforcement Learning Agent

The deep reinforcement learning agent (green module in Figure 1) we developed
focuses on optimizing critical operational decisions for aircraft, with a
primary emphasis on the health state of the battery. In this study, we
explicitly consider two key operational parameters: determining the optimal
altitude and establishing the most efficient sequence for reaching
destinations. While other variables like speed could be a potential additional
parameter, our investigation specifically centers on these two decisions
within the framework of the algorithm. Figure 2 provides a comprehensive
overview of agent-environment interaction in the DRL method.

Figure 2:  The agent-environment interaction in deep reinforcement learning
method

To ensure the successful implementation of DRL, it is essential to define the
learning environment. Within our framework, the environment comprises three
key components: the health-aware discharge prediction module, the flight
simulator, and the adverse weather conditions sampler. The state vector
integrates information from the environment, encompassing the following data:

  * 1.

The predicted voltage discharge curve for the mission up to the next
destination, which is obtained by inputting the simulated current curve from
the flight simulator into Dynaformer.

  * 2.

Implicitly inferred battery health representation, which is derived by
inputting the initial segment of the current/voltage profiles from the flight,
approximately 20 seconds in duration, into the encoder. The encoder’s output
provides the inferred health status.

  * 3.

The next destination locations.

  * 4.

The count of destinations reached so far.

Thus, the state vector can be expressed as follows:

$s=[PC_{1},PC_{2},V,(x,y),i],$ (1)

where $PC_{1}$ and $PC_{2}$ represent the first two principal components of
the Dynaformer’s encoder embedding. These components have been demonstrated to
exhibit strong correlations with degradation parameters $q_{max}$ and $R_{0}$,
capturing the total amount of available active Li-ions and the increase in
internal resistance, respectively [20]. Here, $V$ denotes the voltage of the
battery at the present destination. The aircraft’s coordinates at the current
destination are given by $(x,y)$. The variable $i$ represents the number of
destinations that the aircraft has reached up to the current destination. The
aircraft’s action involves determining the next destination and the altitude
at which it will fly to reach the destination, which can be expressed as
follows:

$a=[(x^{\prime},y^{\prime}),h],$ (2)

where $(x^{\prime},y^{\prime})$ represents the coordinates of the next
destination, and $h$ denotes the altitude at which the aircraft must fly to
reach that destination. In the algorithm proposed in this study, the aircraft
is rewarded upon successfully reaching each destination, with rewards based on
pre-determined priority. However, if the battery voltage drops below the end-
of-discharge level at any destination or before reaching the destination, the
aircraft incurs a significant penalty. Additionally, the aircraft incurs a
penalty upon reaching the charging station, albeit smaller compared to when
the battery voltage falls below the end-of-discharge level. This penalty
structure encourages the aircraft to reach multiple destinations within a
single charging cycle, maximizing both performance and efficiency. Thus, the
reward can be formulated as follows:

$r=\begin{cases}\alpha,&\text{if Battery's voltage$>EOD$},\\\ -\beta,&\text{if
aircraft reaches charging station $\&$ Battery's voltage$>EOD$},\\\
-\gamma,&\text{if Battery's voltage$<EOD$},\end{cases}$ (3)

where $\alpha$, $\beta$, and$\gamma$ are three positive parameters and
$\gamma>>\beta$. The term $EOD$ denotes the end of discharge, defined as the
time when a battery reaches a certain pre-specified voltage cut-off value,
indicating full discharge.

Given the continuous nature of the state representation (where all elements of
the state vector, including the principal components and the battery voltage
in state space, are continuous parameters as per (Equation 1)), and the
discrete nature of the action space (as defined in Equation 2), we employ Deep
Q-Learning (DQL). DQL is a popular method for solving reinforcement learning
problems [31] and serves as a variant of the Q-Learning algorithm, utilizing a
deep neural network to approximate the Q-function. Q-Learning is a model-free
algorithm that learns an optimal policy by iteratively improving the Q-values
of state-action pairs. The Q-value for a state-action pair $(s,a)$ represents
the expected cumulative reward achieved by taking action $a$ in state $s$ and
then following the optimal policy. Q-Learning updates these Q-values using the
Bellman equation:

$\displaystyle Q(s,a)$ $\displaystyle\leftarrow
Q(s,a)+\alpha[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)],$

where $s^{\prime}$ is the next state, $r$ is the reward, $\alpha$ is the
learning rate, and $\gamma$ is the discount factor.

Deep neural networks (DNN) serve as powerful function approximators capable of
learning complex mappings between inputs and outputs. By combining Q-Learning
with DNN, Deep Q-Learning can effectively solve reinforcement learning
problems characterized by high-dimensional state spaces and discrete action
spaces. The Q-values are computed using a deep neural network with state $s$
as the input and action $a$ as the output:

$\displaystyle Q(s,a;\theta)$ $\displaystyle=f_{\theta}(s,a)$

where $f_{\theta}$ is the neural network parameterized by $\theta$. The loss
function corresponds to the mean squared error between the predicted Q-values
and the target Q-values:

$\displaystyle\operatorname{loss}$
$\displaystyle=\mathbb{E}[(y_{t}-Q(s_{t},a_{t};\theta))^{2}]$

To stabilize training, the target Q-values $y_{t}$ are computed using a
separate target network parameterized by $\theta^{-}$:

$\displaystyle y_{t}$
$\displaystyle=r_{t}+\gamma\max_{a^{\prime}}Q(s_{t+1},a^{\prime};\theta^{-})$

The target network parameters $\theta^{-}$ are periodically copied from the
Q-network weights $\theta$. This periodic copying ensures that the target
Q-values remain fixed for a certain duration, enabling the Q-network to adapt
slowly to changing targets. The Deep Q-Learning algorithm uses an
$\epsilon$-greedy exploration strategy to balance exploration and
exploitation. In essence, the algorithm aims to learn an optimal policy by
iteratively improving the Q-values through the integration of experience
replay, target networks, and DNN.

###  2.4 Flight Simulator

To generate distinct current profiles under various battery health conditions
and wind conditions as the input of the health-aware discharge prediction
module, we simulated flights using a NASA conceptual multirotor aircraft model
obtained from hardware-in-the-loop experiments [28] (blue module in Figure 1).
We employ the electro-chemistry-based model of a Li-ion battery pack,
developed by Daigle and Kulkarni [29]. This model, along with associated
prognostics algorithms has undergone rigorous verification and validation
procedures, including previous research conducted on electric unmanned aerial
vehicles [32]. We specifically focus on Li-ion 18650 batteries with an average
nominal voltage of 3.7V and a nominal capacity of 2200mAh. A pack size of
10S50P, as discussed in [28], is used as a reference battery pack. However,
the model is general enough that, with some modifications, it may be applied
to different battery chemistries [28].

In the hardware-in-the-loop experiments, cells, and packs are subjected to
charge/ discharge profiles expected during flights. The MACCOR Battery tester
system is programmed to run a power demand profile similar to the simulated
trajectory for analyzing battery performance at different operating conditions
for comprehensive analysis [28]. The Series 4000 MACCOR Battery Tester system
comprises essential components, including a test cabinet, a computer, and
software for both the tester and data analysis. Inside the test cabinet,
microprocessor controllers efficiently manage tests for dynamic loading of the
battery packs and collect data. Each controller board can support multiple
test channels, from 5V,20A upto 100V, 250A. Individually controllable
Programmable loads and power supplies are integrated into the test cabinet.
This design allows for the independent operation of each test channel,
enabling simultaneous testing of various parameters. Test programming is
performed on the computer through a user-friendly Graphical User Interface
(GUI) [28].

The dataset used for training the models is the High-Intensity Radiated Fields
(HIRF) Data $\\#$ 15 [33]. The dataset utilized in this study comprises
experimental data from HIRF tests conducted on the Edge 540, a small electric
unmanned aerial fixed-wing vehicle (e-UAV). The e-UAV employed in this
research represents a scaled-down version, measuring 33% of the size, of the
Zivko Aeronautics Inc. Edge 540-T tandem seat aerobatic aircraft. To perform
ground-based testing of the Edge 540-T hardware and software, the vehicle was
securely fastened in the HIRF test chamber. For further details regarding the
HIRF Chamber, reference can be made to a previously published report on UAS
radio frequency emissions testing [34]. The aircraft was positioned on
expanded polystyrene blocks, meticulously centered within the chamber. The
aircraft’s powertrain, equipped with a propeller, was operated while the
vehicle remained anchored to the chamber wall using a steel cable. The motor
and actuators of the aircraft were controlled remotely from a separate room,
utilizing the same radio control system employed during flight tests. Data
collection was performed during the experiments with the aircraft operating in
manual/auto control mode. Operating flight profiles, though similar, exhibited
variations based on factors such as wind conditions, and temperature.

##  3 Case Study

In this section, we present the results of the proposed health-aware optimal
operation method designed for urban air mobility outlined in Section 2. Our
evaluation focuses on conceptual multirotor aircraft model simulated flights
obtained from hardware-in-the-loop experiments as detailed earlier. To assess
the performance of the proposed framework under a range of realistic flight
conditions, we conduct simulations for various mission scenarios at various
altitudes, including 500 m (1640 ft), 1000 m (3280 ft), 2000 m (6561 ft), and
3000 m (9842 ft). In addition, different wind speeds are considered, including
headwinds of 13, 26, and 39 kts, as well as tailwinds of 13, 26, and 39 kts.
Further details about the dataset can be found in Section 2.4 and A. In the
following, the performance results of the proposed DRL agent across various
mission scenarios are presented.

###  3.1 Mission Scenarios

We explore various mission scenarios to assess the performance of our proposed
health-aware prescriptive DRL algorithm. These scenarios encompass three
distinct types, each representing a different flight mission. The first
mission consists of a single flight, while the second and third missions
involve multiple flights to various destinations. Each mission will be
discussed in further detail below.

####  3.1.1 Single Flight Mission

In the first scenario, we consider a mission with a single flight, aiming to
determine the optimal altitude as the operating parameter for the aircraft
under different adverse wind conditions and with different health conditions.
Optimizing the flight’s altitude can significantly extend the time to the end
of discharge for the battery, thereby allowing for longer flight times. Figure
3 illustrates the effect of varying altitudes on the corresponding voltage
discharge curve and current profile.

(a) current profile for different altitudes

(b) voltage discharge curves for different altitudes

Figure 3:  Effect of altitude on the voltage discharge curves and current
profile for a specific flight. The orange lines in (a) show the climbing,
cruise, and vertical landing times for a flight at an altitude of 1000 meters.

Nevertheless, the end of discharge is influenced by additional factors,
including adverse weather conditions and the degradation state of the battery.
Adverse weather conditions may lead to increased energy consumption and
diminished battery performance. Since weather conditions are typically
unpredictable, they can lead to unfinished flights or prevent the aircraft
from returning to the charging station. Similarly, the degradation state of
the battery, reflecting the extent to which the battery has lost capacity over
time, significantly impacts the end-of-discharge. Figures 4 and 5 illustrate
the effects of degradation parameters (total available lithium ions i.e.
$q_{max}$ and internal resistance i.e. $R_{0}$) and adverse wind conditions on
the end of the discharge. To ensure an accurate prediction of the end of
discharge for a single flight, it is crucial to consider both the current
degradation state of the battery and the uncertainties arising from adverse
weather conditions. To mitigate potential adverse events, such as unfinished
flights or loss of the aircraft, proactive reactions to both health conditions
and adverse events in real time are essential, as proposed in this algorithm.

In Figure 6, we present the results of the proposed framework for two distinct
single-flight missions: mission #1 and mission #2. The key difference between
these missions is that the second mission is twice as long as the first. The
altitude chosen in each learning episode by the aircraft is illustrated in
Figure 6(b). This figure shows that the aircraft learns to optimize its
altitude choice based on the mission duration. Specifically, in mission #1,
the optimal altitude is determined to be 500 meters, while in mission #2, it
increases to 1000 meters. This behavior can be attributed to the observation
depicted in Figure 3(a), where the climbing time increases with higher flight
altitudes, while the cruise time decreases. Therefore, for shorter flights,
flying at a lower altitude proves to be optimal, while for longer flights, the
aircraft finds it advantageous to endure additional climbing time in exchange
for reduced cruise time at higher altitudes.

(a) Current profile for specific flight ….

(b) Varying $R_{0}$ and keeping $q_{max}$ fixed

(c) Varying $q_{max}$ and keeping $R_{0}$ fixed

Figure 4:  Effect of varying degradation parameters on the voltage discharge
curves (with a constant current profile).

(a) Current profile for different wind condition

(b) Voltage discharge curves for different wind conditions

Figure 5:  Effect of wind condition on the voltage discharge curves and
current profile for a specific flight.

(a) Map of two single missions

(b) Altitude of aircraft in each mission

Figure 6: Altitude of aircraft in each mission during each episode using the
learning algorithm

####  3.1.2 Mission with a Single Charging Cycle

In this scenario, the aircraft is tasked with reaching multiple destinations,
each assigned varying priority levels, and returning to the charging station
before reaching EOD. Wind conditions may vary for reaching each destination in
one mission. The mission’s objective is to reach as many high-priority
destinations as possible before returning to the charging station. To achieve
this, the aircraft must decide on the destinations to reach, the order in
which to reach them, and the altitude at which to fly to each destination.
Figure 7 provides an illustrative example of such a mission, depicting a
scenario where the aircraft faces eight destinations. In this case, the
aircraft strategically selects destinations #1, 2, 3, and 6 based on their
priority levels, the current battery health state, and wind conditions. In the
following, we will discuss interesting observations and results that we
achieved in this mission.

Figure 7:  A mission with a single charge of battery. The number of stars next
to each destination shows the priority of that destination.

Figure 8 provides a more detailed visualization of this mission. As shown in
Figure 8(b), the aircraft can only reach a single destination if it maintains
a cruise altitude of 3000 m. However, during the flight path to the second
destination, the voltage of the aircraft’s battery falls below the EOD
threshold. On the other hand, by adjusting the cruise altitude to 1000 m, the
aircraft becomes capable of reaching an additional destination, allowing it to
visit two destinations before reaching the EOD limit (Figure 8(c)). It is
important to note that in both of these missions, the headwind is set to 39
kts.

(a) Map of an example mission

(b) Voltage discharge curve for cruise altitude=3000 m

(c) Voltage discharge curve for cruise altitude=1000 m

Figure 8:  Impact of flight altitude on voltage discharge curve and the number
of destinations that the aircraft reached before reaching EOD

(a) Number of reached destinations

(b) moving average of the number of reached destinations

Figure 9: Number of reached destinations and moving average of the number of
reached destinations in each episode using a learning algorithm.

To illustrate the learning progress of the aircraft using the proposed DRL
method in this scenario, we plot the number of destinations the aircraft
reaches before returning to the charging station or reaching the EOD in each
learning episode. This is depicted in Figure 9(a). The results demonstrate
that the aircraft gradually improves its performance and achieves more
destinations over subsequent episodes. However, it is important to note that
due to random sampling of the battery’s health and wind conditions for each
flight, the number of destinations that the aircraft reaches may vary even
after the aircraft learns its optimal strategy. To provide a more
comprehensive representation of the learning process, we compute the moving
average of the number of destinations reached for each learning episode, as
depicted in Figure 9(b). It is noteworthy that while Figure 9(a) indicates
instances where the aircraft reaches five destinations, these cases result in
the battery voltage reaching EOD before reaching the charging station. As a
result, the aircraft incurs a substantial negative reward and learns to avoid
such conditions in subsequent episodes.

####  3.1.3 Mission with Multiple-Charging Cycles

In the third scenario, the aircraft’s mission is to reach all the predefined
destinations with a minimum number of charging cycles. To accomplish this, the
aircraft must make decisions regarding the order of destinations to reach, the
timing of returning to the charging station, and the altitude at which to fly
to reach each destination. Figure 10 shows an example of this kind of mission
when the aircraft faced a mission with eight destinations.

Figure 10: A mission with a multi-charge of battery. Each color represents the
path that the aircraft flights in one charging cycle.

To illustrate the learning behavior of the aircraft through the proposed DRL
method in this scenario, we plot the number of destinations reached by the
aircraft before reaching the EOD (Figure 11(a)) and the number of charging
cycles throughout each learning episode in this mission (Figure 11(b)). As
illustrated in these figures, in the initial episodes, the aircraft’s battery
voltage drops below the EOD threshold before reaching all destinations.
However, with training time, the aircraft learns the optimal order of
destinations, the timing for returning to the charging station, and the
optimal flight altitude to reach all destinations without encountering EOD
issues. Given the significant penalty associated with falling below the EOD
threshold, the aircraft prioritizes reaching all destinations without reaching
EOD, even if it requires more charging cycles. Subsequently, the aircraft
shifts its focus towards minimizing the number of charging cycles while still
reaching all destinations. This dynamic is reflected in Figure 11(b), where we
observe an initial increase followed by a decrease in the number of charging
cycles as the learning progresses. It is noticeable that to prevent the
aircraft from learning to immediately return to the charging station after
reaching each destination, it is crucial to finely tune the parameters of the
reward functions, denoted as $\alpha$, $\beta$, and $\gamma$ in Equation 3. By
carefully adjusting these parameters, we can effectively shape the learning
process and optimize the aircraft’s decision-making behavior. In the
following, we investigate the effect of these parameters on the learning
process.

(a) Number of reached destinations in each episode

(b) Number of charging cycles in each episode

Figure 11:  Number of reached destinations and charging cycles in each episode
using learning algorithm

As outlined in Section 2.3, the aircraft faces a substantial penalty should
its battery voltage fall below the EOD threshold level before reaching the
destination. The impact of this penalty, denoted by $\gamma$ in equation 3, on
both the number of reached destinations and the number of charging cycles is
visually demonstrated in Figures 12 and 13. Note that throughout these
figures, $\beta$ which represents the penalty imposed when the aircraft
reaches a charging station, and $\alpha$ which signifies the reward for
reaching each destination, remain fixed. Figure 12 reveals that as $\gamma$
increases, the aircraft becomes more adept at reaching all destinations in
fewer learning episodes without breaching the EOD threshold. As shown in
Figure 13, in the early episodes, the aircraft learns to reach all
destinations by maximizing the number of charging cycles. This strategic
approach ensures the avoidance of penalties associated with the battery
voltage dropping below the EOD level. Subsequently, the aircraft gradually
refines its strategy, learning to reach all destinations with the minimum
number of charging cycles. Moreover, Figure 13 shows that when increasing
$\gamma$, the aircraft’s learning process to reach all destinations with the
minimum number of charging cycles requires more time. Furthermore, with higher
$\gamma$ values, the aircraft tends to adopt a more conservative approach,
opting for additional charging cycles to reach all destinations. This
conservative strategy ensures that irrespective of the battery health state or
wind conditions, the voltage will not fall below the EOD level.

Similarly, the aircraft faces a penalty upon reaching a charging station,
albeit less severe than the penalty incurred when the battery voltage falls
below the EOD level. This penalty structure serves to incentivize the aircraft
to efficiently reach multiple destinations within a charging cycle, thereby
optimizing its overall performance. The impact of this penalty, denoted by
$\beta$ in equation 3, on both the number of reached destinations and the
number of charging cycles is visualized in Figures 14 and 15. It is crucial to
emphasize that throughout these figures, both $\gamma$ and $\alpha$ remain
constant. Figure 15 demonstrates that as the penalty parameter $\beta$
increases, the aircraft progressively refines its strategy, becoming more
proficient at reaching all destinations with the minimum number of charging
cycles. Notably, when $\beta=0$, the aircraft exhibits minimal learning
regarding the optimization of charging cycles. In this case, since the
aircraft returns to the charging station after reaching each destination, the
battery voltage seldom drops below the EOD. Consequently, as depicted in
Figure 14, after the aircraft learns its optimal strategy, the aircraft
consistently reaches all destinations.

Figure 12: Number of reached destinations in each episode for different values
of $\gamma$ in reward function while both $\alpha=1$ and $\beta=1$ remain
fixed. Figure 13: Number of charging cycles in each episode for different
values of $\gamma$ in reward function while both $\alpha=1$ and $\beta=1$
remain fixed.  Figure 14: Number of reached destinations in each episode for
different values of $\beta$ in reward function while both $\alpha=1$ and
$\gamma=5$ remain fixed.  Figure 15: Number of charging cycles in each episode
for different values of $\beta$ in reward function while both $\alpha=1$ and
$\gamma=5$ remain fixed.

##  4 Conclusion

In summary, realizing the transformative potential of Urban Air Mobility
depends on the successful integration of battery-powered aircraft into urban
transportation systems. The widespread adoption of UAM relies on overcoming
challenges associated with battery degradation and unforeseen events, both
critical elements in mission planning and health-aware real-time operational
control.

Our research has addressed the complex task of combining mission planning and
health-aware real-time operational control, with a particular focus on
integrating battery health and degradation state into optimal decision-making.
The proposed DRL algorithm represents a proactive strategy, prescribing
operational parameters that extend the battery’s discharge cycle based on its
current health status, while simultaneously maximizing mission efficiency. In
numerous scenarios simulated with a NASA multirotor aircraft model, the
proposed deep reinforcement learning framework demonstrates its adaptability
and versatility.

The proposed algorithm’s strength lies in its ability to adapt to batteries in
different states of degradation without any prior knowledge of the true
degradation state and effectively handle uncertainties, a critical feature in
the unpredictable UAM environment where factors like adverse weather
conditions can disrupt operations. While predicting the end of discharge can
already be very advantageous, our proposed algorithm goes a step further by
incorporating information on the health condition into its decision-making
process. The algorithm optimizes operations in a health-aware manner, ensuring
near-optimal performance across diverse operational scenarios and efficient
aircraft operation. Beyond individual aircraft optimization, our approach has
the potential to enhance overall system throughput by optimizing fleet
operations. As UAM continues to evolve, the adaptability and robustness of our
algorithm position it as a valuable tool for ensuring the reliable and
efficient integration of battery-powered aircraft into urban transportation
networks. The proposed prescriptive algorithm is characterized by its low
computational cost and fast processing time, enabling seamless integration
into air traffic control systems and facilitating prompt and efficient
decision-making.

Our work can be extended in multiple ways. Firstly, the health-aware control
strategy devised in this study provides a versatile framework applicable
beyond Urban Air Mobility. Future investigations could explore the
adaptability of the proposed framework in electric vehicles, portable
electronic devices, and renewable energy systems. Secondly, the integration of
the proposed algorithm into air traffic management systems has the potential
to alleviate airspace congestion and improve traditional air traffic.
Furthermore, optimal charging station placement in urban areas could be a
subject of exploration utilizing optimization algorithms. Lastly, there is
potential in relaxing some of the limiting assumptions considered in this
research, such as the finite set of wind conditions, the finite set of
altitudes that the aircraft can fly to reach each destination, or the
assumption of constant wind conditions throughout the flight to reach each
destination.

## References

  * [1] P. D. Vascik, R. J. Hansman, and N. S. Dunn, “Analysis of urban air mobility operational constraints,” Journal of Air Transportation, vol. 26, no. 4, pp. 133–146, 2018.
  * [2] S. Sripad and V. Viswanathan, “The promise of energy-efficient battery-powered urban aircraft,” Proceedings of the National Academy of Sciences, vol. 118, no. 45, p. e2111164118, 2021.
  * [3] J. Zhao, W. Wang, C. Yang, Y. Li, L. Yang, and J. Cheng, “A new efficient algorithm for short path planning of the vertical take-off and landing air-ground integrated vehicle,” Engineering Applications of Artificial Intelligence, vol. 127, p. 107386, 2024.
  * [4] M. Liu, H. Hao, Z. Lin, X. He, Y. Qian, X. Sun, J. Geng, Z. Liu, and F. Zhao, “Flying cars economically favor battery electric over fuel cell and internal combustion engine,” PNAS nexus, vol. 2, no. 3, p. pgad019, 2023.
  * [5] P. Pradeep and P. Wei, “Energy-efficient arrival with rta constraint for multirotor evtol in urban air mobility,” Journal of Aerospace Information Systems, vol. 16, no. 7, pp. 263–277, 2019.
  * [6] B. P. Hill, D. DeCarme, M. Metcalfe, C. Griffin, S. Wiggins, C. Metts, B. Bastedo, M. D. Patterson, and N. L. Mendonca, “Uam vision concept of operations (conops) uam maturity level (uml) 4,” 2020.
  * [7] N. M. Guerreiro, R. W. Butler, J. M. Maddalon, and G. E. Hagen, “Mission planner algorithm for urban air mobility–initial performance characterization,” in AIAA Aviation 2019 Forum, p. 3626, 2019.
  * [8] J. Bertram, X. Yang, M. W. Brittain, and P. Wei, “Online flight planner with dynamic obstacles for urban air mobility,” in AIAA Aviation 2019 Forum, p. 3625, 2019.
  * [9] Y.-J. Chen, B.-G. Jhong, and M.-Y. Chen, “A real-time path planning algorithm based on the markov decision process in a dynamic environment for wheeled mobile robots,” in Actuators, vol. 12, p. 166, MDPI, 2023.
  * [10] J. Xue, J. Zhu, J. Du, W. Kang, and J. Xiao, “Dynamic path planning for multiple uavs with incomplete information,” Electronics, vol. 12, no. 4, p. 980, 2023.
  * [11] N. Bernini, M. Bessa, R. Delmas, A. Gold, E. Goubault, R. Pennec, S. Putot, and F. Sillion, “Reinforcement learning with formal performance metrics for quadcopter attitude control under non-nominal contexts,” Engineering Applications of Artificial Intelligence, vol. 127, p. 107090, 2024.
  * [12] G. Lee, K. Kim, and J. Jang, “Real-time path planning of controllable uav by subgoals using goal-conditioned reinforcement learning,” Applied Soft Computing, vol. 146, p. 110660, 2023.
  * [13] C. Huang, X. Zhou, X. Ran, J. Wang, H. Chen, and W. Deng, “Adaptive cylinder vector particle swarm optimization with differential evolution for uav path planning,” Engineering Applications of Artificial Intelligence, vol. 121, p. 105942, 2023.
  * [14] Y. Jia, L. Qu, and X. Li, “A double-layer coding model with a rotation-based particle swarm algorithm for unmanned combat aerial vehicle path planning,” Engineering Applications of Artificial Intelligence, vol. 116, p. 105410, 2022.
  * [15] K. Li, X. Yan, and Y. Han, “Multi-mechanism swarm optimization for multi-uav task assignment and path planning in transmission line inspection under multi-wind field,” Applied Soft Computing, vol. 150, p. 111033, 2024.
  * [16] S. Semanjski, I. Querol Puchal, R. A. Ponguillo Intriago, and I. Semanjski, “Aircraft path planning for uam applications,” in The Eleventh International Conference on Data Analytics DATA ANALYTICS 2022, pp. 8–11, International Academy Research and Industry Association, 2022.
  * [17] W. Zhang, C. Peng, Y. Yuan, J. Cui, and L. Qi, “A novel multi-objective evolutionary algorithm with a two-fold constraint-handling mechanism for multiple uav path planning,” Expert Systems with Applications, vol. 238, p. 121862, 2024.
  * [18] Q. Chen, Q. He, and D. Zhang, “Uav path planning based on an improved chimp optimization algorithm,” Axioms, vol. 12, no. 7, p. 702, 2023.
  * [19] M. Salinas-Camus, C. Kulkarni, and M. Orchard, “Battery state-of-health aware path planning for a mars rover,” in Annual Conference of the PHM Society, vol. 15, 2023.
  * [20] L. Biggio, T. Bendinelli, C. Kulkarni, and O. Fink, “Ageing-aware battery discharge prediction with deep learning,” Applied Energy, vol. 346, p. 121229, 2023.
  * [21] Y. Mazzi, H. B. Sassi, and F. Errahimi, “Lithium-ion battery state of health estimation using a hybrid model based on a convolutional neural network and bidirectional gated recurrent unit,” Engineering Applications of Artificial Intelligence, vol. 127, p. 107199, 2024.
  * [22] J. Meng, G. Luo, M. Ricco, M. Swierczynski, D.-I. Stroe, and R. Teodorescu, “Overview of lithium-ion battery modeling methods for state-of-charge estimation in electrical vehicles,” Applied sciences, vol. 8, no. 5, p. 659, 2018.
  * [23] A. Bartlett, J. Marcicki, S. Onori, G. Rizzoni, X. G. Yang, and T. Miller, “Electrochemical model-based state of charge and capacity estimation for a composite electrode lithium-ion battery,” IEEE Transactions on control systems technology, vol. 24, no. 2, pp. 384–399, 2015.
  * [24] M. Jiao, D. Wang, Y. Yang, and F. Liu, “More intelligent and robust estimation of battery state-of-charge with an improved regularized extreme learning machine,” Engineering Applications of Artificial Intelligence, vol. 104, p. 104407, 2021.
  * [25] M. Aykol, P. Herring, and A. Anapolsky, “Machine learning for continuous innovation in battery technologies,” Nature Reviews Materials, vol. 5, no. 10, pp. 725–727, 2020.
  * [26] P. Pradeep, T. A. Lauderdale, G. B. Chatterji, K. Sheth, C. F. Lai, B. Sridhar, K.-M. Edholm, and H. Erzberger, “Wind-optimal trajectories for multirotor evtol aircraft on uam missions,” in AIAA AVIATION 2020 FORUM, p. 3271, 2020.
  * [27] G. B. Chatterji, “Trajectory simulation for air traffic management employing a multirotor urban air mobility aircraft model,” in AIAA AVIATION 2020 FORUM, p. 3209, 2020.
  * [28] C. S. Kulkarni, P. Pradeep, and G. B. Chatterji, “Simulation studies for an urban air mobility aircraft using hardware-in-loop experiments,” in AIAA AVIATION Forum, 2021.
  * [29] M. Daigle and C. Kulkarni, “Electrochemistry-based battery modeling for prognostics,” in Annual Conference of the Prognostics and Health Management Society 2013, pp. 249–261, Oct. 2013.
  * [30] M. Daigle and C. S. Kulkarni, “End-of-discharge and end-of-life prediction in lithium-ion batteries with electrochemistry-based aging models,” in AIAA Infotech@ aerospace, p. 2132, 2016.
  * [31] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
  * [32] E. Hogge, B. Bole, S. Vazquez, C. Kulkarni, T. Strom, B. Hill, K. Smalling, and C. Quach, “Verification of prognostic algorithms to predict remaining flying time for electric unmanned vehicles,” in International Journal of Prognostics and Health Management, ISSN 2153-2648, 2018 021, 2018.
  * [33] C. Kulkarni, E. Hogge, C. Quach, and K. Goebel, “Hirf battery data set”, nasa prognostics data repository,” 2015.
  * [34] J. J. Ely, S. V. Koppen, T. X. Nguyen, K. L. Dudley, G. N. Szatkowski, C. C. Quach, S. L. Vazquez, J. J. Mielnik, E. F. Hogge, B. L. Hill, et al., “Radiated emissions from a remote-controlled airplane-measured in a reverberation chamber,” tech. rep., 2011.

##  Appendix A Simulation Setup

Table 1 provides an overview of the power required (in kW) during various
phases of the 30 nm flight. These phases encompass vertical takeoff, climbing,
cruise, approach, and vertical landing, with corresponding phase duration
detailed in Table 2.

Table 1: Power required (kW) in different phases of flight of a simulated 30 nm flight of aircraft in 13 kts tailwind  | Power Required (kW)
---|---
Cruise Altitude (m) | Vertical Takeoff | Climb | Cruise | Descent (45 Deg) | Approach (8 Deg) | Vertical Landing
500 | 264.94 | 273.53 | 138.39 | Negligible | Negligible | 139.45
1000 | 264.94 | 272.66 | 141.72 | Negligible | Negligible | 139.45
2000 | 264.94 | 273.03 | 145.68 | Negligible | Negligible | 139.45
3000 | 264.94 | 273.13 | 155.44 | Negligible | Negligible | 139.45
Table 2: Duration (in seconds) of the different phases of flight of a simulated 30 nm flight of aircraft in 13 kts tailwind  | Flight Duration (sec)
---|---
Cruise Altitude (m) | Vertical Takeoff | Climb | Cruise | Descent (45 Deg) | Approach (8 Deg) | Vertical Landing
500 | 15.51 | 85 | 1167 | 0 | 115 | 15.5
1000 | 15.51 | 180 | 1143 | 11.4 | 115 | 15.5
2000 | 15.51 | 369 | 1120 | 32.6 | 115 | 15.5
3000 | 15.51 | 558 | 1056 | 52.1 | 115 | 15.5

In this study, the wind is assumed to be uniform in each flight, i.e.,
constant magnitude and direction. The study assesses how wind magnitude
uncertainties affect both power requirements and the time of the flight’s
cruise phase. Tables 3, 4, 5, and 6 provide a detailed analysis of how wind
affects power requirements and cruise duration at various cruise altitudes.

Table 3: Power required (kW) and duration of flight (sec) in cruise phase when the cruise altitude is 500m  Wind Magnitude (kts) | Cruise Airspeed | Power Required (kW) | Flight Duration (sec)
---|---|---|---
-39 | 38 m/s (73.9 kts) | 130.57 | 959
-26 | 39 m/s (75.8 kts) | 132.92 | 1063
-13 | 41 m/s (79.7 kts) | 138.39 | 1167
13 | 45 m/s (87.5 kts) | 152.18 | 1453
26 | 48 m/s (93.3 kts) | 165.22 | 1608
39 | 52 m/s (101.1 kts) | 186.36 | 1743
Table 4: Power required (kW) and duration of flight (sec) in cruise phase when the cruise altitude is 1000m  Wind Magnitude (kts) | Cruise Airspeed | Power Required (kW) | Flight Duration (sec)
---|---|---|---
-39 | 39 m/s (75.8 kts) | 133.78 | 943
-26 | 40 m/s (77.7 kts) | 136.18 | 1043
-13 | 42 m/s (81.6 kts) | 141.72 | 1143
13 | 46 m/s (89.4 kts) | 155.59 | 1416
26 | 49 m/s (95.2 kts) | 168.61 | 1563
39 | 53 m/s (103.1 kts) | 189.71 | 1690
Table 5: Power required (kW) and duration of flight (sec) in cruise phase when the cruise altitude is 2000m  Wind Magnitude (kts) | Cruise Airspeed | Power Required (kW) | Flight Duration (sec)
---|---|---|---
-39 | 41 m/s (79.7 kts) | 140.75 | 911
-26 | 42 m/s (81.6 kts) | 143.01 | 1005
-13 | 43 m/s (93.3 kts) | 148.68 | 1120
13 | 48 m/s (93.3 kts) | 162.21 | 1347
26 | 51 m/s (99.1 kts) | 174.99 | 1479
39 | 55 m/s (106.9 kts) | 195.3 | 1594
Table 6: Power required (kW) and duration of flight (sec) in cruise phase when the cruise altitude is 3000m  Wind Magnitude (kts) | Cruise Airspeed | Power Required (kW) | Flight Duration (sec)
---|---|---|---
-39 | 43 m/s (83.6 kts) | 147.82 | 882
-26 | 44 m/s (85.5 kts) | 150.07 | 970
-13 | 46 m/s (89.4 kts) | 155.44 | 1056
13 | 50 m/s (97.1 kts) | 172.61 | 1256
26 | 53 m/s (103.0 kts) | 181.03 | 1405
39 | 57 m/s (110.8 kts) | 200.77 | 1507