11affiliationtext: Istituto Dalle Molle di Studi sull’Intelligenza Artificiale
(IDSIA), USI-SUPSI, Lugano (Switzerland)22affiliationtext: Armasuisse Science
+ Technology, Thun (Switzerland)

# Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

Ardian Selmonaj  Oleg Szehr  Giacomo Del Rio  Alessandro Antonucci
Adrian Schneider  Michael Rüegsegger

###### Abstract

The application of artificial intelligence to simulate air-to-air combat
scenarios is attracting increasing attention. To date the high-dimensional
state and action spaces, the high complexity of situation information (such as
imperfect and filtered information, stochasticity, incomplete knowledge about
mission targets) and the nonlinear flight dynamics pose significant challenges
for accurate air combat decision-making. These challenges are exacerbated when
multiple heterogeneous agents are involved. We propose a hierarchical multi-
agent reinforcement learning framework for air-to-air combat with multiple
heterogeneous agents. In our framework, the decision-making process is divided
into two stages of abstraction, where heterogeneous low-level policies control
the action of individual units, and a high-level commander policy issues macro
commands given the overall mission targets. Low-level policies are trained for
accurate unit combat control. Their training is organized in a learning
curriculum with increasingly complex training scenarios and league-based self-
play. The commander policy is trained on mission targets given pre-trained
low-level policies. The empirical validation advocates the advantages of our
design choices.

###### Index Terms:

Hierarchical Multi-Agent Reinforcement Learning, Heterogeneous Agents,
Curriculum Learning, Air Combat.

##  I Introduction

In defense area, complex air-to-air combat scenarios simulation requires
simultaneous real-time control of individual units (troop level) and global
mission planning (commander level). _Deep Reinforcement Learning_ (DRL) has
achieved super-human level in various environments, ranging from discrete
perfect information scenarios (such as games like Chess and Go) to real-time
continuous control and strategic decision-making scenarios with imperfect
information (typical of modern war games). However, conventional DRL ignores
the structural requirements typical of real-world combat scenarios, where
decision-making authority is organized hierarchically. It is crucial for real-
world operations that low-level combat decisions (such as fire/duck) are made
by individual units and executed at low latency, while abstract mission
planning decisions (such as conquer and hold coordinates) at higher hierarchy
levels take account of information from all available units. An example is the
guidance of drones in modern warfare, where individual units act autonomously
even without connection to a centralized intelligent instance. This
information abstraction motivates the investigation of _Multi-Agent Deep
Reinforcement Learning_ (MARL) techniques for creating artificial agents in a
realistic simulation environment. In MARL systems, the hierarchical splitting
of mixed planning and control tasks can be achieved by incorporating dedicated
algorithms at varying levels of abstraction. This allows each agent to control
itself in a decentralized manner while providing sufficient flexibility for
the emergence of targeted group behavior.

###  I-A Contributions

  1. 1.

Considering low latency as crucial for DRL problems, we develop a lightweight
simulation platform suitable for fast simulation of agent dynamics and
interactions.

  2. 2.

We employ a hierarchical framework for simultaneous planning and control to
solve the overall decision-making problem for air-to-air combat scenarios.

  3. 3.

We realize a fictitious self-play mechanism through curriculum learning with
increasing levels of complexity to improve combat performance as learning
proceeds.

  4. 4.

We develop a sophisticated neural network architecture composed of recurrent
and attention units. Coordination is achieved without an explicit
communication channel.

###  I-B Outline

Sect. II summarizes previous contributions and describes how our work extends
the existing literature. Sect. III details air-to-air engagement scenarios and
describes our framework. Our experimental findings are presented in Sect. IV,
while our conclusions and possible future works are discussed in Sect. V.

##  II Related Work

Aerial combat tactics have been discussed extensively in the literature, with
a significant portion of research dedicated to the study of engagements with
small numbers of units (one to two). Research on small engagements typically
focuses on _control_ , i.e., it examines how the maneuvering of individual
units impacts the overall engagement outcome. A frequent focus lies on
achieving an advantage against the opponent: in this position, it is possible
to fire at the opponent with little risk of return fire [1]. Popular methods
include expert systems [2, 3, 4, 5], control laws for pursuit and/or evasion
[6, 7, 8, 9], game-theory [10, 11, 12], but also machine learning [13, 14, 15,
16] and hybrid approaches [17, 18, 19, 20, 21, 22, 23]. Classical research
about larger-scale engagements focuses on weapon-target assignment [24] and
[1], human-pilot-like decision-making [25], and high-level engagement tactical
decisions [26], i.e., on _planning_. _Reinforcement learning_ (RL) techniques
gained increasing interest in this context. [27] train a Recurrent _Deep
$Q$-Network_ (DQN) algorithm [28] and employ a situation evaluation function
to revise the decision-making system. Other approaches use _deep deterministic
policy gradient_ (DDPG) [29, 30] or A3C [31]. _Cascade learning_ approaches
that gradually increase combat complexity are discussed in [32] and [33]. In
[34], the combat strategy is learned through a league system to prevent the
policy from circling around poor local optima.

MARL is currently thriving [35]. In the study of emergent behavior and
complexity from coordinated agents, the introduction of centralized training
decentralized execution actor-critic methods, such as the _Multi-Agent Deep
Deterministic Policy Gradient_ algorithm (MADDPG) [36], has been a milestone
[36, 37, 38]. Such methods train actor policies with critics having access to
information of all other agents. However, they are not structured to account
for the hierarchies present in real-world operations and emerging phenomena
such as agent attrition (exit learning process). [39] uses MADDPG combined
with potential-based reward shaping [40]. A maneuver strategy for _Unmanned
Aerial Vehicles_ (UAV) swarms is developed in [41] using MADDPG, but the
discussion is limited to one-to-one or multi-to-one combat. [42] and [43] use
attention based neural networks. The former uses a two-stage attention
mechanism for coordination. In the latter, the attention layers calculate the
relative importance of surrounding aircraft, where opponents are purely
controlled by scripts.

On the other side stands the concept of _Hierarchical Reinforcement Learning_
(HRL), which divides the overall task into sub-tasks [44]. In HRL, training is
organized in nested loops. Inner loop training controls the aircraft, while
the outer trains a super-policy for guidance and coordination of individual
agents. HRL has been applied in the context of air-to-air combat in [45] and
[46].

There appears to be little research in air-to-air combat by combining HRL and
MARL. A Hierarchical MARL approach to handle variable-size formations is
proposed in [47]. The authors employ an attention mechanism and self-play with
a DQN high-level policy trained with QMIX. An approach similar to the one
presented in this paper, focusing on heterogeneous agents of two types, was
explored in [48]. The high-level target allocation agents are trained using
DQN, and the low-level cooperative attacking agents are based on _independent
asynchronous proximal policy optimization_ (IAPPO). However, they follow the
goal of _suppression of enemy air defense_ (SEAD). SEAD aims to gain air
superiority by targeting and disrupting the enemy’s ability to detect and
engage friendly aircraft. Unlike the concept of SEAD, which focuses on
neutralizing enemy air defense systems, dogfighting is centered around
engaging and defeating enemy aircraft in direct air-to-air combat. This
article investigates air-to-air combat scenarios for coordinated dogfighting
with heterogeneous agents and hierarchical MARL in a cascaded league-play
training scheme. To our knowledge, this setup has yet to occur in publications
for this kind of application.

##  III Method

###  III-A Aircraft Dynamics

We base our modeling on the dynamics of the _Dassault Rafale_ fighter
aircraft.111 dassault-aviation.com/en/defense/rafale. We focus on hierarchical
coordination of multiple heterogeneous agents in 2D (assuming a constant
altitude of our aircraft). There are _beyond_ and _within visual range_ air
combat scenarios [49], where we focus on the latter in this article. Since
real-world combat scenarios frequently involve different types of aircraft, we
add a modified version of the Rafale aircraft with different dynamics: The
original aircraft (AC1) is more agile and equipped with rockets, while the
modified type (AC2) has no rockets but longer cannon range. The dynamics of
AC1 and AC2 are characterized as:

  * •

angular velocity [$deg/s$]: $\omega_{AC1}\in[0,5]$, $\omega_{AC2}\in[0,3.5]$;

  * •

speed [knots]: $v_{AC1}\in[100,900]$, $v_{AC2}\in[100,600]$;

  * •

conical _weapon engagement zone_ (WEZ):
angle [$deg$]: $\omega_{WEZ,AC1}\in[0,10]$, $\omega_{WEZ,AC2}\in[0,7]$, range
[$km$]: $d_{a,AC1}\in[0,2]$, $d_{a,AC2}\in[0,4.5]$;

  * •

hit probability $p_{hit,AC1}=0.70$ and $p_{hit,AC2}=0.85$.

Figure 1: Aircraft attacking mechanisms.

###  III-B Multi-Agent Reinforcement Learning

RL is used to solve sequential decision-making problems. Agents interact with
an environment to learn a behavior that is evaluated based on a reward
function $r_{t}(s_{t},a_{t},s_{t+1})$. The goal of the agent is to maximize
the cumulative reward $\sum_{t}r_{t}(s_{t},a_{t},s_{t+1})$. In MARL, there are
multiple agents interacting in a cooperative or competing fashion, or both.
The decision function, called policy $\pi(a_{t}|s_{t})$, maps states to a
distribution over actions. We model the interactions of individual agents as a
_partially-observable Markov game_ (POMG) defined by a tuple
$(\mathcal{S},\mathcal{O},\mathcal{A}_{1},\ldots,\mathcal{A}_{N},P,R_{1},\ldots,R_{N},\gamma)$,
where: $\mathcal{S}$ is the state-space representing possible configurations
of the environment, $\mathcal{O}\subset\mathcal{S}$ is the set of
observations, $\mathcal{A}_{i}$ is the set of actions for player $i$,
$P(s^{\prime}|s,a_{1},\ldots,a_{N})$ represents the dynamics of the
environment and specifies the probability of transitioning to state
$s^{\prime}$ when players take actions $a_{1},\ldots,a_{N}$ in state $s$,
$R_{i}(s,a_{1},\ldots,a_{N},s^{\prime})$ defines the immediate reward for
player $i$ when the system transitions from state $s$ to state $s^{\prime}$
with players taking actions $a_{1},\ldots,a_{N}$.

We adopt a _Centralized Training and Decentralized Execution_ (CTDE) [36]
scheme for training agents. Our modeled POMG with CTDE scheme is used to train
low-level control policies, from which we define to have two: a fight policy
$\pi_{f}$ and an escape policy $\pi_{e}$. Further on, there is a distinct
policy for each aircraft type. Overall we have four low-level policies:
$[\pi_{f,AC1},\pi_{f,AC2},\pi_{e,AC1},\pi_{e,AC2}]$. Agents of the same type
use the same shared policies. Thus all AC1 use $\pi_{f,AC1}$ and
$\pi_{e,AC1}$, irrespective on the number of agents, and similarly for AC2. In
this way, policies are trained with experiences of all agents of the same type
and ensures a coherent behavior.

###  III-C Hierarchical Reinforcement Learning

HRL employs temporal abstraction by decomposing the overall task into a nested
hierarchy of sub-tasks, enhancing efficiency in learning and decision-making
[44]. Abstract commands are issued from higher hierarchy levels to apply a
control policy (so-called _option_) for a limited amount of time. Symmetries
within a particular (lower) hierarchy level can be exploited by using the same
option for different sub-tasks, e.g. controlling similar airplanes. This
results in better scalability (reducing the effective dimensions of state and
action spaces) and enhances generalization (generating new skills by combining
sub-tasks) [50]. HRL also fits naturally with the hierarchical structure of
defense organizations. Formally, our hierarchical system corresponds to a
_partially observable semi-Markov Decision Process_ (POSMDP) with options as a
tuple $(\mathcal{S},\mathcal{O}_{s},\mathcal{A},R,P,\gamma)$. Similar to the
notions of POMG, $\mathcal{S}$ is the state space, $\mathcal{O}_{s}$ is the
set of sub-strategies (options), $\mathcal{A}$ is the action space, $R$ is the
reward function and the transition function $P(s^{\prime},\tau|s,o)$ defines
the probability of landing in state $s^{\prime}$ from state $s$ after $\tau$
time steps when executing $o$. We again use CTDE to train a single high-level
commander policy $\pi_{h}$ to be used for all agents and aircraft types. Fig.
2 illustrates the relations between high and low-level policies.

Figure 2: Hierarchy of policies.

###  III-D Metrics for Air-to-Air Combat

We now describe observations, actions and rewards in our hierarchical MARL
approach. All observation values are normalized to the range $[0,1]$ and are
based on the metrics shown in Fig. 3 Further observations include map position
($x,y$), current speed ($s$), remaining cannon ammunition ($c_{1}$) and
remaining rockets ($c_{2}$). Indicator ($w$) defines if the next rocket is
ready to be fired and ($s_{r}$) indicates if the aircraft is currently
shooting. Subscript $a$ indicates agent, $o$ opponent and $fr$ friendly
aircraft (i.e. from the same team). A subscript in a value, e.g.,
$\alpha_{off,o}$, defines the angle-off w.r.t. to the opponent. Actions of all
policies are discrete.

(a)

(b)

(c)

(d)

(e)

Figure 3: Aircraft metrics: heading (a), heading off (b), aspect angle (c),
antenna train angle (d), distance (e).

####  III-D1 Fight Policy

$\pi_{f}$ can observe its closest opponent and closest friendly aircraft.

$\displaystyle o_{t,a}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},\alpha_{off,o},\alpha_{AA,o},\alpha_{ATA,o},d_{o},c_{1},\overbrace{c_{2},w}^{\text{AC1}},s_{r}]$
$\displaystyle o_{t,o}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},\alpha_{off,a},\alpha_{AA,a},\alpha_{ATA,a},d_{a},s_{r}]$
$\displaystyle o_{t,fr}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{off,a},\alpha_{ATA,a},\alpha_{ATA,fr},d_{a},s_{r}]$
$\displaystyle o_{t,full}$ $\displaystyle:=$ $\displaystyle
o_{t,a}||o_{t,o}||o_{t,fr}$

The control maneuvers (actions) are:

  * •

relative heading maneuvers: turn in range [-90°, 90°]
($h\in\\{-6,\ldots,6\\}\xrightarrow{}\alpha_{h}=-15\cdot h+\alpha_{h}$);

  * •

velocity: mapping of $v$ to velocity ranges of AC1 or AC2
($v\in\\{0,\ldots,8\\}$);

  * •

shooting with cannon: ($c\in\\{0,1\\}$);

  * •

shooting with rocket (AC1): ($r\in\\{0,1\\}$).

In air-to-air combat, facing the opponent’s tail is a favorable situation for
shooting. We therefore define the reward function based on $\alpha_{ATA,a}$ of
the opponent to the agent. We further encourage the combat efficiency by
incorporating the remaining ammunition ($c_{rem}=c_{1}+c_{2}$):

$r_{t,k}=\alpha_{ATA,a}+\frac{c_{max}-c_{rem}}{c_{max}}\in[1,2]\,.$ (1)

Punishing rewards are given when flying out of environment boundaries
$r_{t,b}=-5$ and when destroying a friendly aircraft $r_{t,f}=-2$. There is no
per-time-step reward given. The total reward is then:
$r_{t}=r_{t,k}+r_{t,b}+r_{t,f}$.

####  III-D2 Escape Policy

$\pi_{e}$ senses two closest opponents and its closest friendly aircraft. The
actions remain same as for $\pi_{f}$.

$\displaystyle o_{t,a}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},c_{1},\overbrace{c_{2}}^{\text{AC1}}]$
$\displaystyle o_{t,o}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},\alpha_{off,a},\alpha_{ATA,a},\alpha_{ATA,o},d_{a}]$
$\displaystyle o_{t,fr}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},\alpha_{ATA,a},\alpha_{ATA,fr},d_{a}]$
$\displaystyle o_{t,full}$ $\displaystyle:=$ $\displaystyle
o_{t,a}||o_{t,o_{1}}||o_{t,o_{2}}||o_{t,fr}$

The per-time-step reward depends on distances to opponents:

$r_{t,e}=\begin{cases}-0.01&d<6km\\\ +0.01&d>13km\\\
0&\mathrm{otherwise}\end{cases}\,.$ (2)

The total reward is finally $r_{t}:=r_{t,e}+r_{t,b}+r_{t,f}$.

####  III-D3 Commander Policy

$\pi_{h}$ is called for every agent separately. The observations are based on
three closest opponents and two closest friendly aircraft.

$\displaystyle o_{t,a}$ $\displaystyle:=$ $\displaystyle[x,y,s,\alpha_{h}]$
$\displaystyle o_{t,o}$ $\displaystyle:=$
$\displaystyle[x,y,s,\alpha_{h},\alpha_{AA,a},\alpha_{AA,o},\alpha_{ATA,a},\alpha_{ATA,o},d_{a}]$
$\displaystyle o_{t,fr}$ $\displaystyle:=$ $\displaystyle[x,y,s,d_{a}]$
$\displaystyle o_{t,full}$ $\displaystyle:=$ $\displaystyle
o_{t,a}||o_{t,o_{1}}||o_{t,o_{2}}||o_{t,o_{3}}||o_{t,fr_{1}}||o_{t,fr_{2}}$

The commander decides the low-level policy to use for each agent. The action
set is $a_{c}\in\\{0,1,2,3\\}$, where $0$ activates $\pi_{e}$ and $\pi_{f}$
otherwise. If $\pi_{f}$ is activated, the commander action ($1$, $2$ or $3$)
determines which of the three observable opponents the agent should attack.
The agent then gets the corresponding observation for its low-level policy. In
our setup, the commander adapts to only these pre-trained low-level policies.
The reward is composed of two parts. First, a killing reward $r_{t,f}=1$ if an
agent with its low-level policy killed an opponent and $r_{t,f}=-1$ if the
agent got killed. The second part, $r_{t,c}$ should encourage the commander to
exploit favorable situations and is defined as follows:

$r_{t,c}=\begin{cases}+0.1&d_{o}\\!\\!<\\!\\!5km\wedge\alpha_{ATA,o}\\!\\!<\\!\\!30^{\circ}\wedge\alpha_{AA,o}\\!\\!<\\!\\!50^{\circ}\wedge
a_{c}\\!\\!>\\!\\!0\\\ 0&\mathrm{otherwise}\end{cases}$ (3)

We also include the out-of-boundary reward $r_{t,b}=-5$. Total reward given to
$\pi_{h}$ is thus $r_{t}:=r_{t,f}+r_{t,c}+r_{t,b}$.

###  III-E Training Structure

The overall training loop for our hierarchical MARL algorithm is split into
two main stages (Fig 4). We first train the low-level policies with
observations $O_{l}$ and rewards $R_{l}$. In the second stage, low-level
policies are fixed (i.e., no learning is done anymore) and serve as options
for the commander. $\pi_{h}$ is then trained with observations $O_{h}$ and
rewards $R_{h}+R_{l}$.

Figure 4: Hierarchical MARL training loop.

The training of low-level policies is done in five levels following a
_curriculum learning_ scheme. The complexity is increased at each level by
making the opponents more competitive. Namely, the opponent behavior of each
level is L1 (static), L2 (random), L3 (scripts), L4 (L3 policy), and L5
(policies L1-4). Scripted opponents are programmed to engage the closest agent
with $\alpha_{ATA}\approx 0$ and to randomly escape. When training is
completed at a level, we transfer the policy to the next level and continue
training.

Our neural network is based on Actor-Critic [51] (Fig. 5). Low-level AC1 and
AC2 agents have distinct neural network instances (with different input and
output dimensions) but share one layer (green box). This layer is further
shared between the actor and critic inside the network. Sharing parameters
improves agents’ coordination [52]. The architecture modifications are marked
for the three policy types. $\pi_{f}$ uses a _self-attention_ (SA) module
[53], $\pi_{h}$ a _Gated-Recurrent-Unit_ (GRU) module [54] and $\pi_{e}$ does
not use any of them. The embedding layer is linear with $100$ neurons and
$tanh$ activation. High-level commander policy has only one instance for both
aircraft types. Since we train the policies with the CTDE scheme, the critic
gets the observations of all interacting agents and their actions (global
information) as input. Besides parameter sharing, a fully observable critic
improves coordination between heterogeneous agents. We update our network
parameters using the Actor-Critic approach of Proximal Policy Optimization
(PPO) [55] (see Alg. 1).

Figure 5: Neural network architecture. Algorithm 1 PPO training procedure for
$\pi_{f}$, $\pi_{e}$ and $\pi_{h}$

1: Set number of episodes $N$, time horizon $T$, batch size $b$ and levels $L$

2: Initialize buffer $D$ $\rightarrow$ $\\{\\}$, policy parameters $\theta$,
value function parameters $\phi$

3: for level $l=1$ to $L$ do

4: for episode $n=0$ to $N$ do

5: Initialize state $S_{0}$

6: for $t=0$ to $T$ do

7: Get agent actions $A_{t,ag}$ by current policy $\pi_{\theta}$

8: Get opp action $A_{t,o}$: script if $l<=3$ else $\pi_{\theta_{l-1}}$

9: Execute $(A_{t,ag},A_{t,o})$, obtain $R_{t}$ and $S_{t+1}$

10: Store $D$ $\leftarrow$ $(S_{t},A_{t,ag},A_{t,o},R_{t},S_{t+1})$

11: end for

12: if $|D|>=b$ then

13: compute advantage estimation $A^{\pi_{\theta}}$

14: for update iteration $k=1$ to $K$ do

15: update policy parameters
$\theta_{k+1}=\underset{\theta}{\arg\max}\mathop{\mathbb{E}}_{\tau\sim
D}[\sum_{t=0}^{T}[\min(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}A^{\pi_{\theta_{k}}},$
$clip(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})},1-\epsilon,1+\epsilon)A^{\pi_{\theta_{k}}})]]$

16: update value function parameters
$\phi_{k+1}=\underset{\phi}{\arg\min}\mathop{\mathbb{E}}_{\tau\sim
D}[\sum_{t=0}^{T}(V_{\phi}(s_{t})-\hat{R}_{t})^{2}]$

17: end for

18: empty buffer $D$ $\rightarrow$ $\\{\\}$

19: set $\theta$ $\leftarrow$ $\theta_{k+1}$

20: set $\phi$ $\leftarrow$ $\phi_{k+1}$

21: end if

22: end for

23: end for

##  IV Experiments

###  IV-A Simulation Settings

We validate our method by simulations.222Code available at
github.com/IDSIA/marl. For this purpose we developed a dedicated 2D (Python)
simulation platform to have full control and low inertia. Our platform is
lightweight, fast and simulates the dynamics of our aircraft (Sect. III-A).
Trajectories of each aircraft can be visualized and a landmark is set at the
position where an aircraft got destroyed (Fig. 9). Map size and number of
interacting aircraft can be specified, highlighting the diversity and
scalability properties of our model. We refer to time step $t$ as one
simulation round. A simulation episode ends when either the time horizon is
reached or there are no alive aircraft of one team. An aircraft is destroyed
when getting hit by cannon or rocket or when hitting the map boundary. For
each episode, a side of the map (left or right half) is chosen at random for
each team, followed by generating random initial positions and headings for
each aircraft. Agent training uses the popular libraries _Ray RLlib_ and
_Pytorch_.

###  IV-B Training and Results

Since we use a shared policy for each aircraft type and the commander, we do
not need to restrict our simulations to a fixed number of agents. For every
episode, aircraft types are randomly selected, having at least one of each
type per group. Map sizes per axis are $30$ km for low-level and $50$ km for
high-level policy training. Learning curves showing mean rewards include the
performance of all agents. Evaluations are done for $1,000$ episodes. _Win_ is
when all opponents are destroyed, _loss_ if all agents got destroyed and
_draw_ if at least one agent per team remains alive after the episode ends.

The PPO parameters are kept constant for all training procedures: learning
rate actor and critic $lr=0.0001$, discount factor $\gamma=0.95$, clip
parameter $\epsilon=0.2$, Adam as optimizer, batch size of $2,000$ for low-
level policies and $1,000$ for high-level policy. We train all policies
according to Alg. 1 and set the five levels as discussed in Sect. III-E. We
have compared the performance of the proposed architecture to a standard RL
system, which showed a very poor performance and is left out.

####  IV-B1 Low-level policies

Since each agent can sense only one opponent and one friendly agent at a time,
we train our low-level policies in a 2vs2 setting. Since our framework allows
an arbitrary number of agents and opponents, we evaluate the performance of
each policy type in different combat scenarios. For every new episode, the
aircraft ammunition is: $200$ cannon shots and $5$ rockets (AC1). We make the
opponents stronger by giving them ammunition of $400$ cannons and $8$ rockets.
As the levels increase, we also increase the time horizon of an episode by
$\Delta T=50$ starting from $T=200$ on L1.

Let us first examine the fight policy $\pi_{f}$. Training starts in L1 and
ends in L5. In the latter, the opponents get assigned one of the previous
learned fight policies at random for every episode. To highlight the strength
of our network architecture, we consider L3, where training is done against
script based opponents. The combat behavior of the opponents at this level is
the most deterministic, therefore allowing to compare the performance of
different architectures (see Fig. 6).

Figure 6: Training performance of $\pi_{f}$ at L3: SA-Net is our self-
attention network (Fig 5), (_no Curr_) is the same network but trained without
curriculum, FC is a fully connected network with two layers of $500$ neurons
and $tanh$ activation.

We evaluate the performance of our agents when training has completed (Fig.
7). We deploy every agent with $\pi_{f}$ of L5 and every opponent with
$\pi_{f}$ of L4. The combat skills of AC1 clearly surpass those of AC2, which
is most likely due to rockets as further equipment and having more agile
dynamics. We infer that our agents could further improve their combat
performance during L5 training and are able to combat in scenarios up to 5vs5,
even though training was conducted in a 2vs2 scheme. However, as the number of
aircraft increases, the portion of draws also rises. This observation may
suggest that the low-level policy has reached its peak learning capacity. An
example of a fight scenario is shown in Fig 9(b), showing the circular
trajectories to reach the tail of the opponents.

k-1 k-2 d-1 d-2 fk-1 fk-2 Draw 0200400600800Evaluation Level 5 (a) $\pi_{f}$
2vs2 statistics

$0\%$$25\%$$50\%$$75\%$$100\%$2vs23vs34vs45vs5Evaluation FightWinLossDraw (b)
$\pi_{f}$ evaluation scenarios

Figure 7: Evaluation of $\pi_{f}$ after finishing L5 training. Destroying an
opponent is abbreviated with _k_ , getting destroyed with _d_ and _fk_
indicates “friendly” kills. The attached numbers indicate the aircraft types,
e.g. $k-1$ kill by AC1, $d-2$ AC2 destroyed.

The escape policy $\pi_{e}$ has the purpose of fleeing from opponents. We
consider only L3 for training and show the training results in Fig. 8(a).
Agents can still fire and destroy opponents, but training results indicate the
correct behavior by exploiting the escaping reward more than the killing
reward. As the number of aircraft increases in evaluation, the agents are less
capable of fleeing successfully from opponents (Fig. 8(b)). Fleeing
trajectories are visualized in Fig 9(c).

(a) $\pi_{e}$ training performance

$0\%$$25\%$$50\%$$75\%$$100\%$2vs23vs34vs45vs5Evaluation
EscapeEscapedKilledKills (b) $\pi_{e}$ evaluation scenarios

Figure 8: Training performance of $\pi_{e}$ at L3 (a) and different combat
scenarios (b). _Escaped_ means no agent got killed (including going out of
boundary), _killed_ is when at least one agent got killed, _kills_ when at
least one opponent got killed. Algorithm 2 Simulation procedure for commander
policy $\pi_{h}$

1: Set high-level time horizon $T_{h}=40$, low-level time horizon $T=10$ and
max number of episodes $N$

2: for episode $n=0$ to $N$ do

3: Sample a 3vs3 scenario

4: for $t_{h}=0$ to $T_{h}$ do

5: Get commander actions for each agent $A_{h,t_{h}}$

6: Activate corresponding low-level policies $\pi_{f}$ or $\pi_{e}$

7: for $t=0$ to $T$ do

8: Execute $\pi_{f}$ or $\pi_{e}$ for each agent and opponent

9: if $t\geq 10$ or event then

10: break

11: end if

12: end for

13: Get reward and next state

14: end for

15: end for

(a) Commander

(b) Fight

(c) Escape

Figure 9: Different policy trajectories in the simulation platform.

####  IV-B2 High-level policy

The purpose of the commander policy $\pi_{h}$ is to provide strategic commands
(attack or escape). Since $\pi_{h}$ can observe three agents and three
opponents per time, we do a 3vs3 combat training. Agents and opponents are
equipped either with $\pi_{f}$ of L5 or $\pi_{e}$. Opponents are mainly set to
fight and randomly to escape. Ammunition is set to $300$ cannons and $8$
rockets. We include the low-level policies as part of the environment (low-
level dynamics in Fig. 2). We run experiments according to the procedure in
Alg. 2. The commander gets invoked dynamically on events or when a low-level
horizon is reached. Events are characterized as:

  * •

any aircraft got destroyed (by shooting or hitting map boundary);

  * •

an agent approaches the map boundary ($d<6km$);

  * •

an agent _or_ an opponent is in favorable situation as described in Eq. (3);

  * •

two opponents are close and face an agent ($d<5km$,
$\alpha_{ATA,a}<30^{\circ}$).

Training results are in Fig 10. The learning curves of all models quickly
saturate (Fig 10(a)), but the GRU module improves the result by storing the
last state. We choose this model as our commander $\pi_{h}$ and evaluate the
performance. We infer that the commander $\pi_{h}$ improves combat performance
for small team sizes. However, as the number of aircraft increases in
evaluation scenarios (Fig 10(b)), the result tends to a draw and an equal win-
to-loss ratio, which we could also expect when no commander would be involved,
since both teams are equally equipped (except for number of AC1 and AC2 per
team). The reason for this is the partial observability of only three
opponents around an agent. Another reason might be the stochasticity involved,
where an opponent suddenly might switch from fight to escape, affecting the
coordination of the commander. A further aspect for not achieving superiority
in larger team sizes is the number of AC1 and AC2 per team, since AC1 has
stronger combat performance as shown in Fig 7(a). An example of a 2vs4 combat
scenario with the commander involved is shown in Fig 9(a), where two opponents
could successfully be destroyed within the time horizon.

(a) $\pi_{h}$ training performance

$0\%$$25\%$$50\%$$75\%$$100\%$2vs23vs34vs45vs5Evaluation CommanderWinLossDraw
(b) $\pi_{h}$ evaluation scenarios

Figure 10: Training performance of $\pi_{h}$ for different architectures. SA-
Net is the network used to train $\pi_{f}$, GRU-Net is the network in Sect.
III-E, FC is a fully connected network with two layers of 500 neurons and
$tanh$ activation.

##  V Conclusion

We presented a hierarchical, heterogeneous, multi-agent reinforcement learning
procedure for air-to-air combat maneuvering. The key ideas are using
curriculum learning, fictitious self-play and sophisticated neural networks.
The empirical validation shows the promising potential of our design. Our
agents can effectively engage in air-to-air combat with solid resilience,
while the commander has difficulties in successfully coordinating larg team
configurations. In future work, we intend further to improve the hierarchical
structure for better tactical decisions, regardless of the team size. We also
plan to incorporate a dedicated communication mechanism as well as to switch
to 3D aircraft models for a more realistic environment and accurate aircraft
dynamics.

## References

  * [1] R. L. Shaw, _Fighter Combat: Tactics and Maneuvering_. Naval Institute Press, 1987.
  * [2] G. Burgin, L. Fogel, and J. P. Phelps, “An adaptive maneuvering logic computer program for the simulation of one-on-one air-to-air combat,” _NASA, Contractor Report CR-2582_ , 1975.
  * [3] G. Burgin and L. Sidor, “Rule-based air combat simulation,” _DTIC Document_ , 1988.
  * [4] R. Jones, J. Laird, P. Nielsen, K. Coulter, P. Kenny, and F. Koss, “Automated intelligent pilots for combat flight simulation,” _AI Mag._ , vol. 20, pp. 27–44, 1999.
  * [5] F. Austin, G. Carbone, M. Falco, H. Hinz, and M. Lewis, “Automated maneuvering decisions for air-to-air comba,” _Guidance, Navigation, and Control and Co-located Conferences, American Institute of Aeronautics and Astronautics_ , 1987\.
  * [6] J. Eklund, J. Sprinkle, and S. Sastry, “Implementing and testing a nonlinear model predictive tracking controller for aerial pursuit/evasion games on a fixed wing aircraft,” in _Proceedings of the 2005, American Control Conference._ IEEE, 2005.
  * [7] K. Virtanen, T. Raivio, and R. P. Hamalainen, “Modeling pilot's sequential maneuvering decisions by a multistage influence diagram,” _Journal of Guidance, Control, and Dynamics_ , vol. 27, no. 4, pp. 665–677, 2004.
  * [8] K. Virtanen, J. Karelahti, and T. Raivio, “Modeling air combat by a moving horizon influence diagram game,” _Journal of Guidance, Control, and Dynamics_ , vol. 29, no. 5, pp. 1080–1091, 2006.
  * [9] D.-I. You and D. H. Shim, “Design of an aerial combat guidance law using virtual pursuit point concept,” _Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering_ , vol. 229, no. 5, pp. 792–813, 2014.
  * [10] B. Jarmark and C. Hillberg, “Pursuit-evasion between two realistic aircraft,” _Journal of Guidance, Control, and Dynamics_ , vol. 7, no. 6, pp. 690–694, 1984.
  * [11] A. W. Merz, “To pursue or to evade - that is the question,” _Journal of Guidance, Control, and Dynamics_ , vol. 8, no. 2, pp. 161–166, 1985.
  * [12] N. Greenwood, “A differential game in three dimensions: The aerial dogfight scenario,” _Dynamics and Control_ , vol. 2, no. 2, pp. 161–200, 1992.
  * [13] J. S. McGrew, J. P. How, B. Williams, and N. Roy, “Air-combat strategy using approximate dynamic programming,” _Journal of Guidance, Control, and Dynamics_ , vol. 33, no. 5, pp. 1641–1654, 2010.
  * [14] X. Ma, L. Xia, and Q. Zhao, “Air-combat strategy using deep q-learning,” in _2018 Chinese Automation Congress (CAC)_. IEEE, 2018.
  * [15] M. Day, L. Strickland, E. Squires, K. DeMarco, and C. Pippin, “Responding to unmanned aerial swarm saturation attacks with autonomous counter-swarms,” in _Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR IX_ , T. Pham, M. A. Kolodny, and D. M. Wiegmann, Eds. SPIE, 2018.
  * [16] B. Vlahov, E. Squires, L. Strickland, and C. Pippin, “On developing a UAV pursuit-evasion policy using reinforcement learning,” in _(ICMLA 2018)_. IEEE, 2018.
  * [17] R. Smith, B. Dike, R. Mehra, B. Ravichandran, and A. El-Fallah, “Classifier systems in combat: two-sided learning of maneuvers for advanced fighter aircraft,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 186, no. 2-4, pp. 421–437, 2000.
  * [18] R. E. Smith, B. A. Dike, B. Ravichandran, A. El-Fallah, and R. K. Mehra, “The fighter aircraft LCS: A case of different LCS goals and techniques,” in _Lecture Notes in Computer Science_. Springer Berlin Heidelberg, 2000, pp. 283–300.
  * [19] R. E. Smith, B. A. Dike, B. Ravichandran, A. El-Fallah, and R. Mehra, “Two-sided, genetics-based learning to discover novel fighter combat maneuvers,” in _Lecture Notes in Computer Science_. Springer Berlin Heidelberg, 2001, pp. 233–242.
  * [20] R. Smith, B. Dike, B. Ravichandran, A. El-Fallah, and K. Mehra, “Discovering novel fighter combat maneuvers,” in _Creative Evolutionary Systems_. Elsevier, 2002, pp. 467–VIII.
  * [21] R. Smith, A. El-Fallah, B. Ravichandran, R. Mehra, and B. A. Dike, “The fighter aircraft LCS: A real-world, machine innovation application,” in _Applications of Learning Classifier Systems_. Springer Berlin Heidelberg, 2004, pp. 113–142.
  * [22] A. Toubman, J. J. Roessingh, P. Spronck, A. Plaat, and J. V. D. Herik, “Transfer learning of air combat behavior,” in _ICMLA 2015_. IEEE, 2015.
  * [23] A. Toubman, J. J. Roessingh, J. van Oijen, R. A. Lovlid, M. Hou, C. Meyer, L. Luotsinen, R. Rijken, J. Harris, and M. Turcanik, “Modeling behavior of computer generated forces with machine learning techniques, the NATO task group approach,” in _2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, 2016\.
  * [24] G. den Broeder, R. Ellison, and L. Emerling, “On optimum target assignments,” _Operations Research_ , vol. 7, 1959.
  * [25] G. Tidhar, C. Heinze, and M. Selvestrel, “Flying together: Modelling air mission teams,” _Applied Intelligence_ , vol. 8, no. 3, pp. 195–218, 1998\.
  * [26] M. A. Day, _Multi-Agent Task Negotiation Among UAVs to Defend Against Swarm At- tacks_. Thesis, Naval Postgraduate School, Monterey, CA,, 2012.
  * [27] J.-D. Zhang, Y.-F. Yu, L.-H. Zheng, Q.-M. Yang, G.-Q. Shi, and Y. Wu, “Situational continuity-based air combat autonomous maneuvering decision-making,” _Defence Technology_ , 2022.
  * [28] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013.
  * [29] J. Guo, Z. Wang, J. Lan, B. Dong, R. Li, Q. Yang, and J. Zhang, “Maneuver decision of uav in air combat based on deterministic policy gradient,” _IEEE 17th International Conference on Control & Automation_, 2022.
  * [30] W. Yuan, Z. Xiwen, Z. Rong, T. Shangqin, Z. Huan, and D. Wei, “Research on ucav maneuvering decision method based on heuristic reinforcement learning,” _Computational Intelligence and Neuroscience_ , vol. 2022, 2022.
  * [31] Z. Fan, Y. Xu, Y. Kang, and D. Lou, “Air combat maneuver decision method based on a3c deep reinforcement learning,” _Machines_ , 2022.
  * [32] Q. Yang, J. Zhang, G. Shi, J. Hu, and Y. Wu, “Maneuver decision of uav in short-range air combat based on deep reinforcement learning,” _IEEE Access_ , vol. 8, 2020.
  * [33] J. H. Bae, H. Jung, S. Kim, S. Kim, and Y.-D. Kim, “Deep reinforcement learning-based air-to-air combat maneuver generation in a realistic environment,” _IEEE Access_ , vol. 11, pp. 26 427–26 440, 2023.
  * [34] Z. Wang, H. Li, H. Wu, and Z. Wu, “Improving maneuver strategy in air combat by alternate freeze games with a deep reinforcement learning algorithm,” _Mathematical Problems in Engineering_ , vol. 2020, 2020.
  * [35] S. Gronauer and K. Diepold, “Multi-agent deep reinforcement learning: a survey,” _Artificial Intelligence Review_ , vol. 55, no. 2, pp. 895–943, 2021.
  * [36] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _Conference on Neural Information Processing Systems_ , 2017.
  * [37] X. Chu and H. Ye, “Parameter sharing deep deterministic policy gradient for cooperative multi-agent reinforcement learning,” 2017.
  * [38] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” 2017.
  * [39] W. Kong, D. Zhou, K. Zhang, and Z. Yang, “Air combat autonomous maneuver decision for one-on-one within visual range engagement base on robust multi-agent reinforcement learning,” _IEEE 16th International Conference on Control & Automation_, 2020.
  * [40] N. Andrew, D. Harada, and S. Russel, “Policy invariance under reward transformations: Theory and applications to reward shaping,” _ICML ’99: Proceedings of the Sixteenth International Conference on Machine Learning_ , 1999\.
  * [41] L. Wang, J. Hu, and C. Zhao, “Autonomous maneuver strategy of swarm air combat based on ddpg,” _Autonomous Intelligent Systems_ , 2021.
  * [42] T. Zhang, T. Qiu, Z. Liu, Z. Pu, J. Yi, J. Zhu, and R. Hu, “Multi-uav cooperative short-range combat via attention-based reinforcement learning using individual reward shaping,” _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022.
  * [43] F. Jiang, M. Xu, Y. Li, H. Cui, and R. Wang, “Short-range air combat maneuver decision of uav swarm based on multi-agent transformer introducing virtual objects,” _Engineering Applications of Artificial Intelligence 123_ , vol. 123, p. 106358, 2023.
  * [44] A. Barto and S. Mahadevan, “Recent advantages in hierarchical reinforcement learning,” _Discrete Event Dynamic Systems_ , pp. 41–77, 2003.
  * [45] A. Pope, J. Ide, D. Micovic, H. Diaz, and D. Rosenbluth, “Hierarchical reinforcement learning for air-to-air combat,” _arXiv:2105.00990v2_ , 2021\.
  * [46] J. Chai, W. Chen, Y. Zhu, Z.-X. Yao, and D. Zhao, “A hierarchical deep reinforcement learning framework for 6-dof ucav air-to-air combat,” _IEEE Transactions on systems, man and cybernetics: systems_ , 2023.
  * [47] W.-r. Kong, D.-y. Zhou, Y.-j. Du, Y. Zhou, and Y.-y. Zhao, “Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat,” _IET Control Theory and Applications_ , 2022.
  * [48] L. Yue, R. Yang, J. Zou, Y. Zhang, Q. Li, and Y. Zhang, “Unmanned aerial vehicle swarm cooperative decision-making for sead mission: A hierarchical multiagent reinforcement learning approach,” _IEEE Access_ , vol. 10, 2022\.
  * [49] J. Hu, L. Wang, T. Hu, C. Guo, and Y. Wang, “Autonomous maneuver decision making of dual-uav cooperative air combat based on deep reinforcement learning,” _Electronics_ , vol. 11, p. 467, 2022.
  * [50] Y. Flet-Berliac, “The promise of hierarchical reinforcement learning,” _The Gradient_ , 2019.
  * [51] V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” _NeurIPS_ , 1999.
  * [52] J. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” _Adaptive Learning Agents Workshop at AAMAS_ , 2017.
  * [53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS 2017_ , 2017.
  * [54] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, and Y. Bengio, “Learning phrase representations using rnn encoder–decoder for statistical machine translation,” _arXiv:1406.1078_ , 2014.
  * [55] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv:1707.06347v2_ , 2017.