revtex4-2Repair the float

#  Laser-target symmetry-breaking in high harmonic generation: from frequency
shift to odd-even intensity modulation

Doan-An Trieu  Computational Physics Key Laboratory K002, Department of
Physics, Ho Chi Minh City University of Education, Ho Chi Minh City 72711,
Vietnam  Van-Hoang Le  Computational Physics Key Laboratory K002, Department
of Physics, Ho Chi Minh City University of Education, Ho Chi Minh City 72711,
Vietnam  Ngoc-Loan Phan<EMAIL_ADDRESS>Computational Physics Key
Laboratory K002, Department of Physics, Ho Chi Minh City University of
Education, Ho Chi Minh City 72711, Vietnam

###### Abstract

Although the frequency shift and odd-even intensity modulation in high-order
harmonic generation (HHG) have both been observed for asymmetric laser-target
systems, they are typically studied as two separate phenomena. In this Letter,
we provide a comprehensive picture of these two nonlinear optical phenomena,
unifying them through a common origin - asymmetry of the laser-target system.
By tuning asymmetric laser-target systems, we discover a transition from the
harmonic frequency shift to the odd-even intensity modulation upon increasing
the duration of the driving laser pulse. Specifically, these phenomena are
observed simultaneously for laser pulses with intermediate pulse duration. For
numerical evidence, we solve the time-dependent Schrödinger equation, while
insight into the underlying physics is obtained from a simplified analytically
tractable model. Understanding the asymmetric characteristics reflected in the
HHG as provided is crucial for retrieving laser-target information, sampling
external fields, and probing molecular dynamics.

Introduction — High-order harmonic generation (HHG) resulting from the laser-
matter interaction is crucial for attosecond technologies, enabling tracking
ultrafast dynamics inside matters with unprecedented temporal resolution [1,
2, 3, 4, 5]. On the other hand, the HHG by itself reflects various properties
of the laser-target system; therefore, it is an effective tool for imaging
molecular structures and dynamics [6, 7, 8, 9, 10]. Fully comprehending HHG
features benefits these important applications in strong-field physics and
attosecond sciences.

A well-known feature of HHG emitted from a symmetric laser-target system is
its spectrum peaks positioning at only odd multiples of the fundamental laser
frequency [11, 12]. This pattern results from the interference of uniform
attosecond bursts emitted every half-cycle time intervals as the electron
recombines with the parent ion after being driven by the laser electric field
following the tunneling ionization [11, 12, 13, 14]. Meanwhile, for a laser-
target system lacking centrosymmetric symmetry, HHG reflects this symmetry-
breaking by producing even-order harmonics in addition to the odd-order ones
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33].
Tuning asymmetry of a laser-target system thus modulates the intensities of
odd and even harmonics [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33].

Recent advances in laser technology have progressively shortened laser pulses
to just a few cycles with stable carrier-envelope phases (CEP) [34, 35, 36].
Interacting with those pulses, targets emit HHG, which no longer follows the
odd- or even-order rules of harmonics due to the nonuniform attosecond bursts
caused by the rapid change of the instantaneous electric field after half of
the laser cycle [8, 37]. Furthermore, adjusting the short-laser-pulse’s
asymmetry shifts the harmonic peaks, which is significant for characterizing
laser CEP [37, 38] and pulse waveform [39, 40], probing nuclear dynamics [8,
41], and controlling the ionization and recombination times [30, 9].

While the odd-even intensity modulation [17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33] and harmonic frequency shift [42, 43, 44, 37,
39, 45, 46, 32, 47, 38, 8, 41, 30, 9, 48, 49, 40] both characterize asymmetric
laser-target systems in two limits of the laser pulse, multi-cycle versus few-
cycle, they are usually studied separately since different laser-target
objects are considered. Remarkably, there is still no direct relationship
between the rules governing these two phenomena. Also, it is intriguing to
identify situations where both effects are simultaneously observable.

In the present Letter, we theoretically study the link between harmonic
frequency shift and odd-even intensity modulation upon increasing the duration
of the laser pulses while interacting with tunable asymmetric laser-target
systems. The two systems with different controlling symmetry-breaking factors
are considered: (i) a symmetric laser-atom system with an additional low-
frequency electric field, and (ii) a planar molecule with varying orientation
angles. We demonstrate that harmonic frequency shift and odd-even harmonic
modulation are both governed by the same symmetry-breaking factor and, indeed,
are the two different manifestations of this symmetry-breaking in HHG. Also,
we observe both effects occurring simultaneously in HHG at intermediate pulse
durations.

Figure 1: Sketch of subcycle and intercycle interference in the time (a) and
frequency (b) domains. The subcycle interference is synthesized from subcycle
emission bursts, emitted as electron recombination after implementing closed
trajectories [solid curves in Fig (a)], resulting in an interference pattern
with flat peaks whose positions may not at integer harmonic orders [red dotted
curve in Fig. (b)]. The intercycle interference resulting from emission bursts
repeated each cycle (a) leads to a comb pattern [blue dashed curves in Fig
(b)]. In a multicycle laser pulse, both subcycle and intercycle are involved,
leading to odd-even intensity modulation [purple solid curves in Fig (b)].

Subcycle and intercycle interferences — Although harmonic frequency shift and
odd-even intensity modulation have been studied separately before (see Refs.
[37, 15, 16, 17, 39]), we broadly review them here to emphasize their
interconnectedness and explain their underlying physics comprehensively.

According to the strong-field approximation, HHG in the time domain is indeed
the interference of attosecond bursts emitted at each recombination event of
electrons to the parent ion after traveling in the laser electric field
following the ionization near the laser intensity peaks [11, 12]. As shown in
Fig. 1, two types of interference exist for asymmetric laser-target systems:
subcycle and intercycle. Subcycle interference arises from adjacent bursts
emitted within one optical cycle. During each half-cycle, two bursts are
generated by short and long trajectories responsible for generating each
harmonics [11, 12]. However, for harmonics near the cutoff or under good
phase-matching, only short trajectories remain, while long trajectories cancel
out each other. Thereby, the subcycle interference can be mathematically
expressed as

$D_{\mathrm{sub}}(N)=D_{\mathrm{{}_{1}}}e^{i\varphi_{\mathrm{{}_{1}}}}\bigg{[}1-\dfrac{D_{\mathrm{{}_{2}}}}{D_{\mathrm{{}_{1}}}}e^{i(\Delta\varphi-N\pi)}\bigg{]},$
(1)

where $N$ is the harmonic order; $D_{\mathrm{j}}$ and $\varphi_{\mathrm{j}}$
are the amplitude and phase of the emission bursts; and the index
$\mathrm{j}=1,~{}2$ means the first and second half-cycles.
$\Delta\varphi=\varphi_{\mathrm{2}}-\varphi_{\mathrm{1}}$ is the phase
difference between the two attosecond bursts. For symmetry-breaking laser-
target systems, the two adjacent bursts with a half-cycle time interval are
asymmetric both in amplitude and phase.

Subcycle interference [Eq. (1)] modulates its intensity as presented by the
red dotted curve in Fig. 1(b), whose maxima locate at harmonic orders

$N_{\mathrm{sub-max}}=2m+1+\dfrac{\Delta\varphi}{\pi},$ (2)

which strongly depend on the phase difference $\Delta\varphi$, characterizing
the asymmetry of the laser-target system. Here, $m$ is an integer. If the
system is centrosymmetric under the half-cycle time translation
($\Delta\varphi=0$), the maxima overlap with odd-order harmonics only.

Subcycle interference occurs when the laser pulse consists of a few optical
cycles, i.e., in the few-cycle limit. However, for a multicycle laser pulse,
the subcycle emission bursts are repeated every cycle but weighted by the
factor $\alpha$ because of the unequal instantaneous ionization rate, unequal
peak amplitudes of electric fields in each optical cycle, or the depletion. In
this case, the intercycle interference between subcycle emissions bursts is
involved as

$D(N)=D_{\mathrm{sub}}(N)~{}D_{\mathrm{inter}}(N).$ (3)

where $D_{\mathrm{inter}}(N)=\sum\limits_{j=0}^{n}\alpha_{j}e^{jiN\omega T}$
characterizes the intercycle interference. Here, $n$ is the number of the
pulse’s optical cycles, $\omega$ and $T$ are the laser carrier frequency and
optical cycle. In the continuous-wave limit or the number cycle is large
enough, every optical cycle is approximately identical, i.e.,
$\alpha_{j}\approx 1$, the intercycle interference has magnitude

$|D_{\mathrm{inter-cw}}(N)|=\dfrac{\sin[(n+1)N\pi]}{\sin(N\pi)}.$ (4)

Equation (4) implies that the peaks are placed at integer numbers of $N$,
i.e., at both odd- and even-order harmonics, as presented by the dashed blue
curve in Fig. 1(b).

Equation (3) demonstrates that when an asymmetric laser-target system
interacts with a multicycle laser pulse, the harmonic peak’s frequency (odd
and even) is governed by $D_{\mathrm{inter}}(N)$, but amplitude is controlled
by $D_{\mathrm{sub}}(N)$. Thereby, the intensity ratio between the even and
odd harmonics is

$\dfrac{I_{\mathrm{even}}}{I_{\mathrm{odd}}}\approx\dfrac{|D_{\mathrm{sub}}(2m)|^{2}}{|D_{\mathrm{sub}}(2m+1)|^{2}}\approx
1-\dfrac{2\kappa\cos\Delta\varphi}{1+\kappa\cos\Delta\varphi},$ (5)

where
$\kappa=\dfrac{2D_{\mathrm{1}}D_{\mathrm{2}}}{|D_{\mathrm{1}}|^{2}+|D_{\mathrm{2}}|^{2}}$
is the intensity imbalance of the two subcycle emission bursts. This odd-even
intensity was well validated for polar molecules in a multicycle laser pulse
[23], or atoms in a combination of a multicycle pulse with a low-frequency
electric pulse [33].

In summary, we have connected two nonlinear optical phenomena, the harmonic
frequency shift and the odd-even intensity modulation, as manifestations of
the phase difference between adjacent attosecond bursts. Their appearance
depends on the duration of the laser pulse. In a few-cycle laser pulse, only
subcycle interference is involved in the HHG of an asymmetric laser-target
system. The symmetry-breaking factor causes a harmonic frequency shift by Eq.
(2). In contrast, in multicycle laser pulse, both subcycle and intercycle
interferences are involved, producing both odd and even harmonics, whose even-
to-odd ratio is modified by changing the asymmetry factor by Eq. (5). The
governing factor of these two effects is the same: the phase difference
between adjacent attosecond bursts, $\Delta\varphi$.

Figure 2: Transition from SEF-induced harmonic frequency shift in single-cycle
(a) and few-cycle (b) regimes to SEF-induced odd-even intensity modulation in
multi-cycle (c) and quasi-cw (d) regimes of HHG emitted from a hydrogen atom
in the combination of driving laser pulse and SEF (e). Upper panels [(a1) -
(d1)] show the electric field of driving laser pulses (grey curves) and the
harmonic emission bursts in the absence of SEF. Panels [(a2) - (d2)]
demonstrate the TDSE-simulated HHG for harmonics near cutoff when varying SEF.
In Figs. [(a2), (b2)], the black dots pick the centroids of harmonic peaks,
which closely follow the analytical predictions (6) presented by solid lines.
The separated comparisons are exhibited in Figs. [(a3), (b3)]. For multi-cycle
and quasi-cw regimes, adding a SEF causes additional harmonics (called
sideband) besides existing peaks without SEF (called mainband) [(c2), (d2)].
Varying SEF, the intensity of the mainband and the sideband [(c3), (d3)] and
the even-to-odd ratios [(c4), (d4)] modulate with the same period, which is
consistent with the analytical prediction (7) (black curves). When using
intermediate pulses [(b), (c)], both harmonic frequency shift and odd-even
intensity modulation are observed. The driving laser with the intensity of
$1.5\times 10^{14}$ W/cm2, and the wavelength of 800 nm is used.

Transition from harmonic frequency shift to odd-even intensity modulation — To
investigate the interconnectedness between the harmonic frequency shift and
odd-even intensity modulation in HHG, we consider laser-target systems with
varying laser pulse durations and tunable symmetry-breaking factors.
Specifically, we show results for driving laser pulses with three, four, and
six optical cycles, called single-cycle, few-cycle, and multi-cycle regimes.
In addition, the quasi-continuous-wave limit (quasi-cw) is also given using a
driving laser pulse of ten optical cycles with the flattop envelope. The
cosine-squared envelope and carrier-envelope phase (CEP) of $-\pi/2$ are
chosen for all the considered driving pulses. Two targets are considered: (i)
a hydrogen atom with an additional weak low-frequency electric field and (ii)
a planar molecule H${}_{3}^{2+}$ with varying orientation angles. We chose
these cases because the symmetry-breaking factors (the external electric field
and the molecular orientation) causing the asymmetry are tunable. In the
adiabatic approximation, a low-frequency electric field (terahertz field) can
be considered as a quasi-static electric field.

We proceed the study with numerical evidence from solving the time-dependent
Schrödinger equation (TDSE). These results can be additionally validated by
the analytical analysis based on simple models. We show the results in the
main text while the details of calculations are provided in supplementary
[50].

(i) Quasi-static electric field as a symmetry-breaking factor

Figure 2 exhibits a scenario of HHG where a hydrogen atom interacts with the
driving laser pulse combined with a tunable quasi-static electric field (SEF).
Following the pulse-duration change from single-cycle to quasi-cw regimes,
this figure demonstrates the transition from SEF-induced harmonic frequency
shift [Panels (a)-(c)] to SEF-induced even-to-odd modulation in the HHG
[Panels (c)-(d)].

Particularly, in single-cycle and few-cycle regimes, there are only two or
three attosecond bursts, as shown in the time domain [Panels (a1)-(b1)],
consequently only subcycle interference occurs. In this case, harmonic peaks
shift because of the varying SEF-induced asymmetry [Panels (a2)-(b2)]. The
figures show that the shift is up to four harmonic orders when SEF varies from
$-20$ to $+20$ MV/cm. Meanwhile, when the driving laser pulse is long enough
(multi-cycle or quasi-cw), both subcycle and intercycle interference are
involved because of the contribution of more emission bursts in the time
domain [Panels (c1)-(d1)]. The intercycle interference sharpens the harmonic
peaks at specific orders for the multi-cycle regimes or integers for the
quasi-cw regime. Moreover, varying SEF leads to the intensity oscillation of
these harmonic peaks, as shown in Panels (c3)-(d3), resulting in the
oscillation of the intensity ratio between the adjacent harmonic orders,
called even-to-odd ratio [Panels (c4)-(d4)].

Specifically, both phenomena are observed in the same HHG pattern in Panels
(b) and (c), where the pulses with intermediate duration are used. The
frequency shift is seen clearly for harmonics near and at the cutoff, while
the odd-even intensity oscillation is found for harmonics below cutoffs. This
simultaneous existence of the two effects can be explained by looking into the
time-frequency profiles in Panels (b1)-(c1). They show that the harmonics near
the cutoff coherently interfere from only two emission bursts, i.e., subcycle
interference, leading to the harmonic frequency shift. Meanwhile, the low
harmonics away from the cutoff result from more than two emission bursts,
i.e., both subcycle and multi-cycle interference, causing odd-even intensity
oscillation.

To continue, we analyze the observed transition between the two effects, from
the harmonic frequency shift to the even-to-odd ratio oscillation. It is well-
known that the asymmetry caused by the SEF leads to the phase distortion of
the attosecond bursts. This SEF-induced phase distortion difference has been
calculated analytically in Refs. [33, 40] within the strong-field
approximation [11, 12]. Substituting it into Eq. (2), we can describe SEF-
induced shift in the form

$\Delta N=\pm 2C\dfrac{E_{m}}{\pi\omega^{3}}E_{T},$ (6)

where $\Delta N$ is the shift of harmonic order; $E_{m}$ is the average
amplitude of the two electric peaks at the two half-cycles at the center of
the driving pulses, which are responsible for the emission bursts; $E_{T}$ is
the SEF; and $C$ is a constant, which is equal to $2.558$ for harmonics near
cutoff. The analytical prediction (6) of SEF-induced frequency shift is fairly
consistent with that from numerical simulation, as shown in Panels (a2), (a3),
(b2), (b3), and (c2).

For the quasi-cw regime, the simulated even-to-odd ratio matches the
analytical description well. This formula, which has the form

$\dfrac{I_{\mathrm{even}}}{I_{\mathrm{odd}}}\approx\tan^{2}\bigg{(}C\dfrac{E_{0}}{\omega^{3}}E_{T}\bigg{)},$
(7)

is a consequence of Eq. (5) incorporated with the analytical THz-induced phase
distortion difference. The demonstration of Eq. (7) in Panels (c4) and (d4)
shows its consistency with numerical results. It is worth noting that the
even-to-odd ratio (7) and the frequency shift (6) are the two manifestations
of the same symmetry-breaking nature. The common factor
$2CE_{0}/\pi\omega^{3}$ is simultaneously the slope of linear frequency shift
in the single- or few-cycle regimes and half of the oscillation frequency in
odd-even intensity oscillation in the multi-cycle and quasi-cw regimes.

(ii) Orientation of planar molecule H${}_{3}^{2+}$ as a symmetry-breaking
factor

Figure 3: Transition from harmonic frequency shift [(a)-(c)] to odd-even
intensity oscillation [(c)-(d)] for a planar molecule H${}_{3}^{2+}$ with
varying orientation angle [model in Fig. (e)], interacting with a linearly
polarized laser pulse in single-, few-, multi-cycle, and quasi-cw regimes.
When using a multi-cycle laser (c), both frequency shift and odd-even
oscillation are simultaneously appeared. In Figs. [(a2), (b2), (c3), (d3)],
the frequency shift and odd-even oscillation numerically simulated from TDSE
(black dots) fairly match with those predicted analytically (solid curves).
The laser with the intensity of $3\times 10^{14}$ W/cm2, and the wavelength of
800 nm is used.

Figure 3 exhibits the numerical simulation of HHG obtained by solving TDSE for
planar molecule H${}_{3}^{2+}$ with varying orientation angles in different
pulse regimes. Because of the C3 symmetry of H${}_{3}^{2+}$, we present the
results only from $0^{\circ}$ to $120^{\circ}$ for the orientation angle. The
first observation from this picture is that, for all regimes, the HHG
intensity is enhanced at orientation angles of $30^{\circ}$ and $90^{\circ}$,
and strongly suppressed at $0^{\circ}$, $60^{\circ}$, and $120^{\circ}$. This
observation is similar to those observed in C3-symmetry monocrystal [51, 52].
The HHG orientation dependence is easily understood by its consistency with
the orientation-dependent electron density, as seen in Panel (e).

The second feature is the transition from the nonlinear frequency shift
[Panels (a)-(c)] into the odd-even intensity modulation [Panels (c)-(d)] with
varying orientation while increasing the pulse duration. Particularly, as a
result of the subcycle interference, the harmonic frequencies nonlinearly
shift with the period of $120^{\circ}$ in the near-cycle and few-cycle regimes
[Panels (a)-(b)]. To the best of our knowledge, most harmonic frequency shifts
observed so far vary linearly with the changing of asymmetry factors [40, 37,
39, 45, 46, 38, 47]. This nonlinear harmonic frequency shift is intriguing. On
the other hand, for harmonics in the quasi-cw regime [Panels (c)-(d)], the
positions of harmonic peaks are fixed, but the harmonic intensity is modulated
with the period of $60^{\circ}$ when varying molecular orientation. This odd-
even intensity modulation is a consequence of both subcycle and intercycle
interferences.

With a laser pulse of intermediate duration, i.e., in the multi-cycle regime
as shown in Panels (c), both frequency shift and odd-even modulation are
observed. Similar to the case of SEF-induced symmetry-breaking, the harmonic
frequency shift is clearly seen for harmonics at the cutoff, while low
harmonics exhibit odd-even intensity modulation.

Our previously mentioned observations can be additionally justified by
analytical means. According to Eqs. (2) and (5), for formulating analytical
formulae of the frequency shift and even-to-odd ratio, we need an analytical
form of the phase difference between the adjacent attosecond bursts
$\Delta\varphi$. However, different from the SEF case where $\Delta\varphi$
results from the SEF-induced distortion of electron trajectories that can
easily be treated by strong-field approximation, in the oriented
H${}_{3}^{2+}$-molecule case, $\Delta\varphi$ is caused by the molecule
itself. Therefore, we adopt the method of linear combination of atomic
orbitals (LCAO) to approximately describe the molecule H${}_{3}^{2+}$. The
details are presented in Supplementary [50].

According to LCAO, the electron recombination to the ground state gives an
emission characterized as $D_{1}\sim d_{\mathrm{1}}e^{i\varphi_{\mathrm{1}}}$,
where the phase $\varphi_{\mathrm{1}}$ satisfies

$\tan\varphi_{\mathrm{1}}=\dfrac{\sin\mathbf{k}\mathbf{R}_{1}+\sin\mathbf{k}\mathbf{R}_{2}+\sin\mathbf{k}\mathbf{R}_{3}}{\cos\mathbf{k}\mathbf{R}_{1}+\cos\mathbf{k}\mathbf{R}_{2}+\cos\mathbf{k}\mathbf{R}_{3}},$
(8)

with $\mathbf{R}_{j}$ ($j=1$, 2, 3) are coordinates of the H atoms with
respect to the center of charge, and $\mathbf{k}$ is the electron wave number.
After half of an optical cycle, the second emission is $D_{2}\sim
d_{\mathrm{2}}e^{i(\varphi_{\mathrm{2}}+N\pi)}$ with
$\varphi_{\mathrm{2}}=-\varphi_{\mathrm{1}}$.

First, we discuss the quasi-cw regime. Substituting
$\Delta\varphi=-2\varphi_{\mathrm{1}}$ and assume
$|d_{\mathrm{{}_{1}}}|\approx|d_{\mathrm{2}}|$ for simplicity, in to Eq. (5),
we have the even-to-odd ratio

$\dfrac{I_{\mathrm{even}}}{I_{\mathrm{odd}}}=\bigg{|}\dfrac{\sin{(y\sin\theta)}+\sin(y\cos\theta_{1})-\sin(y\cos\theta_{2})}{\cos{(y\sin\theta)}+\cos(y\cos\theta_{1})+\cos(y\cos\theta_{2})}\bigg{|}^{2},$
(9)

where $\theta$ is the orientation angle, $\theta_{1(2)}=\theta\pm\pi/6$, and
$y=k|\mathbf{R}_{j}|$ with $k=\sqrt{2N\omega}$. Its plot in Panels (c3)-(d3)
qualitatively follows the oscillation of even-to-odd ratio numerically
simulated from TDSE. The scaling factor between analytical and numerical odd-
even ratio is caused by the approximation
$|d_{\mathrm{{}_{1}}}|\approx|d_{\mathrm{2}}|$, i.e., ignoring the magnitude
asymmetry of the two adjacent attosecond bursts.

To analytically describe the frequency shift, we can not apply Eq. (2)
directly because the phase $\varphi_{1}$ also depends on the harmonic order
$N$ via Eq. (8). This circumstance makes Eq. (2) nontrivial. Therefore, we
approach this problem another way, rewriting Eq. (1) for the subcycle
interference, and find the condition that maximizes

$I_{\mathrm{sub}}=|D_{1}-D_{2}|^{2}.$ (10)

Detailed partition of Eq. (10) is shown in Supplementary [50]. Its solution is
found numerically and presented in Panels (a) and (b), which fairly follows
the results from the TDSE simulation.

Conclusions and discussions— Depending on the laser pulse duration, the
subcycle only or both subcycle and intercycle interferences of attosecond
bursts occur, which is reflected in the HHG spectra via the harmonic frequency
shift or even-to-odd intensity modulation. These non-linear effects are the
two manifestations of one common physics - the asymmetry of laser-target
systems. Although the harmonic shift and the even-to-odd harmonic intensity
modulation have been investigated previously, for the first time we provide a
unified picture for both of them, where the transition is demonstrated by
varying the laser pulse duration from single to multi-cycles. Interestingly,
both effects in HHG spectra are observable for the laser pulse with
intermediate duration.

For the above conclusion, we have utilized the numerical method of solving the
Schrödinger equation combined with the theoretical analysis based on simple
models. This study adopts two laser-target systems with tunable symmetry-
breaking factors; however, the results are general, and the conclusion can be
used for different asymmetric laser-target systems.

We can also discuss the application prospective because the harmonic frequency
shift and even-to-odd intensity modulation clearly encode the laser-target
information. For the first case in this study, where the electric field acts
as a symmetry-breaking factor, the asymmetry manifestations in HHG can be
leveraged to sample the field’s temporal profile or control the subcycle
motion of electron wavepackets. Meanwhile, when asymmetry factors originate
from the targets themselves, the harmonic frequency shift and even-to-odd
harmonic modulation can serve as tools to retrieve molecular structures or to
probe molecular dynamics.

## Acknowledgements

This work was funded by Vingroup and supported by Vingroup Innovation
Foundation (VINIF) under project code VINIF.2021.DA00031. The calculations
were executed by the high-performance computing cluster at Ho Chi Minh City
University of Education, Vietnam.

## References

  * Brabec and Krausz [2000] T. Brabec and F. Krausz, Rev. Mod. Phys. 72, 545 (2000).
  * Corkum and Krausz [2007] P. B. Corkum and F. Krausz, Nat. Phys. 3, 381 (2007).
  * Ramasesha  _et al._ [2016] K. Ramasesha, S. R. Leone, and D. M. Neumark, Annu. Rev. Phys. Chem. 67, 41 (2016).
  * Nisoli  _et al._ [2017] M. Nisoli, P. Decleva, F. Calegari, A. Palacios, and F. Martín, Chem. Rev. 117, 10760 (2017).
  * Li  _et al._ [2020a] J. Li, J. Lu, A. Chew, S. Han, J. Li, Y. Wu, H. Wang, S. Ghimire, and Z. Chang, Nat. Commun. 11, 2748 (2020a).
  * Itatani  _et al._ [2004] J. Itatani, J. Levesque, D. Zeidler, H. Niikura, H. Pépin, J.-C. Kieffer, P. B. Corkum, and D. M. Villeneuve, Nature 432, 867 (2004).
  * Haessler  _et al._ [2010] S. Haessler, J. Caillat, W. Boutu, C. Giovanetti-Teixeira, T. Ruchon, T. Auguste, Z. Diveki, P. Breger, A. Maquet, B. Carré, R. Taïeb, and P. Salières, Nat. Phys. 6, 200 (2010).
  * Lan  _et al._ [2017] P. Lan, M. Ruhmann, L. He, C. Zhai, F. Wang, X. Zhu, Q. Zhang, Y. Zhou, M. Li, M. Lein, and P. Lu, Phys. Rev. Lett. 119, 033201 (2017).
  * Kim  _et al._ [2014] K. T. Kim, D. Villeneuve, and P. Corkum, Nat. Photonics. 8, 187 (2014).
  * Peng  _et al._ [2019] P. Peng, C. Marceau, and D. M. Villeneuve, Nat. Rev. Phys. 1, 144 (2019).
  * Corkum [1993] P. B. Corkum, Phys. Rev. Lett. 71, 1994 (1993).
  * Lewenstein  _et al._ [1994] M. Lewenstein, P. Balcou, M. Y. Ivanov, A. L’Huillier, and P. B. Corkum, Phys. Rev. A 49, 2117 (1994).
  * Mairesse  _et al._ [2003] Y. Mairesse, A. de Bohan, L. J. Frasinski, H. Merdji, L. C. Dinu, P. Monchicourt, P. Breger, M. Kovačev, R. Taïeb, B. Carré, H. G. Muller, P. Agostini, and P. Salières, Science 302, 1540 (2003).
  * Niikura  _et al._ [2005] H. Niikura, D. M. Villeneuve, and P. B. Corkum, Phys. Rev. Lett. 94, 083003 (2005).
  * Ben-Tal  _et al._ [1993] N. Ben-Tal, N. Moiseyev, and A. Beswick, J. Phys. B: At. Mol. Opt. Phys. 26, 3017 (1993).
  * Neufeld  _et al._ [2019] O. Neufeld, D. Podolsky, and O. Cohen, Nat. Commun. 10, 405 (2019).
  * Frumker  _et al._ [2012] E. Frumker, N. Kajumba, J. B. Bertrand, H. J. Wörner, C. T. Hebeisen, P. Hockett, M. Spanner, S. Patchkovskii, G. G. Paulus, D. M. Villeneuve, A. Naumov, and P. B. Corkum, Phys. Rev. Lett. 109, 233904 (2012).
  * Kraus  _et al._ [2012] P. M. Kraus, A. Rupenyan, and H. J. Wörner, Phys. Rev. Lett. 109, 233903 (2012).
  * Chen  _et al._ [2013] Y. J. Chen, L. B. Fu, and J. Liu, Phys. Rev. Lett. 111, 073902 (2013).
  * Nguyen-Huynh  _et al._ [2020] K.-N. Nguyen-Huynh, C.-T. Le, N.-L. T. Phan, H. T. Nguyen, and L.-P. Tran, Comm. Phys. 30, 197 (2020).
  * Phan  _et al._ [2019] N.-L. Phan, C.-T. Le, V.-H. Hoang, and V.-H. Le, Phys. Chem. Chem. Phys. 21, 24177 (2019).
  * Nguyen  _et al._ [2022a] H. T. Nguyen, K.-N. H. Nguyen, N.-L. Phan, C.-T. Le, D. Vu, L.-P. Tran, and V.-H. Le, Phys. Rev. A 105, 023106 (2022a).
  * Nguyen  _et al._ [2022b] K.-N. H. Nguyen, N.-L. Phan, C.-T. Le, D. Vu, and V.-H. Le, Phys. Rev. A 106, 063108 (2022b).
  * Luu and Wörner [2018] T. T. Luu and H. J. Wörner, Nat. Commun. 9, 916 (2018).
  * Uzan-Narovlansky  _et al._ [2023] A. J. Uzan-Narovlansky, G. Orenstein, S. Shames, M. Even Tzur, O. Kneller, B. D. Bruner, T. Arusi-Parpar, O. Cohen, and N. Dudovich, Phys. Rev. Lett. 131, 223802 (2023).
  * Li  _et al._ [2023] S. Li, Y. Tang, L. Ortmann, B. K. Talbert, C. I. Blaga, Y. H. Lai, Z. Wang, Y. Cheng, F. Yang, A. S. Landsman, P. Agostini, and L. F. DiMauro, Nat. Commun. 14, 2603 (2023).
  * Lambert  _et al._ [2015] G. Lambert, B. Vodungbo, J. Gautier, B. Mahieu, V. Malka, S. Sebban, P. Zeitoun, J. Luning, J. Perron, A. Andreev, S. Stremoukhov, F. Ardana-Lamas, A. Dax, C. P. Hauri, A. Sardinha, and M. Fajardo, Nat. Commun. 6, 6167 (2015).
  * Vampa  _et al._ [2018] G. Vampa, T. Hammond, M. Taucer, X. Ding, X. Ropagnol, T. Ozaki, S. Delprat, M. Chaker, N. Thiré, B. Schmidt, F. Légaré, D. D. Klug, A. Y. Naumov, D. M. Villeneuve, A. Staudte, and P. B. Corkum, Nat. Photonics 12, 465 (2018).
  * Luu and Wörner [2018] T. T. Luu and H. J. Wörner, Phys. Rev. A 98, 041802 (2018).
  * Shafir  _et al._ [2009] D. Shafir, Y. Mairesse, D. Villeneuve, P. Corkum, and N. Dudovich, Nat. Phys. 5, 412 (2009).
  * Niikura  _et al._ [2010] H. Niikura, N. Dudovich, D. M. Villeneuve, and P. B. Corkum, Phys. Rev. Lett. 105, 053003 (2010).
  * Li  _et al._ [2020b] B.-Y. Li, J. Zhang, Y. Zhang, T.-M. Yan, and Y. Jiang, Phys. Rev. A 102, 063102 (2020b).
  * Trieu  _et al._ [2023] D.-A. Trieu, N.-L. Phan, Q.-H. Truong, H. T. Nguyen, C.-T. Le, D. Vu, and V.-H. Le, Phys. Rev. A 108, 023109 (2023).
  * Paulus  _et al._ [2003] G. G. Paulus, F. Lindner, H. Walther, A. Baltuška, E. Goulielmakis, M. Lezius, and F. Krausz, Phys. Rev. Lett. 91, 253004 (2003).
  * Wittmann  _et al._ [2009] T. Wittmann, B. Horvath, W. Helml, M. G. Schätzel, X. Gu, A. L. Cavalieri, G. Paulus, and R. Kienberger, Nat. Phys. 5, 357 (2009).
  * Hauri  _et al._ [2004] C. P. Hauri, W. Kornelis, F. Helbing, A. Heinrich, A. Couairon, A. Mysyrowicz, J. Biegert, and U. Keller, Appl. Phys. B 79, 673 (2004).
  * Naumov  _et al._ [2015] A. Y. Naumov, D. Villeneuve, and H. Niikura, Phys. Rev. A 91, 063421 (2015).
  * Hollinger  _et al._ [2020] R. Hollinger, D. Hoff, P. Wustelt, S. Skruszewicz, Y. Zhang, H. Kang, D. Würzler, T. Jungnickel, M. Dumergue, A. Nayak, R. Flender, L. Haizer, M. Kurucz, B. Kiss, S. Kühn, E. Cormier, C. Spielmann, G. G. Paulus, P. Tzallas, and M. Kübel, Opt. Express 28, 7314 (2020).
  * You  _et al._ [2017] Y. S. You, M. Wu, Y. Yin, A. Chew, X. Ren, S. Gholam-Mirzaei, D. A. Browne, M. Chini, Z. Chang, K. J. Schafer, M. B. Gaarde, and S. Ghimire, Opt. Lett. 42, 1816 (2017).
  * Trieu  _et al._ [2024] D.-A. Trieu, T.-T. D. Nguyen, T.-D. D. Nguyen, T. Tran, V.-H. Le, and N.-L. Phan, “Analytically controlling laser-induced electron phase in sub-cycle motion,” (2024), arXiv:2405.11753 [physics.optics] .
  * Bian and Bandrauk [2014] X.-B. Bian and A. D. Bandrauk, Phys. Rev. Lett. 113, 193901 (2014).
  * Zhou  _et al._ [1996] J. Zhou, J. Peatross, M. M. Murnane, H. C. Kapteyn, and I. P. Christov, Phys. Rev. Lett. 76, 752 (1996).
  * Graml  _et al._ [2023] M. Graml, M. Nitsch, A. Seith, F. Evers, and J. Wilhelm, Phys. Rev. B 107, 054305 (2023).
  * Haworth  _et al._ [2007] C. Haworth, L. Chipperfield, J. Robinson, P. Knight, J. Marangos, and J. Tisch, Nat. Phys. 3, 52 (2007).
  * Schmid  _et al._ [2021] C. P. Schmid, L. Weigl, P. Grössing, V. Junk, C. Gorini, S. Schlauderer, S. Ito, M. Meierhofer, N. Hofmann, D. Afanasiev, J. Crewse, K. A. Kokh, O. E. Tereshchenko, J. Güdde, F. Evers, J. Wilhelm, K. Richter, U. Höfer, and R. Huber, Nature 593, 385 (2021).
  * Schubert  _et al._ [2014] O. Schubert, M. Hohenleutner, F. Langer, B. Urbanek, C. Lange, U. Huttner, D. Golde, T. Meier, M. Kira, S. W. Koch, and R. Huber, Nat. Photonics 8, 119 (2014).
  * Mandal  _et al._ [2022] A. Mandal, J. M. Rost, T. Pfeifer, and K. P. Singh, Opt. Express 30, 45020 (2022).
  * Shafir  _et al._ [2012] D. Shafir, H. Soifer, B. D. Bruner, M. Dagan, Y. Mairesse, S. Patchkovskii, M. Y. Ivanov, O. Smirnova, and N. Dudovich, Nature 485, 343 (2012).
  * Silaev  _et al._ [2022] A. Silaev, A. Romanov, and N. Vvedenskii, Journal of Physics: Conference Series 2249, 012004 (2022).
  * sup [2024] See Supplemental Materials for Numerical methods for HHG calculations; Analytical analysis for oriented planar molecule H${}_{3}^{2+}$ (2024).
  * Kong  _et al._ [2022] X.-S. Kong, X.-Y. Wu, L. Geng, and W.-D. Yu, Front. Phys. 10 (2022).
  * Imasaka  _et al._ [2022] K. Imasaka, Y. Shinohara, T. Kaji, K. Kaneshima, N. Ishii, J. Itatani, K. L. Ishikawa, and S. Ashihara, Opt. Continuum 1, 1232 (2022).


# Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement
Learning

Jacob Wiebe1 Ranwa Al Mallah1&Li Li2 1Royal Military College of Canada
2Defence Research and Development Canada
{jacob.wiebe<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent advancements in deep learning techniques have opened new possibilities
for designing solutions for autonomous cyber defence. Teams of intelligent
agents in computer network defence roles may reveal promising avenues to
safeguard cyber and kinetic assets. In a simulated game environment, agents
are evaluated on their ability to jointly mitigate attacker activity in host-
based defence scenarios. Defender systems are evaluated against heuristic
attackers with the goals of compromising network confidentiality, integrity,
and availability. Value-based Independent Learning and Centralized Training
Decentralized Execution (CTDE) cooperative Multi-Agent Reinforcement Learning
(MARL) methods are compared revealing that both approaches outperform a simple
multi-agent heuristic defender. This work demonstrates the ability of
cooperative MARL to learn effective cyber defence tactics against varied
threats.

##  1 Introduction

The pace of technological advancements in modern network infrastructure has
levied demand for highly skilled and experienced cybersecurity professionals
in roles such as penetration testing, threat hunting, and incident response.
Meanwhile, machine learning and deep learning have risen as dominant
technologies in the space of complex problem-solving. The emerging field of
autonomous cyber operations has sought to harness the potential of
Reinforcement Learning (RL), deep learning, and multi-agent systems to model
the complexity of the cyber battlespace. Autonomous agents operating in cyber
defence environments enable rapid tactical-level response to threats given
complex observational inputs Vyas et al. (2023).

Cyber defence analysts typically employ a chain of decisions leading them from
the first discovery of a threat to incident response and mitigation. Human
experts are the best tool available for this role. However, human ability in
cyber defence tasks faces the challenges of, among others, attention
allocation, cognitive load, lack of measurable impact, and reaction time
Gutzwiller et al. (2015). Furthermore, cyber defence is characterized by a
large search space of computer network features with which vulnerabilities may
be exposed and exploited. To provide coverage over this space, teams of
analysts typically coordinate actions. Autonomous RL agents can derive
tactical policies in concert to achieve a coordinated effect, mitigating some
of the limitations of human actors. This multi-agent approach has yet to be
shown in a complex network defence setting.

This work demonstrates the applicability of cooperative Multi-Agent
Reinforcement Learning (MARL) for tactical-level decision-making in cyber
defence. The results of this work show that cooperative MARL can learn to
defend a simulated network environment against three heuristic attack patterns
with an emphasis on host-based identification and mitigation of lateral
movement.

There are numerous advantages to a cooperative MARL approach over single-agent
RL for ACD: (1) input-output spaces can be constrained, avoiding the curse of
dimensionality while enabling a high resolution representation of the
environment, (2) agents can learn specialized roles relating to a specific
function or area of a network, and (3) segregating agents into local areas of
control can allow for improved robustness (e.g., if a particular agent is
compromised an attacker cannot leak observations from other agents/network
segments).

This work makes the following contributions:

  1. 1.

To the best of our knowledge, CyMARL (Cyber MARL) is the first cooperative
MARL training environment for enterprise network defence tasks. It extends the
CybORG simulator with additional game types, actions, and network topologies
and it provides a PyMARL environment interface with which it is possible to
train tens of open-source algorithms.

  2. 2.

A comparison of cooperative MARL approaches to learn independent and
centrally-learned policies in a ACD context.

  3. 3.

A demonstration of the adaptability of cooperative MARL training when
encountering multiple attacker types and action sets.

The remainder of this paper is organized as follows. Section 2 reviews related
work in autonomous and multi-agent applications for cyber defence and
advancements in cooperative MARL. Section 3 discusses the background of RL,
cooperative MARL, and autonomous cyber defence. Section 4 describes the
evaluation of the two cooperative MARL approaches tested. Section 6 discusses
the results. Section 7 suggests future work and concludes.

##  2 Related Work

RL for specific cybersecurity tasks has been studied in the context of DDoS
protection Xu et al. (2007), anomaly-based intrusion detection Utic and
Ramachandran (2022), and penetration testing Chowdhary et al. (2020); Hu et
al. (2020a), among other specific use cases Nguyen and Reddi (2019). RL has
been applied to a variety of decision-making tasks relating to cyber defence
Wang et al. (2022). Although ACD techniques have been evaluated using multiple
game environments and solving methods Vyas et al. (2023), to the best of our
knowledge, this work is the first to consider multi-agent coordination of
tactical decision-making using state-of-the-art RL techniques.

###  2.1 ACD Training Environments

Game environments are commonly modelled as graph-based abstractions of
computer networks. Simple defensive decision-making games can be learned using
tabular RL methods Hu et al. (2020b); Applebaum et al. (2022). In more complex
game environments, deep RL is preferred for its generalizability and
representational capacity. Methods such as DDQN Van Hasselt et al. (2016), A3C
Mnih et al. (2016), and PPOSchulman et al. (2017) have been shown to learn to
minimize attacker propagation in graph-based network games modelled as
Partially-Observable Markov Decision Processes (POMDP) Han et al. (2020);
Nyberg and Johnson (2023).

More realistic behaviours can be trained using more detailed simulated data.
Simulators such as CybORG Standen et al. (2021), CyberBattleSim Team (2021),
and FARLAND Molina-Markham et al. (2021) emphasize host-based features from
which agents learn. Host-based simulations improve task realism by providing
more possible states, represented by host processes, vulnerabilities,
sessions, and opportunities for Red and Blue agents to interact with these
features. In contrast, Yawning Titan Andrew et al. (2022) enables greater
detail for agents to observe and affect network traffic in simulation.

Foley et al. demonstrate how a hierarchical PPO algorithm can learn to adapt
to different attacker types to defend a high value server from a heuristic
attacker in a simulated environment Foley et al. (2022), winning CAGE
Challenge 2 CAGE (2022). Building from this foundation using the CybORG
simulator, Wolk et al. analyze the generalizability of various RL methods used
in CAGE Challenge 2 Wolk et al. (2022).

Emulation has been used to bridge the sim-to-real gap between RL systems that
can be trained in defence games and real network operations Li et al. (2023);
Molina-Markham et al. (2021). Emulated environments are impractical for
training agents in many research settings due to higher requirements for
compute and clock-time than comparable simulations, instead providing a
valuable validation component for agents trained in simulation. Red versus
Blue asymmetric attack-defence games have used self-play to train defenders
against RL-trained attackers Gabirondo-López et al. (2021); Kunz et al.
(2023). RL self-play and adversarial attacks on RL cyber defenders Standen et
al. (2023) are outside the scope of this research.

###  2.2 Multi-Agent Systems for ACD

Communication between heuristic defender agents has been shown to improve
their ability to protect a simulated network from DDoS attacks Kotenko (2007).
Heuristic defender agents programmed to communicate filter configurations
between teams had improved reaction time and effectiveness than those that did
not share information. MARL systems have defended a simulated network against
DDoS attacks by selectively throttling network traffic without the use of
communication Malialis and Kudenko (2015). MARL systems have successfully
performed anomaly-based intrusion detection when framed as a classification
task Servin and Kudenko (2008); Shi and He (2021).

One of the most commonly used cooperative MARL environments for research is
the StarCraft Multi-Agent Challenge (SMAC) Samvelyan et al. (2019). SMAC
provides a variety of scenarios ranging from easy to very hard in which
virtual units learn to defeat each other in a battle video game. A comparative
study of cooperative MARL algorithms across a variety of environments,
including SMAC, showed that CTDE methods generally perform better than
independent learning Papoudakis et al. (2020). SMAC tasks are assumed to have
transferability for general tactical-level decision-making as a result of
agents needing to target actions in a coordinated way toward individual
enemies in order to achieve success. The cyber defence problem is modelled
similarly, using actions to selectively affect host characteristics.

In practice, the QMIX MARL algorithm Rashid et al. (2018) has outperformed
state-of-the-art value-based and policy-based methods at a range of tasks
Papoudakis et al. (2020); Samvelyan et al. (2019). Most notably, QMIX has been
shown, with the help of specific implementation tricks, to discover optimal or
near-optimal policies on all SMAC Samvelyan et al. (2019) scenarios Hu et al.
(2021).

##  3 Background

A useful ACD agent must be able to interpret its environment and derive a
logical chain of decisions. The RL learner, an autonomous agent, takes
sequential actions given inputs from its environment. The agent’s goal is to
choose actions that maximize its cumulative reward, referred to as return,
received from the environment. To learn which series of actions will produce
favourable results, an agent must explore the environment, receive rewards,
and update its understanding. This process occurs iteratively, allowing the
agent to refine its knowledge over many steps and episodes of play. To provide
an analysis of how learning architectures compare when used to learn cyber
defence tasks, this work constrains the approaches studied to model-free,
value-based algorithms.

###  3.1 The RL Framework

In a POMDP, an agent will take action $a_{t}$ causing the environment to step
to the next state $s_{t+1}$ based on a transition probability $p$, where
$p(s_{t+1}|s_{t},a_{t})$ is the transition function. The environment generates
two outputs from the state transition: the reward $r(s,a)$ as a function of
the state and action at time $t-1$, and the observation $o(s)$ as a function
of the state at time $t$. The observation is some subset of state information
determined by the observation function. A game in this context generalizes the
POMDP and refers to an environment with a consistent set of rules in which one
or more agents interact.

An agent’s policy $\pi(a|o)$ is a mapping of actions to the observation space.
The policy, therefore, determines a series of actions for an agent to take. An
agent adjusts its policy over many episodes of a game to optimize for its
cumulative reward in an episode, its return Sutton and Barto (2018).

A value-based agent will predict the value of states or state-action pairs
using its value function and decide on an action that will maximize its
expected future return $\pi(o)=\arg\max_{a\in A}Q(o,a)$. The value function is
calculated from the expected discounted future return:

$Q(o,a)=\alpha[r+\gamma\max_{a_{t+1}}Q(o_{t+1},a_{t+1})]+(1-\alpha)Q(o,a)$ (1)

Where $\alpha$ is the learning rate and the discount rate $(\gamma\in(0,1))$
weights rewards less as the agent looks to further $k$ states. To encourage
exploration, rather than to continue to revisit known high-value states, an
$\epsilon$-greedy strategy is commonly employed which sets a probability
$\epsilon$ that an agent will randomly select an action rather than taking the
action with the highest expected reward.

With deep value-based methods, sampling from experience replay Lin (1992)
often produces more favourable results, as demonstrated by Mnih et. Al with
Deep Q-Networks Mnih et al. (2013). A replay buffer can improve sample
efficiency by exposing the agent to more learning experience without
generating each sample through actions. Recurrent Neural Networks (RNN) allow
RL models to learn long-term dependencies between data samples which is useful
for accurate predictions in partially observable environments Bakker (2001).

###  3.2 Multi-Agent Reinforcement Learning

The cooperative multi-agent game builds upon the POMDP defined previously by
allowing multiple agents to take actions and receive rewards simultaneously
within a single timestep Littman (1994); Buşoniu et al. (2010). Although each
agent, denoted by $i$, receives individual observations $o^{i}$ from the
environment, a joint reward $r$ is determined based on the state and actions
of all agents. Agents collectively seek to optimize the joint reward. Figure 1
provides a cooperative MARL representation of agent-environment interaction.
The game’s global state depends on the vector of actions
$\textbf{a}_{t}=\\{a^{1}_{t},...,a^{n}_{t}\\}$, where $i=1,2,...n$ for $n$
agents. At each timestep, the environment will output a joint reward as a
scalar and an observation vector
$\textbf{o}_{t+1}=\\{o^{1}_{t+1},...,o^{n}_{t+1}\\}$ that includes individual
observations for each agent.

Figure 1: Multi-Agent RL Agent-Environment Framework.

Simultaneous actions in a cooperative MARL setting allow for greater
exploration. Cooperative MARL also has the advantage of dividing large tasks
into manageable goals, thus handling greater task complexity. However, since
the game’s current state depends on the joint actions of all agents, an
agent’s actions can affect other agents’ observations. Since both agents are
learning and updating their policies, this interaction creates multi-agent-
non-stationarity in the environment; the optimal policy is a moving target. In
a single-agent game, the fixed probability of a stochastic transition function
can be incorporated into the learned policy, for example, with the use of a
replay buffer. However, in the multi-agent case, actions of “external agents”
affect observations in a way that cannot be explained by changes in the
observing agent’s policy Lowe et al. (2017). It can therefore be challenging
for an agent to learn a stable policy when the actions of its peers are
silently affecting its reward and observations.

####  3.2.1 Independent Q-Learning

A naive approach to cooperative multi-agent learning systems is to
decentralize prediction and control, referred to as independent learning. This
approach employs self-contained RL agents that act out separate policies based
on independent observations while seeking to maximize a joint reward. Each
agent is responsible for learning a policy from their individual observations.
A decentralized approach allows for greater scalability, at the cost of high
variance due to the unmitigated non-stationarity. Independent Q-Learning
(IQL), as implemented in this work, is equivalent to training independent DQN-
style agents111This work utilizes an implementation of IQL from PyMARL2 Hu et
al. (2021) in which each DQN-style agent uses an RNN in addition to other
implementation tricks to optimize for cooperative play. to learn to take
simultaneous actions to maximize their joint reward.

####  3.2.2 QMIX

QMIX Rashid et al. (2018) is a Centralized Training with Decentralized
Execution (CTDE) Lowe et al. (2017) algorithm that uses a central mixing
architecture to perform value decomposition Sunehag et al. (2017). With CTDE,
semi-independent agents follow separate policies and receive updates
periodically from a central learner. The central learner trains on information
from all agents and provides a learning update, allowing each agent’s policy
to condition on the policies of their peers, mitigating the non-stationarity
problem. The foundational difference between QMIX and IQL implementations in
this work is that QMIX performs learning updates centrally.

###  3.3 Autonomous Cyber Defence

ACD does not seek to replace tools that are successful in the field, such as
anomaly-based intrusion detection. It instead aims to build a tactical-level
decision-making framework to integrate with existing technologies. In this
research, the decision space is modelled as a simulation on top of a tool-
based abstraction of the network state. An RL simulation for ACD must model
the elements of the environment to the level of detail that will allow the
agent to learn tactical-level action chains when presented with a series of
alerts about the underlying network status. A multi-agent framework further
allows for the decentralization of the decision-making processes while
maintaining an input-output space that is constrained to a level that can be
learned effectively by current RL methods.

The high-level actions of the defender in this game represent the choices for
a defender to selectively understand or act against the threat. The tactical
decision-making competency of agents then can be inferred from their ability
to minimize the impact of the attack. Many thousands or millions of iterations
are often required for RL to learn reasonable policies. This problem is
exasperated by a lower sample efficiency of partially-observable environments
Papadimitriou and Tsitsiklis (1987). By simulating ACD, RL systems can be
trained over many iterations in an abstraction of the underlying computer
network environment. Simulation requires less compute than the alternative
emulated approach while still offering enough complexity to challenge current
state-of-the-art designs.

Cyber Operations Research Gym (CybORG) is a framework that provides a
simulated cyber operations environment for training and evaluating RL agents
Standen et al. (2021). CybORG simulates information technology systems related
to network security that a cybersecurity professional may use in network
defence or penetration testing. The simulated environment presented in this
work, CyMARL, adapts CybORG to create a multi-agent host-based monitoring game
for training tactical cyber defence. CyMARL includes a PyMARL222PyMARL is an
open-source MARL research project that allows algorithms to be built from
existing deep RL components such as agents and trainers. It is included as
part of the SMAC environment and paper Samvelyan et al. (2019). The repository
for PyMARL can be found at https://github.com/oxwhirl/pymarl. environment
allowing for integration with the tens of open-source cooperative MARL
algorithms built on the PyMARL framework.

##  4 Experimental Design

To demonstrate the applicability of cooperative MARL for tactical-level
decision-making for ACD, three game scenarios are evaluated against three
attacker types. CAGE Challenge 2 CAGE (2022) is a central contributor to many
aspects of this experiment, including the attacker models, network topology,
and agent actions. CyMARL extends the complexity of CAGE Challenge 2 by
employing new scenarios (with diverse objectives, attacker behaviours, and
reward functions), modified observation and action spaces, and a cooperative
multi-agent framework.

###  4.1 Simulated Network Topology

The simulated game environment is composed of nine hosts, five in a user
subnet and four in an operational subnet. Hosts are a mix of Linux and
Windows, with unique exploits for each OS. Hosts simulate processes, users,
active sessions, and network interfaces. Rewards are calculated from the
importance score of hosts. Hosts in the user subnet have a score of 0.1,
operational subnet hosts, 1, and the operational server (located in the
operational subnet), 10. Each host in a compromised state incurs a negative
reward for the defender team equal to its importance score.

A heuristic attacker agent begins each episode with an initial foothold on a
user host and moves between hosts by scanning for and exploiting vulnerable
services. The attacker can move laterally to a target host if one of its
sessions’ hosts has visibility of the target host through its network
interface. The attacker’s actions follow a hierarchy: it must first discover a
host, then a port, before attempting to move laterally on the network using an
exploit. Escalating session privilege can then be performed. The attacker’s
session cannot be removed from the initially compromised host allowing for
episodic gameplay where both sides have opportunities to influence the reward.
RL defender agents are segregated by subnet, one agent responsible for the
user subnet and the other controlling the operational subnet. An example of
the agent-network interface is shown in Figure 2.

Figure 2: Simple network diagram with defender sensors and targets.

###  4.2 Attack Patterns

A real-world attacker is often not solely motivated to establish communication
channels on a network. The heuristic attacker’s behaviour in this game is
differentiated by its goal, which can be confidentiality, integrity, or
availability. In each case, the attacker takes different actions upon
establishing a session on a host, though the heuristic for lateral movement is
the same across all scenarios. In the confidentiality scenario, the attacker’s
intent is to spy on privileged information within the network but not take
actions to modify it directly. The confidentiality attacker achieves
“compromise” for each host it is connected to at each timestep. The integrity
attacker seeks to modify files stored on a host. At each timestep that
modified data exists, a host is considered compromised. The availability
attacker will spawn a Denial of Service (DoS) malware process that represents
a consumption of CPU cycles, affecting user availability. Similarly, each host
in this scenario with a malware process running is compromised. This does not
affect the defender’s ability to monitor the victim host. Although the
attacker’s behaviour is a simplification of real attack patterns, the
difference in objective affects how the threat appears to the defender and
which actions it should take.

To compromise a host in the integrity and availability scenarios, an
additional action must be taken once a session is established on a victim host
using the exploit action. The tamper action is taken by the integrity attacker
to simulate the modification of important files. The deny action is taken by
the availability attacker to simulate the creation of a DoS process. The two-
step scoring process in the integrity and availability scenarios decouples the
observation of attacker presence on a host from the signal of a host
compromise via the negative reward.

The cumulative reward for each strategy serves to model the risk to the
network while malicious artifacts exist on network hosts. As the number of
timesteps increases, there is a greater probability that the attacker will
have collected privileged information or disrupted normal processes on the
network. Likewise, the severity of the risk is proportional to the importance
of the host that becomes compromised.

###  4.3 Observations

Host-based monitoring systems are a common and effecti tool for cyber defence
operations, particularly in situations where an attacker has made an initial
breach and is attempting to gain Command and Control (C2) channels within a
network. In CyMARL, each defender agent’s observations are modelled as an
abstraction of information that may be provided from a host-based monitoring
service. The defender receives a set of six flags at each turn for each host
(denoted by the variable $x$) within its area of control:

  1. 1.

Host $x$ was scanned last turn.

  2. 2.

An exploit was run on a Host $x$ service last turn.

  3. 3.

There are no malicious sessions on Host $x$.

  4. 4.

Host $x$ status is unknown.

  5. 5.

A malicious session on Host $x$ exists with user privileges.

  6. 6.

A malicious session on Host $x$ exists with root/admin privileges.

The flags are one-hot encoded into a vector that is assigned to an agent ID
for interface with PyMARL. These observations do not specifically alert of
malicious files or processes so the defender must learn patterns between the
information encoded within its observation space and its rewards.

###  4.4 Defender Actions

To mitigate attacker activity, the defender agents select actions to monitor,
remove suspicious processes or files, restore hosts, or gather file
information. monitor returns the observation vector and checks all hosts
within an agent’s vision for new processes that are then added to a suspicious
process list. An agent can then choose to remove suspected malicious processes
from a specific host. As a more aggressive action, an agent can re-image a
host using restore. Restoring is the only way to remove an attacker that has
escalated its privilege on a host. It will also remove any malicious processes
or files. However, the restore action has an associated penalty that scales
with the importance score of the target host.

In the integrity scenario, agents have additional actions to manage the
specific threat type. Agents can take an analyze action to determine the
security characteristics of files on a host, allowing it to identify any
malicious files. To remove the discovered files, the data_repair, is required.
In the availability scenario, the remove action can be used to target DoS
processes and simulated TCP sessions. In the second set of trials, the
misinform action is introduced which generates a service on a host that can be
misinterpreted by an attacker as vulnerable. The attacker may choose to
exploit this process which will always fail. A defender agent may choose to
perform any action on any of its connected hosts. However, some actions are
invalid (e.g., if a host is targetted with the remove action but does not have
a malicious process running) and are penalized with a reward of -0.1.

###  4.5 Heuristic Defender

To validate the performance of MARL systems, a heuristic multi-agent defender
is used as a benchmark for each game scenario. The heuristic defender’s design
assumes it has access to a tailored expert system that provides a precise
signal of malicious process creation. It relies on selected information from
the true state and does not utilize the observation space of the MARL systems.
In the confidentiality scenarios, this policy monitors all hosts within each
agent’s subnet, and restores hosts indicated by the process creation alert.
The restore action was chosen because it scored higher in testing than using
combinations of other actions. In the availability scenario, detection occurs
due to a DoS process being created, not an attacker session. In the integrity
scenarios, the heuristic defender will attempt to remove malware on a host
that it has discovered to have a tampered file with a 50% probability and
restore otherwise. In trials using the misinform action, the defender system
creates a decoy process on a random host for the first five turns and then
proceeds with its standard policy for the remainder of the episode. Since
using monitor to detect and restore to re-image affected hosts takes two
turns, the attacker has the opportunity to stay one step ahead if it continues
to exploit new hosts. Although the heuristic defender has perfect game
information, its policy is restrictive enough to allow the attacker
opportunities to overcome its defences.

###  4.6 Evaluation Metrics

For each experiment, two algorithms are evaluated: IQL and QMIX, implementing
fully-decentralized and CTDE architectures, respectively. Agents do not
directly share information with each other, but in the case of QMIX, a joint
policy representation is learned centrally before being decomposed into
separate behaviour policies. The architecture of cooperative MARL systems has
been shown to significantly affect the learning ability of models at a range
of tasks Papoudakis et al. (2020); Rashid et al. (2018). Learning ability is
the capacity of an RL model to generate a policy from its initial randomized
state. A trained model that achieves a higher evaluation score (i.e., mean
return) at a task has a higher learning ability than a lower-performing model
given the same training opportunity.

MARL systems are trained for two million timesteps. A timestep is represented
in the game as a turn in which each agent takes one action. Each trained model
is evaluated over 1000 timesteps of play without learning. The evaluation
score is the average of the mean return of five evaluation runs using
separately trained static policies for each trial. Evaluation scores are
compared using percent difference. The variance of models is expressed in the
standard deviation of the mean score. Learning speed is an implicit evaluation
criterion to mitigate the impracticality of long training times. Although
there may be greater performance gains outside of the bounds of the training
time for the model, training time is constrained to conserve computational
resources. This allows for greater ease of reproducibility of these results
and a reasonable expectation of performance given a fixed amount of model
experience.

Figure 3: Learning curves of IQL and QMIX compared to heuristic at three scenarios. Figure 4: Learning curves of IQL and QMIX compared to heuristic with the addition of the proactive misinform action. Table 1: Mean return +/- standard deviation of trained iql and qmix policies relative to the heuristic model. Scenario | IQL | QMIX | Heuristic
---|---|---|---
Confidentiality | -26.19 ±5.5 | -23.01 ±2.47 | -65.96 ±24.33
Integrity | -24.38 ±5.44 | -16.8 ±2.73 | -32.97 ±20.82
Availability | -15.94 ±3.78 | -17.5 ±4.68 | -23.11 ±18.88
Confidentiality (Misinform) | -19.76 ±4.18 | -25.07 ±7.1 | -67.64 ±24.03
Integrity (Misinform) | -18.17 ±3.86 | -16.75 ±3.48 | -31.9 ±20.73
Availability (Misinform) | -14.47 ±2.98 | -11.63 ±2.84 | -21.39 ±16.95
*The highest scores for each scenario are given in bold.

##  5 Experiments

Two architectures are trained and compared at controlling two defender agents
in three scenario types each with different attacker behaviour
(confidentiality, integrity, and availability) in two sets of experiments. The
first set uses the standard action set described in Section 4 and the second
adds the misinform action to each scenario. The addition of a proactive action
to the set of reactive actions sets the misinform experiment apart and allows
for an evaluation of how the two MARL architectures perform under varied
conditions.

To set initial conditions, a selection of hyperparameters was varied in a grid
search in which MARL systems were trained to 500,000 timesteps at the
confidentiality scenario evaluating four parameters: batch size (128, 256),
buffer size (5,000, 10,000), learning rate (0.005, 0.01), and TD-$\lambda$
trace decay (None, 0.6). For each set of hyperparameter values, three random
seeds were trained to provide a representative result. The simplest scenario
is confidentiality scenario due to the attacker’s exploit and the reward for
compromise occurring simultaneously. It is assumed that these hyperparameter
values transfer reasonably well to the other scenarios. The hyperparameter
values chosen were based on initial trials with the environment and the
findings in Hu et al. and Hessel et al. that evaluated the effect of RL
implementation designs in QMIX and DQN, respectively Hu et al. (2021); Hessel
et al. (2018).

The training curves of IQL and QMIX are shown in Figure 3 for the three
baseline scenarios and Figure 4 for the misinform scenarios. The shaded area
of each curve is the standard deviation of the average training return. The
dashed line is the mean heuristic defender score for each scenario. Both IQL
and QMIX outperform the heuristic multi-agent defender model in all scenarios.
QMIX has a slight learning speed advantage in the confidentiality and
integrity scenarios but otherwise, both architectures tend towards similar
return scores. Table 1 presents the evaluation scores of IQL, QMIX, and the
multi-agent heuristic defender at each scenario.

The option to generate decoy processes generally improved the performance of
both architectures with a 19.8% and 8.3% average improvement in score for IQL
and QMIX, respectively. However, QMIX had reduced performance in the
confidentiality scenario and a negligible difference in the integrity
scenario. The heuristic model using the misinform action at the start of each
episode performed 2.7% better on average.

##  6 Discussion

The availability and integrity scenario types employ an attacker that must
perform an additional action upon establishing a session to score points. The
attacker in these scenario types is therefore slower and will tend to score
less in a fixed-timestep game. On average the MARL systems trained at the
integrity and availability scenarios respectively scored 19.1% and 36.7%
higher than the confidentiality scenario. This effect is more pronounced in
the heuristic, scoring 51.4% and 66.7% higher in the integrity and
availability scenarios, respectively, than in the confidentiality scenario.
Moreover, the addition of the misinform action narrowed the performance gap
between IQL and QMIX from a 11.2% QMIX advantage to 0.2%. These results
suggest that there is an attractive local minimum for each scenario that both
architectures tend to settle in.

The multi-agent host-monitoring game presented does not require explicit
coordination for a defender team to be successful. Moreover, the segregation
of agents into subnets limits their ability to influence the state of other
agents. As a result, IQL is able to discover policies that score closely to
those of QMIX despite it’s disadvantages of lacking coordination and suffering
from multi-agent non-stationarity. IQL also has the advantage of scalability.
QMIX requires that Q-functions are learned centrally, thus limiting the number
of possible agents since the central learner will eventually create a
bottleneck. IQL does not require any central learning and therefore the number
of agents in a game does not pose a constraint.

##  7 Conclusion and Future Work

Tactical decision-making in cyber defence contends with an expansive problem
space necessitating expert knowledge and, in many cases, decentralization of
effort to provide essential coverage. This work presented the initial evidence
of the success of cooperative MARL to generate of decentralized, tactical-
level control policies for a variety of host-based ACD scenarios. Independent
and CTDE value-based cooperative MARL architectures can learn policies that
outperform a basic heuristic model in this game. In particular, the
performance of IQL at these tasks suggests that this approach may scale well
to larger networks with more agents. Future gains in performance are expected
to be possible both in terms of the learning ability of MARL systems and in
the realism of training games. We suggest the following areas for the future
development of cooperative MARL in the domain of ACD:

  1. 1.

Techniques to approximate greater environmental realism, including the use of
generative programs to increase the sample size of experimentation Ellis et
al. (2022); Molina-Markham et al. (2021),

  2. 2.

Games offering more sophisticated and varied threat types using self-play
Hammar and Stadler (2020) and adversarial machine learning attacks Standen et
al. (2023); Molina-Markham et al. (2021); Han et al. (2020), and

  3. 3.

Leveraging promising MARL techniques such as multi-agent policy-gradient
methods Yu et al. (2021) or hierarchical role assignment Wang et al. (2020a),
transfer learning Wang et al. (2020b), attention mechanisms Yang et al. (2020)
and transformers for value-decomposition Khan et al. (2022).

## References

  * Andrew et al. [2022] Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir.  Developing optimal causal cyber-defence agents via cyber security simulation.  arXiv preprint arXiv:2207.12355, 2022.
  * Applebaum et al. [2022] Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, Adrian Webster, et al.  Bridging automated to autonomous cyber defense: Foundational analysis of tabular q-learning.  In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security, pages 149–159, 2022.
  * Bakker [2001] Bram Bakker.  Reinforcement learning with long short-term memory.  Advances in neural information processing systems, 14, 2001.
  * Buşoniu et al. [2010] Lucian Buşoniu, Robert Babuška, and Bart De Schutter.  Multi-agent reinforcement learning: An overview.  Innovations in multi-agent systems and applications-1, pages 183–221, 2010.
  * CAGE [2022] CAGE.  Ttcp cage challenge 2.  In AAAI-22 Workshop on Artificial Intelligence for Cyber Security (AICS), 2022.
  * Chowdhary et al. [2020] Ankur Chowdhary, Dijiang Huang, Jayasurya Sevalur Mahendran, Daniel Romo, Yuli Deng, and Abdulhakim Sabur.  Autonomous security analysis and penetration testing.  In 2020 16th International Conference on Mobility, Sensing and Networking (MSN), pages 508–515. IEEE, 2020.
  * Ellis et al. [2022] Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Foerster, and Shimon Whiteson.  Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.  arXiv preprint arXiv:2212.07489, 2022.
  * Foley et al. [2022] Myles Foley, Chris Hicks, Kate Highnam, and Vasilios Mavroudis.  Autonomous network defence using reinforcement learning.  In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, pages 1252–1254, 2022.
  * Gabirondo-López et al. [2021] Jon Gabirondo-López, Jon Egana, Jose Miguel-Alonso, and Raul Orduna Urrutia.  Towards autonomous defense of sdn networks using muzero based intelligent agents.  IEEE Access, 9:107184–107199, 2021.
  * Gutzwiller et al. [2015] Robert S Gutzwiller, Sunny Fugate, Benjamin D Sawyer, and PA Hancock.  The human factors of cyber network defense.  In Proceedings of the human factors and ergonomics society annual meeting, volume 59, pages 322–326, 2015.
  * Hammar and Stadler [2020] Kim Hammar and Rolf Stadler.  Finding effective security strategies through reinforcement learning and self-play.  In 2020 16th International Conference on Network and Service Management (CNSM), pages 1–9. IEEE, 2020.
  * Han et al. [2020] Yi Han, David Hubczenko, Paul Montague, Olivier De Vel, Tamas Abraham, Benjamin IP Rubinstein, Christopher Leckie, Tansu Alpcan, and Sarah Erfani.  Adversarial reinforcement learning under partial observability in autonomous computer network defence.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020.
  * Hessel et al. [2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * Hu et al. [2020a] Zhenguo Hu, Razvan Beuran, and Yasuo Tan.  Automated penetration testing using deep reinforcement learning.  In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 2–10. IEEE, 2020.
  * Hu et al. [2020b] Zhisheng Hu, Minghui Zhu, and Peng Liu.  Adaptive cyber defense against multi-stage attacks using learning-based pomdp.  ACM Transactions on Privacy and Security (TOPS), 24(1):1–25, 2020\.
  * Hu et al. [2021] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao.  Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning.  arXiv e-prints, pages arXiv–2102, 2021.
  * Khan et al. [2022] Muhammad Junaid Khan, Syed Hammad Ahmed, and Gita Sukthankar.  Transformer-based value function decomposition for cooperative multi-agent reinforcement learning in starcraft.  In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 18, pages 113–119, 2022.
  * Kotenko [2007] Igor Kotenko.  Multi-agent modelling and simulation of cyber-attacks and cyber-defense for homeland security.  In 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, pages 614–619. IEEE, 2007.
  * Kunz et al. [2023] Thomas Kunz, Christian Fisher, La Novara-Gsell, Christopher Nguyen, Li Li, et al.  A multiagent cyberbattlesim for rl cyber operation agents.  arXiv preprint arXiv:2304.11052, 2023.
  * Li et al. [2023] Li Li, Jean-Pierre S El Rami, Adrian Taylor, James Hailing Rao, and Thomas Kunz.  Enabling a network ai gym for autonomous cyber agents.  arXiv preprint arXiv:2304.01366, 2023.
  * Lin [1992] Long-Ji Lin.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  Machine learning, 8(3):293–321, 1992.
  * Littman [1994] Michael L Littman.  Markov games as a framework for multi-agent reinforcement learning.  In Machine learning proceedings 1994, pages 157–163. Elsevier, 1994\.
  * Lowe et al. [2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  Advances in neural information processing systems, 30, 2017.
  * Malialis and Kudenko [2015] Kleanthis Malialis and Daniel Kudenko.  Distributed response to network intrusions using multiagent reinforcement learning.  Engineering Applications of Artificial Intelligence, 41:270–284, 2015.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * Molina-Markham et al. [2021] Andres Molina-Markham, Cory Miniter, Becky Powell, and Ahmad Ridley.  Network environment design for autonomous cyberdefense.  arXiv preprint arXiv:2103.07583, 2021.
  * Nguyen and Reddi [2019] Thanh Thi Nguyen and Vijay Janapa Reddi.  Deep reinforcement learning for cyber security.  IEEE Transactions on Neural Networks and Learning Systems, 2019\.
  * Nyberg and Johnson [2023] Jakob Nyberg and Pontus Johnson.  Training automated defense strategies using graph-based cyber attack simulations.  arXiv preprint arXiv:2304.11084, 2023.
  * Papadimitriou and Tsitsiklis [1987] Christos H Papadimitriou and John N Tsitsiklis.  The complexity of markov decision processes.  Mathematics of operations research, 12(3):441–450, 1987.
  * Papoudakis et al. [2020] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht.  Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks.  arXiv preprint arXiv:2006.07869, 2020.
  * Rashid et al. [2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.  In International conference on machine learning, pages 4295–4304. PMLR, 2018.
  * Samvelyan et al. [2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson.  The StarCraft Multi-Agent Challenge.  CoRR, abs/1902.04043, 2019.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * Servin and Kudenko [2008] Arturo Servin and Daniel Kudenko.  Multi-agent reinforcement learning for intrusion detection: A case study and evaluation.  In German Conference on Multiagent System Technologies, pages 159–170. Springer, 2008.
  * Shi and He [2021] Guochen Shi and Gang He.  Collaborative multi-agent reinforcement learning for intrusion detection.  In 2021 7th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), pages 245–249. IEEE, 2021.
  * Standen et al. [2021] Maxwell Standen, Martin Lucas, David Bowman, Toby J Richer, Junae Kim, and Damian Marriott.  Cyborg: A gym for the development of autonomous cyber agents.  arXiv preprint arXiv:2108.09118, 2021.
  * Standen et al. [2023] Maxwell Standen, Junae Kim, and Claudia Szabo.  Sok: Adversarial machine learning attacks and defences in multi-agent reinforcement learning.  arXiv preprint arXiv:2301.04299, 2023.
  * Sunehag et al. [2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al.  Value-decomposition networks for cooperative multi-agent learning.  arXiv preprint arXiv:1706.05296, 2017.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * Team [2021] Microsoft Defender Research Team.  Cyberbattlesim.  https://github.com/microsoft/cyberbattlesim, 2021.
  * Utic and Ramachandran [2022] Zheni Utic and Kandethody Ramachandran.  A survey of reinforcement learning in intrusion detection.  In 2022 1st International Conference on AI in Cybersecurity (ICAIC), pages 1–8. IEEE, 2022.
  * Van Hasselt et al. [2016] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * Vyas et al. [2023] Sanyam Vyas, John Hannay, Andrew Bolton, and Professor Pete Burnap.  Automated cyber defence: A review.  arXiv preprint arXiv:2303.04926, 2023.
  * Wang et al. [2020a] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang.  Rode: Learning roles to decompose multi-agent tasks.  arXiv preprint arXiv:2010.01523, 2020.
  * Wang et al. [2020b] Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao.  From few to more: Large-scale dynamic multiagent curriculum learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7293–7300, 2020.
  * Wang et al. [2022] Wenhao Wang, Dingyuanhao Sun, Feng Jiang, Xingguo Chen, and Cheng Zhu.  Research and challenges of reinforcement learning in cyber defense decision-making for intranet security.  Algorithms, 15(4):134, 2022.
  * Wolk et al. [2022] Melody Wolk, Andy Applebaum, Camron Denver, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, et al.  Beyond cage: Investigating generalization of learned autonomous network defense policies.  arXiv preprint arXiv:2211.15557, 2022.
  * Xu et al. [2007] Xin Xu, Yongqiang Sun, and Zunguo Huang.  Defending ddos attacks using hidden markov models and cooperative reinforcement learning.  In Pacific-Asia Workshop on Intelligence and Security Informatics, pages 196–207. Springer, 2007.
  * Yang et al. [2020] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang.  Qatten: A general framework for cooperative multiagent reinforcement learning.  arXiv preprint arXiv:2002.03939, 2020.
  * Yu et al. [2021] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu.  The surprising effectiveness of ppo in cooperative, multi-agent games.  arXiv preprint arXiv:2103.01955, 2021.