<EMAIL_ADDRESS>
Chacon Bauer Mühlbauer Pfleiderer

[a]Physik Department, Technische Universität München, Garching, Germany
[b]Centre for QuantumEngineering (ZQE), Technical University of Munich,
D-85748 Garching, Germany [c]Heinz Maier-Leibnitz Zentrum (MLZ), Technische
Universität München, Garching, Germany [d]Munich Center for Quantum Science
and Technology (MCQST), Technical University of Munich, D-85748 Garching,
Germany

# Optimization strategies and artefacts of time-involved small angle neutron
scattering experiments

Alfonso  Andreas  Sebastian  Christian

###### Abstract

Kinetic small-angle neutron scattering provides access to the microscopic
properties of mesoscale systems under slow, periodic perturbations. By
interlocking the phases of neutron pulse, sample modulation, and detector
signal, so-called Time-Involved Small Angle Neutron scattering Experiments
(TISANE) allow to exploit the neutron velocity spread and record data without
major sacrifice in intensity at time-scales down to micro-seconds. We review
the optimization strategies of TISANE that arise from specific aspects of the
process of data acquisition and data analysis starting from the basic
principles of operation. Typical artefacts of data recorded in TISANE due to
the choice of time-binning and neutron chopper pulse width are illustrated by
virtue of the response of the skyrmion lattice in MnSi under periodic changes
of the direction of the magnetic field stabilizing the skyrmion lattice.

###### keywords:

small angle neutron scattering

###### keywords:

skyrmion

###### keywords:

TISANE

##  1 Introduction

Small-Angle Neutron Scattering (SANS) is widely used as a probe of mesoscopic
length scales up to several hundred nanometers in disciplines as diverse as
material science, physics, chemistry, and biology [mue_RMP2019,
Muehlbauer_NIMPRA297_2016, Dewhurst_JAC49_2016, Wood_JAC51_2018,
Kohlbrecher_JAC33_2000, ISIS_SANS2D, Barker_2022_JAC_55, Glinka_1998_JAC_31].
Providing reciprocal space information, SANS is complementary to real-space
and surface-sensitive imaging techniques, e.g., scanning-tunneling, force,
transmission-electron, and optical microscopy, or even electron holography.
The growing need for information on the dynamic properties of mesoscale
structures has motivated time-resolved SANS, resulting, more recently, in the
development of so-called Time-Involved Small Angle Neutron scattering
Experiments (TISANE) [Wiedenmann_PRL97_2006, Kippling_PLA372_2008]. Targeting
the dynamic response under a periodic drive, TISANE, allow to extend the time
resolution of conventional SANS to well below sub-milliseconds at high
scattering intensities.

In the light of the rapidly growing scientific and technological interest in
mesoscale textures in quantum materials such as superconducting vortex matter,
long-wavelength magnetic modulations, or skyrmion and meron lattices, a large
number of scientific questions have recently emerged that may be addressed by
means of time-resolved SANS studies. For instance, stroboscopic SANS has been
used to study the vortex lattice dynamics in superconducting niobium
[mue_PRB2011], show-casing the determination of the elasticity moduli, as well
as vortex lattice relaxation and diffusion. Similarly, TISANE has been used to
track the unpinning of the skyrmion lattice in MnSi under periodic changes of
field direction [mue_NJP2016]. The excitation of Ni nanorod colloids under
oscillating magnetic fields allowed to identify differences of response as a
function of the amplitude and frequency of the driving field
[Bender_Nanoscale_2015]. Recent studies [Glinka_2020_JAC_53] report
utilization of TISANE for the study of hematite spindles in oscillating
magnetic fields.

In this paper we review basic aspects of time-resolved SANS, focusing on
optimization strategies of TISANE. Starting from the principles of operation,
we discuss key elements of the data acquisition and analysis. Considering the
example of skyrmion lattice kinetics in MnSi, we illustrate prominent
artefacts associated with the choice of time-binning and neutron pulse width.

##  2 Time-resolved small-angle neutron scattering

###  2.1 Time-resolved SANS experiment with a continuous beam

Conventional SANS with a continuous beam permits to resolve slow changes of
the scattering pattern as a function of time. The associated time versus
distance diagram is depicted in Fig. 1. A velocity selector (C) transmits a
quasi-continuous beam of neutrons with a wavelength $\lambda$ and a wavelength
spread $\Delta\lambda/\lambda$. Typical neutron trajectories are represented
by the solid and dashed lines where the slope corresponds to the velocity of
the neutrons, $v_{\mathrm{n}}$, and the gray shaded areas reflect the
distribution of velocities due the wavelength spread $\Delta\lambda/\lambda$.
The neutrons interact with a sample (S) which is subject to a periodic
modulation with period of modulation $T_{\mathrm{S}}$, e.g., due to a AC
magnetic field. Here, the oscillation depicted in green at the sample position
represents the periodic variation of the scattering by the sample as driven by
the external modulation.

Following the scattering process, the neutrons are recorded at a detector (D)
placed at a distance $L_{\mathrm{SD}}$ behind the sample. As a function of
time data recorded at the detector are binned with a period $T_{\mathrm{D}}$.
Neutrons that pass through the sample at a given phase of the sample
modulation contribute to the detector signal at the same phase of the
oscillation. In the idealized case, when all neutrons have the same velocity,
$v_{\mathrm{n}}$, the time dependence of the detector signal follows the time-
dependence of the sample modulation precisely, shifted by the time the
neutrons need to travel the distance $L_{\mathrm{SD}}$. Hence, in order to
collect accurate data, the sample modulation and the detector data acquisition
systems need to be synchronized such that $T_{\mathrm{D}}=T_{\mathrm{S}}$.

Figure 1: Time versus distance diagram of SANS with a continuous neutron beam.
The velocity selector (C) generates a continuous beam with wavelength
$\lambda$ and wavelength spread $\Delta\lambda/\lambda$. Passing through the
sample (S) the neutrons are recorded at the detector (D) as a function time.
The solid and dashed lines represent neutron trajectories, where the slope
corresponds to the velocity of the neutrons. A periodic perturbation at the
sample (green) depicts changes of the sample scattering intensity with period
$T_{\mathrm{S}}$. The oscillation at the detector (red) depicts changes of the
recorded signal with a period $T_{\mathrm{D}}=T_{\mathrm{S}}$. The gray shaded
area represents the distribution of neutron velocities associated with a
wavelength spread $\Delta\lambda/\lambda$, which generates a the time-smearing
of the detector signal characterized by $\Delta t$.

It is now instructive to consider typical velocity selectors used at SANS beam
lines, which generate a triangular velocity distribution
[Wagner_PhysB180_1992]. This velocity distribution results in a spread of
neutron trajectories as depicted by gray shading in Fig. 1, where the limiting
velocities may be denoted $v_{\mathrm{min}}$ and $v_{\mathrm{max}}$. For the
purpose of the discussion marked in gray shading and bounded by dashed lines
are neutrons with velocities that pass through the sample at the same point of
time of the modulation. These neutrons will arrive at the detector at
different points of time. This causes an averaging of the detector signal as a
function of time, where the time difference between the signal produced by
neutrons with speed $v_{\mathrm{n}}$ and those with velocities
$v_{\mathrm{min}}$ or $v_{\mathrm{max}}$ is given by

$\Delta
t=\frac{L_{\mathrm{SD}}}{v_{\mathrm{n}}}\left(\frac{\Delta\lambda}{\lambda}\right)$
(1)

Inserting values of typical SANS beam lines,
$L_{\mathrm{SD}}=10~{}\mathrm{m}$, $\lambda=4.5$ Å, and
$\Delta\lambda/\lambda=10\,\%$, one finds $\Delta t\approx 1.14\,\mathrm{ms}$.
Hence, in order to prevent significant averaging of the data recorded, the
sample modulation should be chosen such that $T_{\mathrm{S}}>10~{}\Delta t$,
corresponding to frequencies $f_{\mathrm{S}}<87.7~{}\mathrm{Hz}$. Reducing the
wavelength spread, $\Delta t$ may be decreased, however, at the expense of
neutron flux.

###  2.2 Time-resolved SANS experiments with a pulsed beam

To improve the resolution in time-resolved SANS, with minimal loss in
intensity, a pulsed-beam technique proposed by R. Gähler
[Kippling_PLA372_2008] known as TISANE may be used. The technique uses
interlocking of the phases for neutron pulse, sample modulation, and detector
signal binning which allows a broad band of wavelengths to be used without
degrading the time resolution.

Shown in Fig. 2 is the schematic layout of a TISANE instrument, e.g., as
implemented the SANS-1 beamline at FRM II. A velocity selector produces a
continuous neutron beam with a wavelength $\lambda$ and a wavelength spread
$\Delta\lambda/\lambda$. Next, a chopper system generates neutron pulses with
a repetition time $T_{\mathrm{C}}$. The system used at SANS-1 consists of two
disks placed at 50 mm distance from each other. Each chopper disk has Boron-
covered blades with 14 windows, each window corresponding to an opening angle
of $9.06^{\circ}$, and a set of magnetic bearings allowing chopper disk
rotational speeds up to 20000 RPM. The system allows to adjust disk rotation
speed, direction, and phase between the disks. Careful selection of these
three parameters provides access to a wide range of $T_{\mathrm{C}}$ and
allows to tune the neutron pulse width while maintaining the desired pulse
repetition time.

Figure 2: Schematic depiction of the experimental setup used for the Time-
Involved Small Angle Neutron scattering Experiment (TISANE) as implemented at
SANS-1 (FRM II). Following the velocity selector a chopper generates neutron
pulses with repetition time $T_{\mathrm{C}}$. The neutrons pass through a
sample subject to a periodic perturbation with period of oscillation
$T_{\mathrm{S}}$. Neutrons are recorded at a detector with with period of time
binning $T_{\mathrm{D}}$. The scattered neutrons are characterised by the
scattering vector $\mathbf{q}$. The chopper control electronics, the sample
modulation system, and the detector are synchronised by the master trigger
generator operating at 10 MHz.

Following the chopper, the neutrons pass through the sample which is placed at
a distance $L_{\mathrm{CS}}$ behind the second chopper disk. The sample is
subject to a periodic modulation with a period of oscillation
$T_{\mathrm{S}}$, e.g., generated by an AC magnetic field. Typical sample
thicknesses are limited to below a few millimeters in order to avoid multiple
scattering events. The neutrons are finally recorded with a period of time
binning $T_{\mathrm{D}}$ at a detector placed at a distance $L_{\mathrm{SD}}$
behind the sample. The resulting data hence form a continuous time-resolved
stream that is binned into a discrete number of time frames.

Choosing the parameters $L_{\mathrm{CS}}$, $L_{\mathrm{SD}}$,
$T_{\mathrm{C}}$, $T_{\mathrm{S}}$, and $T_{\mathrm{D}}$ appropriately, all
neutrons from different chopper pulses that arrive at the sample at the same
point of time of the sample state oscillation, i.e., at the same phase, will
reach the detector within the same phase (corresponding to the detector
frequency) irrespective of their wavelengths. This is known as the TISANE
condition. It requires that the chopper control electronics, the sample
modulation system, and the detector are synchronized at high accuracy, since a
small deviation in the period of the oscillation will cause the phase shift to
build up with increasing measurement time, averaging the scattering
intensities as result. To avoid any problems due to insufficient
synchronization, a master trigger unit is used at SANS-1 (also known as Drive
Reference Unit or DRU). Equipped with a high precision quartz oscillator, the
DRU transmits the reference signal with a base frequency of 10 MHz to the
chopper system acting as a virtual master chopper of the AC magnetic field
generator and the detector. As an alternative approach bespoke electronics,
referred to as the detector trigger generator (DTG), had been used
[Glinka_2020_JAC_53]. It allows continuous monitoring of $f_{\mathrm{S}}$ and
$f_{\mathrm{C}}$, and recalculates $f_{\mathrm{D}}$ required to satisfy the
TISANE condition during the measurement. It is also worth mentioning work,
where the instrument was used in the Time-of-Flight mode at the spallation
neutron source [Adlmann_2015_JAC_48]. There, data were acquired in the event-
mode and synchronized on an absolute time scale with the sample modulation
system. After the experiment, the data was processed using the correlation
between the neutron events and the oscillatory cycle, and the detector signal
rate was adjusted.

The TISANE condition may be illustrated in a time versus distance diagram as
shown in Fig. 3 (a). The continuous neutron beam is transformed into neutron
pulses with repetition time $T_{\mathrm{C}}$ at the chopper (C). Neutrons
passing through the sample (S), which is modulated at a period
$T_{\mathrm{S}}$, are recorded at the detector (D) with a period of time
binning $T_{\mathrm{D}}$. Neutrons that start from the center of a chopper
pulse which arrive at the sample at a given phase of the modulation will
contribute to the same phase of the detector signal irrespective of the
neutron velocity. Graphically this is depicted in Fig. 3 (a) for two neutrons
starting from the same chopper pulse with different velocities (solid lines).
In addition, neutrons from an earlier pulse (dashed line) must reach the
sample and the detector also at the correct phase. Corresponding to this
condition, the relations between $T_{\mathrm{D}}$, $T_{\mathrm{C}}$, and
$T_{\mathrm{S}}$ as determined by the distances between the chopper system,
the sample, and the detector are given by

$\displaystyle\frac{T_{\mathrm{D}}}{T_{\mathrm{S}}}$
$\displaystyle=\frac{L_{\mathrm{CS}}+L_{\mathrm{SD}}}{L_{\mathrm{CS}}}$ (2)
$\displaystyle\frac{T_{\mathrm{C}}}{T_{\mathrm{S}}}$
$\displaystyle=\frac{L_{\mathrm{CS}}+L_{\mathrm{SD}}}{L_{\mathrm{SD}}}$ (3)

Figure 3: Time versus distance diagrams of TISANE. The chopper, sample, and
detector are denoted (C), (S), and (D), respectively. Lines represent neutron
trajectories, where the slope corresponds to the velocity. A periodic
perturbation of the sample (green) will cause a periodic oscillation of the
neutron intensity at the detector (red) (a) Depiction of the TISANE condition.
Neutrons with different velocities that start from the center of a chopper
pulse reach the sample and the detector at the same phase of time dependence.
(b) Signal smearing due to a finite chopper pulse width. The gray shaded area
represents a distribution of neutrons with different velocities which arrive
at the sample at the same phase of the modulation. Neutrons starting with the
time difference $\Delta t_{\mathrm{C}}$ with respect to the center of the
pulse reach the detector with delay $\Delta t$ causing a smearing.

As presented so far, the TISANE condition assumes infinitely narrow chopper
pulses. However, the finite chopper pulse width $\Delta t_{\mathrm{C}}$
encountered in real experiments results in a distortion of the detector signal
as illustrated in Fig. 3 (b). Neutrons starting with a time difference from
the pulse center will reach the detector shifted by this difference as
weighted by $L_{\mathrm{SD}}/L_{\mathrm{CS}}$. Additional contributions
$\Delta t_{\mathrm{S}}$ may arise from the averaging of the signal due to the
finite flight time of the neutrons across the sample. In a similar manner, a
contribution $\Delta t_{\mathrm{D}}$ may be expected from the finite thickness
of the detector. The total time resolution at the detector is then given by
[Wiedenmann_PRL97_2006]

$\Delta t^{2}=\left(\Delta
t_{\mathrm{C}}\cdot\frac{L_{\mathrm{SD}}}{L_{\mathrm{CS}}}\right)^{2}+\left(\Delta
t_{\mathrm{S}}\cdot\frac{L_{\mathrm{CS}}+L_{\mathrm{SD}}}{L_{\mathrm{CS}}}\right)^{2}+\Delta
t^{2}_{\mathrm{D}}.$ (4)

For typical chopper systems the values of $\Delta t_{\mathrm{C}}$ may vary
between $50~{}\mu$s and $500~{}\mu$s. In comparison, typical sample
thicknesses of a few mm correspond to $\Delta t_{\mathrm{S}}\approx 5~{}\mu$s.
Typical diameters of the detector tubes of several mm correspond to $\Delta
t_{\mathrm{D}}\approx 10~{}\mu$s. As a result, the time resolution of TISANE
is essentially determined by the pulse width $\Delta t_{\mathrm{C}}$.

###  2.3 Advantages and limitations or TISANE

A key advantage of the TISANE technique concerns the possibility to adjust the
time resolution and optimize the beam intensity for increasing frequencies of
the sample modulation. The details require careful consideration of the
chopper duty cycle and frame overlap as discussed in this section. Assuming a
chopper system that allows to tune the pulse width independently from the
pulse repetition time, it is convenient to define the ratio between the
chopper pulse width and repetition time as the chopper duty cycle
$D_{\mathrm{C}}$

$D_{\mathrm{C}}=\frac{\Delta t_{\mathrm{C}}}{T_{\mathrm{C}}}$ (5)

Ignoring minor contributions by $\Delta t_{\mathrm{S}}$ and $\Delta
t_{\mathrm{D}}$, the time resolution of the instrument may hence be expressed
as a function of the duty cycle

$\Delta
t=D_{\mathrm{C}}T_{\mathrm{D}}=D_{\mathrm{C}}\frac{L_{\mathrm{CS}}+L_{\mathrm{SD}}}{L_{\mathrm{CS}}}T_{\mathrm{S}}.$
(6)

It is important to keep in mind that the changes of the scattering intensity
observed at the detector are characterized by the period of the detector
signal oscillation $T_{\mathrm{D}}$ rather than the period of the sample
modulation $T_{\mathrm{S}}$. Thus, for a given $T_{\mathrm{D}}$, the signal
quality may be inferred from $D_{\mathrm{C}}$. This way, the frequency range
accessible in the sample modulation is essentially limited by the maximum
rotational speed of the chopper disks (until $\Delta t_{\mathrm{S}}$ and
$\Delta t_{\mathrm{D}}$ become comparable to $\Delta t_{\mathrm{C}}$). The
approach is, however, limited by the neutron flux being proportional to the
chopper duty cycle $D_{\mathrm{C}}$. While increasing $D_{\mathrm{C}}$ past a
certain value results in a decrease of the signal contrast due to insufficient
time resolution, decreasing $D_{\mathrm{C}}$ would be on the expense of the
resolution and require significantly longer measurement times to compensate
for the reduction of intensity. For example, the simulations for the case of
the harmonically modulated sample scattering function yield the optimal
chopper duty cycle value of 11 % [Kippling_PLA372_2008].

In addition, it is also possible to maximise the intensity by exploiting the
spread of neutron velocities, since the time resolution remains unaffected as
long as the TISANE condition is satisfied. Namely, when the spread of neutron
velocity is sufficiently large, overlap between consecutive neutron pulses is
reached such that neutrons that originate in different pulses pass through the
sample at the same point of time. The number of chopper openings contributing
to the sample intensity at a given instant in time is referred to as overlap
factor $N_{\mathrm{OF}}$, given by

$N_{\mathrm{OF}}\cdot
T_{C}=\frac{L_{\mathrm{CS}}}{v_{\mathrm{min}}}-\frac{L_{\mathrm{CS}}}{v_{\mathrm{max}}}.$
(7)

For $N_{\mathrm{OF}}<1$ the flux of neutrons at the sample is not constant. In
this limit the detector signal may exhibits parasitic contributions due to,
e.g., residual Fourier terms of the chopper transmission function and the
fundamental chopper and sample frequencies.

The scattering intensity recorded at the detector may be calculated as the
product of the neutron velocity distribution function, $F$, the chopper
transmission function, $P_{\mathrm{C}}$, and the sample scattering function,
$S$, integrated over all possible neutron velocities [Kippling_PLA372_2008]

$I_{\mathrm{D}}(t,q)=I_{0}\int^{\infty}_{0}F(v)\cdot
P_{\mathrm{C}}(t_{\mathrm{C}})\cdot S(t_{\mathrm{S}},q)~{}dv,$ (8)

where the time at the chopper and the sample is corrected by the offsets
$t_{\mathrm{C}}=t-(L_{\mathrm{CS}}+L_{\mathrm{SD}})/v$ and
$t_{\mathrm{S}}=t-(L_{\mathrm{SD}})/v$, respectively. The detector signal is
here calculated neglecting variations of flight time due to the beam
divergence and due to small angle scattering.

An example for the intensities expected is depicted in Fig. 4 (a). The
distribution of neutron velocities was modelled by a triangular shape with
$\lambda=4.5\,$Å and $\Delta\lambda/\lambda=10\,\%$. The triangular chopper
pulse function with $D_{\mathrm{C}}=0.15$ was approximated by a Fourier
series. For modelling the oscillation of the scattering intensity of the
sample a harmonic motion was assumed with a frequency
$f_{\mathrm{S}}=403.7\,\mathrm{Hz}$. The distances
$L_{\mathrm{CS}}=23.9\,\mathrm{m}$ and $L_{\mathrm{SD}}=10.0\,\mathrm{m}$ were
chosen in agreement with the configuration of the experiment discussed below
and the frame overlap was $N_{\mathrm{OF}}\approx 0.65$.

Keeping the overlap factor $N_{\mathrm{OF}}$ well above unity should reduce
the presence of undesired frequencies in the detector signal, in a typical
TISANE configurations it may be achieved at the values exceeding
$N_{\mathrm{OF}}>10$ [Kippling_PLA372_2008]. If sufficiently large values of
$N_{\mathrm{OF}}$ are not accessible (e.g. for the low sample modulation and
hence chopper frequencies), parasitic signal components may be filtered out
during the process of the detector intensity averaging, calculated as a sum
over several time constants $T_{\mathrm{D}}$

$I_{\mathrm{avg}}(t)=\frac{1}{n_{\mathrm{max}}}\sum^{n_{\mathrm{max}}}_{n=0}I_{\mathrm{D}}(t+nT_{\mathrm{D}}).$
(9)

The results of such an averaging are illustrated in Fig. 4 (b). It is worth
mentioning that in a typical TISANE, data are always collected over a large
number of detector periods for the sake of intensity, which would naturally
filter out these “parasitic” signal components. For the reference considered
here, the measurement times required for the experimental data shown in Fig. 6
were 120 s, with the $f_{\mathrm{D}}=284.7075\,\mathrm{Hz}$ giving
$n_{\mathrm{max}}\approx 3.4\cdot 10^{4}$.

To summarize, the typical time resolution of TISANE is between $0.02$ ms and
$2.00$ ms. For example, SANS-1 permits studies at sample oscillation
frequencies up to $f_{\mathrm{S}}\approx 30\,\mathrm{kHz}$. The technique is
designed to probe periodic changes in the neutron scattering pattern, but does
not permit to resolve individual stochastic events. While there are no strict
limitations at lower frequencies, the TISANE advantage of time resolution
being proportional to the period of detector signal oscillation becomes less
prominent. Considering the intensity losses at the chopper system, one may
argue that conventional time-resolved SANS with a continuous beam might be
preferable for sample frequencies up to few hundreds of Hz. A precise
comparison of the techniques may be possible when comparing Eq. 1 to Eq. 6. It
would depend on various details of implementation, notably beam intensity,
desired $q$-resolution, range of available neutron wavelengths etc.

Figure 4: Effect of the low frame overlap on the detector signal. (a) Time
versus distance diagram for $N_{\mathrm{OF}}\approx 0.65$. Insufficient frame
overlap results in a strongly varying beam intensity at the sample, causing
additional components in the detector signal. (b) Averaged detector signal for
different measurement durations. The dashed black line represents the original
sample signal. The coloured lines show detector signals averaged over multiple
periods $T_{\mathrm{D}}$ as stated in the legend.

##  3 Parasitic signal contributions

###  3.1 TISANE of the skyrmion lattice (SL) motion in chiral magnets

In the following we present TISANE data recorded in a kinetic neutron
scattering study of the skyrmion lattice (SL) motion in chiral magnets.
Skyrmions are topologically non-trivial spin textures that exhibit an
exceptionally efficient coupling to spin currents, notably spin-polarized
charge currents and magnon currents [Schulz_NatPhys8_2021,
Everschor_PRB86_2012, Mochizuki_NatMat13_2014, Zhang_NatComm9_2018]. The data
we report here follow up on an investigation carried out at the SANS beamline
V4 at HZB [mue_NJP2016]. The work reported here concerned the information
content and putative presence of parasitic signal contributions at large
excitation amplitudes, when the skyrmion lattice follows the oscillatory
motion of the field direction.

Figure 5: Basic aspects of TISANE in the skyrmion lattice of MnSi. (a) The
static magnetic field $\mathbf{B}_{\mathrm{dc}}$ and the oscillating magnetic
field $\mathbf{B}_{\mathrm{ac}}$ were generated by Helmholtz coils depicted in
gray shading. The orientation of the resulting magnetic field
$\mathbf{B}_{\mathrm{tot}}$ with respect to $\mathbf{B}_{\mathrm{dc}}$ is
denoted by the angle $\omega_{\mathrm{ac}}$. The orientation of the static
field with respect to the incident neutron beam is denoted by the rocking
angle $\omega$. (b) Typical SANS intensity pattern of the skyrmion lattice
(SL) in MnSi as stabilized by a static magnetic field
$\mathbf{B}_{\mathrm{dc}}$ (no $\mathbf{B}_{\mathrm{ac}}$ present).
Crystallographic $\langle 110\rangle$ and $\langle 100\rangle$ axes were
vertical and horizontal, respectively. Boxes 1, 2, 3, 4 denote detector
segments in which scattering intensity was observed. (c) Typical integrated
scattering intensity recorded in the boxes marked in panel (b) as a function
of $\omega$. (d) Average intensity recorded in Boxes 1 and 2. The red line
represents a Gaussian fit of the data, with $\omega_{0}$ denoting the rest
position of the skyrmion lattice in the absence of an AC field.

The experiment was carried out at the beam-line SANS-1 at FRM-II, Garching,
Germany [Muehlbauer_NIMPRA297_2016]. Unpolarized neutrons were used with a
wavelength of $\lambda=4.5\,$Å and a wavelength spread
$\Delta\lambda/\lambda\sim 10\,\%$ (FWHM). The collimation distance was set to
12 m, the sample-detector distance was $L_{\mathrm{SD}}=10.025\,\mathrm{m}$,
and the chopper-sample distance was $L_{\mathrm{CS}}=23.925\,\mathrm{m}$. The
chopper system was operated in a mode where one chopper disk was spinning
while the second chopper disk was kept in a fixed position. With each chopper
disk having 14 openings and each opening having an angular width of
$9.06^{\circ}$, this resulted in a chopper duty cycle of
$D_{\mathrm{C}}=0.352$.

A spherical single crystal of MnSi with a diameter of 5.8 mm was studied. The
spherical sample shape served to ensure uniformity of both demagnetizing and
hence internal fields [Adams_PRL107_2011]. The oscillatory motion of the
direction of the applied magnetic field was generated by means of the
superposition of crossed DC and AC fields, each produced by a set of Helmholtz
coils. The static magnetic field was $B_{\mathrm{dc}}=170\,\mathrm{mT}$. The
amplitude of the oscillating magnetic field was
$B_{\mathrm{ac}}=6.11\,\mathrm{mT}$ at a frequency
$f_{\mathrm{S}}=403.7075\,\mathrm{Hz}$. In order to account for shielding of
the AC magnetic field by the cryostat and instrument components, the magnetic
field was calibrated with a Hall probe at room temperature. The sample was
cooled to a temperature $T=28.0\,\mathrm{K}$ by means of a closed cycle
cryostat equipped with a quartz vacuum shield. Additional heating effects due
to the AC field were carefully compensated.

A schematic depiction of the experimental setup is shown in Fig. 5 (a),
featuring the direction of the incident neutron beam, the sample, the magnetic
static and oscillating magnetic fields $\mathbf{B}_{\mathrm{dc}}$ and
$\mathbf{B}_{\mathrm{ac}}$, and the detector. A static field,
$\mathbf{B}_{\mathrm{dc}}$, needed to stabilize the skyrmion lattice phase was
generated by a set of Helmholtz coils depicted in gray shading. The direction
$\mathbf{B}_{\mathrm{dc}}$ is tilted with respect to the incident neutron beam
by the rocking angle $\omega$. The oscillation of the direction of the
magnetic field was generated with a small AC field,
$\mathbf{B}_{\mathrm{ac}}$, aligned perpendicular to the dc-field. The
direction of the resulting total magnetic field $\mathbf{B}_{\mathrm{tot}}$
with respect to $\mathbf{B}_{\mathrm{dc}}$ is characterized by the angle
$\omega_{\mathrm{ac}}$. The sample was aligned in a way that the vertical axis
of the system was parallel to a crystallographic $\langle 110\rangle$
direction, and the direction of the dc-field $\mathbf{B}_{\mathrm{dc}}$
coincided with another $\langle 110\rangle$ crystallographic direction.

Shown in Fig. 5 (b) is a typical six-fold intensity pattern of the skyrmion
lattice in the absence of the AC field [Muehlbauer_Sci323_2009]. The pattern
was recorded at $\omega=-0.4^{\circ}$ corresponding to the zero position. The
diffuse intensity contributions in the background are visible on a logarithmic
scale only; they are very weak and may be ignored in what follows. For the
TISANE four peaks were selected as marked by the boxes denoted 1 through 4.
The integrated intensity in these boxes as a function of rocking angle is
shown in Fig. 5 (c). The two remaining peaks at the top and the bottom are not
tracked as they do not depend on $\omega$.

Comparison of the intensities in the four boxes underscores the sensitivity of
the scattering intensity to the precise orientation between the sample,
neutron beam and the magnetic field. In case of a perfectly aligned system,
Figure 5 (c) should exhibit symmetry around $\omega=0$: curves for boxes 1 and
4 and boxes 2 and 3 should cross at $\omega=0$, and respective curves for
boxes 1 and 2 and boxes 3 and 4 should overlap. To compensate for the small
misalignment, we take the sum of the intensity of box 1 and box 2 as a
function of $\omega$ as shown in Fig. 5 (d). The resulting curve may be
described well with a Gaussian featuring a full width at half maximum
$\mathrm{FWHM}=0.68^{\circ}$. The shape and the width of the curve contained
the information about the mosaicity of the SL structure. The area under the
curve is proportional to the volume fraction of SL phase. The position of the
peak center $\omega_{0}=-1.27^{\circ}$ represents the orientation of the SL in
the absence of AC magnetic field. Tracking deviations of the peak position
permitted to track changes of the SL direction.

Figure 6: Acquisition process of time-resolved rocking scans. (a) Sum of the
intensities in box 1 and box 2 in cf. Fig. 5) as a function of time $t/T$ for
fixed positions of rocking angle $\omega-\omega_{0}$ as denoted on the right
hand side. Data have been shifted vertically for clarity. (b) Two-dimensional
time-resolved rocking map inferred from panel (a). Horizontal lines correspond
to specific rocking angles where the color coding is that of panel (a). (c)
Transposed two-dimensional intensity map of the map shown in panel (b).
Horizontal lines denote time frames. (d) Rocking peaks for different time
frames. The dashed line represents the variation of the field direction
$\omega_{\mathrm{ac}}$. The arrows mark the value of $\omega_{\mathrm{ac}}$
for each respective time bin. Data are shifted vertically for clarity.

To determine time-dependent rocking scans we recorded at first the scattering
intensity for selected fixed rocking angles $\omega-\omega_{0}$ as a function
of time. Data were then sorted into a number of intervals according to
different points of time during a full period of the oscillation. Note that
the data were recorded at the detector with the period of oscillation
$T_{\mathrm{D}}$, by plotting the intensity as a function $t/T_{\mathrm{D}}$
it is possible to relate the intensity variations to the modulation of the
sample state characterized by $t_{\mathrm{S}}/T_{\mathrm{S}}$, therefore we
omit the notations and write it as $t/T$. The sum of the intensities of box 1
and 2 (see Fig. 5 (b)) as a function of time is shown in Fig. 6 (a). The color
shading denotes different rocking angles $\omega-\omega_{0}$. Based on these
data the two-dimensional time-resolved rocking map was generated shown in Fig.
6 (b). Here horizontal lines correspond to specific rocking angles where the
colours correspond to Fig. 6 (a).

In order to extract the rocking curves for a sequence of different time
frames, the two-dimensional rocking map was transposed as shown in Fig. 6 (c).
This way data are depicted as a function of $t/T$. The horizontal lines mark
time frames rather than rocking angles. The resulting rocking curves are
depicted in Fig. 6 (d) where the coloured lines denote the time frames marked
in Fig. 6 (c). Taken together these time-resolved TISANE rocking curves are
the result of a comprehensive data set, noting that a correct arrangement of
time and rocking angles is essential.

At first sight Fig. 6 (d) appears to show that the SL follows the direction of
$B_{\mathrm{tot}}$, where the instantaneous field direction is marked by an
arrow. In comparison to a the Gaussian rocking curve observed without
oscillation of the field direction shown in Fig. 5 (d), the intensities are,
however, heavily broadened seemingly shifting between two prominent maxima. As
discussed in the following, both the qualitative and quantitative form of the
TISANE rocking curves are purely parasitic effects that originate in the
specific choice of parameters of data binning and pulse length.

###  3.2 Consequences of data binning and pulse length

To gain insight on the origin of the TISANE rocking curves shown in Fig. 6 (d)
different basic responses of the SL motion were calculated assuming that the
whole SL volume simultaneously and instantly followed changes of the magnetic
field direction as described by $\omega_{\mathrm{ac}}$ (Fig. 7). In addition
it was assumed that the SL order is characterized by a Gaussian distribution
that is not affected by the motion. Quantitative values of the Gaussian
rocking intensity as well as the size of $\mathbf{B}_{\mathrm{ac}}$
corresponded to those observed in the study of MnSi reported in Fig. 5 and
Fig. 6.

It proved to be instructive to consider at first three fundamental oscillatory
time-dependences, namely a sinusoidal, a triangular, and a square motion,
depicted in the top, middle and bottom row of Fig. 7, respectively. As shown
in the panels on the left hand side of Fig. 7, without binning the data as a
function of time, the Gaussian distributions follow any changes of field
direction instantly without change of width or intensity. In comparison,
panels on the right hand side of Fig. 7 depict TISANE rocking scans, when
binning the time dependence into ten time frames. Within each time frame,
$t_{i}$, all scattering data were thereby integrated from $t_{i}$ to
$t_{i+1}$:

$I(\omega-\omega_{0})|_{t=t_{i}}=\frac{1}{t_{i+1}-t_{i}}\int^{t_{i+1}}_{t_{i}}I(\omega-\omega_{0},\tau)~{}d\tau$
(10)

Figure 7: Changes of simulated TISANE rocking intensities due to binning in
time frames. Data have been shifted vertically for clarity. Panels on the left
hand side assume a Gaussian intensity distribution that follows accurately a
given periodic time dependence without binning. Panels on the right hand side
display the effects of time smearing when binning data in ten time frames.
Color shading depicts specific time frames. (a, b) Behaviour for a sinusoidal
oscillation. The binning results in a broadening and concomitant reduction of
peak height. (c, d) Behaviour for a triangular motion. The binning essentially
results in the same broadening and peak reduction for all time frames. (e, f)
Behaviour for a rectangular oscillation. A double peak emerges for time frames
of fast changes.

As evident in the panels on the right hand side of Fig. 7, the binning
generates an averaging and concomitant smearing. For the sinusoidal motion,
shown in Fig. 7 (b), both the peak height and peak width vary periodically as
a function time, where the height and width scale with the rate of change of
the field orientation. Accordingly, for the triangular time-dependence shown
in Fig. 7 (d) the smearing is essentially the same for all time frames, as the
absolute value of the rate of change of the field direction is essentially
constant. Contributions by points of time where changes of the direction of
motion take place are averaged out. The rectangular motion, finally, exhibits
a broadening and a double peak structure in the vicinity of the points of time
of instant changes of field orientation only as shown in Fig. 7 (f). While the
binning into time-frames does not affect the appearance of the TISANE rocking
scans radically, the effect is inevitable and important to keep in mind.
Indeed, the impact of the binning may be reduced by increasing the number of
time frames, however, at a significant cost of intensity for each time frame.
It seems logical to select the number of time bins $N_{\mathrm{bin}}$ at least
high enough so that the duration a single time bin
$T_{\mathrm{D}}/N_{\mathrm{bin}}$ is less than the instrumental resolution
defined in Eq. 6.

Comparing the calculated rocking scans shown in Fig. 7 (b) with the
experimental data shown in Fig. 6 (d), the general trend of a broadening that
scales with the rate of change of the field direction under sinusoidal motion
cannot be ruled out. Yet, the experimental data exhibit a pronounced double
peak structure with maxima that are essentially located at two fixed rocking
angles $\omega-\omega_{0}\approx\pm 1^{\circ}$ and scattering intensity that
appears to be shifted between these two maxima as a function of time.

Figure 8: Changes of simulated TISANE rocking intensities due to binning for
different duty cycles. Data have been shifted vertically for clarity. (a)
Gaussian rocking curves that follow a sinusoidal driving field without binning
and time smearing. (b) Gaussian rocking curves that follow a sinusoidal
oscillation taking into account a TISANE duty cycle $D_{\mathrm{C}}=0.15$ and
binning into ten time frames. A broadening and a reduction of the Gaussian
distribution appears to be present. (c) Gaussian rocking curves that follow a
sinusoidal oscillation taking into account a TISANE duty cycle
$D_{\mathrm{C}}=0.35$ and binning into ten time frames. Besides a seeming
broadening and reduction of the Gaussian distribution a double peak structure
emerges due to the binning.

It transpires that the double peak structure may be fully attributed to the
choice of neutron pulse length affecting the time resolution defined in Eq. 6.
Considering a chopper made of two disks with rectangular windows a triangular
pulse shape with duty cycles typically ranging between 0.10 and 0.35 is
obtained. In case of a chopper pulse that is too wide, details of the signal
will get heavily averaged out and the shape of the TISANE rocking scans will
become symmetric. For a detailed assessment of the influence of the pulse
shape and duty cycle on the appearance of the TISANE rocking scans, the
intensities may be estimated with Eq. 8. For the calculations, we assume the
TISANE condition is strictly followed, hence the wavelength spread does not
impact the time smearing of the detector signal. Additionally we consider the
measurement times to be long enough so that $n_{\mathrm{max}}\to\infty$ when
the averaged detector signal is calculated according to Eq. 9. With that, in
the calculations the wavelength spread can be taken as delta function, and the
TISANE rocking intensity may then be estimated as convolution of the chopper
pulse function, $P_{\mathrm{C}}$, with the sample scattering function, $S$,
namely

$I_{\mathrm{avg}}(\phi)=I_{0}\int^{2\pi}_{0}P_{\mathrm{C}}(\Delta\phi)\cdot
S(\phi+\Delta\phi)~{}d\Delta\phi.$ (11)

Shown in Fig. 8 is an evaluation of the influence of the pulse length for
chopper duty cycles $D_{\mathrm{C}}=0.15$ and 0.35. For ease of comparison
shown in Fig. 8 (a) is the behaviour without binning and a vanishingly small
value of $D_{\mathrm{C}}$. In this limit the TISANE rocking intensity follows
any changes in $\omega_{\mathrm{ac}}$ while maintaining the peak shape and
height as already shown in Fig. 7 (a). Considering now $D_{\mathrm{C}}=0.15$
the combined effect of binning and finite pulse length causes an averaging
akin to Fig. 7 (b). Finally, for $D_{\mathrm{C}}=0.35$, the calculated TISANE
rocking curves shown in Fig. 8 (c) exhibit the double peak behaviour observed
experimentally. As the experimental data shown in Fig. 6 (d) was measured for
a chopper duty cycle $D_{\mathrm{C}}=0.352$ this provides a full account of
the double peak structure.

On an intuitive level the double-peak distribution may be understood as a
parasitic superposition of rocking curves due to the wide chopper pulse width.
Shown by the black line in Fig. 9 is the TISANE rocking curve displayed in
Fig. 8 (c) for $t/T=0.3$. This curve may be understood as a normalized sum of
Gaussian intensity profiles that originate at different times within the
pulse. The color shading here corresponds to the time frames shown in Fig. 8
(c). In other words, the pulse width effectively includes contributions
associated with several time frames weighted by the triangular pulse shape at
the respective moments of time.

Figure 9: Origin of the double peak structure in TISANE rocking scans for a
long neutron pulse corresponding to a duty cycle of $D_{\mathrm{C}}=0.35$. The
black line represents the TISANE rocking scan shown in Fig. 8 (c) for a time
frame $t/T=0.3$. Color-shaded Gaussians represent scattering contributions
scaled by the chopper pulse function as associated with different time frames
due to the long chopper pulse.

##  4 Summary

In summary, we reviewed optimization strategies of TISANE in kinetic SANS
studies of mesoscale textures. To illustrate parasitic effects we considered
the motion of the SL in MnSi under oscillations of the field direction,
focusing on the emergence of pronounced qualitative and quantitative changes
of the intensity distribution, notably strong velocity dependent broadening
and a double peak structure suggestive of periodic shifts of scattering
intensity. Simulating the effects of binning into time frames for different
duty cycles for a Gaussian rocking distribution that is driven by a
sinusoidal, triangular, and square wave excitation, we reproduce the behaviour
observed experimentally. This illustrates the potential of TISANE to obtain
time-resolved information at high scattering intensities, while emphasizing
the need for carefully simulating and choosing key parameters of data
detection and analysis.

Acknowledgements
We wish to thank A. Bezvershenko, P. Böni, A. Heinemann, A. Rosch, and U.
Keiderling for fruitful discussions and the staff at the Heinz Maier-Leibnitz
Zentrum (MLZ) for support. This work has been funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under TRR80 (From
Electronic Correlations to Functionality, Project No. 107745057, Project E1),
the priority program SPP 2137 (Skyrmionics) under grant PF393/19 (project-id
403191981), and the excellence cluster MCQST under Germany’s Excellence
Strategy EXC-2111 (Project No. 390814868). Financial support by the European
Research Council (ERC) through Advanced Grants No. 291079 (TOPFIT) and No.
788031 (ExQuiSid) is gratefully acknowledged.

[tisane]


# CyberForce: A Federated Reinforcement Learning Framework for Malware
Mitigation

Chao Feng∗1, Alberto Huertas Celdrán1, Pedro Miguel Sánchez Sánchez2, Jan
Kreischer1,
Jan von der Assen1, Gérôme Bovet3, Gregorio Martínez Pérez2, and Burkhard
Stiller1 ∗Corresponding author.This work has been partially supported by the
Swiss Federal Office for Defense Procurement (armasuisse) with the DEFENDIS
and CyberForce projects, and the University of Zürich UZH.1Chao Feng, Alberto
Huertas Celdrán, Jan Kreischer, Jan von der Assen, and Burkhard Stiller are
with the Communication Systems Group (CSG) at the Department of Informatics
(IFI), University of Zurich (UZH), 8050 Zürich, Switzerland (e-mail:[cfeng,
huertas, vonderassen<EMAIL_ADDRESS>jan.kreischer@uzh.ch).2Pedro Miguel
Sánchez Sánchez and Gregorio Martínez Pérez are with the Department of
Information and Communications Engineering, University of Murcia, 30100
Murcia, Spain ([pedromiguel.sanchez, gregorio]@um.es).3Gérôme Bovet is with
the Cyber-Defence Campus within armasuisse Science & Technology, 3602 Thun,
Switzerland (gerome.bovet@armasuisse.ch).

###### Abstract

Recent research has shown that the integration of Reinforcement Learning (RL)
with Moving Target Defense (MTD) can enhance cybersecurity in Internet-of-
Things (IoT) devices. Nevertheless, the practicality of existing work is
hindered by data privacy concerns associated with centralized data processing
in RL, and the unsatisfactory time needed to learn right MTD techniques that
are effective against a rising number of heterogeneous zero-day attacks. Thus,
this work presents CyberForce, a framework that combines Federated and
Reinforcement Learning (FRL) to collaboratively and privately learn suitable
MTD techniques for mitigating zero-day attacks. CyberForce integrates device
fingerprinting and anomaly detection to reward or penalize MTD mechanisms
chosen by an FRL-based agent. The framework has been deployed and evaluated in
a scenario consisting of ten physical devices of a real IoT platform affected
by heterogeneous malware samples. A pool of experiments has demonstrated that
CyberForce learns the MTD technique mitigating each attack faster than
existing RL-based centralized approaches. In addition, when various devices
are exposed to different attacks, CyberForce benefits from knowledge transfer,
leading to enhanced performance and reduced learning time in comparison to
recent works. Finally, different aggregation algorithms used during the agent
learning process provide CyberForce with notable robustness to malicious
attacks.

###### Index Terms:

Federated Learning, Reinforcement Learning, Moving Target Defense,
Fingerprinting.

##  I Introduction

The rapid expansion of wireless communication technologies and the emergence
of the Internet-of-Things (IoT) paradigm are leading to a substantial rise in
the quantity of internet-connected devices with restricted capabilities.
Currently, there are approximately 14 billion IoT devices, with projections
indicating a rise to 64 billion by 2025 [1], from healthcare to smart homes
scenarios. These devices enhance human life and optimize productivity while
minimizing costs.

Although utilizing IoT devices brings numerous advantages, it also introduces
specific cybersecurity concerns attributable to well-known and novel
vulnerabilities found in resource-constrained devices [2]. One of the main
concerns pertains to data privacy, as IoT devices typically gather sensitive
information from users, such as location data, personal health records, or
sensor readings. Another issue revolves around data integrity, as IoT devices
rely on data sourced from various devices and users, ensuring the integrity
and authenticity of the collected data is essential. Moreover, IoT devices are
also vulnerable to various forms of attacks, including device compromise,
malware injection, and network attacks. In such circumstances, malware can be
distributed through compromised devices or malicious applications, potentially
leading to data theft, unauthorized access, or control over the device [3].

Addressing all these aspects requires cybersecurity approaches that encompass
secure communication protocols, data encryption, authentication mechanisms,
intrusion detection systems, and continuous monitoring. Unfortunately, the
complexity and resources consumed by these mechanisms, combined with zero-day
vulnerabilities and attacks affecting resource-constrained devices, make them
unsuitable for IoT platforms. Therefore, assuming the impracticality of
perfect security, a pioneering cybersecurity paradigm known as Moving Target
Defense (MTD) was introduced in 2009 [4]. MTD aims to counteract adversaries
by proactively or reactively altering specific system parameters such as IP
addresses, file extensions, or system libraries to impede and safeguard
against attacks [5].

Proactive and reactive MTD approaches present important challenges that must
be tackled. This work focuses on reactive approaches, where reinforcement
learning (RL) has demonstrated its efficacy in learning which MTD mechanisms
are able to mitigate heterogeneous zero-day attacks [6]. More in detail, for
each device and attack, an RL-based agent acquires knowledge through trial and
error, determining the most effective MTD mechanism based on discrepancies in
the IoT device behavior before and after deploying each MTD mechanism.
However, existing solutions require a remarkable amount of time to learn the
right MTD per attack, and it is not scalable when several devices are under
attack, making them unfeasible for many real IoT scenarios. The scalability
issue is becoming even worse with the increasing number of zero-day attacks
being faced by resource-constrained devices. In addition, existing RL-based
solutions follow the traditional Machine Learning (ML) pipeline, where data is
pooled in a central server, thus bringing concerns about data privacy when
dealing with collaborative learning. These limitations could be tackled by
Federated Learning (FL), where devices could collaborate to learn the MTD
mechanisms mitigating each attack in a privacy-preserving manner. However,
there is no work or solution that effectively integrates Federated and
Reinforcement learning for the purpose of countering zero-day attacks with the
utilization of MTD techniques. Therefore, critical performance and scalability
aspects such as MTD selection accuracy, learning time, transfer of knowledge
among IoT devices, and robustness against malicious attacks have not been
compared to centralized RL-based approaches.

To address the previous challenges, the main contributions of the present work
are:

  * •

The design and implementation of CyberForce (source code publicly available in
[7]), a novel cybersecurity framework that combines Federated and
Reinforcement Learning (FRL) to learn optimal MTD techniques mitigating
heterogeneous zero-day attacks in a collaborative and privacy-preserving
fashion. CyberForce considers behavioral fingerprinting and ML-based anomaly
detection to reward the decisions of an FRL-based agent that uses Deep-Q
Learning to learn effective MTD techniques per attack behavior. By using
CyberForce, resource-constrained devices simultaneously train their models and
send only their parameters to the server for aggregation. Therefore,
CyberForce preserves data privacy while optimizing training time by enabling
knowledge sharing among devices.

  * •

The deployment of CyberForce on ten Raspberry Pis 4 acting as resource-
constrained spectrum sensors of a real-world IoT platform called Electrosense
[8]. Each device has been affected by six different malware attacks belonging
to ransomware, Command and Control (C&C), and rootkit families. Then, four
existing MTD mechanisms have been considered to mitigate the previous malware.

  * •

A pool of experiments were carried out to evaluate the effectiveness and
learning time of the CyberForce framework while mitigating the impact of
previous malware attacks on the federation of ElectroSense sensors. These
experiments covered different patterns of data distribution, ranging from
Independent and Identically Distributed (IID) to non-IID, to thoroughly assess
the adaptability and efficiency of CyberForce in various cybersecurity
scenarios.

  * •

A pool of experiments evaluating the robustness of CyberForce when data
poisoning and model poisoning attacks affect the FRL-based agent. In this
context, different aggregation algorithms are used by CyberForce to mitigate
the previous attacks. In conclusion, when the framework suffers from model
poisoning attacks but with IID environment, the Krum aggregation function
enhances the robustness of the models. When CyberForces faces non-IID data,
but in a secure environment, FedAvg optimizes performance, being the best
alternative.

The remainder of this article is structured as follows. Section II gives an
overview of approaches that leverage RL, MTD, and FL. The design of CyberForce
is introduced in Section III, and the experiments using its implementation are
presented in Section IV. Finally, Section V gives an overview of the
conclusions and future research directions.

##  II Related Work

This section reviews existing work focused on providing cybersecurity against
a plethora of threats. TABLE I shows and compares how RL, MTD, and FL have
been considered by recent research to improve the security of different
devices.

In the IoT domain, [9] employed FL for anomaly detection and conducted
experiments to demonstrate its superiority over traditional centralized ML
methods, primarily due to its incorporation of data privacy protection.
However, this solution did not consider malware mitigation and the usage of
RL, as the work at hand does. Another approach that leverages FL is [10],
which advises how to combine FL with RL to collaboratively learn an optimal
control policy. The collaboration amongst devices proved to accelerate the
learning process, mitigate training instability and increase generalization.
The main different with the paper at hand is that in the previous work,
threats and MTD techniques were not considered. Similar to the work proposed
in [10], [11] provided evidence of optimizing defense with RL, however, the
MTD paradigm is not supported.

TABLE I: Defense Approaches Leveraging RL, MTD, or FL

Solution | Scenario | Device | Threat | Env. | RL | MTD | FL | RA
---|---|---|---|---|---|---|---|---
[9] 2022 | Network Security | IoT | Botnets | R | ✗ | ✗ | ✓ | ✗
[10] 2020 | Optimal Control | IoT | None | R | ✓ | ✗ | ✓ | ✗
[11] 2021 | IT Security | IoT | DDoS, Spoof | S | ✓ | ✗ | ✗ | ✗
[12] 2013 | Network Security | Servers | DDoS | S | ✗ | ✓ | ✗ | ✗
[13] 2019 | Policy Planning | Servers | Probing | S | ✓ | ✓ | ✗ | ✗
[14] 2020 | Web Security | Servers | Various | S | ✓ | ✓ | ✗ | ✗
[15] 2021 | Intrusion Prevention | Network | DoS, Scan | H | ✓ | ✓ | ✗ | ✗
[16] 2018 | IT Security | Network | DDoS, Botnet | R | ✓ | ✓ | ✗ | ✗
[17] 2022 | Routing | SDN | Eavesdropping | R | ✓ | ✓ | ✗ | ✗
[18] 2023 | System Security | CPS | From NVD | R | ✓ | ✓ | ✗ | ✗
[19] 2021 | Network Security | CPS | DDoS | S | ✓ | ✓ | ✗ | ✗
[20] 2023 | IoV | IoV | DDoS | S | ✓ | ✓ | ✗ | ✗
[21] 2021 | Network Security | IoV | Various | H | ✓ | ✓ | ✗ | ✗
[6] 2022 | System Security | IoT | Malware | H | ✓ | ✓ | ✗ | ✗
This work | Crowdsensing | IoT | Malware | R | ✓ | ✓ | ✓ | ✓

Robustness Analysis (RA), Real-World (R), Simulated (S), Hybrid (H)

Looking at the combination of RL and MTD, several approaches can be found in
the literature that were implemented and validated in real or simulated
environments. [12, 13, 14] deployed MTD mechanisms on servers to mitigate
various threats such as Distributed Denial-of-Service (DDoS) or Reconnaissance
attacks. Although solutions such as [13] successfully demonstrated that RL can
be used to find the optimal MTD technique, all three approaches were
implemented in a simulated environment and none of them cover the
applicability of FL to further optimize their approach, as the work at hand
does. Moreover, the focus on computationally strong environments may indicate
that they might not be suitable for resource-constrained devices.

With respect to network-based approaches, both generic elements of the attack
surface (e.g., IP addresses, TCP source ports) and Software-defined Networking
(SDN) parameters are exploited to counter an array of threats, including
Botnets, DDoS, and Reconnaissance attacks. [15, 16, 17] all combine RL and MTD
to mitigate the threat vectors described. The validation of their techniques
was conducted in real-world environments. However, none of them considered the
effects of a federated setting with respect to their defense model.

In the cyber-physical systems (CPS) domain, [18] demonstrated that an MTD
framework with RL can be used to pre-train policies by using simulated
environments. More in detail, it was shown that MTD can be optimized to defend
against unknown attacks. [19] proposed an RL-based mobile MTD strategy capable
of balancing system security and system performance. The goal of the defender
is to thwart DDoS attacks by launching a network shuffling MTD before the
attacker completes the reconnaissance phase. Deep-Q learning was used to
optimize and adapt to the evolving strategy of attackers. Experiments in a
simulated environment have shown that this allows the defender to find a
balance between security and performance. Similar approaches were followed by
[20] and [21], using both simulated and real-world Internet-of-Vehicles (IoV)
environments for evaluation. Although the above mentioned approaches
demonstrate that RL-based MTD methods are effective in mitigating the impact
of cyberattacks in multiple scenarios, none of them take into account the
problem of data privacy. Finally, [6] combined RL and existing MTD mechanisms
to optimize the deployment of the techniques for zero-day attacks. An online
RL agent was trained in a real-world scenario, resulting in a realistic
validation scenario that was able to mitigate multiple samples from a wide
range of malware families (e.g., rootkits, ransomware, data extortion, or
botnets). Concerning the learning environment, only an isolated device and
agent were considered. Hence, no federated approaches were considered.

In summary, related approaches can be divided into three main categories. The
first comprises works dealing with either MTD, RL, or FL in isolation. Only a
few papers combine RL and FL to form the second category. Finally, numerous
publications using RL for MTD deployment or optimization can be identified,
constituting the third category. However, there is a gap in these studies
regarding the consideration of data privacy. Additionally, none of the
existing work includes a thorough analysis of the robustness of their proposed
approach. Thus, there is an opportunity to combine RL and FL for the
deployment of MTD and to improve existing limitations. By exchanging the
learned knowledge, devices that have not seen a specific attack can profit
from behavioral learning. Furthermore, such a collaboration could reduce the
overall training time, save resources for the devices, and preserve the
privacy and security of the data.

##  III CyberForce Framework

This section introduces CyberForce, an FRL-based framework that learns optimal
MTD mechanisms to mitigate unseen malware in a collaborative and privacy-
preserving fashion [7]. Figure 1 shows the main elements and lifecycle of
CyberForce. The main actor is the Federated Agent, which combines FL and RL to
learn effective MTD techniques mitigating heterogeneous zero-day attacks
affecting various IoT devices. Each device of the federation hosts a Local
Agent that uses Deep Q-Learning to learn the right MTD action according to the
state of the device (called Environment) and a Reward mechanism that measures
the impact of the MTD action on the environment.

Figure 1: CyberForce Framework Overview

###  III-A Agent and Federated Learning Process

The FRL-based agent is the main novelty and contribution of this work. The
agent learns MTD actions to mitigate attacks in an online fashion by using the
Deep Q-learning algorithm and a federated neural network. First, the federated
agent selects and distributes an initial neural network (see Figure 1 and
TABLE V for hyperparameters configuration) among the devices of the
federation. Then, in each client, the local agent trains its local neural
network by interacting with the environment or client (step 2 in Figure 1). In
this interaction, each local agent takes an MTD action for a given state.
States are explained below and represent the agent’s vision of the client at a
given time. Then the new state (affected by the MTD action) is evaluated by a
reward mechanism focused on anomaly detection (explained below), and the
output is fed to the local agent. After that, the local agent selects the next
MTD action based on this new information, and this loop is repeated. Sequences
of the previous steps are called episodes. After a given number of episodes,
the weights of the neural networks of each local agent are shared with the
Federated Agent (step 3 in Figure 1), which aggregates them to create a global
model (step 4 in Figure 1). CyberForce provides several algorithms, as shown
in TABLE II, that are used for local model aggregation, including FedAvg [22],
Krum [23], and Trimmed Mean [24]. After the aggregation, the global model is
subsequently transmitted to each client, thereby replacing their respective
local models. The previous steps are repeated for a given number of rounds
until the federated neural network converges.

Mathematically, the goal of the FRL agent is to maximize the expected
cumulative discounted future rewards for all clients, as shown in equation 1.

$G_{t}=R_{t+1}+\gamma
R_{t+2}+\gamma^{2}R_{t+3}+...=\sum_{k=0}^{\infty}R_{t+k+1}$ (1)

Where, $R_{t}$ denotes the reward at time step t, and $\gamma$ corresponds to
a discount factor. To maximize this expected return $G_{t}$, the federated
agent needs to learn a policy. Deep FRL is a promising approach, as it
approximates the action-value function via a deep neural network. Therefore,
Federated Deep Q-Learning is an adequate choice as it utilizes Temporal
Difference and accounts for large state spaces.

TABLE II: Aggregation Algorithms for FRL Framework Aggregation |  Description
---|---
FedAvg |  It averages the parameters of all the local models to yield the aggregated global model, with using $w=\sum_{k=1}^{K}\frac{1}{K}w_{k}$, where $w$ represents the weights of the updated global model, $w_{k}$ denotes the parameter of each local model, and $K$ signifies the number of clients.
Krum |  It chooses the global model by identifying the local model with the highest similarity to the rest of the local models. The similarity is determined by calculating the inverse of Euclidean distance with: $d=\sqrt{(w_{i}-w_{j})^{2}}$, where $w_{i}$ and $w_{j}$ denote the parameter of two local models.
| Trimmmed
---
Mean
It eliminates the outliers from the parameters of local models and
subsequently calculates the average of the remaining values to get the global
model.

###  III-B Environment & State

The environment consists of a set of devices affected by zero-day malware
attacks. Particularly, this work considers ten Raspberry Pi 4 devices running
an ElectroSense sensor [8]. ElectroSense is a publicly accessible and open-
source IoT crowdsensing platform that collects data on the radio frequency
spectrum worldwide. In such a scenario, six types of malware, originating from
the following three different families, have been identified as harmful for
crowdsensing devices: C&C, rootkits, and ransomware. TABLE III summarizes the
main aspects of the malware samples, more information can be found in [25].

TABLE III: Behaviors of the Malware Affecting IoT Devices Malware | Family |  Description
---|---|---
The Tick | C&C |  It controls bots from a remote location through a server utilizing a remote shell and retrieving data from targeted devices
Jakoritar | C&C |  It creates client and server components to facilitate the occurrence of data leakage and remote control
Dataleak | C&C |  It deploys a shell script, which utilizes the netcat command to regularly leak confidential data from either files or commands
Beurk | rootkits |  Its features range from hiding pseudo-terminal backdoor clients, files, directories, and real-time log cleanup to concealing processes, logins, and bypassing analysis
Bdvl | rootkits |  Its functionality is immense, and it ranges from hidden backdoors that allow multiple connection methods to keylogging and stealing passwords and files
| Ransom-
---
ware_PoC
Ransomware |  Crypto-ransomware with typical functionality, except that it is not controlled by a C&C server

A state is the agent’s vision of the environment at a given time. CyberForce
uses device behavioral fingerprinting to represent environment states. In
particular, software and kernel tracepoint events are considered because they
cover an extensive range of promising events for attack detection and
representation, as identified in previous works [25]. Initially, over 100
distinct perf events were monitored, encompassing various dimensions such as
system calls, CPU operations, device drivers, scheduler operations, network
activities, file system operations, virtual memory usage, and random number
generation. The selection criterion aimed to encompass a wide range of sources
to identify minor disruptions caused by diverse zero-day attacks effectively.
It is worth noting that the states or fingerprints should possess precise and
stable characteristics over time, and the complexity of state representation
(feature dimensionality) increases with a high number of events or features.
Consequently, the learning process of the agent requires more time to
converge. To address this, all features were monitored within time windows of
5 s over 8 hours (as suggested in [25]), representing the normal behavior of
Raspberry Pis. It is important to mention that previous studies have
demonstrated the appropriateness of the chosen time window and monitoring
duration [25]. Once the dataset was collected, data distributions of all
features were thoroughly examined. Features exhibiting constant or unstable
values, as well as those with a correlation exceeding 90%, were eliminated.
Finally, a subset of 85 events was selected, as shown in Figure 2.

Figure 2: CyberForce State Events

###  III-C MTD Action

The actions undertaken in this work pertain to the implementation of MTD
techniques as a means to mitigate zero-day attacks. The primary objective of
this framework is not to propose novel MTD mechanisms, but rather to establish
a collaborative selection mechanism. Therefore, the framework considers the
MTD techniques outlined in [5]. As shown in TABLE IV, IP shuffling combats C&C
attacks by migrating the private IP address of the targeted victim. Ransomware
trap creates dummy files that are subsequently encrypted by ransomware
attacks. File randomization modifies the file format extension, concealing the
files from manipulation. Finally, Library sanitation shuffles shared system
libraries between different sets and cleans associated links.

TABLE IV: MTD Techniques for Malware Mitigation

MTD Techniques | Mitigated Malware | Malware Family
---|---|---
IP shuffling | The Tick, Jakorita, Dataleak | C&C
Ransomware trap | Ransomware_PoC | Ransomware
File randomization | Ransomware_PoC | Ransomware
Library sanitation | Beurk, Bdvl | rootkits

###  III-D Reward

The learning process of the agent is facilitated by positive and negative
rewards, which provide feedback on the efficacy of selected actions in
different states. This study proposes the utilization of an anomaly detection
(AD) system based on unsupervised ML to automate the reward generation
process. Specifically, when an attack affects a client and a particular MTD
technique is chosen by the local agent, the AD system evaluates the resulting
device behavior. If the AD system predicts a normal behavior, it means that
the deployed MTD technique effectively mitigated the attack, resulting in a
positive reward. In contrast, if the device behavior is deemed abnormal, it
indicates that the selected MTD technique was ineffective against the attack,
leading to a negative reward.

To enable this functionality, an offline process is employed to train one
Autoencoder per client using normal behavior.

The training data is collected by monitoring the previously selected events
over a period of eight days for each Raspberry Pi, which remained unaffected
by any attacks. Subsequently, the datasets undergo several tasks, including:
i) splitting into training and validation sets, ii) normalizing feature
values, and iii) eliminating outliers using the Z-score method. Following
this, individual Autoencoder models are trained for each device using 80% of
the dedicated normal data. The remaining 20% of samples are utilized to
calculate the threshold, determined by the mean predicted Mean Squared Error
(MSE) reconstruction loss plus 2.5 standard deviations. Then, in real-time, an
online process evaluates each environment state. This involves executing
malware samples on each Raspberry Pi and triggering the local agent if the AD
system detects abnormal behavior. The local agent then selects and deploys a
specific MTD technique. After giving the MTD technique two minutes to mitigate
the attack, the AD system re-evaluates the device state. If the behavior is
determined to be normal, a positive reward $(+1)$ is given to the agent. If
the behavior remains abnormal, the reward is negative $(-1)$.

##  IV Experiments

This section performs a pool of experiments to evaluate the learning process,
mitigation performance, and robustness of CyberForce when different attacks
affect various ElectroSense sensors and the federated learning process.

###  IV-A Experiment 1: Anomaly Detection for Rewards

The local RL agent is triggered when the AD determines that the current state
of the device is abnormal. Therefore, the performance of the AD has a crucial
influence on the overall effectiveness of the CyberForce framework. To
identify zero-day attacks, it suffices to train the AD on normal behavioral
patterns, enabling it to identify markedly different malware samples. Then,
the agent is alerted and provided with the relevant state sample to choose an
appropriate MTD.

TABLE V: Hyperparameter Search for the AD System

| Hyperparameter
---
Class
Type | Candidates
Model Hyperparameter | NR_NEURONS_PER_LAYER |  (64, 32), (64, 16), (64, 8)
ACTIVATION_FUNCTION | Sigmoid, Tanh, ReLU, ELU, GELU
BATCH_NORMALIZATION | False, True
Optimization Hyperparameter | LOSS_FUNCTION | MAE, MSE, RMSE
OPTIMIZER | SGD, Adam, RMSprop
LR | 1e-3, 1e-4, 1e-5
L2_REGULARIZATION | 1e-1, 1e-2, 1e-3, 1e-4
EARLY_STOPPING | False, True (patience=5)
BATCH_SIZE |  32, 64
| Prediction
---
Hyperparameter
N_STD | 1, 2, 3

This experiment performs a hyperparameter search for the AutoEncoder used as
an AD model. The complete set of hyperparameters can be seen in TABLE V, where
a GridSearch and five-fold cross-validation were performed to test all
possible hyperparameter combinations. Additionally, early stopping with a
patience of five is employed to prevent overfitting. The hyperparameter
combination used by the best-performing AD model is displayed in bold in TABLE
V, which is used during the following experiments.

TABLE VI: AD Accuracy for States and Afterstates Behavior |  Accuracy |  Target State
---|---|---
Normal |  99.54% |  Normal
Ransomware_PoC (state) |  100.00% |  Abnormal
Ransomware_PoC + Ransomware trap |  100.00% |  Normal
Ransomware_PoC + File randomization |  100.00% |  Normal
Ransomware_PoC + IP shuffling |  100.00% |  Abnormal
Ransomware_PoC + Library sanitation |  100.00% |  Abnormal
Bdvl (state) |  99.52% |  Abnormal
Bdvl + Ransomware trap |  47.69% |  Abnormal
Bdvl + File randomization |  48.29% |  Abnormal
Bdvl + IP shuffling |  59.68% |  Abnormal
Bdvl + Library sanitation |  99.51% |  Normal
Beurk (state) |  99.89% |  Abnormal
Beurk + Ransomware trap |  0.00% |  Abnormal
Beurk + File randomization |  0.10% |  Abnormal
Beurk + IP shuffling |  0.10% |  Abnormal
Beurk + Library sanitation |  100.00% |  Normal
The Tick (state) |  99.01% |  Abnormal
The Tick + Ransomware trap |  0.09% |  Abnormal
The Tick + File randomization |  0.10% |  Abnormal
The Tick + IP shuffling |  99.61% |  Normal
The Tick + Library sanitation |  0.00% |  Abnormal
Jakoritar (state) |  99.76% |  Abnormal
Jakoritar + Ransomware trap |  0.00% |  Abnormal
Jakoritar + File randomization |  0.32% |  Abnormal
Jakoritar + IP shuffling |  100.00% |  Normal
Jakoritar + Library sanitation |  0.00% |  Abnormal
Dataleak (state) |  99.56% |  Abnormal
Dataleak + Ransomware trap |  0.00% |  Abnormal
Dataleak + File randomization |  0.00% |  Abnormal
Dataleak + IP shuffling |  99.51% |  Normal
Dataleak + Library sanitation |  0.00% |  Abnormal

This experiment evaluates the performance of the AD system in two aspects: (i)
on the behavior of the device which it is affected by attacks; and (ii) when
subsequent deployed the MTD techniques, including both correct ones and
incorrect ones. If the deployment of the appropriate MTD for each attack leads
to the device returning to a normal state, it indicates that the AD system
effectively provides precise feedback to the agent. Similarly, if the AD
system detects abnormal behavior in cases where an incorrect MTD strategy is
applied to an attack, this also implies that the AD is capable of providing
accurate feedback to the agent. TABLE VI shows the performance with the best
hyperparameter combination of the AD for all states. The results present the
effectiveness of the AD in identifying anomalies when considering the current
states alone, as well as when taking into account the subsequent outcomes of
implementing MTD techniques. The first column displays the behavior of the
device and the applied MTD technique. The accuracy of detecting normal or
abnormal behavior is reflected in the second and third columns. All states
that adhere to proper MTD techniques for a specific attack should be
identified as normal, whereas state that employ incorrect or ineffective MTD
strategies should be abnormal. These results are obtained evaluating
approximately 1000 samples per behavior.

Overall, the model achieves a 99.54% accuracy for normal behavior and gets
more than a 99% successful detection rate for all kinds of malware through the
analysis of current device behavior. When it comes to accuracy of AD system
with device behavior after the implementation of MTD techniques, two primary
observations can be made. Firstly, the accuracy is comparable to that attained
for states. However, if an incorrect MTD technique is implemented, the
recognition of the state is notably poor in the case of Beurk, Dataleak,
Jakoritar, and The Tick, with an accuracy of almost 0, while an acceptable
accuracy score for Ransomware_PoC and Bdvl. It implies that the AD can offer
precise feedback to the agent by detecting abnormal behavior when an incorrect
MTD strategy is used against an attack. The second aspect is that, by
implementing appropriate MTD techniques, all behaviors are correctly
identified with a precision that surpasses 99%. The AD system effectively
provides precise feedback to the agent, as each attack is countered with the
appropriate MTD, resulting in the device returning to its normal state. In
conclusion, these results indicate that the AD system demonstrates its
effectiveness in several crucial aspects. It can accurately identify whether
the device is currently under attack, promptly trigger the agent to select the
right strategy in time, and provide precise feedback regarding the actions
decided by the agent.

###  IV-B Experiment 2: FRL-based Agent for MTD Selection

The Deep Q-Learning model holds utmost importance within the proposed
CyberForce framework, as its ability to make accurate decisions based on the
current device state is crucial. The performance of the model is greatly
influenced by the hyperparameters, therefore, identifying the optimal
combination of hyperparameters would prove advantageous for future
experiments. This experiment lists 17 different hyperparameters from three
aspects of the federation, neural network, and training strategy to find the
most suitable combination, as shown in TABLE VII.

TABLE VII: Hyperparameter Search for FRL-based Agent

Element | Hyperparameter | Values
---|---|---
Federation | NR_CLIENTS | 10
NR_ROUNDS | 30
NR_EPISODES_PER_ROUND | 100
NR_EPISODES_PER_CLIENT | 3,000
TOTAL_NR_EPISODES | 30,000
Neural Network | NR_NEURONS_PER_LAYER |  (128, 64), (128, 64, 32), (128, 64, 32, 16)
ACTIVATION_FUNCTION | Sigmoid, Tanh, ReLU, SELU
DROPOUT |  0, 0.2, 0.5
Training | OPTIMIZER | SGD, Adam, RMSprop, Adagrad
LOSS_FUNCTION | MAE, MSE, RMSE
GAMMA | 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9
LEARNING_RATE | 1e-2, 1e-3, 1e-4, 1e-5
L2_REGULARIZATION | 0, 1e-1, 1e-2, 1e-3, 1e-4
EPSILON_START | 1.0
EPSILON_DEC | 0.8/NR_EPISODES_PER_CLIENT
EPSILON_END | 0.01
AGGREGATION_STRATEGY | FedAvg

Through the five-fold cross-validation strategy, this work tests all possible
combinations. For this, 30,000 training samples are assigned to 10 clients and
subsequently, each client is trained with 3,000 episodes distributed over 30
rounds of FL with 100 episodes per round. The best hyperparameter combination
are bolded in TABLE VII. They are used for the following experiments.

####  IV-B1 IID Scenario

To measure the effectiveness of the FRL training approach, the malware
mitigation evaluation begins with the ideal scenario where all clients have
the same and balanced data of each malware type (data follows an IID
distribution across the clients). In this scenario, the training data is
divided into ten uniform and balanced parts for each client. As a comparison,
this work conducts a Centralized Baseline model according to [6], where all
the data is located in a single client and the RL model is trained based on a
traditional ML pipeline. The accuracy score of Centralized Baseline and
CyberForce with IID are shown in Figure 3. As can be seen, CyberForce has a
comparable malware mitigation efficiency with Centralized Baseline, both
achieve more than 98% accuracy at the end of the learning process. However,
the federated global model of CyberForce achieves 96% accuracy in the 4th
round, i.e., 400 episodes, and the Centralized Baseline achieves 96% accuracy
at 1300 episodes. Thus, it can be seen that CyberForce is able to save about
two-thirds of the learning time compared to the traditional Centralized
training approach, such as the one proposed in [6].

Figure 3: Test Accuracy of Centralized Baseline [6] and CyberForce with IID
Data Distribution

####  IV-B2 Non-IID Scenarios

More realistically, data is not uniformly distributed across clients, not all
clients are exposed to all malware, as it is essential to evaluate the
performance of the framework in the non-IID case. This work designs two
different non-IID setups: weak non-IID and strong non-IID.

In the weak non-IID setup, only one type of malware does not appear in the
training dataset for one client at a time. For example, in a federation of ten
clients, only client one has not been exposed to The Tick malware, while the
remaining nine clients are exposed to it. To verify the impact of such a non-
IID setup on the CyberForce framework, this experiment evaluates the accuracy
of the absent malware. In other words, the mitigation accuracy for The Tick in
the aforementioned example. Moreover, to validate the impact of the extent of
malware-missing clients on the collaboration of CyberForce, the experiment
sequentially increases the number of malware-missing clients from one to ten.

TABLE VIII presents the test accuracy of absent malware data for weak non-IID
configured clients with three aggregation algorithms. In general, FedAvg
demonstrates superior performance in addressing weak non-IID scenarios,
specifically when the number of malware-missing clients is below ten. In such
cases, FedAvg effectively learns all absent malware behavior. This can be
attributed to FedAvg exceptional capacity to collectively acquire and
distribute knowledge from all participating clients. Krum, however, is unable
to effectively deal with the challenges posed by non-IID. When the number of
malware missing clients increases, Krum has a higher chance of selecting a
client that is not exposed to the absent malware, resulting in an overall
federation that lacks data for that absent malware and cannot effectively
respond to absent attacks. Additionally, Trimmed Mean shows a balanced
performance. When the percentage of malware missing client is less than 40%,
Trimmed Mean is almost equivalent to FedAvg. However, when the percentage of
malware missing client is greater than 70%, the normal clients who contain all
the malware behavior instead become outliers and are excluded by the Trimmed
Mean, rendering the system incapable of effectively responding to a absent
malware attack.

TABLE VIII: Absent Malware Accuracy for Weak Non-IID

|  | No. of Malware Missing Clients
---|---|---
Absent Malware | Algorithm | 1 | 4 | 7 | 10
The Tick | FedAvg | 100.00% | 100.00% | 100.00% | 100.00%
Krum | 100.00% | 100.00% | 100.00% | 100.00%
Trimmed Mean | 100.00% | 100.00% | 100.00% | 100.00%
Jakoritar | FedAvg | 99.74% | 99.48% | 100.00% | 100.00%
Krum | 97.40% | 0.00% | 0.00% | 0.00%
Trimmed Mean | 98.70% | 33.50% | 0.00% | 0.00%
Dataleak | FedAvg | 95.68% | 92.15% | 89.21% | 94.11%
Krum | 100.0% | 14.37% | 6.53% | 10.45%
Trimmed Mean | 99.01% | 98.03% | 80.39% | 9.60%
Beurk | FedAvg | 96.15% | 95.05% | 96.93% | 94.21%
Krum | 96.71% | 0.00% | 0.00% | 0.00%
Trimmed Mean | 95.60% | 4.24% | 0.00% | 0.00%
Bdvl | FedAvg | 97.52% | 96.68% | 97.95% | 94.93%
Krum | 99.34% | 0.32% | 0.00% | 0.00%
Trimmed Mean | 100.0% | 98.62% | 72.74% | 0.00%
Ransomware-PoC | FedAvg | 99.40% | 98.81% | 98.81% | 33.25%
Krum | 100.0% | 99.40% | 32.54% | 0.00%
Trimmed Mean | 99.40% | 99.40% | 98.10% | 0.59%

From the malware point of view, Ransomware can be effectively mitigated with
FedAvg by leveraging its two MTD actions and the cooperative learning aspect.
However, if none of the clients have seen Ransomware data, the likelihood of
agents making the correct choice diminishes to 33.25%. In contrast, the
malware belonging to the C&C family (The Tick, Jakoritar, and Dataleak) as
well as the rootkits family (Beurk and Bdvl) exhibit an overall mitigation
success rate exceeding 94% with FedAvg, despite the number of weak non-IID
clients varies. This can be attributed to the collaborative learning mechanism
of the FRL architecture, where even clients lacking specific malware data can
still benefit from the knowledge shared by other clients possessing that data.
Another reason is that similar behaviors and MTD action decisions in the same
malware family enable the agent to learn appropriate actions transferred from
other malware data. Thus, further experiments are conducted to verify this
hypothesis.

To validate the transfer learning capability of CyberForce, experiments are
conducted where one family of malware is not seen for one client at a time. In
this experiment, FedAvg is used as the aggregation algorithm. As illustrated
in Figure 4, Beurk is not seen by any client and Bdvl is missing by one client
(a), and seven clients (b). In the scenario depicted in Figure 4 (a), where
there is only one client without rootkit data, the global model demonstrates a
convergence with an accuracy exceeding 98% for Bdvl. On the opposite, when the
number of clients without rootkit data increases to seven, as shown in Figure
4 (b), the global model could not converge effectively due to the absence of
transfer learning from similar behaviors. Similarly, the federated global
model is able to converge with an accuracy of approximately 96% when only one
client does not see the C&C malware family data, as shown in Figure 4 (c).
However, as the number of clients lacking the C&C malware data increases to
seven, the accuracy of the global model significantly drops, approaching zero,
as illustrated in Figure 4 (d). This experiment provides an explanation for
the noteworthy performance of FedAvg presented in TABLE VIII, as all the
clients have only one absence malware data, allowing the agents to acquire the
knowledge through the observation of similar behavior exhibited by the same
malware family.

(a)

(b)

(c)

(d)

Figure 4: (a) and (b): Bdvl Test Accuracy with Absence of Rootkits Malware
Family Data; (c) and (d): JAKORITAR Test Accuracy with Absence of C&C Malware
Family Data

In the strong non-IID scenario each client is not fully exposed to all six
malware attacks. In this setup, three varieties of malware are randomly absent
on each client. Compared to the weak non-IID scenarios (TABLE VIII), the
results of strong non-IID demonstrate a significant decline in accuracy in all
three aggregation algorithms, fluctuating between 60% and 85%, as illustrated
in Figure 5, which demonstrates the constraints of transfer learning and the
collaborative learning capabilities of the FRL framework.

Figure 5: Test Accuracy of CyberForce with Strong Non-IID

###  IV-C Experiment 3: Robustness Analysis of CyberForce

This experiment evaluates the robustness of the CyberForce framework against
adversary attacks affecting FL models. In particular the following two attacks
have been designed: (i) Data poisoning attack, in which all normal data labels
are manipulated as malware labels and all malware labels are randomly changed
to normal labels, with the aim of undermining the effectiveness of the AD
system in delivering undesirable rewards; and (ii) Model poisoning attack,
where 50% Gaussian noise is injected into the neural network after each round
of training, thus affecting the performance of the agent.

By progressively increasing the number of data poisoned clients, the accuracy
of all three aggregation algorithms declines, as illustrated in Figure 6 (a).
FedAvg and Trimmed Mean showed better resistance to data poisoning attacks,
with their accuracy starting to decline only when the share of poisoned
clients is greater than half. However, Krum performs the worst in model
poisoning, and its accuracy drops significantly to 60% when the number of
poisoned clients is greater than three.

(a)

(b)

Figure 6:  Test Accuracy of Different Aggregation Algorithms with Malicious
Attacks

In contrast, Krum performs the best when confronting model poisoning attacks,
as shown in Figure 6 (b), maintaining an accuracy of over 95% even when 90% of
the clients have been poisoned. It is observed that the Euclidean distance
between poisoned clients and the rest of the clients is consistently greater
than the distance between benign clients. Consequently, the Krum algorithm can
detect and choose the benign model as the global model. FedAvg exhibits the
least resilience against model poisoning attacks, as evidenced by a
significant decrease in its overall accuracy to nearly 50% when only one
client has been poisoned. Trimmed Mean is effective in mitigating low level
mod poisoning attacks, specifically those involving only one poisoned client.
Nevertheless, when the number of affected clients surpasses three, the
performance of Trimmed Mean aligns with that of the FedAvg method.

(a)

(b)

Figure 7: Learning Curve in FedAvg with Malicious Attacks

While closely inspecting FedAvg, it can be noticed that when exposed to model
poisoning attacks, the learning curve of the poisoned client is completely
destroyed, as shown in Figure 7 (a), which indicates that the client is unable
to learn any meaningful information.

(a)

(b)

Malicious Client Model (Mi), Benign Client Model (Bj), Global Model (G)

Figure 8: Model Similarity in FedAvg with Data Poisoning

Nevertheless, FedAvg presents resilience against data poisoning attacks, as
shown in Figure 7 (b), where malicious agents exhibit learning curves
comparable to benign agents. Figure 8 portrays the model similarity between
different agents under the same scenario. When there is only one being
attacked, as shown in Figure 8 (a), the global model is closer to the benign
models. Whereas, when malicious clients increase to 5, the global model gets
closer to the malicious model, as shown in Figure 8 (b), the accuracy of the
global model starts to decrease.

In conclusion, in IID scenarios, CyberForce significantly optimizes training
time while maintaining over 98% accuracy, outperforming centralized RL-based
solutions. Its adaptability shines with aggregation algorithms like Krum,
FedAvg, and Trimmed Mean, addressing diverse cyberattack challenges. Krum
strengthens resilience against model poisoning attacks in IID environments,
while FedAvg enhances non-IID data performance. Trimmed Mean offers balanced
performance, effectively managing limited non-IID conditions and resisting
minor malicious attacks.

##  V Conclusions and Future Work

This work presents CyberForce, an FRL-based framework able to select and
deploy MTD mechanisms mitigating diverse zero-day attacks on IoT devices.
CyberForce incorporates behavioral fingerprinting and ML-based AD methods to
identify zero-day attacks. Furthermore, it employs a federated agent that
utilizes Deep-Q Learning to learn the most effective MTD technique per attack.
The framework effectiveness was evaluated by deploying it on ten Raspberry Pi
4 devices, which acted as sensors of an IoT crowdsensing platform named
Electrosense. Each device was attacked by six distinct malware samples (from
the ransomware, C&C, and rootkit families), and four heterogeneous and
existing MTD mechanisms were considered to mitigate them. A series of
experiments were conducted to assess the CyberForce selection performance,
learning time, and robustness against attacks. With the aim of showcasing the
suitability of CyberForce in diverse scenarios, the experiments encompassed
the previous malware affecting multiple devices and involved varying data
distributions (ranging from IID to non-IID). The results showed that in the
IID scenario, CyberForce significantly reduces training time/episodes of
existing centralized RL-based solutions by two-thirds while maintaining an
accuracy rate of over 98%. The CyberForce framework offers remarkable
adaptability and flexibility by providing various aggregation algorithms such
as Krum, FedAvg, and Trimmed Mean to address diverse cyberattack challenges.
In scenarios involving substantial model poisoning attacks but an IID
environment, the inclusion of Krum can bolster the resilience of the system.
Conversely, when confronted with a non-IID situation within a secure setting,
FedAvg can enhance performance by leveraging the collaborative mechanism.
Meanwhile, Trimmed Mean provides a balanced performance. It is capable of
effectively managing scenarios with limited non-IID conditions, while also
displaying resilience against minor malicious attacks.

As future work, it is planned to extend the deployment of the CyberForce
framework to additional device types and diverse scenarios in order to
evaluate its efficacy and scalability. Furthermore, it is intended to augment
the framework and its evaluation by incorporating additional mitigation
mechanisms, as well as new malware families such as cryptominers,
infostealers, and botnets. Besides, the exploration of a combined approach
involving vertical FL and RL is being considered.

## References

  * [1] K. Riad, T. Huang, and L. Ke, “A Dynamic and Hierarchical Access Control for IoT in Multi-authority Cloud Storage,” _Journal of Network and Computer Applications_ , vol. 160, p. 102633, 2020.
  * [2] F. Meneghello, M. Calore, D. Zucchetto, M. Polese, and A. Zanella, “IoT: Internet of Threats? A Survey of Practical Security Vulnerabilities in Real IoT Devices,” _IEEE Internet of Things Journal_ , vol. 6, no. 5, pp. 8182–8201, 2019.
  * [3] P. M. S. Sánchez, J. M. J. Valero, A. H. Celdrán, G. Bovet, M. G. Pérez, and G. M. Pérez, “A Survey on Device Behavior Fingerprinting: Data Sources, Techniques, Application Scenarios, and Datasets,” _IEEE Communications Surveys & Tutorials_, vol. 23, pp. 1048–1077, 2021.
  * [4] G.-l. Cai, B.-s. Wang, W. Hu, and T.-z. Wang, “Moving Target Defense: State of the Art and Characteristics,” _Frontiers of Information Technology & Electronic Engineering_, vol. 17, no. 11, pp. 1122–1153, 2016.
  * [5] J. von der Assen, A. H. Celdrán, P. M. S. Sánchez, J. Cedeño, G. Bovet, G. M. Pérez, and B. Stiller, “A Lightweight Moving Target Defense Framework for Multi-purpose Malware Affecting IoT Devices,” in _IEEE International Conference on Communications_ , Rome, Italy, 2023.
  * [6] A. H. Celdrán, P. M. S. Sánchez, J. von der Assen, T. Schenk, G. Bovet, G. M. Pérez, and B. Stiller, “Rl and fingerprinting to select moving target defense mechanisms for zero-day attacks in iot,” 2022.
  * [7] J. Kreischer, “FRL for Crowdsensing Security,” https://github.com/jan-kreischer/FedRL-for-IoT-Security, 2023, [Accessed August 2023].
  * [8] S. Rajendran, R. Calvo-Palomino, M. Fuchs, B. V. den Bergh, H. Cordobés, D. Giustiniano, S. Pollin, and V. Lenders, “Electrosense: Open and Big Spectrum Data,” _IEEE Communications Magazine_ , vol. 56, no. 1, pp. 210–217, January 2018.
  * [9] V. Rey, P. M. Sánchez Sánchez, A. Huertas Celdrán, and G. Bovet, “Federated learning for malware detection in IoT devices,” _Computer Networks_ , vol. 204, p. 108693, 2022.
  * [10] H.-K. Lim, J.-B. Kim, J.-S. Heo, and Y.-H. Han, “Federated Reinforcement Learning for Training Control Policies on Multiple IoT Devices,” _Sensors_ , vol. 20, p. 1359, 03 2020.
  * [11] A. Uprety and D. B. Rawat, “Reinforcement learning for iot security: A comprehensive survey,” _IEEE Internet of Things Journal_ , pp. 1–1, 11 2020\.
  * [12] Q. Jia, K. Sun, and A. Stavrou, “Motag: Moving target defense against internet denial of service attacks,” in _22nd International Conference on Computer Communication and Networks (ICCCN)_ , 2013, pp. 1–9.
  * [13] T. Eghtesad, Y. Vorobeychik, and A. Laszka, “Adversarial Deep Reinforcement Learning Based Adaptive Moving Target Defense,” in _Decision and Game Theory for Security_. Springer International Publishing, 2020, pp. 58–79.
  * [14] S. Sengupta and S. Kambhampati, “Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense,” _ArXiv_ , 07 2020.
  * [15] S. Kim, S. Yoon, J.-H. Cho, D. Kim, T. Moore, F. Free-Nelson, and H. Lim, “Divergence: Deep reinforcement learning-based adaptive traffic inspection and moving target defense countermeasure framework,” _IEEE Transactions on Network and Service Management_ , pp. 1–1, 01 2021.
  * [16] M. Albanese, S. Jajodia, and S. Venkatesan, “Defending from stealthy botnets using moving target defenses,” _IEEE Security & Privacy_, vol. 16, pp. 92–97, 02 2018.
  * [17] X. Xu, H. Hu, Y. Liu, T. Jinglei, and H. Zhang, “Moving target defense of routing randomization with deep reinforcement learning against eavesdropping attack,” _Digital Communications and Networks_ , vol. 8, 01 2022.
  * [18] H. Li and Z. Zheng, “Robust moving target defense against unknown attacks: A meta-reinforcement learning approach,” in _Decision and Game Theory for Security: 13th International Conference, GameSec 2022, Pittsburgh, PA, USA, October 26–28, 2022, Proceedings_. Berlin, Heidelberg: Springer-Verlag, 2023, p. 107–126.
  * [19] C. Gao and Y. Wang, “Reinforcement learning based self-adaptive moving target defense against DDoS attacks,” _Journal of Physics: Conference Series_ , vol. 1812, p. 012039, 02 2021.
  * [20] T. Zhang, C. Xu, P. Zou, H. Tian, X. Kuang, S. Yang, L. Zhong, and D. Niyato, “How to mitigate ddos intelligently in sd-iov: A moving target defense approach,” _IEEE Transactions on Industrial Informatics_ , pp. 1–10, 01 2022\.
  * [21] S. Yoon, J.-H. Cho, D. Kim, T. Moore, F. Free-Nelson, and H. Lim, “Desolater: Deep reinforcement learning-based resource allocation and moving target defense deployment framework,” _IEEE Access_ , vol. PP, pp. 1–1, 04 2021\.
  * [22] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [23] H. McMahan, E. Moore, D. Ramage, and B. Agüera y Arcas, “Federated learning of deep networks using model averaging,” 02 2016.
  * [24] D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5650–5659.
  * [25] A. Huertas Celdrán, P. M. Sánchez Sánchez, M. zorín Castillo, G. Bovet, G. Martínez Pérez, and B. Stiller, “Intelligent and behavioral-based detection of malware in IoT spectrum sensors,” _International Journal of Information Security_ , pp. 1–21, 2022.

|  Chao Feng received the MSc degree in Informatics from the University of
Zurich, Switzerland. He is currently pursuing his Ph.D. in computer science at
the Communication Systems Group, Department of Informatics at the University
of Zurich. His scientific interests include IoT, cybersecurity, data privacy,
machine learning, and computer networks.
---|---
|  Alberto Huertas Celdrán is senior researcher at the Communication Systems
Group CSG, Department of Informatics IfI, University of Zurich UZH. He
received the MSc and PhD degrees in Computer Science from the University of
Murcia, Spain. His scientific interests include cybersecurity, machine and
deep learning, continuous authentication, and computer networks.
---|---
|  Jan Kreischer received his BSc in Informatics from Karlsruhe Institute of
Technology and recently finished his master degree in Artificial Intelligence
at the University of Zurich (UZH). His research interests lie in IoT,
Cybersecurity, Machine Learning and Trustworthy AI.
---|---
|  Pedro M. Sánchez Sánchez is pursuing his PhD in computer science at the
University of Murcia. He received the MSc degree in Computer Science from the
University of Murcia, Spain. His research interests focus on continuous
authentication, networks, 5G, cybersecurity, and machine learning and deep
learning.
---|---
|  Jan von der Assen received his MSc degree in Informatics from the
University of Zurich, Switzerland. Currently, he is pursuing his Doctoral
Degree under the supervision of Prof. Dr. Burkhard Stiller at the
Communication Systems Group, University of Zurich. His research interest lies
at the intersection between risk management and the mitigation of cyber
threats.
---|---
|  Gérôme Bovet received his Ph.D. in networks and computer systems from
Telecom ParisTech, France, in 2015, and an Executive MBA from the University
of Fribourg, Switzerland in 2021. He is the head of data science for the Swiss
Department of Defense, where he leads a research team and portfolio of about
30 Cyber-Defence projects. His work focuses on ML and DL approaches, with an
emphasis on anomaly detection, adversarial and collaborative learning applied
to data gathered by IoT sensors.
---|---
|  Gregorio Martínez Pérez is Full Professor in the Department of Information
and Communications Engineering of the University of Murcia, Spain. His
scientific activity is mainly devoted to cybersecurity and networking. He is
working on different national (14 in the last decade) and European IST
research projects (11 in the last decade) related to these topics, being
Principal Investigator in most of them. He has published 200+ papers in
international conference proceedings, magazines and journals.
---|---
|  Burkhard Stiller received his MSc degree in Computer Science and the PhD
degree from the University of Karlsruhe, Germany, in 1990 and 1994. Since 2004
he chairs the Communication Systems Group CSG, Department of Informatics IfI,
University of Zürich UZH, Switzerland as a Full Professor. His main research
interests are published in +300 papers and include decentralized systems with
fully control, network and service management, IoT, and telecommunication
economics.
---|---