# Bayesian Optimization for Robust State Preparation in Quantum Many-Body
Systems

Tizian Blatz<EMAIL_ADDRESS>Department of Physics and
Arnold Sommerfeld Center for Theoretical Physics, Ludwig-Maximilians-
Universität München, Munich D-80333, Germany  Munich Center for Quantum
Science and Technology (MCQST), Munich D-80799, Germany  Joyce Kwan
Department of Physics, Harvard University, Cambridge, MA 02138, USA  Julian
Léonard  Vienna Center for Quantum Science and Technology, Atominstitut, TU
Wien, Vienna 1020, Austria  Annabelle Bohrdt<EMAIL_ADDRESS>regensburg.de Munich Center for Quantum Science and Technology (MCQST), Munich
D-80799, Germany  Institute of Theoretical Physics, University of Regensburg,
Regensburg D-93053, Germany

###### Abstract

New generations of ultracold-atom experiments are continually raising the
demand for efficient solutions to optimal control problems. Here, we apply
Bayesian optimization to improve a state-preparation protocol recently
implemented in an ultracold-atom system to realize a two-particle fractional
quantum Hall state. Compared to manual ramp design, we demonstrate the
superior performance of our optimization approach in a numerical simulation –
resulting in a protocol that is $10\times$ faster at the same fidelity, even
when taking into account experimentally realistic levels of disorder in the
system. We extensively analyze and discuss questions of robustness and the
relationship between numerical simulation and experimental realization, and
how to make the best use of the surrogate model trained during optimization.
We find that numerical simulation can be expected to substantially reduce the
number of experiments that need to be performed with even the most basic
transfer learning techniques. The proposed protocol and workflow will pave the
way toward the realization of more complex many-body quantum states in
experiments.

Introduction – Recent advances in quantum simulation platforms based on
ultracold atoms in optical lattices Bloch  _et al._ (2012); Gross and Bloch
(2017) have allowed the creation of increasingly complex quantum many-body
states in larger systems, necessitating improvements in state-preparation
quality and speed. Well-established approaches like adiabatic ramping Eckstein
and Kollar (2010), manual mapping of the quantum many-body gap Léonard  _et
al._ (2023), and optimal control techniques Doria  _et al._ (2011); Glaser
_et al._ (2015); Koch (2016); Koch  _et al._ (2022) have allowed experimental
access to exciting new phenomena, such as magnetic and topological states
Simon  _et al._ (2011); van Frank  _et al._ (2016); de Léséleuc  _et al._
(2019); Léonard  _et al._ (2023).
In recent years, machine learning methods, such as reinforcement learning Niu
_et al._ (2019); Paparelle  _et al._ (2020), have emerged as a powerful tool
for quantum optimal control (QOC) applications. Bayesian optimization (BO)
Shahriari  _et al._ (2016); Garnett (2023), an alternative machine-learning
approach, has been used to generate efficient protocols for fast creation of
high-particle number Bose-Einstein condensates Wigley  _et al._ (2016);
Vendeiro  _et al._ (2022), cooling in magneto-optical traps Tranter  _et al._
(2018), and state preparation Mukherjee  _et al._ (2020a); Xie  _et al._
(2022). In these works, an online optimization loop is formed between the
learner, and either an experimental setup or numerical simulation. The
resulting optimized protocols have been demonstrated to compare favorably to
manual ramp design Vendeiro  _et al._ (2022) and other optimization
techniques, such as differential evolution and Nelder-Mead Xie  _et al._
(2022); Lagarias  _et al._ (1998); Das and Suganthan (2011). BO is well suited
to these tasks featuring experimental disorder or low repetition rates, as it
typically requires fewer iterations, than e.g., methods based on gradient
descent, and training is robust to moderate noise.
In this work, we use an exact-diagonalization (ED) simulation and apply BO to
prepare a few-body fractional quantum Hall (FQH) state which has recently been
realized in an ultracold-atom experiment Léonard  _et al._ (2023). This allows
us to compare the optimized result to a protocol that is manually designed and
has been implemented in practice. We extensively investigate how taking
disorder in the experimental system into account changes the optimal strategy
and how to engineer effective control paths in a noisy environment. Based on
this analysis, we explore how exploiting classical simulation resources can
boost efficiency by reducing experimental costs. From this, we motivate a
general optimization strategy, which we believe to be applicable to any
experimental system in which classical simulation is available in some
capacity. These include not only the preparation of the Laughlin state
Laughlin (1983), which acts as our benchmark, but also other current issues
including challenging state preparation, such as the realization of quantum
spin liquid states Broholm  _et al._ (2020), and cooling in Fermi-Hubbard type
systems Mazurenko  _et al._ (2017); Chiu  _et al._ (2018).
In contrast to other optimal-control applications realizing a phase transition
Xie  _et al._ (2022), e.g., the superfluid to Mott-insulator transition Rosi
_et al._ (2013); van Frank  _et al._ (2016); Sørensen  _et al._ (2019);
Mukherjee  _et al._ (2020a), the FQH problem features a topologically non-
trivial target state prepared by tuning a larger number of control fields (4
vs 1-2) to be optimized over time. This more demanding problem can shed light
on the prospects of machine-learning-optimized state preparation, entering a
regime where simple linear ramping is impossible and an optimized protocol’s
robustness and generality become primary concerns. To determine whether BO is
a tool fit to tackle these challenges, our work proceeds as follows: We
discuss our optimization strategy and investigate the experimental setting
motivating our work by comparing experimental data to ED simulation. In the
core part of our work, we apply BO to the simulated system and thoroughly
investigate the performance of optimized ramping protocols regarding fidelity,
speed, and robustness to disorder. We conclude with a proposal that we believe
will facilitate the creation of more complex states in the system under
investigation and also act as a general tool to boost efficiency in QOC
applications going beyond this specific system.

Optimization Strategy – In this work, we implement a BO training loop as
illustrated in Fig. 1.

Figure 1: Optimization Strategy: (top) Illustration of the Bayesian
optimization training loop. The experiment or numerical simulation acts as a
black-box function, providing a quality measure for a given ramping protocol.
Based on evaluations of this function, the BO learner builds a surrogate model
of the cost landscape and suggests the next protocol to implement. (bottom)
Proposed optimization strategy exploiting numerical resources to boost
optimization efficiency in the experiment. Depending on the task at hand,
several avenues with varying experimental costs may be explored. The most
straightforward way is to directly implement the optimization result obtained
from simulation in the experiment, requiring $\mathcal{O}(1)$ evaluations of
the protocol’s quality. On top of this, the BO surrogate model can provide a
sensitivity analysis that can be exploited to manually calibrate only the most
sensitive protocol parameters in $\mathcal{O}(10)$ experimental evaluations.
Finally, when substantial adaptation to the experimental setting is necessary,
online optimization can be performed in $\mathcal{O}(100)$ iterations directly
with the experiment. Transfer-learning approaches can leverage numerical
resources to accelerate training significantly, e.g., by initializing the
surrogate model used for training in the experiment with training data from
the simulation. In this work, we highlight and motivate these three scenarios
by comparing the numerical simulation of an idealized system to a more
realistic description of the experimental setting that takes experimental
disorder into account.

Gaussian processes (GP) Rasmussen and Williams (2005) are used as a surrogate
model of the cost function or figure of merit of ramping protocols over the
entire parameter space. For details about BO and our implementation, the
reader is referred to Appendix A.
Fig. 1 also shows the proposed optimization strategy to be used in
experiments. The strategy is focused on making use of the trained surrogate
model and numerical simulation to reduce experimental costs. We develop this
strategy in the following sections.

Experimental Setting – Our application of BO is focused on the state-
preparation problem of a bosonic two-particle FQH state recently realized in
experiment Léonard  _et al._ (2023). The experiment features a ramp protocol,
adiabatically 111In the context of optimization, we consider an adiabatic
protocol to be any strategy aiming to minimize losses of the target-state
population by limiting the ramp speed to the maximum set by the inverse of the
many-body gap. transforming an initial state of two pinned particles into the
delocalized FQH state.

Figure 2: Experimental Ramp: (a) Experimental ramping protocol for the $4$
lattice parameters adapted from Léonard  _et al._ (2023). (b) Fidelity w.r.t.
the instantaneous ground state throughout the ramp. The ED simulation is
compared to experimental data Léonard  _et al._ (2023). The color-coded lines
in the background show the energy levels $E-E_{0}$ of $\hat{H}(t)$ and their
respective occupation during the simulated ramping protocol. Here, $E_{0}$ is
the instantaneous ground-state energy. Before $t_{x}$ is turned on at
$T=60\>\tau$, the fidelity is defined in the 1D system formed by the 1st
column of the $4\times 4$ lattice, instead of the full 2D system. This is done
to avoid spurious drops to $F\sim 0$ when $\Delta_{x}<\Delta_{y}$ caused by
the initial-state geometry.  Figure 3: Optimized Ramp: (a) Optimized ramp
protocol for the $4$ lattice parameters. White dots show the parameter values
controlled by the optimizer at fixed points in time (vertical dashed lines).
(b) Evaluated cost function at each iteration of the BO training. The visible
oscillations in the evaluated fidelity follow sweeps of the BO hyperparameter
$\kappa$, which tunes the strategy between exploration and exploitation (see
Appendix A for optimization details). (c) Fidelity w.r.t. the instantaneous
ground state throughout the ramp. The color-coded lines in the background show
the energy levels of $\hat{H}(t)$ and their respective occupation during the
simulated ramping protocol. All energies are w.r.t. the ground state energy
which sets $E=0$. Before $t_{x}$ is turned on at $T=60\>\tau$, the fidelity is
defined in the 1D system formed by the 1st column of the $4\times 4$ lattice,
instead of the full 2D system. This is done to avoid spurious drops to $F\sim
0$ when $\Delta_{x}<\Delta_{y}$ caused by the initial-state geometry.

To achieve this, four global lattice parameters in the interacting Harper-
Hofstadter model Tai  _et al._ (2017); Harper (1955); Hofstadter (1976)

$\displaystyle\begin{split}\hat{H}=\ &-t_{x}\sum_{x,y}(e^{i\phi
y}\hat{a}^{\dagger}_{x,y}\hat{a}^{\vphantom{\dagger}}_{x+1,y}+\mathrm{H.c.})+\Delta_{x}\sum_{x,y}x\,\hat{n}_{x,y}\\\
&-t_{y}\sum_{x,y}(\hat{a}^{\dagger}_{x,y}\hat{a}^{\vphantom{\dagger}}_{x,y+1}+\mathrm{H.c.})+\Delta_{y}\sum_{x,y}y\,\hat{n}_{x,y}\\\
&+\frac{U}{2}\sum_{x,y}\hat{n}_{x,y}(\hat{n}_{x,y}-1)\end{split}$ (1)

are varied independently over time: (i, ii) the tunneling strengths $t_{x,y}$
in x- and y-direction and (iii, iv) the chemical potential gradients
$\Delta_{x,y}$. Here, $\hat{a}^{(\dagger)}_{x,y}$ is the bosonic annihilation
(creation) operator at site $(x,y)$ and
$\hat{n}^{\vphantom{\dagger}}_{x,y}=\hat{a}^{\dagger}_{x,y}\hat{a}^{\vphantom{\dagger}}_{x,y}$
is the particle number operator. The tunneling phase $\phi/2\pi=0.27$ is
constant throughout this work, corresponding to the Laughlin-like state
Laughlin (1983) at the fractional filling $\nu=1/2$, where $\nu$ is the
filling factor. This target state is realized in the flat lattice
$\Delta_{x}=\Delta_{y}=0$, at equal tunneling amplitudes
$t_{x}=t_{y}=\hbar/\tau$. We fix the value of the tunneling amplitudes in the
target state as our unit of energy while the tunneling time $\tau$ acts as the
unit of time. In the experiment, $\tau=4.3(1)\>\mathrm{ms}$ Léonard  _et al._
(2023). Finally, the on-site repulsion is fixed to $U=8\>\hbar/\tau$.
The ramping protocol used in the experiment was devised by mapping out the
many-body gap over the four-dimensional parameter space and manually
identifying a path that avoids a gap closing. In addition, the tunneling
$t_{x}$ in x-direction is only turned on in the latter half of the
experimental protocol to reduce the effects of Floquet heating Sträter and
Eckardt (2016) resulting from the synthetic magnetic field engineered in the
experiment. The protocol is illustrated in Fig. 2. It serves as the primary
benchmark in this work – representing the capabilities of sophisticated
adiabatic ramp design.
The quality of a ramp is quantified by the fidelity

$\displaystyle F=|\langle\psi_{\mathrm{final}}|\psi_{0}\rangle|^{2}\ ,$ (2)

where $\ket{\psi_{\mathrm{final}}}$ is the state after the preparation
protocol, and $\ket{\psi_{0}}$ is the ground state of $\hat{H}$ at the target
parameters $t_{x}=t_{y}=\hbar/\tau$ and $\Delta_{x}=\Delta_{y}=0$. This is a
good figure of merit since it is experimentally accessible by inverting the
ramp and measuring the fraction that has returned to the initial state. For
the experimental protocol it was measured to be $F=43(6)\%$ Léonard  _et al._
(2023). In Fig. 2, we compare the experimental data to an ED time-evolution
simulation. Even though the simulation is performed at zero temperature, does
not take into account any lattice defects, and does not model the Floquet
sequence employed in the experiment – instead simulating the unperturbed
effective Hamiltonian – the fidelity-loss over time (see Fig. 2b)) shows
qualitative agreement with the experiment. The fidelity of $F=53.5\%$ observed
in the simulation is not much higher than the value reported in the
experiment. This supports non-adiabatic losses to excited states during the
state-preparation protocol as the leading source of fidelity loss, providing a
promising foundation for ramp optimization.

Optimization – For each lattice parameter, we parametrize the time-dependent
ramp by $N_{s}$ linear segments of equal length. Fixing the initial and target
parameters to be the same as in the experiment leads to a BO problem with
$4\times(N_{s}-1)$ parameters. Motivated by the experimental conditions and
concerns about Floquet heating, we limit the search space to

$\displaystyle 0\leq t_{x/y}\leq 1.2\>\hbar/\tau\ \mathrm{and}\
-4\>\hbar/\tau\leq\Delta_{x/y}\leq 4\>\hbar/\tau\ ,$ (3)

which corresponds to a Floquet regime that minimizes excitations to higher
bands and achieves efficient dynamics. Specifically, the range for $t_{x/y}$
is chosen to limit the broadening of heating channels Sträter and Eckardt
(2016), as its strength is determined by the driving amplitude. We start
ramping $t_{x}$ and $\Delta_{x}$ only after passing the time $T_{x}=0.6\>T$
where $T$ is the total ramp time. After training with $500$ iterations, BO
identifies an optimal ramp with $F=94.5\%$ as illustrated in Fig. 3.
Interestingly, the protocol is qualitatively similar to the one used in the
experiment. As depicted in Fig. 3, it achieves near-perfect fidelity due to
significant gains in the ground-state population (w.r.t. the instantaneous
ground state) near the final protocol time. Such a resurgence of fidelity has
also been reported in other simulation works on QOC applications Mukherjee
_et al._ (2020b). While captured by optimal control theory, it is outside the
scope of manual design principles, such as adiabatic ramp design, which aim to
minimize fidelity losses, but do not take processes that can return population
to the ground state into account. We now turn to whether this feature,
observable in simulations, can be expected to be robust in experimental
conditions.

Training with Noise – To characterize robustness, and estimate the real-world
fidelity we would expect from optimized protocols, we now model experimental
disorder in our simulation by the sum of random and harmonic chemical
potential offsets (see Appendix C). The disorder $\mu$ is taken to be
Gaussian-distributed with mean $0$ and standard deviation
$\sigma=0.1\>\hbar/\tau$ which is an experimentally realistic disorder level.
For each numerical evaluation of a ramp’s fidelity, we perform time evolution
with a newly sampled realization of the disorder.
Optimization is now performed in the presence of this disorder to represent
training in an experimental setting. To mitigate fluctuations, we take the
average fidelity of $5$ disorder realizations as the cost function, which is a
typical value used in the literature Vendeiro  _et al._ (2022); Xie  _et al._
(2022). After training for $1000$ iterations, we compare the optimized
protocol to the manually designed ramp and the optimum found from training
without disorder in Fig. 4.

Figure 4: Experimental Noise: Expected fidelity $F$ of different ramping
protocols at varying standard deviation $\sigma$ of the experimental disorder.
The shaded areas show the low-disorder (blue) and the experimentally realistic
(orange) regime. Solid lines show ramps trained in the presence of disorder,
dashed lines show ramps designed in the ideal system. The error bars indicate
the standard error of the mean fidelity obtained from averaging over $100$
disorder realizations. The highest fidelity at $\sigma=0$ is achieved by
optimization in the clean system. The highest fidelity at $\sigma\geq
0.1\>\hbar/\tau$ is achieved by training in the disordered system, initialized
from training in the ideal system (‘Transfer’).

The comparison is performed over a range of disorder strengths, covering both
the clean as well as the experimentally realistic case. Firstly, this analysis
highlights the competitiveness of the manually designed protocol and the
complexity of the state preparation problem at hand. The experimentally
implemented ramp is remarkably robust and evaluates to $F=49(2)\%$ at
$\sigma=0.1\>\hbar/\tau$, even closer to the experimentally measured ramp
fidelity and more than $10\times$ the fidelity expected from simple linear
ramping.
The fidelity of the ramp optimized in noisy training beats the fidelity of the
manually designed protocol at all investigated disorder levels. However, it is
inferior to the optimum found in training without disorder. We conclude that
the noisy evaluation of the cost function makes training significantly more
challenging.
We confirm this by significantly increasing the number of disorder
realizations averaged over for an evaluation of the cost function to an
adaptive number between $10$ and $50$ – taking more samples when the fidelity
is high (see Appendix D for more details). Doing so boosts BO performance but
still does not reach the fidelity achieved in training without disorder.
To reach optimum performance in the noisy environment, we turn to our
knowledge of the ideal system. Usually, the surrogate model in BO is
initialized by picking $\mathcal{O}(10)$ points in parameter space at random
and evaluating the fidelity at these points. As a rudimentary implementation
of transfer learning, we replace this random selection with a subset of the
points visited during training in the ideal system. The most relevant points
for training are chosen by a simple heuristic algorithm according to their
fidelity (which is known for the clean system) and Euclidean distance in
parameter space. For details about the algorithm, see Appendix E. The cost
function at the chosen points is subsequently reevaluated in the presence of
disorder to initialize the surrogate model.
With this most basic transfer-learning approach, we reach the highest fidelity
achieved in the noisy environment with a fraction of the necessary number of
runs in the disordered system. Comparing this protocol with the optimized
result from the clean system tells us something about the changes in the cost
landscape when disorder is included. The new optimum in the disordered system
is very close in parameter space to that in the clean system, so we find
disorder to only lead to a slight shift in the ideal strategy. This scenario
is distinct from one realized in a previous study Mukherjee  _et al._ (2020b)
where the optimum location drastically changed with the onset of disorder. For
a more detailed discussion of these two scenarios, see Appendix F which
confirms our findings as a property of the system in question – not an
artifact of optimization – via a brute-force search and examines a special
case in our system that mirrors the scenario in the literature.

Short Ramps – Thus far, we have aimed to produce ramping protocols that
achieve high fidelity both in an ideal as well as a noisy setting. The total
ramp time $T$ has been kept fixed to that of the adiabatic experimental
protocol. However, a major advantage of QOC techniques is that they can
significantly reduce the total protocol time while retaining high fidelities
Wigley  _et al._ (2016); van Frank  _et al._ (2016). To leverage this
advantage, we run optimization with reduced final times $T=10\>\tau$, and
$T=20\>\tau$, i.e., $10\times$, or $5\times$ shorter than the protocols
previously considered. Based on our findings in the previous sections, we
expect the global optimum of the optimization landscape to move only a little
when introducing experimental disorder, so we chose to work in the ideal
system.
Running the optimization for $500$ iterations, we discover ramping protocols
with $F=53\%$, and $F=78\%$, respectively.

Figure 5: Short Ramps: (a) Ramp protocols for the optimized ramps with final
times $T=20\>\tau$ and $T=100\>\tau$. The ramp shown for the final time
$T=100\>\tau$ is the most robust one considered in the previous section, i.e.,
the ‘Transfer’ ramp from Fig. 4. White dots show the parameter values
controlled by the optimizer at fixed points in time. (b) Fidelities of the
ramps optimized for various final times, as well as the experimental ramping
protocol (which has final time $T=100\>\tau$) at varying disorder level
$\sigma$. The error bars indicate the standard error of the mean fidelity
obtained from averaging over $100$ disorder realizations.

The shorter protocols are compared to the ones with $T=100\>\tau$ in Fig. 5.
We find the shortest ramp to be remarkably close to the experimental protocol
in terms of the resulting fidelity – both in the clean as well as the
disordered system. That is, at this fixed level of fidelity, optimization
reaches a speedup by a factor of $10$.
We consider the ramp resulting from the ramping time $T=20\>\tau$ to be
optimal for an experimental implementation. While the fidelity in the
disorderless scenario is still significantly lower than for optimized ramps
with longer ramp times, we find the protocol to be extremely robust under the
introduction of disorder. In fact, the fidelity at the strongest considered
level of disorder $\sigma=0.2\>\hbar/\tau$ is significantly higher than for
any other protocol considered in this work. Similar to previous studies van
Frank  _et al._ (2016); Mukherjee  _et al._ (2020b), we attribute this result
to the shorter protocol time presenting less of an opportunity for the
accumulation of errors. This interpretation is supported by the significantly
lower standard deviation of the fidelity in the noisy setting. At the
experimental disorder level $\sigma=0.1\>\hbar/\tau$, while the mean fidelity
of the $T=20\>\tau$ ramp is similar to that of the ramp optimized for
$T=100\>\tau$, the standard deviation is lower by more than a factor of $4$.

Experimental Proposal – Based on the findings of the previous sections, we
propose the following strategy and best practices for the realization of the
BO approach in an experiment:
(i) Implement suggestion: Due to the advantages in terms of speed and
robustness of shorter ramping protocols, we suggest the $T=20\>\tau$ ramp from
the previous section for an experimental realization. Even though the training
was performed on the ideal system, we believe the scheme to be directly
implementable, as we found the global optimum location to be relatively
insensitive to disorder.
(ii) Sensitivity analysis: Going beyond this innate robustness, we argue that
a sensitivity analysis based on the BO surrogate model can reduce manual
calibration in the experiment to a few parameters. Since BO takes an increased
number of samples around the discovered optimum, the model is very accurate in
this region and can faithfully predict the optimum’s shape in parameter space.
Thus, the surrogate model contains accurate information about the optimum’s
sensitivity to the different parameters. When the optimum is only slightly
shifted in parameter space by the onset of disorder, good performance can be
recovered at moderate experimental cost by manually tuning only the most
sensitive parameters.
The sensitivity analysis drawn from training the proposed $T=20\>\tau$ ramp is
depicted in Fig. 6.

Figure 6: GP Model: Gaussian-process model of the cost landscape around the
global optimum trained during optimization of the $T=20\>\tau$ ramp. (a) The
optimum ramping protocol identified by optimization – characterized by $4$
control fields at $2$ time points (white dots). (b) Single-parameter
variations around the global optimum of the modeled cost landscape with the
remaining parameters fixed to their optimum value. The shaded regions depict
the model uncertainty (one standard deviation), which is generally the lowest
around the global optimum. The sensitivity of the model to changes in
different parameters varies considerably. Grey, shaded regions link the least
(most) sensitive protocol parameter $t_{y}^{(1)}$ ($\Delta_{x}^{(2)}$) to the
corresponding modeled optimum shape.

By varying single parameters around the optimum, we identify the fidelity to
be much more sensitive to tuning the global tilts than to changes in hopping
amplitudes. In particular, the GP model of the cost function is almost
completely insensitive to the value $t_{y}^{(1)}$ at the first time point
$T=20/3\>\tau$ but sharply peaked around the optimum value of
$\Delta_{x}^{(2)}$ at the second fixed time point. By taking advantage of the
trained learner in this way, manual calibration in the experiment can be
reduced from full functional dependencies to only a few parameters.
(iii) Transfer learning: In case sensitivity analysis fails to recover the
optimum, it is favorable to train on the experiment directly. In this
scenario, we have demonstrated that making use of simulation data in a simple
transfer-learning approach can significantly boost performance while reducing
experimental costs.

Discussion & Outlook – We applied a Bayesian optimization approach to optimize
parameter ramps for state preparation. The optimization scheme was
demonstrated to be efficient at finding protocols that outperform manual
design not only in an ideal system but also in the presence of significant
disorder – identifying a ramp that is both $5\times$ faster as well as twice
as close to ideal fidelity as the manual reference. The superior performance
holds when the training data itself is noisy, e.g., when training directly on
an experiment, but at an increased cost of training.
We propose several approaches to decrease experimental costs in cases where
numerical simulation of the system is available. Namely, we demonstrated that
optimized solutions for the ideal system can be intrinsically robust due to a
significant reduction in protocol duration and the surrogate model of the
fidelity-landscape naturally provides information about the most sensitive
parameters to tune in an experiment. When learning directly on the experiment,
prior training in a simulated environment was shown to significantly reduce
the total cost. This also gives hope for scenarios where simulation is
available only in small systems, where transfer learning techniques may still
be able to extract an advantage.
In more general terms, we believe a further reduction in the number of
experimental runs by means of transfer learning and by improving the ramps’
parametrization to be the most pressing issues.
We expect the optimized protocols developed here to be instrumental in
realizing larger FQH states – e.g., by merging several $4\times 4$ plaquettes
into a larger system Palm  _et al._ (2023) – and different filling factors. In
particular, the substantial speedup and robustness offered by protocols
optimized using machine learning can allow the preparation of these more
complex quantum states within the already available coherence times of
contemporary ultracold-atom experiments.
The more general optimization strategy, we believe to be widely applicable to
boost experimental efficiency and improve the interface between classical and
quantum simulation.

Acknowledgments – We are very grateful to F. Grusdt, F. Palm, and C.
Weitenberg for fruitful discussions, and we wish to thank M. Greiner, B.
Bakkali-Hassani, P. Segura, and Y. Li for valuable insights regarding the
experimental system. This research was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s
Excellence Strategy—EXC-2111—390814868. J.L. acknowledges support by the FWF
grant no. Y-1436 and the FFG grant no. F0999896041.

##  Appendix A Bayesian Optimization

Bayesian optimization (BO) Shahriari  _et al._ (2016); Garnett (2023) is an
optimization method designed to find the optimum of a – possibly noisy and
expensive to evaluate – black box function. In our case, the task is finding
the optimal ramping protocol $\hat{H}(t)$ for the time-dependent Hamiltonian
that transforms a given initial state $\ket{\psi_{\mathrm{initial}}}$ into the
target state $\ket{\psi_{\mathrm{target}}}$. The quality of a given ramping
protocol is quantified by the fidelity

$\displaystyle F=|\braket{\psi(T)}{\psi_{\mathrm{target}}}|^{2}\ ,$ (4)

where $\ket{\psi(T)}$ is the initial state evolved according to $\hat{H}(t)$
up to the final ramp time $T$.
To phrase this as a standard optimization problem, we consider Hamiltonians
$\hat{H}(t)$ characterized by a set of parameters $\\{h_{i}(t)\\}_{i}$. Then,
we take each Hamiltonian parameter $h_{i}(t)$ as a piece-wise linear function
characterized by a set of time-value tuples $\\{x_{j}=(t_{j},v_{j})\\}_{j}$.
The optimization task then becomes minimizing the cost function

$\displaystyle C(\\{x_{j}\\}_{j})=1-\sqrt{F}\ .$ (5)

To find optimal values for the parameter set $\mathbf{x}=\\{x_{j}\\}_{j}$, a
BO loop, based on the open-source scikit-optimize package, is run in the
following way: First, the search is initialized by randomly choosing some
$\mathbf{x}_{i}$ – typically $\mathcal{O}(10)$ points – and evaluating the
corresponding cost function values $C(\mathbf{x}_{i})$ – either by running an
experiment or, in our case, via numerical simulation. We then use Gaussian
processes (GP) Rasmussen and Williams (2005) to build a surrogate model
$\tilde{C}(\mathbf{x})$ of the cost function over the entire parameter space,
based on the set of evaluated points. A GP is a generalization of the Gaussian
distribution, defining a distribution over functions by replacing the mean
vector and covariance matrix by the mean function $m(\mathbf{x})$, and the
kernel $k(\mathbf{x},\mathbf{x^{\prime}})$ Rasmussen and Williams (2005).
Given a set of previously evaluated points, the GP model predicts the value of
the cost function over the entire parameter space and gives a measure of the
model’s uncertainty in less-explored areas. Key properties of the GP model,
such as the smoothness of and variation between sampled functions are
controlled via a set of hyperparameters, which are continually adjusted during
training via maximum-likelihood estimation. These hyperparameters include the
noise level $\sigma_{F}$ of the data as well as the length scales $l$ which
give the characteristic scale on which functions sampled from the GP change
their value. Control over these parameters can be a valuable tool to achieve
optimum BO performance. For example, manually setting $\sigma_{F}=0$ for
training in the clean system can boost convergence and contribute to the low
number of iterations needed compared to training in the noisy setting. We also
demonstrate how limiting the lower bound of $l$ – constraining the maximum
complexity of the constructed model – can improve robustness while speeding up
convergence in Appendix G.

After the surrogate model is initialized, a new point $\mathbf{x}_{i+1}$ is
suggested based on the existing cost-function model, and the cost function is
subsequently evaluated at this new point. The new tuple

$\displaystyle(\mathbf{x}_{i},C(\mathbf{x}_{i}))$ (6)

is then used to update $\tilde{C}(\mathbf{x})$, completing the loop.
The suggestion $\mathbf{x}_{i}$ is made by the acquisition function, which can
prioritize either exploitation – choosing an $\mathbf{x}_{i}$ which minimizes
$\tilde{C}(\mathbf{x})$, or exploration – choosing an $\mathbf{x}_{i}$ where
$\tilde{C}(\mathbf{x})$ has high uncertainty. In the lower confidence bound
(LCB) Srinivas  _et al._ (2012) acquisition function

$\displaystyle\mathrm{LCB}(x)=m(x)+\kappa\>\sigma_{\mathrm{GP}}(x)$ (7)

used in this work, this tradeoff is tuned via the hyperparameter $\kappa$,
which sets the weight of the model uncertainty $\sigma_{\mathrm{GP}}$. In
practice, we sweep between these two strategies, starting with a focus on
exploration ($\kappa\sim 20$) and prioritizing exploitation ($\kappa\sim 0.2$)
during the final iterations of the optimization. Fig. 3b) of the main text
showcases $4$ sweeps of $\kappa$ and the corresponding evaluations of the cost
function for a training run with a total of $500$ iterations.
To transfer results from a completed optimization run to a new one, e.g., to
save cost when training on a noisy system, we replace random initialization
with selected parameter-space points evaluated in the completed run. The
points are selected by an algorithm detailed in a separate section. The
corresponding values of the cost function are reevaluated in the new (e.g.,
noisy) environment, and the resulting (point, cost) tuples are used to
initialize the surrogate model for the new optimization run.

##  Appendix B Exact Diagonalization Simulation

Exact-diagonalization (ED) simulations are used to perform ground-state search
and time evolution for the Harper-Hofstadter Hamiltonian Harper (1955);
Hofstadter (1976) on a $4\times 4$ lattice with $N=2$ bosons. The parameters
$t_{x/y}$ and $\Delta_{x/y}$ are the control fields tunable in the experiment
– representing tunneling amplitudes and global lattice tilts respectively.
Site-dependent chemical potentials $\mu_{x,y}$ may be introduced to model
disorder in the experiment.
For efficiency, we exploit the $\mathrm{U}(1)$ symmetry Zhang and Dong (2010),
resulting from particle number conservation. As in the experiment, the initial
state is prepared by manually placing the 2 bosonic particles in the bottom-
left corner of the $4\times 4$ lattice, i.e.,

$\displaystyle\ket{\psi_{\mathrm{initial}}}=\hat{a}^{\dagger}_{0,0}\hat{a}^{\dagger}_{0,1}\ket{0},$
(8)

where $\ket{0}$ is the $0$-particle state.
Time evolution is performed using the Runge-Kutta method of order $\geq 4$
with a maximum step-size $\delta t_{\mathrm{max}}=0.1$.
All computations for this work were run on a commercial laptop (Apple M1 Pro
processor), where a single time evolution to $T=100\>\tau$ takes $\sim
3\>\mathrm{s}$. When running BO, fitting the surrogate model becomes more
expensive than time-evolution simulation after about $200$ iterations. A
typical optimization run to $500$ iterations takes about $1\>\mathrm{h}$.

##  Appendix C Experimental Noise Model

The most important source of disorder in the experimental system are static
chemical-potential offsets. We consider two separate contributions to this
offset potential.

Figure 7: Experimental Disorder: Sample of a disorder realization with
$\sigma=0.1\>\hbar/\tau$ composed of a random and a harmonic component.

Firstly, we consider random offsets with a maximum strength of $\sim
10-15\mathrm{Hz}$ that are uncorrelated and follow the same distribution on
each lattice site. Secondly, we incorporate a chemical-potential offset on the
edge of the system caused by the digital micromirror devices (DMDs) projecting
the walls of the lattice. The offset is modeled as having strength
$2\mu_{\mathrm{DMD}}$ in the corners of the lattice and $\mu_{\mathrm{DMD}}$
everywhere else on the edges, equivalent to a harmonic potential affecting the
entire $4\times 4$ lattice. The strength $\mu_{\mathrm{DMD}}$ of this type of
disorder is taken to follow the same distribution as the random offsets, that
is $\mu_{\mathrm{DMD}}$ may have either sign and the two types of disorder are
similarly strong. As we expect the disorder to have mean $0$ and maximum
strength $\sim 10-15\mathrm{Hz}$ which is about a quarter of the hopping
amplitude $\hbar/\tau\sim 40\>\mathrm{Hz}$ in the final FQH state, which sets
the unit of energy, we model the disorder $\mu\ [\hbar/\tau]$ as being
Gaussian-distributed with mean $0$ and standard deviation
$\sigma=0.1\>\hbar/\tau$. For each numerical time evolution, we sample
$4\times 4+1$ independent values from this distribution – one for each single-
site offset and one for the global harmonic one.

##  Appendix D Adaptive Sampling

To keep the total number of samples used for training low, while reducing the
noise of cost-function evaluations near the optima, we use a simple adaptive
sampling technique – increasing the number of samples based on the fidelity.
Starting from a minimum number of $5$ samples, more samples are added if the
averaged fidelity is above a threshold value. The procedure continues with
higher and higher fidelity thresholds until a predefined maximum number of
$50$ samples. We believe this technique to be beneficial as it allows us to
more accurately map out the shape of discovered optima while saving cost by
sacrificing accuracy in regions where the fidelity is low.

##  Appendix E Initialization Algorithm

To select the most valuable points in parameter space to transfer to a new
optimization run, we use a simple heuristic algorithm. Iterating through all
evaluated points, a point $\mathbf{x}$ is accepted, if it is more than a
predefined distance $r$ away from all previously accepted points (in terms of
the Euclidean distance in parameter space). If the closest accepted point
$\mathbf{x}_{\mathrm{acc.}}$ is closer than $r$, between
$\mathbf{x}_{\mathrm{acc.}}$ and $\mathbf{x}$, the point that yielded a higher
fidelity is accepted. The radius $r$ is chosen such that $\sim 100$ points are
accepted in total. The $10-100$ points with the highest fidelity are then used
to initialize the new optimization run.
This basic algorithm ensures that the point that yielded the highest fidelity
in the BO run used for initialization is accepted and reevaluated for the new
optimization. Apart from that, the algorithm prioritizes keeping points spread
out over the parameter space (due to the division in ‘bubbles’ of size $r$)
while keeping points with high fidelity. We therefore expect this algorithm to
succeed in selecting the most important local optima alongside the global
optimum. We expect it to work well, not only in our situation, where the
location of the global optimum does not change much when introducing disorder
to the system but also in more challenging situations, where the location
changes significantly. This is the case since, in general, we believe the
global optimum in a disordered system to very likely have been a local optimum
in the ideal system. We discuss this in more detail in the following sections.

##  Appendix F Brute-Force Search

In the main text, using ramps with $3$ linear segments, we find the best
performance in the disordered system with a ramp very similar to the optimum
obtained in the clean system. To confirm that this is a feature of the problem
setting and not a result of poor optimization performance, we perform a brute-
force search in the simpler four-dimensional parameter space characterizing
ramps with only $2$ linear segments. The fidelity with and without
experimental disorder is performed within the search area

$\displaystyle t_{x/y}\in[0.0,\,1.2],\ \Delta_{x/y}\in[0.0,\,4.0],$ (9)

stepping through a grid with step sizes $\delta
t_{x/y}=\delta\Delta_{x/y}=0.2$. We motivate this uniform grid size, taking
more than $3$ times more points for the tilts $\Delta$ than for the hopping
amplitudes $t$ by the increased sensitivity to tilts suggested by the GP
models trained during BO (see main text). Even if optimization performance
were to be poor, we trust this general trend of sensitivities, as it is
consistent in all BO training performed in this work. To evaluate the fidelity
in the presence of disorder, we use an adaptive sampling technique similar to
the one used during noisy training – ensuring high accuracy in high-fidelity
regions while cutting down on the total number of evaluations. Still, at least
$10$ independent disorder realizations are evaluated for each point in the
search grid, so more than $200\,000$ time-evolution simulations are carried
out for the full brute-force search in the disordered setting.

Figure 8: Brute-Force Search: The optimization landscape around the global
optimum mapped out in the brute force search. (a, b) Ramp fidelity for
variations of the tilts (a), and tunneling amplitudes (b) with the $2$
remaining parameters fixed to their optimum values (grey text). The value of
each pixel is obtained from a time-evolution simulation of the corresponding
ramping protocol. The white dots indicate the location of the optimum
identified by BO. (c, d) The same as (a, b), but for the brute-force search
carried out in the disordered system with disorder strength
$\sigma=0.1\>\hbar/\tau$. Each pixel represents the mean fidelity obtained
from evaluating at least $10$ independent disorder realizations.

Optimal fidelity is achieved at

$\displaystyle[\>t_{x}=1.0,\ t_{y}=0.6,\ \Delta_{x}=0.2,\ \Delta_{y}=2.2\>],$
(10)

determined to be $F=85.8\%$ in the ideal system, and $F=66(2)\%$ in the system
including disorder. Crucially, optimum fidelity with- and without disorder is
reached at the same point in parameter space, so – within the finite
resolution provided by the grid – the two optima coincide.
BO trained on the clean system with $500$ iterations finds the optimum to be
at

$\displaystyle[\>t_{x}=1.04,\ t_{y}=0.66,\ \Delta_{x}=0.23,\
\Delta_{y}=2.21\>],$ (11)

with $F=87.8\%$. These findings confirm that, in our problem setting, the
global optimum of the cost landscape does indeed not move much with the
introduction of disorder, and BO can efficiently and correctly identify said
global optimum.

##  Appendix G Making Use of the Trained Model

Here, we perform a closer investigation of how the surrogate model trained
during a BO run can be used to make predictions about robustness and how
training simpler models can lead to protocols that are more robust against
unknown types of noise.
Since the parameter space in the optimization directly translates to changes
in Hamiltonian parameters over time – just like many types of noise or
disorder do – we think it is reasonable to believe that the width of an
optimum in parameter space can predict its performance against unknown types
of noise.
If this relation between disorder in the experiment and noise in parameter
space holds true, there is a simple intuitive picture of what is happening to
the cost function when introducing noise to the system. In common with other
noisy signals, peaks in the optimization landscape would be expected to become
wider and flatter with increasing disorder. Depending on the type and strength
of the noise, two distinct scenarios may occur:

  1. (i)

The global optimum widens but remains the highest peak, thus the location of
the global optimum does not change much.

  2. (ii)

Due to the loss in fidelity, a wider – more robust – local optimum becomes the
new global optimum, thus the location of the global optimum in parameter space
may jump dramatically.

Based on the analysis presented in the main text, as well as the brute-force
mapping of the cost landscape detailed above, we are confident that our system
and disorder level realize scenario (i) – both for long ($T=100\>\tau$) and
short ($T\leq 20\>\tau$) ramping protocols.

Figure 9: GP Model: The optimization landscape of the global tilts (hopping
amplitudes) with the remaining 2 parameters fixed to their optimum value is
shown in the left (right) plot. The red dot marks the location of the optimum
identified by BO

However, dramatic changes in the global optimum’s location with the
introduction of noise have been reported in the literature Mukherjee  _et al._
(2020a) indicating scenario (ii). To look for (ii) in our system and test our
conjecture, we turn to the intermediate ramp time $T=50\>\tau$. Here, in the
clean system, BO identifies an optimum that it predicts to be extremely narrow
in parameter space. The corresponding control path and GP model are
illustrated in Fig. 9. The figure also shows the model’s uncertainty, which
becomes high, even very close to the discovered optimum. To accommodate the
narrow optimum, the length-scale hyperparameters of the GPs become very low
during training, making the uncertainty away from evaluated points high.

Figure 10: Robust Models: Fidelities of ramping protocols obtained from
constrained and unconstrained training at varying disorder strength. The error
bars indicate the standard error of the mean fidelity obtained from averaging
over $100$ disorder realizations. While the ramp trained without any
restrictions features the highest fidelity in the ideal system, it rapidly
loses fidelity with the onset of experimental noise – even falling below the
manually designed protocol when crossing the experimentally realistic noise
strength. The ramp trained with a simpler BO model with limited length scales
is substantially more robust and retains its performance advantage over manual
design in the presence of strong disorder.

If there is indeed a direct relation between different types of noise, because
the optimum is so narrow, we would expect this control path to be sensitive to
unknown noises, such as the experimental potential noise used previously. This
is confirmed in Fig. 10. While the protocol evaluates to $F=81\%$ in the
noiseless system, the fidelity quickly drops with the onset of noise and falls
below that of the manually designed protocol for moderate to strong noise.

We can go even further, and ask whether the GP model can not only predict poor
performance against an unknown source of noise but if training can be actively
shaped to avoid unsuitably narrow optima. GPs provide natural approaches to
tackle this issue, because they allow bounding the complexity of the model
being constructed. In our example, by limiting the lower bound of the length
scale hyperparameter, BO identifies an optimum that lies lower in terms of
fidelity but is much wider in parameter space.
As hypothesized – despite only knowing about the ideal system – this new
protocol is significantly more robust against the modeled experimental noise
(see Fig. 10). It outperforms the more narrow optimum at experimentally
realistic noise strengths in a meaningful way. The optimum strategy jumps from
the narrow to the wider optimum between $\sigma=0.05\>\hbar/\tau$ and
$\sigma=0.1\>\hbar/\tau$ which we interpret as a realization of (ii).
We conclude that the surrogate model has predictive power regarding an
optimum’s robustness to unknown sources of noise reiterating our recommended
strategy to exploit the information contained in this costly model. We also
wish to highlight that limiting the model’s complexity – as has been done
above using length scale bounds – can not only produce more robust protocols
but also improve the speed of convergence significantly, thus reducing the
cost of training. This is apparent in the uncertainties of the models compared
in Fig 9 where the restricted model features substantially lower uncertainty
after the same number of iterations.

We conclude this discussion with some technical details about the occurrence
of (ii) in our system (only) at intermediate ramp times. We attribute this
curious behavior to a qualitative transition in the optimal ramping protocol
causing issues due to the simple parametrization using linear segments between
fixed points in time. In the main text – for both the experimental and the
optimized protocol – we observed a crossing in the control fields $t_{x}(t)$
and $t_{y}(t)$, where $t_{x}$ is ramped up from $t_{x}=0.0$ to more than unity
and subsequently decreased back to $t_{x}=1.0\>\hbar/\tau$. This feature is
absent in the $T=20\>\tau$ ramp – presumably due to a lack of time.
Consequently, there must be a crossover at an intermediate total time
$20\>\tau\leq T\leq 100\>\tau$ where both strategies are equally good. This
complicates training and is likely exacerbated by the rigid parametrization
which does not allow ‘doing nothing’. Similar to longer protocols, the
optimized result for $T=50\>\tau$ shown in Fig. 9 features a crossing
$t_{x}(t)=t_{y}(t)$ at $0<t<T$ but has to speed up delocalization in the
y-direction by ramping $\Delta_{y}$ to a value significantly below $0.0$.
Consequently, the control path becomes extremely sensitive to this
$\Delta_{y}$ value which turns out to be the optimization parameter that is
assigned the shortest length scale.

## References

  * Bloch  _et al._ (2012) Immanuel Bloch, Jean Dalibard, and Sylvain Nascimbène, “Quantum simulations with ultracold quantum gases,” Nature Physics 8, 267–276 (2012).
  * Gross and Bloch (2017) Christian Gross and Immanuel Bloch, “Quantum simulations with ultracold atoms in optical lattices,” Science 357, 995–1001 (2017).
  * Eckstein and Kollar (2010) Martin Eckstein and Marcus Kollar, “Near-adiabatic parameter changes in correlated systems: Influence of the ramp protocol on the excitation energy,” New Journal of Physics 12, 055012 (2010).
  * Léonard  _et al._ (2023) Julian Léonard, Sooshin Kim, Joyce Kwan, Perrin Segura, Fabian Grusdt, Cécile Repellin, Nathan Goldman, and Markus Greiner, “Realization of a fractional quantum Hall state with ultracold atoms,” Nature 619, 495–499 (2023).
  * Doria  _et al._ (2011) Patrick Doria, Tommaso Calarco, and Simone Montangero, “Optimal Control Technique for Many-Body Quantum Dynamics,” Physical Review Letters 106, 190501 (2011).
  * Glaser  _et al._ (2015) Steffen J. Glaser, Ugo Boscain, Tommaso Calarco, Christiane P. Koch, Walter Köckenberger, Ronnie Kosloff, Ilya Kuprov, Burkhard Luy, Sophie Schirmer, Thomas Schulte-Herbrüggen, Dominique Sugny, and Frank K. Wilhelm, “Training Schrödinger’s cat: Quantum optimal control,” The European Physical Journal D 69, 279 (2015).
  * Koch (2016) Christiane P. Koch, “Controlling open quantum systems: Tools, achievements, and limitations,” Journal of Physics: Condensed Matter 28, 213001 (2016).
  * Koch  _et al._ (2022) Christiane P. Koch, Ugo Boscain, Tommaso Calarco, Gunther Dirr, Stefan Filipp, Steffen J. Glaser, Ronnie Kosloff, Simone Montangero, Thomas Schulte-Herbrüggen, Dominique Sugny, and Frank K. Wilhelm, “Quantum optimal control in quantum technologies. Strategic report on current status, visions and goals for research in Europe,” EPJ Quantum Technology 9, 1–60 (2022).
  * Simon  _et al._ (2011) Jonathan Simon, Waseem S. Bakr, Ruichao Ma, M. Eric Tai, Philipp M. Preiss, and Markus Greiner, “Quantum simulation of antiferromagnetic spin chains in an optical lattice,” Nature 472, 307–312 (2011).
  * van Frank  _et al._ (2016) S. van Frank, M. Bonneau, J. Schmiedmayer, S. Hild, C. Gross, M. Cheneau, I. Bloch, T. Pichler, A. Negretti, T. Calarco, and S. Montangero, “Optimal control of complex atomic quantum systems,” Scientific Reports 6, 34187 (2016).
  * de Léséleuc  _et al._ (2019) Sylvain de Léséleuc, Vincent Lienhard, Pascal Scholl, Daniel Barredo, Sebastian Weber, Nicolai Lang, Hans Peter Büchler, Thierry Lahaye, and Antoine Browaeys, “Observation of a symmetry-protected topological phase of interacting bosons with Rydberg atoms,” Science 365, 775–780 (2019).
  * Niu  _et al._ (2019) Murphy Yuezhen Niu, Sergio Boixo, Vadim N. Smelyanskiy, and Hartmut Neven, “Universal quantum control through deep reinforcement learning,” npj Quantum Information 5, 1–8 (2019).
  * Paparelle  _et al._ (2020) Iris Paparelle, Lorenzo Moro, and Enrico Prati, “Digitally stimulated Raman passage by deep reinforcement learning,” Physics Letters A 384, 126266 (2020).
  * Shahriari  _et al._ (2016) Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas, “Taking the Human Out of the Loop: A Review of Bayesian Optimization,” Proceedings of the IEEE 104, 148–175 (2016).
  * Garnett (2023) Roman Garnett,  _Bayesian Optimization_ (Cambridge University Press, Cambridge, 2023).
  * Wigley  _et al._ (2016) P. B. Wigley, P. J. Everitt, A. van den Hengel, J. W. Bastian, M. A. Sooriyabandara, G. D. McDonald, K. S. Hardman, C. D. Quinlivan, P. Manju, C. C. N. Kuhn, I. R. Petersen, A. N. Luiten, J. J. Hope, N. P. Robins, and M. R. Hush, “Fast machine-learning online optimization of ultra-cold-atom experiments,” Scientific Reports 6, 25890 (2016).
  * Vendeiro  _et al._ (2022) Zachary Vendeiro, Joshua Ramette, Alyssa Rudelis, Michelle Chong, Josiah Sinclair, Luke Stewart, Alban Urvoy, and Vladan Vuletić, “Machine-learning-accelerated Bose-Einstein condensation,” Physical Review Research 4, 043216 (2022).
  * Tranter  _et al._ (2018) A. D. Tranter, H. J. Slatyer, M. R. Hush, A. C. Leung, J. L. Everett, K. V. Paul, P. Vernaz-Gris, P. K. Lam, B. C. Buchler, and G. T. Campbell, “Multiparameter optimisation of a magneto-optical trap using deep learning,” Nature Communications 9, 4360 (2018).
  * Mukherjee  _et al._ (2020a) Rick Mukherjee, Harry Xie, and Florian Mintert, “Bayesian optimal control of GHZ states in Rydberg lattices,” Physical Review Letters 125, 203603 (2020a), arxiv:2005.05802 [physics, physics:quant-ph, stat] .
  * Xie  _et al._ (2022) Yan-Jun Xie, Han-Ning Dai, Zhen-Sheng Yuan, Youjin Deng, Xiaopeng Li, Yu-Ao Chen, and Jian-Wei Pan, “Bayesian learning for optimal control of quantum many-body states in optical lattices,” Physical Review A 106, 013316 (2022).
  * Lagarias  _et al._ (1998) Jeffrey C. Lagarias, James A. Reeds, Margaret H. Wright, and Paul E. Wright, “Convergence Properties of the Nelder–Mead Simplex Method in Low Dimensions,” SIAM Journal on Optimization 9, 112–147 (1998).
  * Das and Suganthan (2011) Swagatam Das and Ponnuthurai Nagaratnam Suganthan, “Differential Evolution: A Survey of the State-of-the-Art,” IEEE Transactions on Evolutionary Computation 15, 4–31 (2011).
  * Laughlin (1983) R. B. Laughlin, “Anomalous Quantum Hall Effect: An Incompressible Quantum Fluid with Fractionally Charged Excitations,” Physical Review Letters 50, 1395–1398 (1983).
  * Broholm  _et al._ (2020) C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, “Quantum spin liquids,” Science 367, eaay0668 (2020).
  * Mazurenko  _et al._ (2017) Anton Mazurenko, Christie S. Chiu, Geoffrey Ji, Maxwell F. Parsons, Márton Kanász-Nagy, Richard Schmidt, Fabian Grusdt, Eugene Demler, Daniel Greif, and Markus Greiner, “A cold-atom Fermi–Hubbard antiferromagnet,” Nature 545, 462–466 (2017).
  * Chiu  _et al._ (2018) Christie S. Chiu, Geoffrey Ji, Anton Mazurenko, Daniel Greif, and Markus Greiner, “Quantum State Engineering of a Hubbard System with Ultracold Fermions,” Physical Review Letters 120, 243201 (2018).
  * Rosi  _et al._ (2013) S. Rosi, A. Bernard, N. Fabbri, L. Fallani, C. Fort, M. Inguscio, T. Calarco, and S. Montangero, “Fast closed-loop optimal control of ultracold atoms in an optical lattice,” Physical Review A 88, 021601 (2013).
  * Sørensen  _et al._ (2019) J. J. Sørensen, J. H. M. Jensen, T. Heinzel, and J. F. Sherson, “QEngine: A C++ library for quantum optimal control of ultracold atoms,” Computer Physics Communications 243, 135–150 (2019).
  * Rasmussen and Williams (2005) Carl Edward Rasmussen and Christopher K. I. Williams,  _Gaussian Processes for Machine Learning_ (The MIT Press, 2005).
  * Note (1) In the context of optimization, we consider an adiabatic protocol to be any strategy aiming to minimize losses of the target-state population by limiting the ramp speed to the maximum set by the inverse of the many-body gap.
  * Tai  _et al._ (2017) M. Eric Tai, Alexander Lukin, Matthew Rispoli, Robert Schittko, Tim Menke, Dan Borgnia, Philipp M. Preiss, Fabian Grusdt, Adam M. Kaufman, and Markus Greiner, “Microscopy of the interacting Harper–Hofstadter model in the two-body limit,” Nature 546, 519–523 (2017).
  * Harper (1955) P. G. Harper, “The General Motion of Conduction Electrons in a Uniform Magnetic Field, with Application to the Diamagnetism of Metals,” Proceedings of the Physical Society. Section A 68, 879 (1955).
  * Hofstadter (1976) Douglas R. Hofstadter, “Energy levels and wave functions of Bloch electrons in rational and irrational magnetic fields,” Physical Review B 14, 2239–2249 (1976).
  * Sträter and Eckardt (2016) Christoph Sträter and André Eckardt, “Interband Heating Processes in a Periodically Driven Optical Lattice,” Zeitschrift für Naturforschung A 71, 909–920 (2016).
  * Mukherjee  _et al._ (2020b) Rick Mukherjee, Frédéric Sauvage, Harry Xie, Robert Löw, and Florian Mintert, “Preparation of ordered states in ultra-cold gases using Bayesian optimization,” New Journal of Physics 22, 075001 (2020b).
  * Palm  _et al._ (2023) Felix A. Palm, Joyce Kwan, Brice Bakkali-Hassani, Markus Greiner, Ulrich Schollwöck, Nathan Goldman, and Fabian Grusdt, “Growing Extended Laughlin States in a Quantum Gas Microscope: A Patchwork Construction,”  (2023), arxiv:2309.17402 [cond-mat] .
  * Srinivas  _et al._ (2012) Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger, “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design,” IEEE Transactions on Information Theory 58, 3250–3265 (2012), arxiv:0912.3995 [cs] .
  * Zhang and Dong (2010) J. M. Zhang and R. X. Dong, “Exact diagonalization: The Bose–Hubbard model as an example,” European Journal of Physics 31, 591 (2010).


# Adaptive Background Music for a Fighting Game: A Multi-Instrument Volume
Modulation Approach

Ibrahim Khan  Graduate School of Information Science and Engineering
Ritsumeikan University
Kusatsu, Shiga, Japan
<EMAIL_ADDRESS>Thai Van Nguyen  Chollakorn Nimpattanavong
Graduate School of Information Science and Engineering
Ritsumeikan University
Kusatsu, Shiga, Japan
<EMAIL_ADDRESS>Graduate School of Information Science and
Engineering
Ritsumeikan University
Kusatsu, Shiga, Japan
<EMAIL_ADDRESS>Ruck Thawonmas  College of Information Science and
Engineering
Ritsumeikan University
Kusatsu, Shiga, Japan
<EMAIL_ADDRESS>
###### Abstract

This paper presents our work to enhance the background music (BGM) in
DareFightingICE by adding an adaptive BGM. The adaptive BGM consists of five
different instruments playing a classical music piece called “Air on
G-String.”. The BGM adapts by changing the volume of the instruments. Each
instrument is connected to a different element of the game. We then run
experiments to evaluate the adaptive BGM by using a deep reinforcement
learning AI that only uses audio as input (Blind DL AI). The results show that
the performance of the Blind DL AI improves while playing with the adaptive
BGM as compared to playing without the adaptive BGM.

###### Index Terms:

adaptive BGM, Rule-based adaptive background music, background music.

##  I Introduction

As the popularity of video games keeps growing, video game developers are
trying to improve their games to leave a positive impression on the players by
enhancing their experience [1]. These improvements are in many different forms
like gameplay-related changes or visually pleasing games or games with very
enjoyable music or sound effects. Regardless of the type of improvement, the
ultimate goal is to enhance the overall experience of the players.

Our work is based on DareFightingICE Competition, which has two tracks: Sound
Design Track and AI Track [2]. It ran at the 2022 IEEE Conference on Games
(CoG 2022). The platform for the competition is DareFightingICE – an enhanced
version of FightingICE [3] – with an enhanced sound design and a new AI
interface that can give audio data to AIs.

This research focuses on video game music and how to use it to enhance
players’ experience. An immersive gaming experience is greatly aided by BGM
[5, 6]. In fighting games, BGM should change in response to the dynamic
action, creating suspense and excitement as the players engage in combat.
Fighting game BGM is typically implemented using pre-composed songs that loop,
which fall short of accurately capturing the intensity of combat.

As a result, the target genre in this research is fighting games. In fighting
games, players go against another player or a computer player in a one-versus-
one fight, using different attacks and abilities to overcome the opponent.
These fighting games are two-and-a-half-dimensional, which means that the
players can only move in two dimensions (left or right, while in some fighting
games the players can perform an attack or dodge that takes them out of 2D).
In addition, because fighting games are fairly simple compared to other genres
of video games, they are open to a larger audience. This research uses
DareFightingICE.

The goal of this research is to create adaptive BGM by modifying the BGM of
DareFightingICE, and evaluating the performance of the adaptive BGM by using a
deep learning AI that only uses audio as input (Blind DL AI) [4]. The
contributions of our work are as follows:

  1. 1.

One we are the first group to focus on adaptive music in fighting games.

  2. 2.

we are also the first group to use multiple instruments in the adaptation of a
BGM.

  3. 3.

our research focuses on giving players information about the state of the game
through the BGM.

##  II Related Work

Since our research touches on two topics which are audio in video games and
adaptive audio in video games, this section is divided into two parts.

###  II-A Sound in Video games

The significance of music in video games has been extensively studied. Video
game music has been proven to have an impact on everyday living, attitude, and
other aspects [7]. A person’s attachment to the music featured in a game can
result in these effects. It has been established that voice-over in video
games and audio dialogue acted by the player or by non-playable characters is
more entertaining for the players [8]. The information given to the game’s
players is also made easier to recall thanks to these voiceovers. A
significant part of the game is also performed by the background soundtrack.
It was found that players did better when there was background music playing
than when there wasn’t any [9].

Similarly, it has been noted that for video games to be engaging for players,
the music performed within them must match the atmosphere or tone of the game
[10]. A poor illustration of this would be to play soothing music as the game
reaches its conclusion. Moving from background music to more focused sound
effects and auditory signals, it has been observed that directional or 3D
sound effects give players more information about their surroundings,
including where other players are located [11, 12]. Steps and other types of
action are represented in these 3D sound elements.

This section’s focus thus far has been on video games’ audio. There has been
very little study on sound designs in video games, which is a change from
sound in video games. A sound design refers to a collection of sound effects
together with its source code for its timing-control algorithm. Even fewer
have produced findings that are helpful in developing an effective sound
design. Despite the lack of study, sound designs for game categories like
first-person shooters and real-time strategy have been suggested [13]. For
their respective game categories, these two sound designs have distinctive
sound effects.

###  II-B Adaptive Audio in Video Games

There has been some research on adaptive music in video games in the past. A
study made an adaptive music system keeping players’ emotions in mind and it
was shown to be better than the original music in the game [14]. Another study
on adaptive music found that players value adaptive music over linear music
[15]. Lastly, another research on adaptive music changed the tempo of the
music by taking into account players’ actions and the state of the game and
found that it enhanced players’ experience by making the game more immersive
or enjoyable [16].

Previous research has shown the importance of both music and adaptive music in
video games. However, there has been little to no research when it comes to
adaptive music in fighting games.

###  II-C DareFightingICE Platforms

The adaptive BGM is created in the aforementioned fighting game platform
DareFightingICE. Since there has been a recent update to the DareFightingICE
platform from version 5.2 (used in the 2022 DarefightingICE Competition) to
version 6.0, we include both versions in our research. The reason is to show
that our adaptive BGM works well on both of them.

####  II-C1 DareFightingICE 5.2

DareFightingICE version 5.2 was the official version of the DareFightingICE
Competition which was first held at CoG 2022. This version was an upgrade from
the FightingICE platform and added an enhanced sound design keeping players
without vision in mind. The platform also provided audio data to AIs, leading
to the new Blind AI Track of the competition.

Version 5.2 added a new function getAudioData to the original interface that
gives audio data at each frame of the game to make it easier for AIs to access
audio data. At each frame, audio data are sampled with a length of 16.67 ms;
notice that DareFightingICE has a frame per second of 60, as in FightingICE.
The stereo sound in the 2D format used in DareFightingICE helps accurately
depict the location of both players. Consequently, a two-channel sound format
is offered to enable AIs to perceive audio that might aid them in locating the
positions of both players and those of projectiles.

####  II-C2 DareFightingICE 6.0

This version of DareFightingICE has revamped the communication interface
between AIs and the game system by using the open-source remote procedure call
gRPC instead of Py4J. This change has resulted in up to 65% reduced latency,
improved stability, and the elimination of missed frames compared to the Py4J
interface [17]. This version will be used in the 2023 DareFightingICE
Competition with another editable source code file called “Play.java” included
for the Sound Design Track. This file is made available to give the users
ability to modify the BGM at run time.

##  III Rule-Based Adaptive BGM

In this research, we propose an adaptive BGM that adapts to player actions and
the players’ in-game position. The proposed adaptive BGM consists of five
different instruments playing a classical music piece called “Air on
G-String.” This music was selected for this research because we found the
contrast of fast-paced actions with calming music to be a good combination, a
technique used in some famous and popular movies such as X-Men: Days of the
Future Past (QuickSilver-stopping-bullets scene). The five different
instruments are the piano, the cello, the flute, the violin, and the ukulele.
The BGM adjusts by altering the loudness of the instruments. This adjustment
happens when the game elements, these instruments are connected to, change.
The elements in use are both players’ health points (HP), energy points (EP),
and the distance between the two players (PD). The HP is the number of hits a
player can take before losing, and the EP is the energy in point that the
players need to perform different attacks. The design of the proposed adaptive
BGM is illustrated in Fig. 1.

Figure 1: Rule-Based Adaptive BGM.

In Fig. 1, there are five instruments playing together to compose the BGM, and
every instrument is connected to a different game element. The violin and the
piano are connected to player one’s HP and EP, respectively; and the flute and
the ukulele are connected to player two’s HP and EP, respectively. Lastly, the
cello is connected to PD.

⬇

For Violin:

if (P1.HP decreases)

Violin.volume.decreaselevel

For Flute:

if (P2.HP decreases)

Flute.volume.decreaselevel

For Cello:

if(Pos.diff decreases)

Cello.volume.increaselevel

Else

Cello.volume.decreaselevel

For Piano:

if(P1.EP decreases)

Piano.volume.decreaselevel

Else

Piano.volume.increaselevel

For Ukulele:

if(P2.EP decreases)

Ukulele.volume.decreaselevel

Else

Ukulele.volume.increaselevel

Figure 2: Adaptive BGM Rules

In Fig. 2, the instruments connected to HP of both players are at maximum
volume when HP is maximum, and the volume is turned down as the HP gets lower.
The ones connected to players’ EP work in the same way, i.e., when EP is low
the volume is low and when EP is high the volume is high. For the cello, the
volume is at maximum when PD is close to zero meaning they are very close to
each other, and it becomes lower the further away the players move from each
other. The proposed adaptive BGM is designed this way to give useful
information to both human and AI players.

The maximum HP for one player per round is 400, EP is 300, and the PD is 800
pixels horizontally. The HP and EP of both players and their PD are divided
into different levels to decrease or increase the volume of the instruments
gradually. The levels of not only instruments but also players’ HP, EP, and PD
are empirically selected due to the lack of existing research when it comes to
adaptive BGM using volume modulation.

The levels for HP are 400, 300, 250, 200, 150, 100, and 50, which are
connected to the instrument’s volume levels of 75%, 60%, 55%, 40%, 35%, 25%,
and 10% respectively. For EP of both players, the instruments’ volume levels
are the same as HP, but EP levels are 300, 250, 200, 150, 100, and 50. For PD,
the levels are 800, 600, 500, 400, 300, 60, and 0, which are connected to the
instrument’s volume levels of 75%, 60%, 50%, 40%, 30%, 20%, and 10%. The
reason for the abrupt change from 300 to 60 is for the players without vision
to know without a doubt that the opponent player is near them.

⬇

public void playBGM() {

play2(this.Violin, getSoundBuffers().get("bach_V.wav"), 350, 0, true);

play2(this.Piano, getSoundBuffers().get("bach_P.wav"), 350, 0, true);

play2(this.Flute, getSoundBuffers().get("bach_F.wav"), 350, 0, true);

play2(this.Ukulele, getSoundBuffers().get("bach_U.wav") , 350, 0, true);

play2(this.Cello, getSoundBuffers().get("bach_C.wav"), 350, 0, true);

}

Figure 3: Adaptive BGM Code Example

Lastly, for the 2023 DareFightingICE Competition participants, if they want to
modify or change the adaptive BGM111code of the adaptive BGM:
https://tinyurl.com/adaptiveBGM. They just have to change the five instruments
playing the BGM and if they want to use fewer than five instruments, they can
decide which rules to ignore and which to select. Figure 3 shows a code
snippet that they can modify. In Fig. 3, the play2() function takes five
parameters as follows:

  1. 1.

The first parameter is the audio source of the instrument.

  2. 2.

The second parameter is the audio buffer that stores the audio file that will
play on the audio source.

  3. 3.

The third parameter is the horizontal location in the game, where the audio
source will play.

  4. 4.

The fourth parameter is the vertical location in the game.

  5. 5.

The last parameter is to check if the audio source should be looped or not.

##  IV Evaluation

For the evaluation of the adaptive BGM, we did an objective evaluation, the
details for which are following.

For the objective evaluation, we use the aforementioned Blind DL AI. Our
method for this evaluation is to train the Blind DL AI with and without the
adaptive BGM and then compare the performance of the AIs. We hypothesize that
if the performance of the Blind DL AI is better with the adaptive BGM then the
adaptive BGM is giving useful information, and since the AI we are using can
only take audio data as input we believe that players without vision will also
be able to use the same information for their advantage.

To test our adaptive BGM we used the submissions to the 2022 DareFightingICE
Competition Sound Design Track and train the Blind DL AI as mentioned above.
We trained the Blind DL AI for 900 rounds on DareFightingICE 5.2 (1 game has 3
rounds) against MCTSAI65, a weaken version of a sample Monte-Carlo [18]. For
DareFightingICE 6.0, we implement a new weak version of MCTS AI, which fixed
the problem of MCTSAI65’s performance difference on different environments,
and call it MCTSAI23i. This version of MCTS AI has similar performance as the
previous MCTSAI65 against our Blind DL AI as shown in Table I. We trained the
Blind DL AI against the MCTSAI23i for 900 rounds for each sound design.

TABLE I: Performance of our blind DL AI against MCTSAI65 and MCTSAI23i. Opponent | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
MCTSAI65 | 1D-CNN | 0.33 | -28.83
MCTSAI65 | FFT | 0.37 | -40.5
MCTSAI65 | Mel-spectrogram | 0.63 | 37.07
MCTSAI23i | 1D-CNN | 0.53 | 12.92
MCTSAI23i | FFT | 0.37 | -30.08
MCTSAI23i | Mel-spectrogram | 0.54 | 28.18

The difference between MCTSAI65 and MCTSAI23i is that for MCTSAI65 the MCTS
execution time was changed to 6.5 ms and for MCTSAI23i we limit the number of
iterations MCTS can do for each frame. We introduce this change because we
found that MCTSAI65 performed differently in different environments.

For the subjective evaluation or evaluation of our adaptive BGM by human
players, since the performance of Blind DL AI was similar to the performance
of human players in previous work [2], we decided to only go with the
objective evaluation. We believe that both players with and without vision
will also be able to use the same information to their advantage.

We used the same environments as in the 2022 DareFightingICE Competition. More
specifically, six computers were used that have the same specification, i.e.,
CPU: Intel(R) Xeon(R) W-2135 CPU@ 3.70GHz 3.70 GHz, RAM: 16 GB, GPU: NVIDIA
Quadro P1000 4GB VRAM, and OS: Windows 10.

###  IV-A Results

We conducted experiments on three different sound designs; The default,
runner-up, and winner sound design of the 2022 DareFightingICE Competition on
both versions of DareFightingICE.

The experiment structure is that the Blind DL AI fights against MCTSAI65 for
version 5.2 and MCTSAI23i for version 6.0 for each sound design, first with
linear BGM and then with the BGM being replaced with the adaptive one. The AI
is trained from scratch for 900 rounds and since the Blind DL AI had three
different audio encoders [3] as an option, the experiment was run on each
encoder per one sound design; one-dimensional Convolutional Neural Network
(1D-CNN), Fast Fourier Transform (FFT), and Mel-Spectrogram (Mel-Spec).

We then evaluate the performance of each trained Blind DL AI by making it
fight against each of the aforementioned opponent AIs for 90 rounds. The ratio
of the number of wins222In the game, the round winner is either the one with a
non-zero HP while its opponent’s HP has reached zero or the one with the
higher HP when the round-length limit of 60 s has reached. over 90 rounds,
Eqn. (1), and the average HP difference at the end of a round between the
trained AI and its opponent, Eqn. (2), are then calculated. The equations and
details above are taken from previous work [4].

$win_{ratio}=\frac{\textit{winning rounds}}{\textit{total rounds}}$ (1)
$avgHP_{diff}=\frac{\textit{sum of }HP_{r}^{self}-HP_{r}^{opp}\textit{ for all
r}}{\textit{total rounds}}$ (2)

TABLE II: Performance of our blind DL AI with default sound design in DareFighingICE 5.2. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Default | 1D-CNN | 0.33 | -28.83
Default | FFT | 0.37 | -40.5
Default | Mel-spectrogram | 0.63 | 37.07
Default Adaptive | 1D-CNN | 0.79 | 81.98
Default Adaptive | FFT | 0.92 | 118.93
Default Adaptive | Mel-spectrogram | 0.87 | 105.61
TABLE III: Performance of our blind DL AI with runner-up sound design in DareFighingICE 5.2. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Runner-up | 1D-CNN | 0.6 | 42.91
Runner-up | FFT | 0.24 | -55.13
Runner-up | Mel-spectrogram | 0.0 | -282.86
Runner-up Adaptive | 1D-CNN | 0.92 | 102.78
Runner-up Adaptive | FFT | 0.81 | 82.04
Runner-up Adaptive | Mel-spectrogram | 0.91 | 121.86
TABLE IV: Performance of our blind DL AI with Winner sound design in DareFighingICE 5.2. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Winner | 1D-CNN | 0.47 | -5.73
Winner | FFT | 0.80 | 75.54
Winner | Mel-spectrogram | 0.0 | -169.41
Winner Adaptive | 1D-CNN | 1 | 142.61
Winner Adaptive | FFT | 0.87 | 112.31
Winner Adaptive | Mel-spectrogram | 0.71 | 68.88
TABLE V: Performance of our blind DL AI with default sound design in DareFighingICE 6.0. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Default | 1D-CNN | 0.53 | 19.92
Default | FFT | 0.37 | -30.08
Default | Mel-spectrogram | 0.54 | 28.18
Default Adaptive | 1D-CNN | 0.59 | 31.54
Default Adaptive | FFT | 0.67 | 29.59
Default Adaptive | Mel-spectrogram | 0.64 | 41.22
TABLE VI: Performance of our blind DL AI with runner-up sound design in DareFighingICE 6.0. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Runner-up | 1D-CNN | 0.56 | 31.68
Runner-up | FFT | 0.64 | 47.73
Runner-up | Mel-spectrogram | 0.60 | 35.02
Runner-up Adaptive | 1D-CNN | 0.68 | 66.12
Runner-up Adaptive | FFT | 0.77 | 98.23
Runner-up Adaptive | Mel-spectrogram | 0.61 | 56.19
TABLE VII: Performance of our blind DL AI with winner sound design in DareFighingICE 6.0. Sound design | Encoder | $win_{ratio}$ | $avgHP_{diff}$
---|---|---|---
Winner | 1D-CNN | 0.54 | 12.66
Winner | FFT | 0.52 | 27.03
Winner | Mel-spectrogram | 0.56 | 34.23
Winner Adaptive | 1D-CNN | 0.73 | 71.91
Winner Adaptive | FFT | 0.66 | 32.03
Winner Adaptive | Mel-spectrogram | 0.65 | 45.62

As shown from the results in Tables II,IV,III,V,VII, and VI, the Blind DL AI
performs better with the adaptive BGM in all sound designs across both
versions of DareFighingICE. For each encoder, Blind DL AI also performs better
with our adaptive BGM. This proves our hypothesis that the adaptive BGM is
giving useful information.

For version 5.2 the best encoder is 1D-CNN for the winner sound design. It is
undefeated in testing. This encoder is also the most stable and consistent
among all other encoders for all the sound designs for version 5.2.

Version 6.0’s best encoder is FFT for the runner-up sound design as it has the
highest performance among all encoders for all sound designs. We call this
encoder best for version 6.0 because it is also consistent in testing.

###  IV-B Behavior of Blind DL AIs

This section describes the behavior333Link to the videos of Blind DL AIs:
https://tinyurl.com/abm4fg. of different Blind DL AIs for different sound
designs in both versions. Only the behavior of Blind DL AIs trained on the
adaptive BGM is described. For each sound design, the encoder with the best
result is chosen. The summary of different behaviors is shown in Table VIII.

Figure 4 shows a sample of different time steps from a fight between the best
Blind DL AI from version 6.0 against MCTSAI23i, and it also shows the change
in volume of different instruments at these time steps.

Figure 4: A series of fight scenes at different time steps and the level of
different instrument volume at these time steps. TABLE VIII: Difference
between the different Blind DL AIs.

Version | Sound Design | Playing Style | Favorite Move(s) | Avoid Projectiles | Lighting Uppercut
---|---|---|---|---|---
5.2 | Default | Moderate | Uppercut | Yes | Not used
5.2 | Runner-up | Very defensive | Crouch punches | No | Not used
5.2 | Winner | Defensive | Crouch kicks | Yes | Used
6.0 | Default | Aggressive | Heavy punch | No | Not used
6.0 | Runner-up | Aggressive | Punch & kick | No | Used
6.0 | Winner | Defensive | Uppercut | No | Rarely used

####  IV-B1 DareFightingICE 5.2

The behavior of the Blind DL AIs trained on DareFightingICE version 5.2 is
described in this section.

Default Sound Design – For the default sound design, the best encoder is FFT.
This Blind DL AI tends to play a moderate role in terms of attack or defense.
It tries to jump to avoid the fireball attack from the opponent AI. It attacks
by jumping and trying to hit the opponent with kicks. The best move this Blind
AI does is that it counters MCTSAI65’s jump attacks by performing the uppercut
move. This uppercut move seems to be the most used move by this AI.

Runner-up Sound Design – For the runner-up sound design, the best encoder is
1D-CNN. This Blind DL AI plays a very defensive style. It tends to not move
from its location much. It lets the opponent come closer and then attacks by
crouching and punching. This combo of crouch punches seems to be the most used
attack by this AI. It seems that through training, the AI has learned that
crouch punching is effective against MCTSAI65. Although it does not try to
avoid the fireball attack from the opponent, it can still overwhelm the
opponent by trapping them in a corner and spamming crouch punches.

Winner Sound Design – For the winner sound design, the best encoder is 1D-CNN.
Like the runner-up sound design AI, this AI tends to play defensively. It does
not move much apart from some jump attacks. Its favorite combination of attack
is crouch kicks as it is the most used attack by the AI. This AI is the
strongest among all of the Blind DL AIs trained. It tries to avoid the
fireball attack by jumping. Also, it counters MCTSAI65’s jumping attack with
its own jumping attack, or if it has enough energy, it performs the lighting
uppercut. This AI is undefeated in testing.

####  IV-B2 DareFightingICE 6.0

The behavior of Blind DL AIs trained on DareFightingICE version 6.0 is
described in this section.

Default Sound Design – For the default sound design in DareFightingICE version
6.0, the best encoder is FFT. This Blind DL AI is the weakest among the
selected AIs. It tends to play a more aggressive role. It attacks by jumping
toward the opponent. The most used attack by this AI is the heavy punch. It
tends to do a quick sit-stand movement repeatedly. It also does not try to
avoid the fireball attack. The uniqueness of this AI is that it performs the
throw attack which lifts the opponent in the air.

Runner-up Sound Design – For the runner-up sound design in DareFightingICE
version 6.0, the best encoder is FFT. This Blind DL AI is the best for
DareFightingICE version 6.0. Its behavior is similar to the default sound
design AI of DareFightingICE version 6.0. The difference is that this AI tends
to attack slowly as compared to the other Blind DL AIs. The most used attack
combination of this AI is a punch followed by a kick. It also does the throw
move and the lighting uppercut move. It does not try to avoid the fireball
attack.

Winner Sound Design – For the winner sound design in DareFightingICE version
6.0, the best encoder is 1D-CNN. This Blind DL AI is also defensive. Its most
used attack is the uppercut. This AI tries to defend itself from MCTSAI23i’s
jump attacks with either uppercuts or jumping punches. It scarcely uses the
lighting uppercut attack. It also does not try to avoid the fireball attack.

##  V Conclusions

This paper presented a rule-based adaptive background music (BGM) system that
consists of five different instruments playing a classical music piece called
“Air on G-String.” The proposed adaptive BGM adapts by changing the volume of
the instruments. Each instrument is connected to a different element of the
game. The paper also showed that the performance of a deep reinforcement
learning AI using only audio data as its input, called Blind DL AI, improved
while playing with the adaptive BGM as compared to playing without it. We
believe that both players with and without vision will also be able to use the
same information given by the proposed adaptive BGM to their advantage, just
like the Blind DL AI does.

In the future, we are planning to use a deep learning approach to the
adaptation of BGM. This is because in the current work, we used the
performance of the Blind DL AI when fighting against a version of a Monte-
Carlo tree search AI to assess our adaptive BGM and we did not do any
aesthetic evaluation of it. We believe that the adaptive BGM using deep
learning would result in achieving a better performance from the Blind DL AI
as compared to the rule-based, as we plan to use the Blind DL AI’s performance
as a part of the reward function. As for aesthetic evaluation, since our
current work is a rule-based adaptive BGM decided the rules taking into
account the resulting BGM’s aesthetics; However, by using a deep learning
approach we anticipate that the aesthetic evaluation of the BGM would be
needed.

## References

  * [1] B. Berg Marklund, H. Engström, M. Hellkvist, and P. Backlund, “What Empirically Based Research Tells Us About Game Development,” in _The Computer Games Journal_ , vol. 8, no. 3-4, pp. 179–198, 2019.
  * [2] I. Khan, T. V. Nguyen, X. Dai, and R. Thawonmas, “DareFightingICE Competition: A Fighting Game Sound Design and AI Competition,” _2022 IEEE Conference on Games (CoG)_ , pp. 478–485, 2022.
  * [3] F. Lu, K. Yamamoto, L. H. Nomura, S. Mizuno, Y. M. Lee, and R. Thawonmas, “Fighting Game Artificial Intelligence Competition Platform,” _2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE)_ , pp. 320–323, Oct. 2013. (https://www.ice.ci.ritsumei.ac.jp/ ftgaic/index.htm)
  * [4] T. Van Nguyen, X. Dai, I. Khan, R. Thawonmas, and H. V. Pham, “A Deep Reinforcement Learning Blind AI in DareFightingICE,” _2022 IEEE Conference on Games (CoG)_ , pp. 632–637, 2022.
  * [5] C. Klimmt, D. Possler, N. May, H. Auge, L. Wanjek, and A.-L. Wolf, “Effects of soundtrack music on the video game experience,” in _Media Psychology_ , vol. 22, no. 5, pp. 689–713, 2018.
  * [6] E. de la Mora Velasco and A. Hirumi, “The effects of background music on learning: A systematic review of literature to guide future research and practice,” _Educational Technology Research and Development_ , vol. 68, no. 6, pp. 2817–2837, 2020.
  * [7] K. Tuuri, O. Koskela, J. Vahlo, and H. Tissari, “Identifying the Impact of Game Music both Within and Beyond Gameplay,” _Entertainment Computing – ICEC 2021_ , pp. 411–418, Nov. 2021.
  * [8] J. Byun, and C. S. Loh, “Audial engagement: Effects of game sound on learner engagement in digital game-based learning environments,” in _Computers in Human Behavior_ , vol.46, pp.129-138, 2015.
  * [9] J. Zhang, and X. Fu, “The Influence of Background Music of Video Games on Immersion,” in _Journal of Psychology and Psychotherapy_ , vol. 5, no. 4, p.1, 2015.
  * [10] F. Andersen, C. L. King, and A. A. Gunawan, “Audio Influence on Game Atmosphere during Various Game Events,” in _Procedia Computer Science_ , vol.179, pp.222-231, 2021.
  * [11] C. Johanson, and R. L. Mandryk, “Scaffolding Player Location Awareness through Audio Cues in First-Person Shooters,” in _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ , pp. 3450-3461, May 2016.
  * [12] P. Ng, and K. Nesbitt, “Informative Sound Design in Video Games,” in _Proceedings of The 9th Australasian Conference on Interactive Entertainment: Matters of Life and Death_ , pp. 1-9, Sep. 2013.
  * [13] Xiaoxu Li, Marcel Wira and Ruck Thawonmas, “Toward Dynamic Difficulty Adjustment with Audio Cues by Gaussian Process Regression in a First-Person Shooter,” _Proc. of the 21st International Conference on Entertainment Computing (ICEC 2022)_ , Bremen, Germany, Nov. 1-3, 2022, published in Lecture Notes in Computer Science, Gobl et al. (Eds.), vol. 13477, pp. 154-161, 2022.
  * [14] P. E. Hutchings and J. McCormack, “Adaptive Music Composition For Games,” _IEEE Transactions on Games_ , vol. 12, no. 3, pp. 270–280, 2020.
  * [15] C. Plut and P. Pasquier, “Music Matters: An empirical study on the effects of adaptive music on experienced and perceived player affect,” _2019 IEEE Conference on Games (CoG)_ , 2019.
  * [16] M. Makhmutov, J. A. Brown, M. Surkov, A. Timchenko, and K. Timchenko, “Adaptive Game Soundtrack Tempo Based on Players’ Actions,” _2022 IEEE Conference on Games (CoG)_ , pp. 441–448, 2022.
  * [17] C. Nimpattanavong, I. Khan, T. V. Nguyen, R. Thawonmas, W. Choensawat, K. Sookhanaphibarn “Improving Data Transfer Efficiency for AIs in the DareFightingICE using gRPC,” _arXiv preprint arXiv:2303.10001_ , 2023 (accepted for oral representation at 2023 8th International Conference on Business and Industrial Research, May 2023)
  * [18] M. Ishihara, T. Miyazaki, C. Y. Chu, T. Harada, and R. Thawonmas, “Applying and Improving Monte-Carlo Tree Search in a Fighting Game AI,” in _Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology_ , pp. 1-6, Nov. 2016.