# Topological melting of the metastable skyrmion lattice in the chiral magnet
Co9Zn9Mn2

Victor Ukleev<EMAIL_ADDRESS>Laboratory for Neutron Scattering and
Imaging (LNS), Paul Scherrer Institute (PSI), CH-5232 Villigen, Switzerland
Helmholtz-Zentrum Berlin für Materialien und Energie, D-14109 Berlin, Germany
Daisuke Morikawa  RIKEN Center for Emergent Matter Science (CEMS), Wako
351-0198, Japan  Institute of Multidisciplinary Research for Advanced
Materials (IMRAM), Tohoku University, Sendai 980-8577, Japan  Kosuke Karube
RIKEN Center for Emergent Matter Science (CEMS), Wako 351-0198, Japan  Akiko
Kikkawa  RIKEN Center for Emergent Matter Science (CEMS), Wako 351-0198, Japan
Kiyou Shibata  RIKEN Center for Emergent Matter Science (CEMS), Wako 351-0198,
Japan  Institute of Industrial Science, The University of Tokyo, Tokyo
153–8505, Japan  Yasujiro Taguchi  RIKEN Center for Emergent Matter Science
(CEMS), Wako 351-0198, Japan  Yoshinori Tokura  RIKEN Center for Emergent
Matter Science (CEMS), Wako 351-0198, Japan  Department of Applied Physics,
University of Tokyo, Tokyo 113-8656, Japan  Tokyo College, University of
Tokyo, Tokyo 113-8656, Japan  Taka-hisa Arima  RIKEN Center for Emergent
Matter Science (CEMS), Wako 351-0198, Japan  Department of Advanced Materials
Science, University of Tokyo, Kashiwa 277-8561, Japan  Jonathan S. White
Laboratory for Neutron Scattering and Imaging (LNS), Paul Scherrer Institute
(PSI), CH-5232 Villigen, Switzerland

###### Abstract

In a $\beta$-Mn-type chiral magnet Co9Zn9Mn2, we demonstrate that the magnetic
field-driven collapse of a room temperature metastable topological skyrmion
lattice passes through a regime described by a partial topological charge
inversion. Using Lorentz transmission electron microscopy, the magnetization
distribution was observed directly as the magnetic field was swept
antiparallel to the original skyrmion core magnetization, i.e. negative
magnetic fields. Due to the topological stability of skyrmions, a direct
transition of the metastable skyrmion lattice to the equilibrium helical state
is avoided for increasingly negative fields. Instead, the metastable skyrmion
lattice gradually transforms into giant magnetic bubbles separated by $2\pi$
domain walls. Eventually these large structures give way to form a near-
homogeneously magnetized medium that unexpectedly hosts a low density of
isolated skyrmions with inverted core magnetization, and thus a total
topological charge of reduced size and opposite sign compared with the initial
state. A similar phenomenon has been observed previously in systems hosting
ordered lattices of magnetic bubbles stabilized by the dipolar interaction and
called “topological melting”. With support from numerical calculations, we
argue that the observed regime of partial topological charge inversion has its
origin in the topological protection of the starting metastable skyrmion
state.

skyrmions, topological transition, Lorentz microscopy, far-from-equilibrium

##  I Introduction

A magnetic skyrmion is a topological object with vortex-like swirling spin
configuration, which is characterized by an integer termed the topological
number:

$n={\tfrac{1}{4\pi}}\int\mathbf{M}\cdot\left(\frac{\partial\mathbf{M}}{\partial
x}\times\frac{\partial\mathbf{M}}{\partial y}\right)dxdy.$ (1)

Here $\mathbf{M}$ is a unit vector describing the direction of a smoothly
varying local magnetization. A skyrmion crystal (SkX) is a periodic, usually
triangular-coordinated arrangement of skyrmion. It is a non-trivial collective
topological magnetic state that is robust against perturbations, such as
magnetic fields and thermal agitation, since it cannot be continuously
transformed into an alternative state with smoothly varying magnetization
which is described by a different topology [1, 2]. Indeed, in real systems the
stability of skyrmions is limited due to the violation of the continuous limit
at the atomic scale, the presence of defects or edges, and temperature
fluctuations. In the past decade, several kinds of magnets have been found to
host skyrmion ($n=-1$) [3, 4, 5, 6, 7, 8], bi-skyrmion ($n=-2$) [9, 10, 11,
12], or anti-skyrmion ($n=+1$) [13] textures. Note that since $n$ reverses its
sign under time reversal, the above topological numbers are given for a common
polarity of the core [14].

Skyrmions can be manipulated by ultra-low density electric currents, making
them promising for spintronics applications [2, 15]. In bulk materials,
skyrmions have been studied extensively in chiral cubic magnets such as the
$B20$-type compounds [3, 4, 16, 17, 18, 5] and $\beta-$Mn-type Co-Zn-Mn alloys
[6, 19, 20, 21, 22, 23, 24]. In these compounds, due to the interplay between
the exchange interaction and the Dzyaloshinskii-Moriya interaction (DMI), a
helical magnetic ordering becomes stable upon cooling below a critical
temperature ($T_{c}$) in zero field. By the application of a moderate magnetic
field just below $T_{c}$, the helical structure transforms into the ordered
triangular SkX. At the mean-field level [25], both the characteristic helical
pitch $\lambda$ and skyrmion size are determined by the ratio of the exchange
stiffness $A_{ex}$ to the Dzyaloshinskii constant $D$, according to
$\lambda=4\pi A_{ex}/D$.

Figure 1: (a) Schematic illustration of the measurement process. On the left
side of (a), the labels (b-k) describes the points in the schematic phase
diagram where the data shown in the corresponding panels were collected. The
middle of (a) show a sketch the field-driven magnetic texture reversal of the
equilibrium phases just below $T_{c}$ [26]. The right side of (a) shows the
crystallographic orientation of the thin plate with respect to the applied
magnetic field before (top panel) and after sample reversal (bottom panel),
respectively (see text for details). (b) Over-focused LTEM images on the (001)
plane after the FC process from 390 K to 290 K at 70 mT. The crystal
orientation and field direction are indicated in (a). (c) LTEM image of the
zero-field SkX. Merged skyrmions are indicated by dashed blue lines. (d – k)
Under-focused LTEM images of the sample after reversal. Each panel corresponds
to the evolution of the magnetic structure upon application of the magnetic
field in the negative direction compared to the original state. Skyrmions with
reversed cores are indicated in panels (i – j) by dashed red circles.The scale
bar shown in (b) is 200 nm. Figure 2: Micromagnetic simulations of the SkX in
a Co9Zn9Mn2 thin plate in (a) zero and negative magnetic fields (b) $B=-280$
mT, (c) $B=-285$ mT, (d) $B=-290$ mT, (e) $B=-295$ mT, and (f) $B=-310$ mT.
Detailed illustrations of the magnetization distributions for the original
zero-field and inverted skyrmions shown in panels (a–f) are given in panels
(g,h), respectively. The in-plane magnetization direction is indicated by the
colors shown in panels (g,h).

In contrast, periodic lattices of magnetic bubbles can be stabilized in thin
films of centrosymmetric magnets with easy axis anisotropy [27, 28, 29]. The
in-plane rotation sense of the magnetization around the bubble core is non-
degenerate, thus leading to a cancellation of the total magnetic chirality and
therefore the topological charge of the bubble lattice [30, 31]. This is
different from skyrmion systems where the magnetic chirality is determined by
the sign of DMI which itself is determined by the handedness either the chiral
crystal, [32, 33] or the interfacial symmetry breaking in magnetic multilayer
systems. The research of magnetic bubble domains has existed since the second
half of the 20th century, leading to many intriguing observations and
theoretical developments. In particular, a coherent inversion of the bubble
lattice in garnet films has been experimentally observed upon reversing the
direction of applied magnetic field and called "topological switching" [34,
35, 28, 36]. An alternative mechanism of magnetic bubble lattice collapse via
the formation of cellular domains was called "topological melting" [37, 38].
Both scenarios were successfully reproduced numerically via a fine-tuning of
short-range ferromagnetic and long-range antiferromagnetic dipolar
interactions [39]. According to a more recent theory that includes the DMI, it
is suggested that the magnetization switching processes of both magnetic
skyrmions in chiral magnets, [14] and films with the interfacial DMI and
perpendicular magnetic anisotropy (PMA), [40] are accompanied by the
nucleation of skyrmion-antiskyrmion or meron-antimeron pairs during the
reversal process [41]. It is also proposed that for increasingly negative
magnetic fields, a defect-free triangular SkX will undergo a first-order
topological phase transition to an inverted skyrmion phase via a regime of
transient anti-skyrmions [40]. This scenario is analogous to the "topological
switching" phenomena in the bubble lattice. In the presence of defects, the
transition becomes smeared so that it appears continuous [40] and thus
ascribed as "topological melting".

Recently, the $\beta$-Mn-type chiral magnet Co9Zn9Mn2, with Curie temperature
$T_{c}=400$ K, was shown to exhibit a robust metastable SkX state at room
temperature after field-cooling (FC) through the equilibrium SkX phase that is
stable just below $T_{c}$ [20]. At room temperature, the practically infinite
relaxation time of this metastable state makes this compound very promising
for certain applications, and a model system for investigating the topological
stability of metastable magnetic skyrmions. Indeed the enhanced stability of
the metastable SkX in Co9Zn9Mn2 at room temperature allows for the
investigation of the field-induced transformations of the skyrmion spin
texture with a reduced influence of the thermal fluctuation effect [42]. Here
we exploit the robust metastable room-temperature SkX in thin plate samples of
Co9Zn9Mn2 to address the magnetic field-driven magnetization reversal process
experimentally, and investigate the "topological switching" or "topological
melting" scenarios.

##  II Results and discussion

Figure 3: Magnified view of the magnetization patterns obtained from
micromagnetic simulations in (a) positive field $B=+225$ mT, (b) zero field,
and negative fields (c) $B=-275$ mT, (d) $B=-285$ mT, (e) $B=-290$ mT, (f)
$B=-295$ mT, (g) $B=-300$ mT, and (h) $B=-310$ mT. Figure 4: Plot of (a) the
total topological charge $Q$, and (b) the total energy of the system against
external negative field (i.e. a field anti-parallel to the cores of the
skyrmions initially stabilised in the case of SkX inversion) derived the
micromagnetic simulations. The cases of SkX inversion and SkX nucleation from
the helical phase are indicated respectively in each panel by blue and red
curves.

A schematic illustration of both the magnetic phase diagram of Co9Zn9Mn2 and
the measurement protocol are given in Figure 1a. A sketch of the magnetic
texture evolution upon magnetic field reversal just below $T_{c}$ is shown
schematically in the middle of the panel. In this case, the thermal
equilibrium SkX transitions to the helical structure through the merging of
skyrmions firstly into elongated bands, followed by the coherent nucleation of
a hexagonal lattice of the reversed skyrmions. This scenario has been
confirmed for a 100 nm-thick plate of FeGe at $T=240$ K [26]. We note that in
the current setup, it was not possible to test the same protocol of SkX
reversal in the thermal equilibrium SkX phase of Co9Zn9Mn2 at $T\sim 370$ K,
[19] since the magnetic field direction provided by the objective lens of the
TEM could not be reversed. Therefore, the metastability and significantly long
lifetime of the room-temperature and zero-field skyrmions in Co9Zn9Mn2 was of
a critical importance.

The main experimental finding of the present work is shown in Figures 1b–k.
Initially, a triangular-lattice SkX state with a lattice constant
$a_{Sk}\approx 150$ nm is formed as the equilibrium state at approximately 370
K and 70 mT [20]. Upon FC the sample in $B=70$ mT from the equilibrium SkX
phase to room temperature (290 K), a robust metastable SkX is achieved which
serves as our starting state (Figure 1b). As discussed previously for a bulk
sample of Co9Zn9Mn2 [20], the metastable SkX at 290 K persists even after the
magnetic field is reduced to zero. We find however, that already at zero field
the expansion of skyrmion cores and the merging of some skyrmions into larger
structures indicated by dashed blue lines in Figure 1c. These effects lead to
a reduction of the overall topological charge (i.e. number of skyrmions).
Notably, no transformation from the SkX to either a helical state [26, 43] or
meron-antimeron lattice [44] is observed in weak magnetic fields.

Since the magnetic field direction provided by the objective lens of the TEM
cannot be reversed, the investigation of the negative field evolution of the
magnetization distribution was done by removing the sample from the TEM at
zero field, and turning it upside-down before re-installing it back into the
TEM. The defocus distances for the corresponding LTEM images before and after
the $180^{\circ}$ reversal of the sample were reset from positive (+288
$\mu$m) to negative (-288 $\mu$m) values (Figure 1d), and show almost the same
magnetic contrast at zero field. Note that after the sample rotation, the
applied external magnetic field is in the same direction in the laboratory
frame of reference, i.e. parallel to the skyrmion core magnetization of the
starting metastable state. Hereafter, we denote the original direction of the
field during FC as positive, and the field applied after the sample reversal
as negative.

By increasing the magnitude of the field in the negative direction, skyrmions
continue to merge with each other and the total topological charge decreases
further (Figure 1e). Upon sweeping the magnetic field from 0 to -50 mT, the
merging of skyrmions results in the creation and inflation of large skyrmion
bubbles with a size of $5-10a_{Sk}$ (Figure 1e). This phenomenon is consistent
with a theoretically predicted scenario in which the SkX transforms into
highly unstable large hexagonal cells, which eventually elongate into spiral
states [45].

On further increment of the negative magnetic field the bubbles expand, and we
observe a peculiar web of skyrmions and bubbles connected to each other by
$2\pi$ domain wall strings (Figure 1g-h). In this configuration, the total
topological charge still has the same sign as the original SkX state, though
the magnitude is much smaller. Skyrmion core reversal occurs for negative
magnetic fields below -100 mT (Figure 1i-j). The inversion of the skyrmion
core magnetization is accompanied by the reversal of the rotation sense of the
in-plane magnetization, which shows up experimentally by a change in LTEM
contrast: skyrmions with core magnetizations aligned antiparallel to the
magnetic field appear as white dots (indicated by dashed red circles in
Figures 1i-j). Interestingly, the reversed skyrmion core is much smaller than
the original one at 0 T, and its profile changes from hexagonal to circular.
This can be understood as a field-driven core collapse, similar to a previous
observation of the metastable SkX in FeGe under a high positive magnetic field
[46]. This core collapse behavior has been predicted generally to take place
in chiral magnets with uniaxial anisotropy, [1] and is also observed in a
skyrmion-hosting monolayer system [47]. Crucially, in the reversed state the
total topological charge $Q$ (number of reversed skyrmions) has changed sign,
while the magnitude evaluated for the presently studied viewing region of the
sample is $\sim 1\%$ of its original value. Finally, all skyrmions are
destroyed by a magnetic field of $B=-461$ mT and the sample crosses over to a
conical or induced ferromagnetic state (Figure 1k).

An interesting question concerns the detailed magnetic structure of the
inverted skyrmions and, especially, the $2\pi$ domain walls appearing upon the
magnetization reversal. In principle this could be determined by means of a
transport-of-intensity equation (TIE) analysis when the timescale of the
magnetic dynamics are slower than that of the relevant LTEM measurements.
Unfortunately, due to the metastable character of the observed textures, and,
consequently, their dynamic behaviour at the timescale of tens of minutes, it
was not possible to measure the series of images with different defocus
distances required for a TIE analysis. However, this might be possible in
future experiments at lower temperatures where the lifetime of metastable
states exceeds the time scale of the experiment.

Next we describe how the experimentally observed negative magnetic field-
induced collapse of the SkX agrees with the results of Landau-Lifshitz-Gilbert
(LLG) simulations. Our calculations were performed using the MuMax3 package
[48], and appropriate magnetic parameters for a Co9Zn9Mn2 thin plate [49]. The
details of the micromagnetic simulation and the simulation code are given in
the Supplementary materials. The topological charge on a discrete lattice was
quantified using the definition of Berg and Lüscher [50].

To follow the experimental FC protocol, we started the simulation with a
random initial magnetization and applied a magnetic field in the positive
direction. After 2 ns of relaxation, the magnetic field was removed and the
resultant zero-field SkX had a negative field imposed. The magnetization of
the system in negative magnetic fields was recorded every 5 mT with a
relaxation time of 2 ns between the steps. The results are shown in Figure 2.
Although the magnitudes of the magnetic fields in the simulation differ from
the experimental values, the process of the skyrmion collapse is qualitatively
reproduced. The discrepancy between the measured and simulated negative
magnetic field ranges of skyrmion stability can originate from thermal
fluctuations, the presence of defect sites, or the role of demagnetization
fields in the real experiment.

Consistent with our real-space observations, the simulations show that the
collapse process of the SkX starts from elongated (merged) skyrmions that are
already present in the sample at zero field (Figures 2a–c). Furthermore, the
expansion of skyrmion cores and their merging behavior results in the
formation of large skyrmion bubbles (Figure 2c). The annihilation of these
bubbles is accompanied by a contraction of $2\pi$ domain walls that separate
the inside part of the bubble from the outside ferromagnetic medium (Figure
2d). These quasi-one-dimensional topological domain walls tend to collapse
into small skyrmions with core magnetizations anti-aligned with both the
applied field and the original skyrmion core magnetization (Figure 2e).
Eventually, the domain walls vanish, and only a low density of core-inverted
skyrmions is observed for sufficiently negative magnetic fields (Figure 2f).
The in-plane magnetization direction of the skyrmions in Figures 2a–f is given
by the colors shown in Figures 2g,h. A magnified schematic showing the
reversal of the skyrmion internal structure in more detail is shown in Figures
3a-h. Recent theoretical [51] and experimental [52] works have identified the
large change in skyrmion size under positive magnetic field, and formation of
the large patch-like structures for negative magnetic fields, as signatures of
predominantly dipolar-stabilised spin textures in magnetic thin films and
multilayers with the interfacial DMI. We note that in Co9Zn9Mn2 the spiral
pitch and the skyrmion size are independent from the sample dimensions, and
show similar periodicities in bulk crystals and a few-hundred nanometer thick
plates [19]. Therefore, while the dipolar interaction may impact the behaviour
of the SkX under positive or negative magnetic fields, the ground state
periodicity $\lambda$ is determined by the dominant DM and exchange
interactions.

For comparison, we applied the same protocol to determine the total
topological charge and system energy for equilibrium SkX nucleation from a
starting zero-field helical state. In this case, the magnetization evolves
from a single-domain helix to helicoidal structure, and finally, to a a SkX
pattern upon increasing the field.

The field-dependence of the calculated topological charge for the processes of
both SkX inversion and SkX nucleation from helical order are shown in Figure
4a. The field region $300<|B|<420$ mT over which the SkX inversion takes place
is highlighted.. The simulation results for SkX collapse are qualitatively
consistent with the LTEM data: the gradual expansion of the metastable
skyrmions takes place upon sweeping the field from 0 to $\sim 290$ mT,
followed by inversion of the total topological charge $Q$ from negative to
positive. Experimentally, we find the magnitude of topological charge reversal
to be $~{}1\%$ which is in good agreement with a reversed skyrmion fraction of
2.8% suggested by the simulation. Surprisingly, we find SkX nucleation from
the helical phase requires a much higher negative magnetic field than that
required for the inversion of metastable skyrmions (Figure 4a). A small hump
in the magnetic field dependence of the total $Q$ during the nucleation
process at $|B|=280-320$ mT (see Figure S1 in the Supplementary material [53])
is possibly due to the annihilation of meron defects and single helicoids
followed by the creation of isolated skyrmions. At higher field, skyrmions
emerge coherently from the rest of the helicoidal texture. The absolute values
of the final topological charge in the SkX inversion and SkX nucleation
scenarios are not the same due to the difference in the initial number of
skyrmions in the starting condition (random spin orientation vs helical
pattern). Figure 4b shows the magnetic field dependence of the total system
energy. In the whole field range from zero to $|B|\approx 300$ mT the helical
(helicoidal) state is energetically favorable, thus emphasizing the metastable
character of the starting SkX considered for the SkX collapse process.
Crucially, we find the system achieves an energy gain through the
stabilization of skyrmions with inverted core magnetizations. This underlines
the energetic advantage of the direct skyrmion inversion scenario compared to
a SkX to helicoid transition under the application of negative magnetic
fields. Interestingly, this energy gain is not reproduced in a simulation with
of a sample of smaller dimension (see Supplementary material [53]) due to
finite-size effects and the formation of skyrmion bags in this case. A similar
phenomena was observed in a 150 nm-thick FeGe thin plate at low temperature,
when field-cooled metastable skyrmions were packed into a bundle by a field
cycling procedure [54].

Overall, our experimental results are consistent with a field-driven, second-
order "topological melting" transition of the metastable SkX in the presence
of defects described in Ref. 40. In contrast to the "topological switching"
phenomena mediated by anti-skyrmions, the melting scenario does not include
the conservation of the total topological charge in the system $|Q|=|n|\cdot
N_{sk}$, where $n$ is the topological number as defined in the Equation 1, and
$N_{sk}$ is the number of skyrmions in the sample. As the total number of
skyrmions $N_{sk}$ is reduced in the melting process to $\sim 1\%$ of its
original value, only partial topological charge inversion takes place (Figure
4a). The described process of the SkX inversion is qualitatively similar to
the classical process of magnetic bubble domain array reversal in PMA systems
without DMI, also known as "topological melting" [37, 38, 39].

In Ref. 40 the presence of defects was found essential to induce skyrmion
melting process, where the defect site acts as a "topological-charge sink".
Note, that our simulations did not imply any artificially introduced defects,
such as randomly distributed non-magnetic sites or regions with pinned
magnetization or anisotropy axes. Instead, the distorted SkX as seen in Figure
3a is formed by the relaxation of random magnetization pattern in an applied
field (see Supplementary materials for simulation details). The packing
defects of this lattice are similar to the ones observed in the experimental
data (Figure 1)b and assist the "topological melting" scenario.

##  III Conclusions

Taken together, our LTEM experiments and micromagnetic simulations clearly
demonstrate for the first time a successful inversion of the topological
charge in a thin plate of a chiral magnet. The negative field-driven
transformation of the SkX is conceivably due to the enhanced energetic
stability (so-called topological protection) of zero-field skyrmions at room
temperature compared with the thermodynamically stable helical phase. This
scenario contrasts strongly with the direct skyrmion-to-helical conversion
reported for metastable skyrmions in FeGe and Fe0.5Co0.5Si [26, 43, 55]. Our
micromagnetic simulations of both the skyrmion inversion and skyrmion
nucleation processes qualitatively explain our results from the point of view
of the energetics. The time resolution of present LTEM observation was not
sufficient to describe this process dynamically and identify anti-skyrmion
creation that possibly accompanies the skyrmion merging process [14, 40]. This
issue should be further addressed by ultrafast pump-probe LTEM and x-ray
imaging studies [56, 57, 58]. Finally, we have shown that the negative field-
driven merging of metastable skyrmions leads to the formation of large
skyrmion bubbles separated by $2\pi$ domain walls. The realization of such
large skyrmion bubbles in Co9Zn9Mn2 is furthermore a promising step towards
the engineering of novel chiral magnetic particles with any integer
topological charge ("skyrmion bags" and "skyrmion bundles" ) considered in the
theory of Refs. [59, 60], and recently realized experimentally in FeGe [54].
In this instance, it is expected that smaller skyrmions can be nucleated
inside the large bubbles by electric current [61], optical and x-ray light
pulses [56, 62, 63] or a scanning tunneling microscope tip [64], thus
providing new and novel means for topological charge modulation and control.

##  IV Experimental Section

### Samples

The details of sample preparation and the structure of the magnetic phase
diagrams in both bulk and thin plate samples are given in Ref. 20.

### Real-space imaging

The real-space imaging of the negative magnetic field-driven collapse of the
metastable SkX in a $\sim 150$ nm-thick Co9Zn9Mn2 plate was carried out by
means of Lorentz transmission electron microscopy (LTEM). Measurements were
performed with a transmission electron microscope (JEM-2100F) at an
acceleration voltage of 200 kV, and the defocus distances of $\pm 288$ $\mu$m
as described in the main text.

## Supplementary Information

Supporting Information is available from the Wiley Online Library at the link
[53] or from the author.

## Acknowledgements

This research was supported in part by JSPS Grant-in-Aids for Scientific
Research (Grant No. 20K15164). V.U. and J.S.W. acknowledge funding from the
SNSF Sinergia CRSII5_171003 NanoSkyrmionics.

## References

  * Bogdanov and Hubert [1994] A. Bogdanov and A. Hubert, Thermodynamically stable magnetic vortex states in magnetic crystals, Journal of Magnetism and Magnetic Materials 138, 255 (1994).
  * Nagaosa and Tokura [2013] N. Nagaosa and Y. Tokura, Topological properties and dynamics of magnetic skyrmions, Nature Nanotechnology 8, 899 (2013).
  * Mühlbauer  _et al._ [2009] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion lattice in a chiral magnet, Science 323, 915 (2009).
  * Yu  _et al._ [2011] XZ. Yu, N. Kanazawa, Y. Onose, K. Kimoto, W. Zhang, S. Ishiwata, Y. Matsui, and Y. Tokura, Near room-temperature formation of a skyrmion crystal in thin-films of the helimagnet fege, Nature Materials 10, 106 (2011).
  * Seki  _et al._ [2012] S. Seki, XZ. Yu, S. Ishiwata, and Y. Tokura, Observation of skyrmions in a multiferroic material, Science 336, 198 (2012).
  * Tokunaga  _et al._ [2015] Y. Tokunaga, XZ. Yu, J. White, H. M. Rønnow, D. Morikawa, Y. Taguchi, and Y. Tokura, A new class of chiral materials hosting magnetic skyrmions beyond room temperature, Nature Communications 6, 7638 (2015).
  * Kézsmárki  _et al._ [2015] I. Kézsmárki, S. Bordács, P. Milde, E. Neuber, L. Eng, J. White, H. M. Rønnow, C. Dewhurst, M. Mochizuki, K. Yanai, _et al._ , Néel-type skyrmion lattice with confined orientation in the polar magnetic semiconductor gav 4 s 8, Nature Materials 14, 1116 (2015).
  * Kurumaji  _et al._ [2017] T. Kurumaji, T. Nakajima, V. Ukleev, A. Feoktystov, T.-h. Arima, K. Kakurai, and Y. Tokura, Néel-type skyrmion lattice in the tetragonal polar magnet vose 2 o 5, Physical Review Letters 119, 237201 (2017).
  * Yu  _et al._ [2014] XZ. Yu, Y. Tokunaga, Y. Kaneko, W. Zhang, K. Kimoto, Y. Matsui, Y. Taguchi, and Y. Tokura, Biskyrmion states and their current-driven motion in a layered manganite, Nature Communications 5, 3198 (2014).
  * Lee  _et al._ [2016] J. T. Lee, J. Chess, S. Montoya, X. Shi, N. Tamura, S. Mishra, P. Fischer, B. McMorran, S. Sinha, E. Fullerton, _et al._ , Synthesizing skyrmion bound pairs in fe-gd thin films, Applied Physics Letters 109, 022402 (2016).
  * Wang  _et al._ [2016] W. Wang, Y. Zhang, G. Xu, L. Peng, B. Ding, Y. Wang, Z. Hou, X. Zhang, X. Li, E. Liu, _et al._ , A centrosymmetric hexagonal magnet with superstable biskyrmion magnetic nanodomains in a wide temperature range of 100–340 k, Advanced Materials 28, 6887 (2016).
  * Takagi  _et al._ [2018] R. Takagi, XZ. Yu, J. White, K. Shibata, Y. Kaneko, G. Tatara, H. Rønnow, Y. Tokura, and S. Seki, Low-field bi-skyrmion formation in a noncentrosymmetric chimney ladder ferromagnet, Physical Review Letters 120, 037203 (2018).
  * Nayak  _et al._ [2017] A. K. Nayak, V. Kumar, T. Ma, P. Werner, E. Pippel, R. Sahoo, F. Damay, U. K. Rößler, C. Felser, and S. S. Parkin, Magnetic antiskyrmions above room temperature in tetragonal heusler materials, Nature 548, 561 (2017).
  * Koshibae and Nagaosa [2016] W. Koshibae and N. Nagaosa, Theory of antiskyrmions in magnets, Nature Communications 7, 10542 (2016).
  * Fert  _et al._ [2017] A. Fert, N. Reyren, and V. Cros, Magnetic skyrmions: advances in physics and potential applications, Nature Reviews Materials 2, 17031 (2017).
  * Adams  _et al._ [2012] T. Adams, A. Chacon, M. Wagner, A. Bauer, G. Brandl, B. Pedersen, H. Berger, P. Lemmens, and C. Pfleiderer, Long-wavelength helimagnetic order and skyrmion lattice phase in cu 2 oseo 3, Physical Review Letters 108, 237204 (2012).
  * Münzer  _et al._ [2010] W. Münzer, A. Neubauer, T. Adams, S. Mühlbauer, C. Franz, F. Jonietz, R. Georgii, P. Böni, B. Pedersen, M. Schmidt, _et al._ , Skyrmion lattice in the doped semiconductor fe 1- x co x si, Physical Review B 81, 041203 (2010).
  * Kanazawa  _et al._ [2012] N. Kanazawa, J.-H. Kim, D. Inosov, J. White, N. Egetenmeyer, J. Gavilano, S. Ishiwata, Y. Onose, T.-h. Arima, B. Keimer, _et al._ , Possible skyrmion-lattice ground state in the b 20 chiral-lattice magnet mnge as seen via small-angle neutron scattering, Physical Review B 86, 134425 (2012).
  * Karube  _et al._ [2016] K. Karube, J. White, N. Reynolds, J. Gavilano, H. Oike, A. Kikkawa, F. Kagawa, Y. Tokunaga, H. M. Rønnow, Y. Tokura, _et al._ , Robust metastable skyrmions and their triangular–square lattice structural transition in a high-temperature chiral magnet, Nature Materials 15, 1237 (2016).
  * Karube  _et al._ [2017] K. Karube, J. White, D. Morikawa, M. Bartkowiak, A. Kikkawa, Y. Tokunaga, T.-h. Arima, H. Rønnow, Y. Tokura, and Y. Taguchi, Skyrmion formation in a bulk chiral magnet at zero magnetic field and above room temperature, Physical Review Materials 1, 074405 (2017).
  * Morikawa  _et al._ [2017] D. Morikawa, XZ. Yu, K. Karube, Y. Tokunaga, Y. Taguchi, T.-h. Arima, and Y. Tokura, Deformation of topologically-protected supercooled skyrmions in a thin plate of chiral magnet co8zn8mn4, Nano Letters 17, 1637 (2017).
  * Karube  _et al._ [2018] K. Karube, J. S. White, D. Morikawa, C. D. Dewhurst, R. Cubitt, A. Kikkawa, XZ. Yu, Y. Tokunaga, T.-h. Arima, H. M. Rønnow, _et al._ , Disordered skyrmion phase stabilized by magnetic frustration in a chiral magnet, Science Advances 4, eaar7043 (2018).
  * Ukleev  _et al._ [2019] V. Ukleev, Y. Yamasaki, D. Morikawa, K. Karube, K. Shibata, Y. Tokunaga, Y. Okamura, K. Amemiya, M. Valvidares, H. Nakao, _et al._ , Element-specific soft x-ray spectroscopy, scattering, and imaging studies of the skyrmion-hosting compound co 8 zn 8 mn 4, Physical Review B 99, 144408 (2019).
  * Nagase  _et al._ [2021] T. Nagase, Y.-G. So, H. Yasui, T. Ishida, H. K. Yoshida, Y. Tanaka, K. Saitoh, N. Ikarashi, Y. Kawaguchi, M. Kuwahara, _et al._ , Observation of domain wall bimerons in chiral magnets, Nature Communications 12, 1 (2021).
  * Bak and Jensen [1980] P. Bak and M. H. Jensen, Theory of helical magnetic structures and phase transitions in mnsi and fege, Journal of Physics C: Solid State Physics 13, L881 (1980).
  * Kovács  _et al._ [2016] A. Kovács, Z.-A. Li, K. Shibata, and R. E. Dunin-Borkowski, Lorentz microscopy and off-axis electron holography of magnetic skyrmions in fege, Resolution and Discovery 1, 2 (2016).
  * Thiele [1969] A. Thiele, The theory of cylindrical magnetic domains, The Bell System Technical Journal 48, 3287 (1969).
  * Malozemoff and Papworth [1975] A. Malozemoff and K. Papworth, High-speed photography of topological switching and anisotropic saturation velocities in a misoriented garnet film, Journal of Physics D: Applied Physics 8, 1149 (1975).
  * Bogatyrëv and Metlov [2018] A. B. Bogatyrëv and K. L. Metlov, What makes magnetic skyrmions different from magnetic bubbles?, Journal of Magnetism and Magnetic Materials 465, 743 (2018).
  * Yu  _et al._ [2017a] XZ. Yu, Y. Tokunaga, Y. Taguchi, and Y. Tokura, Variation of topology in magnetic bubbles in a colossal magnetoresistive manganite, Advanced Materials 29, 1603958 (2017a).
  * Loudon  _et al._ [2019] J. C. Loudon, A. C. Twitchett-Harrison, D. Cortés-Ortuño, M. T. Birch, L. A. Turnbull, A. Štefančič, F. Y. Ogrin, E. O. Burgos-Parra, N. Bukin, A. Laurenson, _et al._ , Do images of biskyrmions show type-ii bubbles?, Advanced Materials 31, 1806598 (2019).
  * Grigoriev  _et al._ [2009] S. Grigoriev, D. Chernyshov, V. Dyadkin, V. Dmitriev, S. Maleyev, E. Moskvin, D. Menzel, J. Schoenes, and H. Eckerlebe, Crystal handedness and spin helix chirality in fe 1- x co x si, Physical Review Letters 102, 037204 (2009).
  * Morikawa  _et al._ [2013] D. Morikawa, K. Shibata, N. Kanazawa, XZ. Yu, and Y. Tokura, Crystal chirality and skyrmion helicity in MnSi and (Fe,Co)Si as determined by transmission electron microscopy, Physical Review B 88, 024408 (2013).
  * O’Dell [1973] T. O’Dell, The dynamics of magnetic bubble domain arrays, Philosophical Magazine 27, 595 (1973).
  * Papworth [1974] K. Papworth, The topological switching of magnetic bubble domain arrays, IEEE Transactions on Magnetics 10, 638 (1974).
  * Randoshkin  _et al._ [1978] V. Randoshkin, L. Ivanov, and R. Telesnin, „dynamics of domains in iron garnet films in a homogeneous magnetic field”, Sov. Phys. JETP 48, 486 (1978).
  * Babcock and Westervelt [1989] K. Babcock and R. Westervelt, Topological “melting”of cellular domain lattices in magnetic garnet films, Physical Review Letters 63, 175 (1989).
  * Zablotskii and Mamalui [1995] V. Zablotskii and Y. A. Mamalui, Ordering and" melting" of domain lattices in thin magnetic films, Journal of Physics: Condensed Matter 7, 5271 (1995).
  * Jagla [2004] E. A. Jagla, Numerical simulations of two-dimensional magnetic domain patterns, Physical Review E 70, 046204 (2004).
  * Pierobon  _et al._ [2018] L. Pierobon, C. Moutafis, Y. Li, J. Löffler, and M. Charilaou, Collective antiskyrmion-mediated phase transition and defect-induced melting in chiral magnetic films., Scientific Reports 8, 16675 (2018).
  * Heo  _et al._ [2016] C. Heo, N. S. Kiselev, A. K. Nandy, S. Blügel, and T. Rasing, Switching of chiral magnetic skyrmions by picosecond magnetic field pulses via transient topological states, Scientific Reports 6, 27146 (2016).
  * Wild  _et al._ [2017] J. Wild, T. N. Meier, S. Pöllath, M. Kronseder, A. Bauer, A. Chacon, M. Halder, M. Schowalter, A. Rosenauer, J. Zweck, _et al._ , Entropy-limited topological protection of skyrmions, Science Advances 3, e1701704 (2017).
  * Peng  _et al._ [2018] L. Peng, Y. Zhang, L. Ke, T.-H. Kim, Q. Zheng, J. Yan, X.-G. Zhang, Y. Gao, S. Wang, J. Cai, _et al._ , Relaxation dynamics of zero-field skyrmions over a wide temperature range, Nano Letters 18, 7777 (2018).
  * Yu  _et al._ [2018a] XZ. Yu, W. Koshibae, Y. Tokunaga, K. Shibata, Y. Taguchi, N. Nagaosa, and Y. Tokura, Transformation between meron and skyrmion topological spin textures in a chiral magnet, Nature 564, 95 (2018a).
  * McGrouther  _et al._ [2016] D. McGrouther, R. Lamb, M. Krajnak, S. McFadzean, S. McVitie, R. Stamps, A. Leonov, A. Bogdanov, and Y. Togawa, Internal structure of hexagonal skyrmion lattices in cubic helimagnets, New Journal of Physics 18, 095004 (2016).
  * Yu  _et al._ [2018b] XZ. Yu, D. Morikawa, T. Yokouchi, K. Shibata, N. Kanazawa, F. Kagawa, T.-h. Arima, and Y. Tokura, Aggregation and collapse dynamics of skyrmions in a non-equilibrium state, Nature Physics 14, 832 (2018b).
  * Romming  _et al._ [2015] N. Romming, A. Kubetzka, C. Hanneken, K. von Bergmann, and R. Wiesendanger, Field-dependent size and shape of single magnetic skyrmions, Physical Review Letters 114, 177203 (2015).
  * Vansteenkiste  _et al._ [2014] A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. Van Waeyenberge, The design and verification of mumax3, AIP Advances 4, 107133 (2014).
  * Takagi  _et al._ [2017] R. Takagi, D. Morikawa, K. Karube, N. Kanazawa, K. Shibata, G. Tatara, Y. Tokunaga, T.-h. Arima, Y. Taguchi, Y. Tokura, _et al._ , Spin-wave spectroscopy of the dzyaloshinskii-moriya interaction in room-temperature chiral magnets hosting skyrmions, Physical Review B 95, 220406 (2017).
  * Berg and Lüscher [1981] B. Berg and M. Lüscher, Definition and statistical distributions of a topological number in the lattice o (3) $\sigma$-model, Nuclear Physics B 190, 412 (1981).
  * Büttner  _et al._ [2018] F. Büttner, I. Lemesh, and G. S. Beach, Theory of isolated magnetic skyrmions: From fundamentals to room temperature applications, Scientific Reports 8, 1 (2018).
  * Birch  _et al._ [2022] M. Birch, L. Powalla, S. Wintz, O. Hovorka, K. Litzius, J. Loudon, L. Turnbull, V. Nehruji, K. Son, C. Bubeck, _et al._ , History-dependent domain and skyrmion formation in 2d van der waals magnet fe3gete2, Nature Communications 13, 1 (2022).
  * sup [2022] See Supplemental Material at https://… for more details on micromagnetic simulation. (2022).
  * Tang  _et al._ [2021] J. Tang, Y. Wu, W. Wang, L. Kong, B. Lv, W. Wei, J. Zang, M. Tian, and H. Du, Magnetic skyrmion bundles and their current-driven dynamics, Nature Nanotechnology 16, 1086 (2021).
  * Milde  _et al._ [2013] P. Milde, D. Köhler, J. Seidel, L. Eng, A. Bauer, A. Chacon, J. Kindervater, S. Mühlbauer, C. Pfleiderer, S. Buhrandt, _et al._ , Unwinding of a skyrmion lattice by magnetic monopoles, Science 340, 1076 (2013).
  * Berruto  _et al._ [2018] G. Berruto, I. Madan, Y. Murooka, G. Vanacore, E. Pomarico, J. Rajeswari, R. Lamb, P. Huang, A. Kruchkov, Y. Togawa, _et al._ , Laser-induced skyrmion writing and erasing in an ultrafast cryo-lorentz transmission electron microscope, Physical Review Letters 120, 117201 (2018).
  * Büttner  _et al._ [2015] F. Büttner, C. Moutafis, M. Schneider, B. Krüger, C. Günther, J. Geilhufe, C. v. K. Schmising, J. Mohanty, B. Pfau, S. Schaffert, _et al._ , Dynamics and inertia of skyrmionic spin structures, Nature Physics 11, 225 (2015).
  * Litzius  _et al._ [2017] K. Litzius, I. Lemesh, B. Krüger, P. Bassirian, L. Caretta, K. Richter, F. Büttner, K. Sato, O. A. Tretiakov, J. Förster, _et al._ , Skyrmion hall effect revealed by direct time-resolved x-ray microscopy, Nature Physics 13, 170 (2017).
  * Rybakov and Kiselev [2019] F. N. Rybakov and N. S. Kiselev, Chiral magnetic skyrmions with arbitrary topological charge, Physical Review B 99, 064437 (2019).
  * Foster  _et al._ [2019] D. Foster, C. Kind, P. J. Ackerman, J.-S. B. Tai, M. R. Dennis, and I. I. Smalyukh, Two-dimensional skyrmion bags in liquid crystals and ferromagnets, Nature Physics , 1 (2019).
  * Yu  _et al._ [2017b] XZ. Yu, D. Morikawa, Y. Tokunaga, M. Kubota, T. Kurumaji, H. Oike, M. Nakamura, F. Kagawa, Y. Taguchi, T.-h. Arima, _et al._ , Current-induced nucleation and annihilation of magnetic skyrmions at room temperature in a chiral magnet, Advanced Materials 29, 1606178 (2017b).
  * Je  _et al._ [2018] S.-G. Je, P. Vallobra, T. Srivastava, J.-C. Rojas-Sánchez, T. H. Pham, M. Hehn, G. Malinowski, C. Baraduc, S. Auffret, G. Gaudin, _et al._ , Creation of magnetic skyrmion bubble lattices by ultrafast laser in ultrathin films, Nano Letters 18, 7362 (2018).
  * Guang  _et al._ [2020] Y. Guang, I. Bykova, Y. Liu, G. Yu, E. Goering, M. Weigand, J. Gräfe, S. K. Kim, J. Zhang, H. Zhang, _et al._ , Creating zero-field skyrmions in exchange-biased multilayers through x-ray illumination, Nature Communications 11, 1 (2020).
  * Wieser  _et al._ [2017] R. Wieser, R. Shindou, and X. Xie, Manipulation of magnetic skyrmions with a scanning tunneling microscope, Physical Review B 95, 064417 (2017).


ifaamas [AAMAS ’23]Proc. of the 22nd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2023)May 29 – June 2, 2023 London, United
KingdomA. Ricci, W. Yeoh, N. Agmon, B. An (eds.) 2023 2023 136 Instadeep Ltd
Johannesburg South Africa The University of the Witwatersrand Johannesburg
South Africa

# The challenge of redundancy on multi-agent value factorisation

Extended Abstract

Siddarth Singh<EMAIL_ADDRESS>and  Benjamin Rosman
<EMAIL_ADDRESS>
###### Abstract.

In the field of cooperative multi-agent reinforcement learning (MARL), the
standard paradigm is the use of centralised training and decentralised
execution where a central critic conditions the policies of the cooperative
agents based on a central state. It has been shown, that in cases with large
numbers of redundant agents these methods become less effective. In a more
general case, there is likely to be a larger number of agents in an
environment than is required to solve the task. These redundant agents reduce
performance by enlarging the dimensionality of both the state space and and
increasing the size of the joint policy used to solve the environment. We
propose leveraging layerwise relevance propagation (LRP) to instead separate
the learning of the joint value function and generation of local reward
signals and create a new MARL algorithm: relevance decomposition network
(RDN). We find that although the performance of both baselines VDN and Qmix
degrades with the number of redundant agents, RDN is unaffected.

###### Key words and phrases:

Machine Learning, Reinforcement Learning, Multi-Agent Reinforcement Learning,
Multi-Agent Systems

##  1\. Introduction

For most multi-agent tasks in a practical setting, we would not know the
precise number of agents required to optimally solve the problem. In general,
there is likely to be a larger number of agents in an environment than is
required to solve the task. As the number of independent agents increases so
would the size of the state space that most problems require.

In complex environments constructing an accurate ground truth representation
becomes difficult and in practical applications, the state representation data
collected is often noisy or incomplete. Given the limitations of this space,
it is not reliable to assume that the state space is reliable. When there is a
large number of agents, many of them may be redundant for achieving an optimal
policy. These redundant agents exacerbate the issue of the state space growth
in the multi-agent setting. Therefore it is important to develop algorithms
that can effectively separate agents that are essential to solving a task from
the agents that are redundant to allow MARL algorithms to be deployed into
more realistic and eventually real-world challenges.

We consider the idea of a collaborative task with a small margin of error like
the traditional Piano Movers problem schwartz1983piano . However, we consider
the case where only $n$ of $m$ total agents in the environment are required to
effectively complete the task. In the Piano Movers problem we therefore
consider the case where an arbitrarily large number of agents are required to
re-position the piano with each agent occupying a fixed amount of space in the
environment along with the piano. Realistically with a large number of agents
only a small subgroup $n$ of $m$ will be required to solve the task under the
joint optimal policy. Ultimately during training, the policy will update to
minimise the reliance on certain agents on the overall outcome as the joint
optimal policy only require that they do not act in a manner that is
destructive to the actions of the smaller group of required agents. In the
Qatten qatten  paper the idea is put forward that both VDN and Qmix exhibit
poor performance in environments with a high number of redundant agents as it
is difficult to assign credit for task completion accurately when a
disproportionately high credit is assigned to only a small number of the
agents.

In this paper, we propose a method to resolve this problem, relevance
decomposition network (RDN) which makes use of layerwise relevance propagation
(LRP) LRP  as an alternative to learned value decomposition only using local
agent observations. By not using learned decomposition we can separate the
learning of the joint value function from the training of the independent
agents and make full use of the relationships between the local observations
of the agents, their identities and their actions at each timestep to
decompose the relationship between the global and local rewards.

##  2\. Relevance Decomposition Network (RDN)

We propose RDN to perform credit assignment between ad-hoc agents in the
cooperative multi-agent setting under the assumption of a linear relationship
between the individual local rewards and the shared global reward. However,
unlike most central training decentralised execution (CTDE) methods which use
learned decomposition as an end-to-end system, RDN separates the learning of
the global reward function from the local Q-values of the agents MARL_OVERVIEW
.

The main motivation for RDN is that in difficult settings, where agents are
not all similarly responsible for the total reward, Qmix and VDN see decreased
performance Qmix VDN . In these cases to achieve accurate reward assignment,
some agents must be weighted much higher than others when performing value
decomposition qatten . In the case of many redundant agents, Qmix and VDN have
difficulties decomposing the relationship between the high and low-value
agents SMAC . Due to this, the decomposed rewards have high variance during
training which makes discovering stable policies difficult. LRP is more
effective at determining this relationship, which stabilises the training
process.

For RDN our independent agents are modelled as Deep Q Networks (DQNs) DQN .
These DQNs gather data where $Q^{i}(o_{t,i,.})$ is the $Q$ value of
independent agent $i$ at timestep $t$, $h^{i}_{t}$ is the hidden state of
agent $i$ at timestep $t$, $o_{t,i,.}$ is the local observation of agent $i$
at timestep $t$,$a_{t,i}$ is the action taken by agent $i$ at timestep $t$ and
$\pi_{i}(Q^{i}(o_{t,i,.}),\epsilon(e))$ is the policy $\pi$ of agent $i$
dictated by a Q value function and an $\epsilon-greedy$ exploration strategy.
We use the critic network to calculate the expected total Q value at each
timestep parameterised by $\theta^{c}$ and a target Q value using a target
critic parameterised by $\tilde{\theta^{c}}$ whose parameters are copied over
from the critic at fixed intervals. The critic network is updated using loss
$L(\theta^{c})=E_{\vv{o},\vv{a},r,\vv{o}^{\prime}}[(Q^{\theta^{c}}_{tot}(o_{1},...,o_{n},a_{1},...,a_{n})-y)^{2}]$
where
$y=r+\gamma(Q^{{\theta^{\prime}}^{c}}_{tot}(o^{{}^{\prime}}_{1},...,o^{{}^{\prime}}_{n},a^{{}^{\prime}}_{1},...,a^{{}^{\prime}}_{n}))$
and $\theta^{c}$ is the critic’s parameters and $\theta^{{}^{\prime}c}$ is the
target critic parameters, which are reset every $C$ training epochs. To train
the agents, we calculate the Q value at the current timestep for each
independent agent. This is done in the same manner as standard independent
Q-learning where we calculate $Q(o,a_{i})$ for each agent based on their local
observations and actions. We calculate the total target Q-value for the
current timestep $t$ using a separate critic network. This critic takes in the
concatenation of local agent observations and actions to predict the target
total Q-value of the global state at the current timestep then we use LRP to
decompose the global reward calculated into local rewards for each agent.
Essentially we assign relevance scores to all data points in the concatenated
observation space and then separate the scores for each index based on the
agent they were collected or generated from. The concatenated observation
space is a concatenation of the local agent observations and the actions of
each agent at each timestep. The relevance scores for each agent are then
summed per agent and are used as the target Q-values to train the independent
learners as if performing standard independent Q learning. Finally, the total
expected Q value for each timestep is decomposed into independent target
Q-values $\tilde{\theta^{i}}$ and the DRQNs drqn  which act as the agent
networks are trained using the loss
$L(\theta^{i})=E_{\vv{o},\vv{a},r,\vv{o}^{\prime}}[(Q^{i,\theta^{i}}(o_{i},a_{i})-\tilde{Q}^{i})^{2}]$
sandholm1995multiagent  and $\tilde{Q}^{i}=\sum_{i}R_{in}$ where
$\tilde{Q}^{i}$ is the $Q$ target for agent $i$ and $\sum_{i}R_{in}$ is the
sum of all relevance values associated with agent $i$.

A characteristic of LRP is it maintains a conservative calculation between
layers of the neural network LRP . As such the relevance values from later
layers are included completely in the calculation of the relevance values of
the layers closer to the input. Essentially we can assume that the total sum
of the relevance values is approximately the same as the output of the NN when
LRP is used. Therefore we can equate $Q_{tot}$ to the sum of relevance scores
as $Q_{tot}\approx R_{in}$.

##  3\. Results

The pre-existing map from the Starcraft Multi-Agent Challenge (SMAC) we make
use of is the bane_vs_bane map. In this map, each side has 20 zerglings and 4
banelings. The most optimal policy for this environment has the zerglings move
out of the way to not obstruct the banelings’ movement. We make use of 3
additional variants of the original map. bane_med which only has 15 zerglings,
bane_small with has 10 zerglings and  bane_no_z with has no zerglings. We vary
the number of redundant agents (the zerglings) to show the effect of
redundancy on performance.

From figure 1 we can see that across all maps RDN outperforms all baselines
although the performance of VDN improves as the number of redundant agents is
reduced. Interestingly QMIX is outperformed by QMIX_NS which does not use the
global state in bane_med and in  bane _no_z indicating that in cases where
there are an intermediate number of redundant agents the central state already
begins to become uninformative making accurate multi-agent credit assignment
difficult.

In the case where all redundant agents are removed VDN performs similarly to
RDN however, we also find that when no redundant agents are present at all in
the  bane _no_z map performance decreased across all algorithms when compared
to bane_small. We suspect that having a small number of extra agents may aid
in exploration early in the training regime.

Figure 1. Percentage winrates from highest number of redundant agents (left)
to least redundant agents (right) for all tested algorithms

##  4\. Conclusion

We show how increasing numbers of redundant agents makes reaching stable
convergence to an optimal joint policy difficult for both monotonic
factorisation methods like QMIX and linear factorisation methods like VDN
where only $n$ of $m$ total agents are required for an environment to be
solved. We then propose RDN as a method that uses LRP to perform more optimal
credit assignments in environments with high numbers of redundant agents using
only local agent observation. RDN can reach near-optimal convergence on all
environments used with similar overall winrates without the use of the ground
truth state information. With VDN and QMIX we see a gradual decay in
performance and, an increase in variance as the number of redundant agents
increases.

## References

  * (1) J. T. Schwartz and M. Sharir, “On the “piano movers’” problem i. the case of a two-dimensional rigid polygonal body moving amidst polygonal barriers,” _Communications on pure and applied mathematics_ , vol. 36, no. 3, pp. 345–398, 1983.
  * (2) Y. Yang, J. Hao, B. Liao, K. Shao, G. Chen, W. Liu, and H. Tang, “Qatten: A general framework for cooperative multiagent reinforcement learning,” _arXiv preprint arXiv:2002.03939_ , 2020.
  * (3) G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R. Müller, “Layer-wise relevance propagation: an overview,” _Explainable AI: interpreting, explaining and visualizing deep learning_ , pp. 193–209, 2019.
  * (4) K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _Handbook of Reinforcement Learning and Control_ , pp. 321–384, 2021.
  * (5) T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. N. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in _ICML_ , 2018.
  * (6) P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel, “Value-decomposition networks for cooperative multi-agent learning based on team reward,” in _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , ser. AAMAS ’18. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2018, p. 2085–2087.
  * (7) M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson, “The StarCraft Multi-Agent Challenge,” _CoRR_ , vol. abs/1902.04043, 2019\.
  * (8) S. M. Schulze C., “Vizdoom: Drqn with prioritized experience replay, double-q learning and snapshot ensembling,” in _Intelligent Systems Conference 2018 (IntelliSys 2018)_. London: Springer, Cham, 6-7 Sept 2018.
  * (9) M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” _arXiv preprint arXiv:1507.06527_ , 2015.
  * (10) T. W. Sandholm and R. H. Crites, “On multiagent q-learning in a semi-competitive domain,” in _International Joint Conference on Artificial Intelligence_. Springer, 1995, pp. 191–205.