# Metallic conductivity on Na-deficient structural domain walls in the spin-
orbit Mott insulator Na2IrO3

Franziska A. Breitner<EMAIL_ADDRESS>Julian Kaiser
Anton Jesche  Philipp Gegenwart<EMAIL_ADDRESS>Experimental Physics VI, Center for Electronic Correlations and Magnetism,
University of Augsburg, Universitätsstr. 1, 86159 Augsburg, Germany

###### Abstract

Honeycomb Na2IrO3 is a prototype spin-orbit Mott insulator and Kitaev magnet.
We report a combined structural and electrical resistivity study of Na2IrO3
single crystals. Laue back-scattering diffraction indicates twinning with $\pm
120^{\circ}$ rotation around the $c^{*}$-axis while scanning electron
microscopy displays nanothin lines parallel to all three b-axis orientations
of twin domains. Energy dispersive x-ray analysis line-scans across such
domain walls indicate no change of the Ir signal intensity, i.e. intact
honeycomb layers, while the Na intensity is reduced down to $\sim 2/3$ of its
original value at the domain walls, implying significant hole doping.
Utilizing focused-ion-beam micro-sectioning, the temperature dependence of the
electrical resistance of individual domain walls is studied. It demonstrates
the tuning through the metal-insulator transition into a correlated-metal
ground state by increasing hole doping.

††preprint: APS/123-QED

##  I Introduction

Spin-orbit (SO) coupling describes the interaction between the spin and the
orbital motion of electrons. In condensed matter composed of light elements,
SO coupling is very weak and often neglected. However, for heavy atoms with
high nuclear charge, a substantial magnetic field in the reference frame of
moving electrons couples to their spins and SO coupling is the driving source
for novel quantum states, such as topologically protected conducting surface
states in topological insulators (TIs) [1]. Established TIs, based on
semiconductors with s- and p-orbitals, are well described by the band theory
of non-interacting electrons. Challenging for theory and experiment are
various topological phases in highly correlated electron systems [2, 3, 4, 5].

Iridates are promising in this respect: due to the large atomic mass of
iridium ions, a strong SO coupling of order 0.4 eV has been found [6, 7, 8],
reaching the typical size of the Coulomb repulsion for 5d electrons. For the
5d5 electrons of Ir4+ ions in an octahedral crystal electric field the strong
SO coupling breaks the sixfold degeneracy of the t2g states leading to filled
$j_{\rm eff}=3/2$ and half-filled $j_{\rm eff}=1/2$ bands. Since the latter
band is narrow, even a small Coulomb repulsion could then open a Mott gap
leading to a $j_{\rm eff}=1/2$ SO Mott insulator. The corresponding wave
function is different from that of ordinary $s=1/2$ Mott insulators as it
contains an equal mixture of xy, yz and zx orbitals with mixed spin up and
down states [6]. One of the wave function’s components is purely complex.
Thus, 5d electrons will acquire a complex phase upon hopping via oxygen
p-orbitals. The experimental proof for the complex $j_{\rm eff}=1/2$ SO Mott
state has been obtained by resonant inelastic x-ray scattering [9]. The
complex transfer integral for the hopping could lead, in suitable crystal
structures, to topologically nontrivial states. Such scenarios have been
discussed for pyrochlore and honeycomb iridates [2, 10, 4]. The $j_{\rm
eff}=1/2$ spins in Na2IrO3 also give rise to a bond-dependent magnetic
exchange that can be described by an extended Kitaev model [11]. Though these
moments display a zigzag antiferromagnetic order at $\sim 15$ K [12, 13, 14],
there is clear evidence for the presence of a dominating Kitaev interaction
[15, 16, 17, 18] at elevated temperatures and excitation energy.

Tuning SO Mott insulators by charge carrier doping into the metallic state
seems interesting and promising. It is motivated, for instance, by the analogy
between the perovskite $j_{\rm eff}=1/2$ SO Mott insulator Sr2IrO4 and the
isostructural $s=1/2$ Mott insulator La2CuO4, parent compound for high-$T_{c}$
superconductivity. While angle-resolved photoemission spectroscopy (ARPES) and
scanning tunneling microscopy (STM) indeed recorded signatures of a pseudogap
in electron-doped (Sr1-xLax)2IrO4 [19, 20], a fully metallic or
superconducting electrical resistance behavior has not yet been reported for
doped or pressurized Sr2IrO4. The $j_{\rm eff}=1/2$ SO Mott state in honeycomb
Na2IrO3 has been confirmed by the analysis of resonant inelastic x-ray
scattering spectra [21] with charge gap of 0.34 eV, determined by ARPES and
optical conductivity [22]. While Shitade et al. treated Na2IrO3 in the weak
correlation limit as layered quantum spin Hall insulator [10], it is now
clear, that strong correlations are needed to account for its magnetism [17]
and the size of the charge gap [22]. In the absence of a clear hierarchy of
the energy scales for SO coupling, Coulomb repulsion, one-electron hopping and
trigonal crystal field splitting, the band structure of Na2IrO3 has been
proposed to sensitively depend on small structural variations and quantum
phase transitions, separating trivial normal, metallic and topological phases
as function of hopping strength were proposed earlier [23]. Even the
possibility of topological spin-triplet superconductivity in charge carrier
doped honeycomb iridates was considered theoretically [24].

While the bulk electrical resistance of Na2IrO3 displays three-dimensional
variable-range hopping of carriers localized by disorder [12, 25, 26], similar
as in other iridates, different experiments suggested the possibility of
enhancing the conductivity at the surface. An ARPES study reported an in-gap
feature hinting at a metallic surface state [27], that was later attributed to
dispersive states approaching the Fermi level in the Na terminated cleavage
planes [28]. Indeed atomic resolution STM found different Na-deficient surface
reconstructions [29]. Careful analysis of tunneling spectra at differently
reconstructed surfaces revealed defect states in the bulk gap, as well as a
V-shaped bandgap closing that was associated to surface states [30]. A reduced
and temperature independent electrical resitivity, measured under UHV
conditions on a freshly cleaved crystal, was related to such a surface
conductivity channel [30]. On the other hand, argon plasma etching the surface
of Na2IrO3 crystals after cleaving in air revealed first-order density-wave-
like transitions in the electrical resistance around $\sim 220$ K and a Fermi
liquid behavior below 20 K [31].

Motivated by these hints at a high tunability of the electronic properties we
decided to perform a combined structural and electrical resistivity study on
Na2IrO3. We started with carefully searching for possible structural domains.
Indeed, Laue backscattering diffraction indicates three different orientations
of the honeycomb lattice with mutual $\pm 120^{\circ}$ rotations.
Interestingly, these domain boundaries are seen as straight long white lines
by scanning electron microscopy and feature a significant Na deficiency.
Focused-ion-beam (FIB) micro-sectioning of the domain walls allowed us to
study their electrical resistance behavior. We find a clear tendency that
increasing Na deficiency reduces the resistance and finally induces a
completely metallic state down to lowest temperatures, indicating that Na2IrO3
has been tuned locally across the metal-insulator transition by hole doping.

##  II Methods

High quality single crystals of Na2IrO3 of several mm diameter were grown via
solid state reaction as described in [12]. For Laue back-scattering
measurements a Philips PW1830/40 x-ray generator with tungsten anode and
Photonic Science detector were utilized. The diffraction patterns were
recorded with voltages of 10, 15 and 20 kV and a current of 30 mA. An exposure
time of 10 minutes was deemed sufficient, as prolonged exposure did not yield
significant improvement of the diffraction patterns for crystals of good
quality. Attempts were made to further improve the quality of the measured
Laue patterns by polishing or cleaving the samples. However, no significant
improvement was achieved. Data analysis of the Laue patterns was performed
using the Crystal Maker software Single Crystal as well as our own program
which allows to consider additive overlap of reflections as well as
reflections due to proximity to the characteristic wavelengths of tungsten. A
scanning electron microscope (SEM) Merlin Gemini 2 from Zeiss, equipped with
an energy dispersive x-ray analysis (EDX) probe with X-Max 80N SDD detector
from Oxford Instruments was utilized for structural and compositional
investigation of cleaved Na2IrO3 surfaces. Silver epoxy was used to mount the
crystals onto the sample holder. The dual beam FIB-SEM Crossbeam 550 from
Zeiss was used for microstructuring and electrically contacting small surface
areas for electrical resistivity measurements. The measurements were performed
between 2 and 300 K in the Dynacool PPMS with ETO option from Quantum Design.
All measurements were done with an excitation current of $3\cdot 10^{-5}$ mA.

##  III Results and Discussion

###  III.1 Structural domains in Na2IrO3

As the crystal structure is relevant for the following discussion of our Laue
data, we start with a brief overview. Na2IrO3 crystallizes in a monoclinic
C2/m structure with one Ir and three Na positions which are listed in Tab. 1.
The lattice constants are $a=5.427(1)$ Å, $b=9.395(1)$ Å and $c=5.614(1)$ Å
[12, 32]. Due to its layered structure, which can be seen in Fig. 1(a),
consisting of alternating Na and NaIr2O6 layers, Na2IrO3 is prone to stacking
faults. Within the $ab$-plane, as shown in Fig. 1(b), the Iridium atoms form a
honeycomb structure where each Ir atom is surrounded by six Oxygen atoms in
octahedral configuration and each honeycomb is filled by one Na atom.

Na2IrO3 crystals grow as thin platelets with the $c^{*}$-axis perpendicular to
the $a$-$b$-plane surface. The honeycomb structure displayed in Fig. 1(c) has
almost perfect threefold rotational symmetry around the $c^{*}$-axis. It is
slightly distorted due to trigonal distortions of the individual IrO6
octahedra which change the Ir-O-Ir bond angles from the ideal value of 90∘ to
somewhere in the range of 98-99.4∘ [13]. This results in a variation of the
Ir-Ir distances ranging from 3.130 to 3.138 Å.

Figure 1:  Crystal structure of Na2IrO3. Na atoms are colored yellow, Ir atoms blue and O atoms red. (a) shows the layered structure perpendicular to the $c$-axis which consists of alternating Na and NaIr2O6 layers. The honeycomb structure within the $ab$-plane is depicted in (b). (c) depicts the honeycomb structure for three orientations differing by rotation of $\pm 120^{\circ}$ $a^{*}_{0}$ (black), $a^{*}_{+}$ (blue) and $a^{*}_{-}$ (magenta). Due to the slight difference in Ir-Ir-distances (given in Å [13]), there is no perfect overlap of the Ir atoms for the three orientations. atom | site | $x$ | $y$ | $z$ |  $U$Å2)
---|---|---|---|---|---
Ir | 4$g$ | 0.5 | 0.167(1) | 0 | 0.001(1)
Na1 | 2$a$ | 0 | 0 | 0 | 0.001(6)
Na2 | 2$d$ | 0.5 | 0 | 0.5 | 0.009(7)
Na3 | 4$h$ | 0.5 | 0.340(2) | 0.5 | 0.009(6)
O1 | 8$j$ | 0.748(6) | 0.178(2) | 0.789(6) | 0.001(6)
O2 | 4$i$ | 0.711(7) | 0 | 0.204(7) | 0.001(7)
Table 1:  Structural parameters of Na2IrO3 obtained from single-crystal x-ray
data at 300 K [13].

Fig. 2(a) displays a representative measured Laue pattern of one of our
crystals. After background subtraction it is first analyzed using the
simulation program Single Crystal. The pattern matches reasonably well all
three different orientations with mutual $\pm 120^{\circ}$ rotations around
the $c^{*}$-axis, one being displayed in panel (b). These three orientations
shall from now on be referred to as $a^{*}_{0}$ ($0^{\circ}$), $a^{*}_{+}$
(rotation of +120∘) and $a^{*}_{-}$ (rotation of $-120^{\circ}$). Thus, at
first glance, there appears to be no obvious preferred orientation.

Figure 2:  (a) Laue pattern of Na2IrO3 recorded with an acceleration voltage
of 20kV after background subtraction. (b) The simulated pattern is shown for
one of three possible orientations ($a^{*}_{0}$).

Taking a closer look at the reflections 1-6 depicted in Fig. 3(a), it becomes
apparent that not all reflections can be described by the simulation of just
one of the three orientations shown in Fig. 3(b). Here, neither reflection 5
nor 6 is matched by the simulation. One possible explanation for this would be
that the detector sensitivity was underestimated. Indeed, a higher sensitivity
yields a match for all observed reflections, as shown in Fig. 3(c), where the
previously unexplained reflections can now be indexed as
$\overline{6}\,\overline{2}\,\overline{13}$ and
$\overline{6}\,2\,\overline{13}$. However, several additional reflections
appear in the simulation many of which are not detectable in the recorded Laue
pattern. Furthermore, while the simulated reflections disappear for a lower
acceleration voltage of 15 kV, both reflections 5 and 6 remain visible in the
corresponding Laue pattern (see Fig. 3(d)). Thus, another explanation is
needed.

Figure 3:  Zoomed-in section of the previously depicted Laue pattern recorded
at 20kV (a). The relevant reflections are labeled 1-6 in order to be able to
distinguish them in the following discussion. (b) shows the simulated
reflections (red dots) for one direction ($a^{*}_{0}$), (c) those for a
simulation with higher detector sensitivity. Here, all detected reflections
can be matched, but the simulation also shows several additional signatures
which were not detected in the measurement. (d) is the Laue pattern recorded
at 15 kV along with the corresponding simulation. Both reflections 5 and 6 are
still visible in the measurement yet have disappeared in the simulation. The
simulation for simultaneous directions $a^{*}_{0}$, $a^{*}_{+}$ and
$a^{*}_{-}$ (red, blue and magenta) in (e) shows a match for all reflections.

Rotating the honeycomb around 120∘ in either direction (blue or magenta in
Fig. 1(c)) leads in some of the overlapping Ir-Ir-bonds to a mismatch in
length. Looking at the individual Laue patterns for all three directions, this
distortion results in most simulated reflections along (h 0 l) overlapping for
$a^{*}_{0}$, $a^{*}_{+}$ and $a^{*}_{-}$. However, this is not the case for
the (0 k l) direction. This leads to clearly visible reflections along (h 0 l)
whereas those for (0 k l) appear smeared. Reflections where $a^{*}_{0}$,
$a^{*}_{+}$ and $a^{*}_{-}$ overlap would show comparatively higher
intensities than expected if only one orientation would be present. Indeed,
the simultaneous existence of $a^{*}_{0}$, $a^{*}_{+}$ and $a^{*}_{-}$ offers
the best match for the recorded data so far as can be seen in Fig. 3(e). Here,
reflection 5 is explained by the 2 $\overline{4}$ $\overline{8}$ reflection of
$a^{*}_{+}$ and reflection 6 by the 2 4 $\overline{8}$ reflection of
$a^{*}_{-}$. For signature 1 the reflections for all directions overlap, which
explains the increased brightness of this reflection.

Looking again at the reflections $\overline{3}\overline{1}\overline{6}$,
$\overline{3}1\overline{6}$, $2\overline{4}\overline{8}$ and $24\overline{8}$,
we find all of them disappear for an acceleration voltage of 10 kV (not
displayed). For 15 kV, however, the reflections are still detectable. The
wavelength of these reflections is close to that for the L${}_{\beta_{1}}$ and
L${}_{\beta_{2}}$ characteristic lines of tungsten [33] which can be detected
for voltages above 12.1 kV (LI absorption edge) and thus are absent at lower
voltages.

Furthermore, as the intensity for the characteristic lines is far higher than
that of the bremsspectrum, reflections in the immediate vicinity of those
lines might also be visible, while other reflections disappear in the
background noise.

As the software Single Crystal does not have the capacity for pattern
simulation taking into account the above mentioned issues, a self written
program considering the characteristic lines as well as the additive overlap
of individual reflections was utilized for simulation of such a Laue pattern.
Here, we assume a direct proportionality of the intensity to the area of the
reflections. Only the characteristic lines for L${}_{\alpha_{1}}$,
L${}_{\beta_{1}}$ and L${}_{\beta_{2}}$ are taken into consideration, as we
found those to be the only ones significantly contributing to the diffraction
pattern. Furthermore, only those reflections for which the wavelength deviates
less than $\Delta\lambda=0.02\,\mathrm{\AA}$ were assigned to the
characteristic lines. Looking at the resulting Laue pattern, shown in Fig. 4,
where both additive overlap and proximity to the characteristic lines of
tungsten are taken into consideration for the simulation, we find this to be a
good match for all detected reflections. Additionally, here the analysis was
done separately for both detector halves in order to correct the slight
mismatch of both pattern halves stemming from the detector halves not being
aligned perfectly.

Figure 4:  Simulation of the Laue pattern of Na2IrO3 showing reflections
visible due to additive overlap in blue, due to proximity to the
characteristic lines of tungsten in red and reflections where both occur in
magenta. The simulation data was adjusted separately for both detector halves
to allow for an optimal result.

For Li2IrO3, another member of the honeycomb iridates, stacking faults are
attributed to occasional in-plane shifts of the layers by $\pm\vec{b}/3$ [13,
34]. While, however, for a domain containing only a small number of layers, a
rotation by 120∘ yields results similar to those for a translation by
$\pm\vec{b}/3$, for larger domains the results for translation and rotation
deviate. Thus, to the best of our knowledge, attributing the Laue reflections
to the simultaneous presence of three crystal orientations with 120∘ rotation
offers the best explanation, not only explaining the position of reflections
but also the increased brightness of some reflections as well as the smearing
of those reflections attributed to the (0 k l) direction.

###  III.2 Imaging domain walls by EDX

Figure 5:  SEM image of the surface of two samples of Na2IrO3 recorded using a
material contrast sensitive detector. The insets on the right shows the
orientation of the $b$-axes. While sample 5 (a) shows a more disordered
arrangement, the lines in sample 4 (b) are straight and parallel. Despite
these obvious differences, the Na deficient lines appear to be roughly
oriented along the $b$-axes determined by Laue analysis for both samples. As
Laue measurements were performed before EDX, the depicted samples have been
exposed to air for several hours at this point. The dark spots in (a) as well
as the bright dots in (b) are likely due to some contamination on the surface
or small cracks which can create charging effects appearing bright in the
image. This was also observed on other samples previously exposed to air, but
is not found on fresh surfaces. Figure 6:  Elemental distribution of Na
determined via EDX for a single line using different magnifications on sample
1. The mapping clearly shows the occurrence of Na deficiency in the lines
appearing bright in the SEM image. Here green areas indicate a Na percentage
which is diminished compared to the red areas. A line scan of (c) is depicted
on the right side to give a quantification of the Na deficiency.

SEM images recorded using a material contrast sensitive NTS-BS (Nano
Technology Systems - Backscattered Electrons) detector show a series of
brighter lines on the surface with a width in the range of up to $2~{}\mu$m,
as shown in Fig. 5. These lines can be observed on untreated or polished
crystals as well as on freshly cleaved surfaces and could also be found using
other detectors, such as HE-SE2 (High Efficiency Secondary Electron) or AsB
(Angular selective Backscattered electron)), where they appeared, however, far
less prominent. While some of the lines are arranged parallel to each other,
others are rotated by either 60∘ or 120∘. Comparing the lines to the
orientation of the crystal previously determined by Laue, many lines seem to
be running along at least one of the three $b$-axes. While some samples show
more irregularly distributed and overall less straight lines, a rough
assignment to the $b$-axes remains possible for all investigated crystals for
at least one direction. Through thorough investigation of the surface, we
exclude topographical features such as steps, cracks or depositions as the
origin of these lines. Combining SEM and Laue data we thus propose to assign
these lines to structural domain walls separating differently oriented domains
of Na2IrO3. Note that in case of (Na1-xLi${}_{x})_{2}$IrO3 for $x>0.25$ in the
phase separated regime, also line structures were found in EDX, which however,
can be associated with Li2IrO3 [35].

Fig. 6 shows an enlarged image of these lines along with elemental
distribution maps for sodium. Here green areas show a lower Na concentration
compared to red ones. A linescan taken perpendicular to one of the lines
clearly shows a dip in Na concentration which coincides with the brighter
region in the SEM image.

STM measurements previously showed Na2IrO3 to be Na deficient on the surface
with up to 2/3 of the Na atoms missing [29]. However, this method is limited
to the surface, whereas EDX can probe a volume of several hundred nanometers
into the crystal [36]. All EDX data was recorded using a voltage of 20 keV.
Tab. 2 summarizes the Na to Ir intensity ratios on the lines (i.e. on the
domain walls), as well as the surrounding area for 6 different samples. First,
we note that even away from the lines the ratio differs from the
stoichiometric value of 2. A reduction of the intensity ratio below 2 even
away from domain boundaries on freshly cleaved samples likely points to the
occurrence of re-absorption of the heavy Iridium atoms during EDX. Therefore,
the intensity ratio does not directly equal the atomic ratio, though it is a
measure of it. Clearly it is drastically reduced on the lines compared to the
surrounding regions. The Na deficiency effectively hole-dopes the iridate
layers, whose impact on electrical transport is addressed below.

sample | (Na/Ir)line | (Na/Ir)s
---|---|---
1 | 1.5(2) | 1.8(1)
2 | 1.5(2) | 1.8(1)
3 | 1.3(1) | 1.8(1)
4 | 1.3(1) | 1.7(1)
5 | 1.2(2) | 1.6(2)
6 | 1.1(2) | 1.6(2)
Table 2:  Na/Ir ratio determined by EDX for several samples at the detected
lines and on the surrounding area (s).

The lower Na concentration along the structural domain walls also explains
their bright appearance in the SEM image, as within them the average atomic
number is increased. It is not yet clear how deep into the crystal the lines
penetrate. The fact that upon cleaving a crystal, lines could instantly be
found on the fresh surfaces implies either an instantaneous formation upon
exposition or a penetration depth reaching far into the crystal.

###  III.3 Electrical resistivity of domain walls

Figure 7:  Temperature dependence of the electrical transport of Na deficient
lines on the surface of Na2IrO3 crystals with varying Na content. Decreasing
Na concentration lowers the resistance due to hole doping. The sample with
lowest Na content is completely metallic (red line). The inset shows a
microstructure prepared and electrically contacted by FIB. Figure 8:
Temperature dependence of the electrical resistance along a structural domain
wall in sample 6 measured directly after contacting (red curve, from Fig.7),
as well as after 2 days (light gray) and 5 days (gray) of air exposure. Figure
9:  Low temperature electrical resistivity along a structural domain wall in
sample 6. The solid line represents correlated metallic behavior according to
$\rho(T)=\rho_{0}+aT^{2}+bT^{5}$ with $\rho_{0}=1.13\cdot 10^{4}\mu\Omega$cm,
$a=6.18\cdot 10^{-2}~{}\mu\Omega$cm/K2 and $b=1.11\cdot
10^{-5}~{}\mu\Omega$cm/K5.

In order to study the electrical resistance of Na-deficient structural domain
walls, FIB-assisted microsectioning and electrical contacting was utilized.
Crystals were glued onto sapphire substrates using epoxy resin, cleaved inside
an Argon filled glovebox, transferred to a sputter chamber without air
exposure and partially coated with gold. The samples were then structured
using a FIB-SEM in a way that allowed for a four probe measurement along a
defined area by contacting a line using gas-injection system (GIS) deposited
Pt within the FIB chamber and cutting trenches to define the desired geometry
[37], as can be seen in the inset of Fig. 7. Contacting was done in the same
way for all depicted samples. Air contact of the gold covered sample surface
was limited to the transfer from the sputter chamber to the FIB. EDX was used
to determine the Na-Ir ratio along the lines compared to the surrounding
surface. Resistivity data for three such lines, each of which was recorded on
a different crystal, can be seen in Fig. 7.

The absolute resistance was scaled with the geometrical parameters, such as
the width of the lines from SEM, the distance of the voltage contacts, as well
as the total sample thickness, to obtain the resistivity. This gives us an
upper bound of the resistivity, since less Na-deficiency in the inner parts,
deeper below the surface, may reduce the effective thickness of the conducting
regions. While absolute values are already rather small compared to previous
measurements, we observe a further decrease for lower Na content of the lines.
Sample 6 shows metallic behavior ($d\rho/dT>0$) down to lowest temperatures.
To check, whether the metallic behavior is preserved over time, we re-measured
the electrical resistance of sample 6 two subsequent times after air exposure.
Prolonged contact with CO2 and H2O is known to lead to sample degradation
[38]. As shown in Fig. 8, after two days of exposure to air, the resistivity
still appeared metallic, yet the absolute values increased. After five days
the resistivity shows entirely insulating behavior. The metallic behavior of
sample 6 before air exposure cannot be explained by degradation, which by
contrast enhances the resistivity. Instead, the metallic resistance indicates
that the metal-insulator transition has been passed for Na deficient, i.e.
hole-doped Na2IrO3.

Finally, we analyze the low-temperature data, cf. Fig. 9. The temperature
dependence can be described using a sum of the residual resistance,
electron–electron and electron–phonon scattering contributions according to
$\rho=\rho_{0}+aT^{2}+bT^{5}$ [39]. The fitted coefficient
$a=0.062\mu\Omega$cm/K2 lies in between values for bilayer strontium iridate
Sr2RuO4 along the c- and a-axis and it is comparable to values for moderately
heavy 4f- and 5f- Kondo systems [40]. But of course this relies on the above
assumption of a homogeneous Na deficiency across the entire sample thickness
of $\sim 20-40$ $\mu$m.

##  IV Conclusion

In conclusion, Laue pattern of Na2IrO3 indicate structural domains and are
perfectly well described by additive overlap of reflections from three
different structural twins with 120∘ rotated $b$-axes around the $c^{*}$-axis.
Complementary scanning electron microscopy (SEM) energy dispersive x-ray
analysis on cleaved crystal surfaces revealed lines roughly along the
different b-axes with significant (up to 1/3) Na deficiency compared to the
surrounding areas. We thus associate these lines to structural domain walls.
Resistivity measurements along individual domain walls in FIB-assisted
microstructured samples reveal a dependence of the resistance on their Na to
Ir ratio, with lower Na content and thus increased hole doping corresponding
to lower resistivity. For the domain wall with highest Na deficiency a
completely metallic behavior down to 2 K has been found, indicating that
Na2IrO3 can be tuned locally across the metal-insulator transition. The
observed coefficient of $T^{2}$ resistivity behavior suggests a moderately
correlated metal state, though the absolute values of the resistance are
governed by the uncertainty of the effective thickness of the domain
boundaries.

At first glance, this behavior reminds to conductive domain walls observed in
other otherwise insulating materials, such as ferroelectrics [41, 42, 43]
where conductivity can be tuned by the application of an electric field. This
leads to non-Ohmic I-V-characteristics which have also been observed for
Na2IrO3 [26, 30]. However, in the case of Na2IrO3 it is known from ARPES
measurements that Na deficiency or additional deposited Na atoms influence the
charge carrier density and thus the Fermi level [27, 28, 26] and our analysis
reveals a clear Na deficiency at the domain walls, which is thus at the origin
of the conducting behavior.

## Acknowledgements

Technical support by Klaus Wiedenmann is gratefully acknowledged. We thank Dr.
Toni Helm for advise regarding the FIB preparation of samples. This work was
supported by the German Research Foundation project 492547816 (TRR 360).

## References

  * Moore [2010] J. E. Moore, The birth of topological insulators, Nature 464, 194 (2010).
  * Pesin and Balents [2010] D. Pesin and L. Balents, Mott physics and band topology in materials with strong spin-orbit interaction, Nat. Phys. 6, 376 (2010).
  * Dzero  _et al._ [2010] M. Dzero, K. Sun, V. Galitski, and P. Coleman, Topological Kondo insulators, Phys. Rev. Lett. 104, 106408 (2010).
  * Rau  _et al._ [2016] J. G. Rau, E. K.-H. Lee, and H.-Y. Kee, Spin-orbit physics giving rise to novel phases in correlated systems: Iridates and related materials, Annu. Rev. Condens. Matter Phys. 7, 195 (2016).
  * Tokura [2022] Y. Tokura, Quantum materials at the crossroads of strong correlation and topology, Nat. Mater. 21, 971 (2022).
  * Kim  _et al._ [2008] B. J. Kim, H. Jin, S. J. Moon, J.-Y. Kim, B.-G. Park, C. S. Leem, J. Yu, T. W. Noh, C. Kim, S.-J. Oh, J.-H. Park, V. Durairaj, G. Cao, and E. Rotenberg, Novel ${J}_{\mathrm{eff}}=1/2$ Mott State Induced by Relativistic Spin-Orbit Coupling in ${\mathrm{Sr}}_{2}{\mathrm{IrO}}_{4}$, Phys. Rev. Lett. 101, 076402 (2008).
  * Watanabe  _et al._ [2010] H. Watanabe, T. Shirakawa, and S. Yunoki, Microscopic Study of a Spin-Orbit-Induced Mott Insulator in Ir Oxides, Phys. Rev. Lett. 105, 216410 (2010).
  * Clancy  _et al._ [2012] N. Clancy, J. P. adn Chen, C. Y. Kim, W. F. Chen, K. W. Plumb, B. C. Jeon, T. W. Noh, and Y.-J. Kim, Spin-orbit coupling in iridium-based 5d compounds probed by x-ray absorption spectroscopy, Phys. Rev. B 86, 195131 (2012).
  * Kim  _et al._ [2009] B. Kim, H. Ohsumi, T. Komesu, S. Sakai, T. Morita, H. Takagi, and T.-H. Arima, Phase-sensitive observation of a spin-orbital Mott state in Sr2IrO4, Science 323, 1329 (2009).
  * Shitade  _et al._ [2009] A. Shitade, H. Katsura, J. Kuneš, X.-L. Qi, S.-C. Zhang, and N. Nagaosa, Quantum Spin Hall Effect in a Transition Metal Oxide ${\mathrm{Na}}_{2}{\mathrm{IrO}}_{3}$, Phys. Rev. Lett. 102, 256403 (2009).
  * Chaloupka  _et al._ [2010] J. c. v. Chaloupka, G. Jackeli, and G. Khaliullin, Kitaev-Heisenberg Model on a Honeycomb Lattice: Possible Exotic Phases in Iridium Oxides ${A}_{2}{\mathrm{IrO}}_{3}$, Phys. Rev. Lett. 105, 027204 (2010).
  * Singh and Gegenwart [2010] Y. Singh and P. Gegenwart, Antiferromagnetic Mott insulating state in single crystals of the honeycomb lattice material ${\text{Na}}_{2}{\text{IrO}}_{3}$, Phys. Rev. B 82, 064412 (2010).
  * Choi  _et al._ [2012] S. Choi, R. Coldea, A. Kolmogorov, T. Lancaster, I. Mazin, S. Blundell, P. Radaelli, Y. Singh, P. Gegenwart, K. Choi, S.-W. Cheong, P. J. Baker, C. Stock, and J. Taylor, Spin waves and revised crystal structure of honeycomb iridate Na2IrO3, Phys. Rev. Lett. 108, 127204 (2012).
  * Ye  _et al._ [2012] F. Ye, S. Chi, H. Cao, B. C. Chakoumakos, J. A. Fernandez-Baca, R. Custelcean, T. Qi, O. Korneta, and G. Cao, Direct evidence of a zigzag spin-chain structure in the honeycomb lattice: A neutron and x-ray diffraction investigation of single-crystal Na2IrO3, Phys. Rev. B 85, 180403 (2012).
  * Singh  _et al._ [2012] Y. Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, W. Ku, S. Trebst, and P. Gegenwart, Relevance of the Heisenberg-Kitaev Model for the Honeycomb Lattice Iridates ${A}_{2}{\mathrm{IrO}}_{3}$, Phys. Rev. Lett. 108, 127203 (2012).
  * Hwan Chun  _et al._ [2015] S. Hwan Chun, J.-W. Kim, J. Kim, H. Zheng, C. C. Stoumpos, C. Malliakas, J. Mitchell, K. Mehlawat, Y. Singh, Y. Choi, T. Gog, A. Al-Zein, M. Moretti Sala, M. Krisch, J. Chaloupka, G. Jackeli, G. Khaliullin, and B. Kim, Direct evidence for dominant bond-directional interactions in a honeycomb lattice iridate Na2IrO3, Nat. Phys. 11, 462 (2015).
  * Winter  _et al._ [2017] S. M. Winter, A. A. Tsirlin, M. Daghofer, J. van den Brink, Y. Singh, P. Gegenwart, and R. Valentí, Models and materials for generalized Kitaev magnetism, J. Phys.: Condens. Matter 29, 493002 (2017).
  * Revelli  _et al._ [2020] A. Revelli, M. M. Sala, G. Monaco, C. Hickey, P. Becker, F. Freund, A. Jesche, P. Gegenwart, T. Eschmann, F. Buessen, S. Trebst, P. van Loosdrecht, J. van den Brink, and M. Grüninger, Fingerprints of Kitaev physics in the magnetic excitations of honeycomb iridates, Phys. Rev. Research 2, 043094 (2020).
  * Kim  _et al._ [2016] Y. Kim, N. Sung, J. Denlinger, and B. Kim, Observation of a $d$-wave gap in electron doped Sr2IrO4, Nat. Phys. 12, 37 (2016).
  * Battisti  _et al._ [2017] I. Battisti, K. Bastiaans, V. Fedeseev, A. de la Torre, N. Iliopousol, A. Tamai, E. Hunter, R. Perry, J. Zaanen, F. Baumberger, and M. Allan, Universality of pseudogap and emergent order in lightly doped Mott insulators, Nat. Phys. 13, 21 (2017).
  * Gretarsson  _et al._ [2013] H. Gretarsson, J. Clancy, X. Liu, J. Hill, E. Bozin, Y. Singh, S. Manni, P. Gegenwart, J. Kim, A. Said, D. Casa, T. Gog, M. Upton, H.-S. Kim, J. Yu, V. M. Katukuri, L. Hozoi, J. van den Brink, and Y.-J. Kim, Crystal-Field Splitting and Correlation Effect on the Electronic Structure of A2IrO3, Phys. Rev. Lett. 110, 076402 (2013).
  * Comin  _et al._ [2012] R. Comin, G. Levy, B. Ludbrook, Z.-H. Zhu, C. N. Veenstra, J. A. Rosen, Y. Singh, P. Gegenwart, D. Stricker, J. N. Hancock, D. van der Marel, I. S. Elfimov, and A. Damascelli, Na2IrO3 as a novel relativistic Mott insulator with a 340 meV gap, Phys. Rev. Lett. 109, 266406 (2012).
  * Kim  _et al._ [2012] C. H. Kim, H. S. Kim, H. Jeong, H. Jin, and J. Yu, Topological Quantum Phase Transition in $5d$ Transition Metal Oxide Na2IrO3, Phys. Rev. Lett. 108, 106401 (2012).
  * You  _et al._ [2012] Y.-Z. You, I. Kimchi, and A. Vishwanath, Doping a spin-orbit Mott insulator: Topological superconductivity from the Kitaev-Heisenberg model and possible application to (Na2/Li2)IrO3, Phys. Rev. B 86, 085145 (2012).
  * Jenderka  _et al._ [2013] M. Jenderka, J. Barzola-Quiquia, Z. Zhang, H. Frenzel, M. Grundmann, and M. Lorenz, Mott variable-range hopping and weak antilocalization effect in heteroepitaxial Na2IrO3 thin films, Phys. Rev. B 88, 045111 (2013).
  * Rodriguez  _et al._ [2020] J. Rodriguez, G. Lopez, F. Ramirez, N. P. Breznay, R. Kealhofer, V. Nagarajan, D. Latzke, S. Wilson, N. Marrufo, P. Santiago, J. Lara, A. Diego, E. Molina, D. Rosser, H. Tavassol, A. Lanzara, J. G. Analytis, and C. Ojeda-Aristizabal, Competition between magnetic order and charge localization in Na2IrO3 thin crystal devices, Phys. Rev. B 101, 235415 (2020).
  * Alidoust  _et al._ [2016] N. Alidoust, C. Liu, S.-Y. Xu, I. Belopolski, T. Qi, M. Zeng, D. S. Sanchez, H. Zheng, G. Bian, M. Neupane, Y.-T. Liu, S. D. Wilson, H. Lin, A. Bansil, G. Cao, , and M. Z. Hasan, Observation of metallic surface states in the strongly correlated Kitaev-Heisenberg candidate Na2IrO3, Phys. Rev. B 93, 245132 (2016).
  * Moreschini  _et al._ [2017] L. Moreschini, I. L. Vecchio, N. Breznay, S. Moser, S. Ulstrup, R. Koch, J. Wirjo, C. Jozwiak, K. Kim, E. Rotenberg, A. Bostwick, J. G. Analytis, and A. Lanzara, Quasiparticles and charge transfer at the two surfaces of the honeycomb iridate Na2IrO3, Phys. Rev. B 96, 161116 (2017).
  * Lüpke  _et al._ [2015] F. Lüpke, S. Manni, S. Erwin, I. Mazin, P. Gegenwart, and M. Wenderoth, Highly unconventional surface reconstruction of Na2IrO3 with persistent energy gap, Phys. Rev. B 91, 041405 (2015).
  * Dziuba [2022] T. Dziuba, Combined Study of Local Spectroscopy and Macroscopic Electronic Transport of the Correlated Honeycomb Spin-Orbit Mott Insulator Na2IrO3, Phys. Status Solidi B 258, 2000421 (2022).
  * Mehlawat and Singh [2016] K. Mehlawat and Y. Singh, First-order density-wave-like transitions in surface-doped Na2IrO3, Phys. Rev. B 94, 041109(R) (2016).
  * Moskvin  _et al._ [2003] A. Moskvin, J. Málek, M. Knupfer, R. Neudert, J. Fink, R. Hayn, S.-L. Drechsler, N. Motoyama, H. Eisaki, and S. Uchida, Evidence for Two Types of Low-Energy Charge Transfer Excitations in Sr2CuO3, Phys. Rev. Lett. 91, 037001 (2003).
  * [33] R. Deslattes, E. Kessler Jr, O. Indelicato, and E. Lindroth, International Tables for Crystallography (2006). Vol. C, Chapter 4.2, pp. 200–212., .
  * Freund  _et al._ [2016] F. Freund, S. Williams, R. Johnson, R. Coldea, P. Gegenwart, and A. Jesche, Single crystal growth from separated educts and its application to lithium transition-metal oxides, Sci. Rep. 6, 1 (2016).
  * Manni  _et al._ [2014] S. Manni, S. Choi, I. Mazin, R. Coldea, M. Altmeyer, H. O. Jeschke, R. Valentï, and P. Gegenwart, Effect of isoelectronic doping on the honeycomb-lattice iridate A2IrO3, Phys. Rev. B 89, 245113 (2014).
  * Castaing [1960] R. Castaing, Electron probe microanalysis, Adv. Electron. Electron Phys. 13, 317 (1960).
  * Moll [2018] P. J. Moll, Focused ion beam microstructuring of quantum matter, Annu. Rev. Condens. Matter Phys. 9, 147 (2018).
  * Krizan  _et al._ [2014] J. Krizan, J. Roudebush, G. Fox, and R. Cava, The chemical instability of Na2IrO3 in air, Mater. 52, 162 (2014).
  * Ashcroft and Mermin [1976] N. W. Ashcroft and N. D. Mermin, Solid state physics, Thomson Learning, Inc (1976).
  * Maeno  _et al._ [1997] Y. Maeno, K. Yoshida, H. Hashimoto, S. Nishizaki, S.-i. Ikeda, M. Nohara, T. Fujita, A. P. Mackenzie, J. G. Hussey, Nigel E.and Bednorz, and F. Lichtenberg, Two-Dimensional Fermi Liquid Behavior of the Superconductor Sr2RuO4, J. Phys. Soc. Jpn. 66, 1405 (1997).
  * Seidel  _et al._ [2009] J. Seidel, L. W. Martin, Q. He, Q. Zhan, Y.-H. Chu, A. Rother, M. E. Hawkridge, P. Maksymovych, P. Yu, M. Gajek, N. Balke, S. V. Kalinin, S. Gemming, F. Wang, G. Catalan, J. F. Scott, N. A. Spaldin, J. Orenstein, and R. Ramesh, Conduction at domain walls in oxide multiferroics, Nature Mater. 8, 229 (2009).
  * Maksymovych  _et al._ [2011] P. Maksymovych, J. Seidel, Y. H. Chu, P. Wu, A. P. Baddorf, L.-Q. Chen, S. V. Kalinin, and R. Ramesh, Dynamic conductivity of ferroelectric domain walls in BiFeO3, Nano Lett. 11, 1906 (2011).
  * Ghara  _et al._ [2021] S. Ghara, K. Geirhos, L. Kuerten, P. Lunkenheimer, V. Tsurkan, M. Fiebig, and I. Kézsmárki, Giant conductivity of mobile non-oxide domain walls, Nat. Commun. 12, 3975 (2021).


ifaamas [AAMAS ’23]Proc. of the 22nd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2023)May 29 – June 2, 2023 London, United
KingdomA. Ricci, W. Yeoh, N. Agmon, B. An (eds.) 2023 2023 443 KAIST Daejeon
Korea KAIST Daejeon Korea

# Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep
Reinforcement Learning

Woojun Kim<EMAIL_ADDRESS>and  Youngchul Sung<EMAIL_ADDRESS>
###### Abstract.

Handling the problem of scalability is one of the essential issues for multi-
agent reinforcement learning (MARL) algorithms to be applied to real-world
problems typically involving massively many agents. For this, parameter
sharing across multiple agents has widely been used since it reduces the
training time by decreasing the number of parameters and increasing the sample
efficiency. However, using the same parameters across agents limits the
representational capacity of the joint policy and consequently, the
performance can be degraded in multi-agent tasks that require different
behaviors for different agents. In this paper, we propose a simple method that
adopts structured pruning for a deep neural network to increase the
representational capacity of the joint policy without introducing additional
parameters. We evaluate the proposed method on several benchmark tasks, and
numerical results show that the proposed method significantly outperforms
other parameter-sharing methods.

###### Key words and phrases:

Multi-agent Reinforcement Learning, Parameter Sharing, Scalability, Neural
Network Pruning

##  1\. Introduction

Multi-agent reinforcement learning (MARL) is gaining attention as an important
research direction for tackling many real-world decision-making tasks such as
traffic management and autonomous driving. Recently, many issues for MARL have
vigorously been studied to advance MARL theories and algorithms, including
value factorization Rashid et al. (2018); Sunehag et al. (2018), communication
Foerster et al. (2016); Kim et al. (2020); Kim et al. (2019), subgoal
generation Jeon et al. (2022), and correlated exploration Mahajan et al.
(2019); Kim et al. (2023). In order to apply these MARL algorithms to real-
world problems with many agents and constrained resources, one needs to handle
the problem of scalability and sample inefficiency in most cases. This is
because these algorithms typically have a high variance for gradient
estimation and require increasing computational cost and memory as the number
of agents increases. One way to address the problem of scalability and sample
inefficiency is parameter sharing, commonly adopted in most MARL algorithms
Rashid et al. (2018); Foerster et al. (2016, 2018); Jeon et al. (2022).
Parameter sharing uses only one parameterized function such as a deep neural
network for all agents’ policies or critics, and increases sample efficiency
yielding fast training. However, for naive parameter sharing using a single
common function, agents choose the same action if their observations are the
same and critics estimate the expected return to be the same even if agents
receive individual rewards. As a result, naive parameter sharing has a limited
representational capacity of the joint policy and critics, and this can
degrade performance in heterogeneous multi-agent environments or environments
that require different behaviors for different agents. Thus, to allow multiple
agents even with the same observations to perform different actions, the
concatenation of the agent’s observation and a one-hot vector encoding agent
indication is widely used for recent MARL algorithms Rashid et al. (2018);
Mahajan et al. (2019); Jeon et al. (2022). Although this method improves
performance compared with naive parameter sharing, the representational
capacity is still limited because the neurons of a deep neural network for
function approximation are shared across all agents but only agent-specific
perturbation is added to a hidden layer. Furthermore, it is difficult for this
one-hot-vector-based method to control the amount of parameter sharing among
agents, and it even introduces an additional parameter of one hot vector.

In this paper, we propose a novel structured network pruning method for
parameter sharing (SNP-PS) in multi-agent deep reinforcement learning so as to
increase the representational capacity of neural networks parameterizing
actors and critics without introducing additional parameters while keeping
high sample efficiency. Inspired by the lottery ticket hypothesis Su et al.
(2020), we conjecture a new hypothesis named the lottery group ticket
hypothesis, which assumes the existence of a group of pruned subnetworks that
allow multiple agents to be identifiable and improve performance compared with
the naive parameter sharing. Then, we provide a neural network pruning method
to obtain such a group of subnetworks called winning group tickets. The
obtained group of subnetworks is used for the policies or critics of multiple
agents. Since the policies or critics of the proposed method shares only
subsets of the parameters, they can yield diverse joint actions. The proposed
method is very easy to implement and can be combined with other parameter-
sharing methods. Furthermore, the proposed method can control the number of
shared parameters by adjusting the pruning ratio without introducing
additional parameters. We evaluated the proposed parameter-sharing method on
top of QMIX and multi-agent advantage actor-critic in several multi-agent
environments. Numerical results show that the proposed method noticeably
enhances both training speed and the final performance compared with other
parameter-sharing methods.

##  2\. Background

###  2.1. Environment Model

A Partially Observable Markov Game A general multi-agent task is typically
modeled as a Partially Observable Markov Game (POMG), also known as a
partially observable stochastic game. A POMG is represented by the tuple
$<\mathcal{N},\mathcal{S},\\{\mathcal{A}_{i}\\}_{i=1}^{N},\\{\Omega_{i}\\}_{i=1}^{N},\mathcal{T},\mathcal{O},$
$\\{\mathcal{R}_{i}\\}_{i=1}^{N}>$, where $\mathcal{N}$ is the set of agents,
$\mathcal{S}$ is the state space of the environment,
$\bm{\mathcal{A}}=\prod_{i=1}^{N}\mathcal{A}^{i}$ is the joint action space,
and $\bm{\Omega}=\prod_{i=1}^{N}\Omega^{i}$ is the joint observation space. At
each time step $t$, Agent $i$ perceives local observation
$o_{t}^{i}\in\Omega_{i}$, which is determined from the global state
$s_{t}\in\mathcal{S}$ according to the observation probability
$\mathcal{O}:\mathcal{S}\times\bm{\mathcal{A}}\times\bm{\Omega}\rightarrow[0,1]$,
and determines and executes action $a_{t}^{i}$ based on its local history. The
joint action $\bm{a_{t}}=(a_{t}^{1},\cdots,a_{t}^{N})$ yields next state
$s_{t+1}$ according to the transition probability
$\mathcal{T}:\mathcal{S}\times\bm{\mathcal{A}}\times\mathcal{S}\rightarrow[0,1]$.
Then, Agent $i$ receives local reward $r_{t}^{i}$ according to its own reward
function
$\mathcal{R}_{i}:\mathcal{S}\times\bm{\mathcal{A}}\rightarrow{\mathbb{R}}$ and
next observation $o_{t+1}^{i}$. This procedure is repeated and each agent aims
to find its own policy that maximizes the expected return defined as
$R_{t}^{i}=\sum_{t^{\prime}=t}^{\infty}\gamma^{t^{\prime}}r_{t^{\prime}}^{i}$
where $\gamma\in[0,1]$ is the discounting factor. Here, Agent $i$’s policy is
conditioned on its own observation-action history
$\tau^{i}\in(\Omega_{i}\times\mathcal{A}_{i})^{*}$. The expected return of one
agent depends on other agents’ policies, and consequently, the notion of
equilibrium, e.g. Nash equilibrium naturally arises in the objective function
of all agents.

If all agents receive the same reward, i.e. team reward, the task is
considered as a fully cooperative setting and the model reduces to a
decentralized partially observable Markov decision process (Dec-POMDP). In
Dec-POMDP, all agents optimize their policies to maximize the common objective
function $J(\bm{\pi})=\mathbb{E}_{\pi}(\sum_{t=0}^{\infty}\gamma^{t}r_{t})$,
where $\bm{\pi}=(\pi^{1},\cdots,\pi^{N})$ is the joint policy.

###  2.2. Multi-Agent Reinforcement Learning

One simple approach to MARL is using a centralized controller which basically
represents the joint policy. This approach requires communication channels
among agents to handle partial observability and suffers from the curse of
dimensionality, i.e., the volume of the input space grows exponentially as the
number of agents increases. An alternative approach is independent learning in
which each agent has a decentralized policy and learns the policy
independently while treating other agents as a part of the environment. The
representative example is independent Q-learning (IQL), which is an extension
of Q-learning to multi-agent settings Tan (1993). However, IQL limits the
learning of coordinated behaviors due to the negligence of the influence of
other agents, and thus the centralized action-value function conditioned on
the environmental state and the joint action has been adopted Lowe et al.
(2017); Foerster et al. (2018); Rashid et al. (2018). A well-known example is
QMIX, which decomposes the joint action-value function
$Q_{JT}(s,\bm{\tau},\bm{a})$ into a non-linear combination of individual
action-value functions with a monotonic constraint as follows:

$\displaystyle Q_{JT}(s,\bm{\tau},\bm{a})=f_{mixing}(s,$ $\displaystyle
Q^{1}(\tau^{1},a^{1}),\cdots,Q^{N}(\tau^{N},a^{N})),$
$\displaystyle\mbox{s.t.}\leavevmode\nobreak\ \frac{\partial
Q_{JT}(s,\bm{\tau},\bm{a})}{\partial Q^{i}(\tau^{i},a^{i})}\geq
0,\leavevmode\nobreak\ \forall i\in\mathcal{N},$ (1)

where $Q_{i}(\tau^{i},a^{i})$ is the individual action-value function of Agent
$i$ and $f_{mixing}$ is the mixing network which is trained to satisfy the
monotonic constraint by imposing non-negativity on its weights. The individual
action-value functions are parameterized by multi-layer perceptrons (MLPs) and
a gated recurrent unit (GRU). For practical implementation and improved sample
efficiency, the parameters are shared across all agents and an encoded one-hot
vector is added to the input for agent indication. The joint action-value
function is parameterized by $\theta$ and trained to minimize the temporal
difference error with the loss function, given by

$\displaystyle\mathcal{L}(\theta)=\mathbb{E}\Big{[}(r+\gamma\max_{\bm{a^{\prime}}}Q_{JT}(s^{\prime},\bm{\tau}^{\prime},\bm{a}^{\prime};\theta^{-})-Q_{JT}(s,\bm{\tau},\bm{a};\theta))^{2}\Big{]}$
(2)

where $s^{\prime},\bm{\tau}^{\prime}$ and $\theta^{-}$ are the next state, the
next joint history $\bm{\tau}^{\prime}=(\tau^{1},\cdots,\tau^{N})$, and the
parameter of the target network, respectively. In this paper, we consider QMIX
as the baseline algorithm to test the proposed parameter-sharing method. In
addition to QMIX, which is a value-based approach, we also consider an actor-
critic-based multi-agent algorithm which extends advantage actor-critic (A2C)
to multi-agent setting considered in Christianos et al. (2021).

|  |
---|---|---
(a) | (b) | (c)
Figure 1. (a) An original dense neural network, (b) a weight-pruned network,
and (c) a neuron-pruned network

###  2.3. Neural Network Pruning

Neural network pruning removes parameters such as weights and/or neurons to
build a smaller neural network without performance degradation. Recently,
neural network pruning has widely been studied for its applicability to
resource-constrained tasks. The typical process of neural network pruning
consists of three steps: 1) training a dense network, 2) pruning the network,
and 3) fine-tuning the pruned network. By iterating this procedure, we can
obtain a pruned network, which is often a sparse neural network. This method
is called pruning after training (PaT) Wang et al. (2021). In order to make
the procedure simpler and reduce the computational cost, pruning at
initialization (PaI) is gaining increasing attention under the lottery ticket
hypothesis (LTH), which assumes that a randomly initialized dense network
contains a subnetwork (called a winning ticket) that can achieve comparable
performance to that of the original dense network Frankle and Carbin (2018).
Under the hypothesis, to find a winning ticket, Su et al. (2020) proposed
using a random ticket that prunes the weights of a deep neural network
randomly while keeping the ratio of pruned weights per layer. Surprisingly, it
is shown that the performance of such a randomly pruned network on VGG-19 on
CIFAR-10 degrades only $1\%$ when the pruning ratio of the randomly
initialized network is $98\%$, i.e., the weights are removed by $98\%$ Su et
al. (2020). Although the lottery ticket hypothesis has not been proved yet due
to its theoretical difficulty, practical applications of random pruning indeed
yield a subnetwork with comparable performance and hint at the possible
validity of the hypothesis.

Neural network pruning methods can be divided into two categories:
unstructured pruning and structured pruning. Unstructured pruning removes
weights individually by fixing them to zero. On the other hand, structured
pruning removes a group of weights together. For example, structured pruning
removes all weights that are connected to a neuron or to filters or channels
in a convolutional neural network. Fig. 1 illustrates network pruning. Fig.
1(a), (b) and (c) show an original dense network, weight-pruned network, and
neuron-pruned network, respectively. A Weight-pruned network and a neurons-
pruned network are examples of unstructured pruning and structured pruning,
respectively.

Notations We denote a deep neural network as a parameterized function
$f(x;\theta)$, where $x$ is the input and $\theta$ is the trainable parameters
including weights and biases in the neural network. Then, for an original
dense deep neural network $f(x;\theta)$, network pruning produces a new neural
network model $f(x;\theta\odot M)$, where $M=\\{0,1\\}^{|\theta|}$ is a binary
mask for pruning and $\odot$ denotes the Hadamard product, i.e., elementwise
product.

##  3\. Related Works

###  3.1. Scalability in Multi-Agent Reinforcement Learning

MARL algorithms often encounter the curse of dimensionality, which arises from
the increasing input space of the parameterized function as the number of
agents increases. One example is the use of centralized critic, which is
conditioned on the joint action and the environment state or the joint
observation. It has widely been considered to address the non-stationarity
problem of MARL Lowe et al. (2017); Foerster et al. (2018). Here, the volume
of the joint action and observation spaces grows exponentially, and
consequently, it is difficult to train a deep neural network approximating the
centralized critic with a large number of agents Kim et al. (2019). To address
this problem, Iqbal and Sha (2019) proposed an attention-based centralized
critic which can be scalable to the number of agents. Yang et al. (2018)
proposed mean-field actor-critic which uses the mean-field theory to estimate
a centralized critic based on local critics capturing interaction. The curse
of dimensionality also arises when inter-agent communication is allowed. In
this setting, each agent generates and sends a message and learns coordinated
action based on received messages Kim et al. (2020); Foerster et al. (2016).
Then, the dimension of received messages increases as the number of agents
increases. Kim et al. (2019) proposed an efficient training method to handle
the increasing dimension of messages and it can also handle the aforementioned
problem of the centralized critic.

---
Figure 2. Neural network architecture for parameter sharing with one-hot
vector (the same network is used for all agents except the one-hot vector
value change): Note that a different one-hot vector only affects the input to
the first hidden-layer through the link weights from the activated one-hot
vector unit to the first hidden layer, and the network from the observation
input to the output layer is the same for all agents.

In addition to the curse of dimensionality, the increase in the number of
trainable parameters with respect to (w.r.t.) the number of agents is also an
issue. For an $N$-agent actor-critic algorithm with deep neural network-based
function approximation, we need $2N$ deep neural networks to parameterize all
policies and critics. Thus, the required memory size grows linearly w.r.t. the
number of agents. Furthermore, the separate approximation of all agents’
networks limits the sample efficiency of learning, because in the separate
approximation case, each agent’s critic and policy can only use its own
observation for training. To address the aforementioned increasing hardware
and sample inefficiency problem, parameter sharing has widely been used in
MARL research Rashid et al. (2018); Kim et al. (2023). By sharing the
parameters across agents, the required memory size does not grow w.r.t. the
number of agents and the training time can be reduced. However, parameter
sharing limits the representational capacity of the joint policy and critic,
and consequently, agents with the same observation-action histories will act
in the same way. This can cause performance degradation in multi-tasks that
require diverse behaviors, e.g., different behaviors for different agents. To
avoid this limitation of simple parameter sharing, most recent MARL algorithms
combine parameter sharing with agent indication to the observation Rashid et
al. (2018); Kim et al. (2023). For agent indication, one-hot vector is
commonly used, as shown in Fig. 2. It was shown that due to the impact of the
one-hot vector on the first hidden layer, parameter sharing using one-hot
vector yields better performance than simple parameter sharing. Terry et al.
(2020) proved that agent indication allows agents’ policies to converge to
optimal ones.

###  3.2. Network Pruning in Reinforcement Learning

Graesser et al. (2022) investigated the application of existing network
pruning techniques to reinforcement learning. They found that sparse training
based on network pruning often performs better than the dense network with the
same number of parameters and increases the robustness against observation
noise. Livne and Cohen (2020) proposed transfer-learning-based network pruning
to prevent the pruned network from performance degradation. They first train a
teacher network, i.e., a dense neural network, and then prune and regenerate
the weights. Sokar et al. (2021) proposed a dynamic sparse training method to
reduce the required resources, e.g. memory and computational cost, while
improving the performance. However, these works are proposed for single-agent
reinforcement learning and are based on unstructured network pruning to reduce
the trainable parameters with keeping performance degradation.

---
(a) (b) (c)
Figure 3. The procedure of generating $N=3$ subnetworks: (a) randomly-
initialized dense neural network. (b) $N=3$ subnetworks are generated from the
same dense network independently. Each colored neural network represents
different agents’ subnetworks. (c) the generated subnetworks share some
subsets of parameters across multiple agents. Each neuron is colored with the
colors of agents who share the corresponding neuron.

##  4\. Methodology

We consider neural network pruning to reinforce agent indication for learning
diverse actions in multi-agent reinforcement learning. First, we present our
motivation. Then, inspired by Su et al. (2020), we provide our conjecture
named the lottery group ticket hypothesis relevant to multi-agent deep
reinforcement learning network pruning, and propose a structured network
pruning-based parameter sharing method to identify winning group tickets of
the conjecture.

###  4.1. Motivation

To increase the representational capacity of simple parameter sharing, the
concatenation of agents’ observation and an agent-specific one-hot vector,
called one-hot encoding agent indication, has been used due to its simplicity
and performance improvement. However, the one-hot encoding agent indication
still has limited representational capacity for different agents. Fig. 2 shows
the neural network architecture of one-hot encoding agent indication. The
input layer consists of one-hot vector of size $N$ and observation input. As
seen in the figure, the weight and bias parameters are the same for all agents
except the weights from the one-hot vector input unit to the first hidden
layer. For Agent $i$, $o^{i}$ is applied to the observation input, and the
$i$th element of the one-hot vector input is activated to one and all other
elements of the one-hot vector input are deactivated as zero. Thus, the first
hidden-layer feature vector is the sum of the observation feature and the
agent indication feature. One can think the first hidden-layer vector is the
observation feature added by an agent-specific perturbation. Different agent
representational capacity comes only from this perturbation vector. No neurons
are agent-specific and the common network is shared by all agents except the
one-hot input and corresponding links. Another problem of the one-hot encoding
agent indication is that it is difficult to explicitly control the amount of
parameter sharing due to its fixed architecture. Depending on tasks, one may
want similar or quite disparate actions by agents for similar observations.
For this, it is necessary to control the amount of parameter sharing.
Furthermore, the input size of the one-hot encoding agent indication method
linearly increases as the number of agents increases. This may be problematic
for MARL with a huge number of agents.

Instead of adding one-hot vector, Christianos et al. (2021) proposed a
parameter-sharing method named selective parameter sharing (SePS). SePS
divides all agents into subgroups that share parameters based on a clustering
algorithm before training. In SePS, unlike one-hot encoding agent indication,
the agents only in the same cluster share parameters, and this can increase
representational capacity. For this, they encode agent indication based on a
variational autoencoder (VAE) and then use the encoded hidden vector of VAE
for the clustering algorithm. However, SePS requires networks as many as the
number of clusters, and this decreases sample efficiency and increases the
required memory resources. In addition, the method depends on the performance
of clustering since the clustered agents are fixed during training.

Circumventing the limitations of the previous methods, we here aim to devise a
parameter-sharing method that increases the representational capacity and the
sample efficiency without introducing additional parameters. For this, we
exploit structured neural network pruning. We start with our conjecture named
the lottery group ticket hypothesis.

###  4.2. The Lottery Group Ticket Hypothesis

Inspired by the lottery ticket hypothesis (LTH) Su et al. (2020), we
conjecture the following hypothesis.

The Lottery Group Ticket Hypothesis (LGTH): A randomly-initialized dense
network with sufficient size contains a group of subnetworks that allow
multiple agents to be identifiable and yields the return performance
comparable to the full dense network with simple parameter sharing if
subnetworks in place of the full dense network are used for policies or
critics in multi-agent reinforcement learning.

Formally speaking, the hypothesis can be stated as follows. Suppose that MARL
agents parameterize their policies based on the common function $f(x;\theta)$
and their own masks $\\{M_{i}\\}_{i=1}^{N}$. We obtain Agent $i$’s policy as
$f(x;\theta\odot M_{i})$, where $\odot$ is element-wise product. Then, the
lottery group ticket hypothesis conjectures that for a sufficient-size network
and up to a certain $N$, there exist masks $\\{M_{i}\\}_{i=1}^{N}$ such that
$\\{f(x;\theta\odot M_{i})\\}_{i=1}^{N}$ are different for the same input $x$
(identifiability) and $J(\prod_{i=1}^{N}f(x;\theta_{final}))\leq
J(\prod_{i=1}^{N}f(x;\theta_{final}\odot M_{i}))$ (comparable performance to
the full dense network), where $\theta_{final}$ is the parameters after
training finishes and $J(\cdot)$ is the objective function of MARL. We refer
to a group of subnetworks $\\{M_{i}\\}_{i=1}^{N}$ that satisfy the LGTH as
winning group tickets.

Like LTH Su et al. (2020), we do not have proof for LGTH, and both LTH and
LGTH are mere conjectures at this point. However, our parameter sharing method
developed hereafter under the conjecture indeed yields a group of subnetworks
satisfying both identifiability and comparable performance conditions,
indicating the possible validity of the conjecture, and proof of LTH and LGTH
remains as a nontrivial future work.

Algorithm 1 Generate Winning Group Tickets

Initialize parameters of root networks for actor and critic

for $Agent=1,2,\cdots$ do

for $Layer=1,2,\cdots$ do

Select the fixed number of neurons to be pruned randomly

Generate the corresponding binary masks for the actor and critic

end for

Initialize neural networks for the policy and critic based on the binary masks

end for

###  4.3. Structrued Network Pruning with Parameter Sharing

We now propose our pruning-at-initialization (PaI)-based structured network
pruning method for parameter sharing (SNP-PS) to construct a collection of
winning group tickets for MARL. Before training, we first build a randomly-
initialized dense neural network and then randomly prune a fixed number of
neurons per layer to generate a subnetwork. We repeat this pruning $N$ times
independently to generate $N$ subnetworks from the same dense network, as seen
in Figs. 3 (a) and (b). Random network pruning at initialization with a fixed
pruning ratio was successfully used to obtain a winning ticket for a
classifier under LTH Su et al. (2020). Whereas Su et al. (2020) adopted
unstructured pruning, we use structured pruning suitable to obtain multiple
policies or critic subnetworks for MARL. In our pruning method, we remove the
group of weights that are connected to the same neuron altogether.
Furthermore, we generate $N$-subnetworks so that they share some subsets of
parameters across multiple agents, and consequently, the parameters connected
to a neuron are trained based on the gradients of all the agents who share
that neuron if the neuron is shared. This enhances sample efficiency. On the
other hand, the parameters connected to a neuron are trained based only on the
gradient of one agent if only one agent owns the neuron. Thus, the obtained
subnetworks as such allow multiple agents to learn common, locally common, and
individual features when they are assigned to different agents. Such learning
can improve overall representational capacity by reinforcing agent indication
compared with simple parameter sharing or parameter sharing with one-hot
vector and also maintains sample efficiency through partial parameter sharing.
In addition, we can handle the amount of parameter-sharing across agents by
adjusting the pruning ratio of each layer without introducing additional
parameters. By controlling the pruning ratio, we can find a soft spot
achieving both high overall representational capacity and sample efficiency
for high performance. Note that such joint control of overall representational
capacity and sample efficiency for MARL has not been achieved by the previous
MARL parameter-sharing methods. We summarized the procedure for generating
winning group tickets based on the proposed structured network pruning method
in Algorithm 1, and Fig. 3 illustrates the overall procedure.

After obtaining the winning group tickets, we start training based on a MARL
algorithm. Note that we need only one dense network to represent all $N$
subnetworks. With the assumption of centralized training with decentralized
execution (CTDE), the gradients of the parameters are averaged over multiple
agents. As aforementioned, this allows $N$-subnetworks to learn both common
and individual features.

|  |  |  |
---|---|---|---|---
(a) 5m vs 6m (SMAC) | (b) 8m vs 9m (SMAC) | (c) MMM2 (SMAC) | (d) LBF1 | (e) LBF2
Figure 4. Performance of FuPS (green), FuPS+id (blue), SePS (black), SNP-PS (red, ours), and SNP-PS+id (red-dotted, ours) on SMAC environments and LBF environments. More results are provided in Appendix. Table 1. Environment information and pruning ratio of the proposed method in each environment. The pruning ratio ’0-0.2-0.6’ means that the ratios of pruned neurons in the first, second, and third layers are $0\%$, $20\%$, and $60\%$, respectively. In the LBF environments, we used the same pruning ratio for actors and critics. A and C denote actors and critics, respectively Environment |  $\\#$ Agents |  $\\#$ Type | Pruning ratio
---|---|---|---
5m vs 6m | 5 | 1 | 0.1-0.1
8m vs 9m | 8 | 1 | 0-0.1
27m vs 30m | 27 | 1 | 0-0.1
MMM2 | 10 | 3 | 0-0.1
LBF1 | 6 | 2 | 0-0.1-0.9 (C) 0-0.1-0.1 (A)
LBF2 | 6 | 3 | 0-0.1-0.9 (C) 0-0.1-0.1 (A)

##  5\. Experiments

In order to compare the proposed parameter-sharing method with other
parameter-sharing methods, we implemented the proposed method and other
baselines on top of QMIX and multi-agent A2C Rashid et al. (2018); Christianos
et al. (2021). We considered three parameter-sharing baselines: 1) FuPS -
simple full parameter sharing in which all agents share the same parameters
without agent indication. 2) FuPS+id - full parameter sharing with one-hot
encoding agent indication. 3) SePS Christianos et al. (2021) \- parameter
sharing within each subgroup and the subgroups are constructed based on a
clustering algorithm. We also considered the combination of our SNP-PS and
one-hot encoding agent indication, which is denoted as SNP-PS+id.

###  5.1. Environments

Starcraft II We evaluated the proposed method on the StarcraftII
micromanagement (SMAC) environment, which is commonly used as a benchmark of
Dec-POMDP Samvelyan et al. (2019). We consider four tasks in SMAC: 5m vs 6m,
8m vs 9m, 27m vs 30m and MMM2. MMM2 is a heterogeneous task consisting of
three different types of agents, whereas the others are homogeneous tasks. For
these tasks, we evaluate all parameter-sharing algorithms on top of QMIX. QMIX
originally uses parameter sharing with one-hot encoding for individual action-
value functions. For a fair comparison, we replace the parameter-sharing
method for the individual action-value functions with the considered methods.
The individual action-value functions have two 64-dimensional hidden vectors
with three layers consisting of a GRU layer and two fully-connected layers
before and after the GRU layer.
Level-based Foraging We also evaluated the proposed parameter-sharing method
on the level-based forging (LBF) environment Albrecht and Ramamoorthy (2015)
in which multiple agents forage randomly generated foods on a two-dimensional
grid. The agents and foods have their own level and the agents can forage a
food if the sum of levels of the agents who are adjacent to the food is equal
to or larger than the level of the food. The agents receive a reward when they
forage a food. We consider two LBF tasks that have different levels of agent
and food. LBF1 consists of six agents whose levels are $(1,1,1,2,2,2)$ and six
foods whose level is $3$. LBF2 consists of six agents whose levels are
$(1,1,2,2,3,3)$ and six foods whose level is $4$. The agent and food levels
here were slightly modified from those in Christianos et al. (2021) to make a
more difficult environment. In this environment, we implemented the proposed
method on top of the multi-agent A2C algorithm considered in Christianos et
al. (2021). We applied the parameter-sharing methods including the proposed
method to both actor and critic. The actor and the critic consist of three
128-dimensional hidden vectors with four fully connected layers.

Table 1 provides the details regarding the environments and the pruning ratio
of the proposed method which is a hyperparameter.

|  |
---|---|---
(a) 5m vs 6m | (b) 8m vs 9m | (c) MMM2
Figure 5. Comparison with other pruning methods: unstructured network pruning
with parameter sharing (USNP-PS, purple) and structured network pruning with
native parameter sharing (SNP-NPS, orange). USNP-PS removes the weights
randomly while keeping the pruning ratio per layer. SNP-NPS generates one
subnetwork based on the structured pruning method and then all agents share
the subnetwork.

###  5.2. Results

Starcraft II Figs. 9 (a)-(d) show the performance of the proposed method and
the considered baselines on the SMAC environments. Parameter sharing with one-
hot encoding agent indication (denoted as FuPS+id) performs better than simple
parameter sharing (denoted as FuPS) in 5m vs 6m and 8m vs 9m tasks, which
consist of a relatively small number of agents. However, agent indication
based on one-hot encoding does not improve performance in the 27m vs 30m task
consisting of many agents and the heterogeneous task (MMM2), and even degrades
the training speed. SePS improves performance in 5m vs 6m compared with FuPS,
but performs equally or worse in the other tasks. It seems that the sample
inefficiency of SePS degrades the performance. The proposed method outperforms
all the considered baselines. Especially in 5m vs 6m, which requires high-
quality coordination, the performance improvement of our SNP-PS is significant
due to the high representational capacity of the proposed method, resulting in
diverse behaviors.
Level-based Foraging Figs. 9 (e)-(f) show the performance of the proposed
method and the considered baselines on the LBF environments. In terms of the
final performance and training speed, the proposed method outperforms the
considered baselines.

###  5.3. Analysis: Construction of the Winning Group Tickets

We proposed using randomly neuron-pruned subnetworks as winning group tickets
for the lottery group ticket hypothesis. As seen in Fig. 9, the proposed
method outperforms the simple parameter sharing, which implies it succeeds in
finding winning group tickets. To see the effectiveness of structured pruning,
we examine whether other pruning methods such as unstructured network pruning
can obtain winning group tickets or not.

Unstructured pruning. We compared our structured network pruning method with
the unstructured pruning method (denoted as USNP-PS in Fig. 5, which removes
the weights randomly with the same pruning ratio. As seen in Fig. 5, the
unstructured pruning method performs worse than the simple parameter sharing,
which implies it fails to find winning group tickets.

Parameter sharing with one pruned subnetwork. The proposed pruning method
generates $N$ different subnetworks that come from a root network. It enables
agents to share parts of the parameters. In order to see if performance
improvement comes from the use of the pruned subnetwork, we compared the
proposed method with the simple parameter sharing based on one selected pruned
subnetwork, which is denoted as SNP-NPS in Fig. 5. That is, SNP-NPS generates
one subnetwork based on the structured pruning method instead of generating
$N$ subnetworks independently, and then all agents use the same pruned network
for naive parameter sharing. As seen in Fig. 5, SNP-NPS is not a winning group
ticket. It is seen that the two considered methods are not an appropriate way
to find winning group tickets, but the proposed method is an adequate way to
construct winning group tickets outperforming the naive parameter-sharing
method in the considered environments.

###  5.4. Analysis: Pruning Ratio

We provided the pruning ratio we used in Table 1. In the SMAC environment, we
found that pruning the first hidden neurons performs better in hard tasks such
as 5m vs 6m, but keeping all neurons in the first hidden neurons performs
better in the other relatively easy environments. In the LBF environment, we
conducted experiments to show the effect of the pruning ratio on the actor and
critic networks, respectively.

The effect of pruning on actor network. We conducted experiments by fixing the
pruning ratio of the critic neural network and changing that of the actor
neural network. The result is shown in Fig. 6 (a). It is seen that the
performance is the highest when the pruning ratio is the smallest, but the
steady-state performance is not sensitive to the pruning ratio of the actor
neural network.

The effect of pruning on critic network. We conducted experiments by fixing
the pruning ratio of the actor neural network and changing that of the critic
neural network. The result is shown in Fig. 6 (b). It is seen that the
training speed and the final performance increase as the pruning ratio of the
critic network increases. Since agents are heterogeneous and receive
individual rewards in the LBF environment, each agent should estimate its own
return and thus the critic should be more identifiable. Highly identifiable
critics can be obtained by increasing the pruning ratio of the critic network.

Note that the best pruning ratio can vary across environments and thus we
leave how to choose the proper pruning ratio as future work.

|
---|---
(a) | (b)
Figure 6. Performances with respect to the pruning ratio (PR) of actor networks and critics networks. (a) the PR of critic networks is fixed as ’0-0.1-0.9’ and that of the last hidden vector in actor networks varies from $0.1$ to $0.9$. (b) the PR of actor networks is fixed as ’0-0.1-0.1’ and that of the last hidden vector in critic networks varies from $0.1$ to $0.9$. |  |  |
---|---|---|---
|  |  |
Figure 7. T-SNE plot of hidden features: 1st row - FuPS+id and 2nd row - SNP-
PS. Each column corresponds to the result of 2nd hidden feature of actor, 3rd
hidden feature of actor, 2nd hidden feature of critic, and 3rd hidden feature
of critic, respectively. We consider three agents in the LBF1 environment,
with the blue, red, and green circles denoting each respective agent.

###  5.5. Analysis: Hidden Representations

The proposed method allows agents to perform different actions even with the
same input by increasing the representational capacity of naive parameter
sharing. To analyze this, we visualize the hidden features of three agents’
networks with the same observation. Note that native parameter sharing yields
identical hidden features for all agents given the same observation. We here
investigate the hidden features generated by the proposed method and the one-
hot encoding agent indication, as shown in Fig. 7. The hidden features of
multiple agents generated by the one-hot encoding agent indication display a
significant degree of overlap or minimal variation, whereas those generated by
the proposed method comprise both partially overlapped features and non-
overlapping features, which shows the agents with the same observation have
either distinct or similar hidden representation. Thus, in Fig. 7, it is shown
that the proposed method has a high degree of representational capacity
compared to the one-hot encoding agent indication.

###  5.6. Analysis: Resource

In real-world decision-making problems, environmental resources such as
samples, memory size, and computational cost are constrained. For the
applicability of MARL algorithms to practical problems, MARL algorithms should
be able to perform in resource-constrained environments. We investigate the
considered algorithms from the perspective of resource constraints.

Sample efficiency.  Sample inefficiency is one notorious limitation of RL, and
the problem aggravates in MARL due to increasing variance for gradient
estimation. Since the proposed method learns the common and individual
features effectively, it is seen that the training speed of the proposed
method is the highest among the considered parameter-sharing algorithms. Thus,
the proposed method performs better than other baselines in terms of sample
efficiency.

Memory size. Without a parameter-sharing method, the required memory size
increases linearly as the number of agents increases. Parameter-sharing
methods address this problem, but one-hot encoding agent indication and SePS
still need more parameters than simple parameter sharing. However, the
proposed method requires the same number of parameters as the simple parameter
sharing and learns the common and individual features based on a structured
pruning method. Table 2 shows the required number of parameters for the
considered algorithms in the LBF environment.

Computational cost. We provided the running time during training normalized by
the maximum value in Fig. 8. Since the proposed method computes the
multiplication of the hidden vector and the generated binary mask, it requires
slightly more computation compared with the naive parameter sharing as seen in
Fig. 8. However, multiplication with binary mask is trivial. The proposed
method has a lower computational cost than SePS, which uses a clustering
algorithm and several neural networks for parameterizing the actor and critic.

Table 2. The required number of parameters in the LBF environment. We consider $K=3$ clusters for SePS. Algorithm | FuPS | FuPS+id | SePS | SNP-PS
---|---|---|---|---
$\\#$ parameters | 76k | 78k | 229k | 76k
---
Figure 8. Normalized running time during training.

##  6\. Conclusion

We have proposed a simple yet effective technique for parameter sharing for
multi-agent deep reinforcement learning to improve the representational
capacity of the joint policy and/or critic while keeping high sample
efficiency. The proposed method generates multiple subnetworks for multi-agent
policies and/or critics by randomly and structurally pruning a common dense
network without introducing additional parameters. Subnetworks possess
overlapping shared parameters and individual parameters. Due to this mixture
property, the proposed structured pruning-based parameter-sharing method
achieves both high joint representational capacity and sample efficiency with
a proper pruning ratio. We have evaluated the proposed method on several
benchmark tasks, and numerical results show that the proposed method
noticeably outperforms other parameter-sharing methods. The practical
effectiveness of the proposed method is verified by experiments but
theoretical proof of the lottery ticket hypothesis and lottery group ticket
hypothesis remains, which is a meaningful work in general deep learning
theory.

##  7\. Acknowledgements

This work was partly supported by Institute of Information $\&$ communications
Technology Planing $\&$ Evaluation (IITP) grant funded by the Korea
government(MSIT) (No.2022-0-00124), Development of Artificial Intelligence
Technology for Self-Improving Competency-Aware Learning Capabilities) and the
National Research Foundation of Korea(NRF) grant funded by the Korea
government(MSIT). (NRF-2021R1A2C2009143)

## References

  * (1)
  * Albrecht and Ramamoorthy (2015) Stefano V Albrecht and Subramanian Ramamoorthy. 2015.  A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems.  _arXiv preprint arXiv:1506.01170_ (2015).
  * Christianos et al. (2021) Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. 2021.  Scaling multi-agent reinforcement learning with selective parameter sharing. In _International Conference on Machine Learning_. PMLR, 1989–1998.
  * Foerster et al. (2016) Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. 2016\.  Learning to communicate with deep multi-agent reinforcement learning.  _Advances in neural information processing systems_ 29 (2016).
  * Foerster et al. (2018) Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018\.  Counterfactual multi-agent policy gradients. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 32.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In _International Conference on Learning Representations_.
  * Graesser et al. (2022) Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. 2022.  The State of Sparse Training in Deep Reinforcement Learning. In _International Conference on Machine Learning_. PMLR, 7766–7792.
  * Iqbal and Sha (2019) Shariq Iqbal and Fei Sha. 2019.  Actor-attention-critic for multi-agent reinforcement learning. In _International conference on machine learning_. PMLR, 2961–2970.
  * Jeon et al. (2022) Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. 2022\.  MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer. In _International Conference on Machine Learning_. PMLR, 10041–10052.
  * Kim et al. (2019) Woojun Kim, Myungsik Cho, and Youngchul Sung. 2019.  Message-dropout: An efficient training method for multi-agent deep reinforcement learning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 6079–6086.
  * Kim et al. (2023) Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. 2023.  A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning.  (2023).  arXiv:2303.00451 [cs.MA]
  * Kim et al. (2020) Woojun Kim, Jongeui Park, and Youngchul Sung. 2020.  Communication in multi-agent reinforcement learning: Intention sharing. In _International Conference on Learning Representations_.
  * Livne and Cohen (2020) Dor Livne and Kobi Cohen. 2020.  Pops: Policy pruning and shrinking for deep reinforcement learning.  _IEEE Journal of Selected Topics in Signal Processing_ 14, 4 (2020), 789–801.
  * Lowe et al. (2017) Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Advances in neural information processing systems_ 30 (2017).
  * Mahajan et al. (2019) Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. 2019.  Maven: Multi-agent variational exploration.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Rashid et al. (2018) Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2018.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In _International conference on machine learning_. PMLR, 4295–4304.
  * Samvelyan et al. (2019) Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019.  The starcraft multi-agent challenge.  _arXiv preprint arXiv:1902.04043_ (2019).
  * Sokar et al. (2021) Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. 2021\.  Dynamic sparse training for deep reinforcement learning.  _arXiv preprint arXiv:2106.04217_ (2021).
  * Su et al. (2020) Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. 2020.  Sanity-checking pruning methods: Random tickets can win the jackpot.  _Advances in Neural Information Processing Systems_ 33 (2020), 20390–20401.
  * Sunehag et al. (2018) Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018\.  Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In _AAMAS_.
  * Tan (1993) Ming Tan. 1993.  Multi-agent reinforcement learning: Independent vs. cooperative agents. In _Proceedings of the tenth international conference on machine learning_. 330–337.
  * Terry et al. (2020) Justin K Terry, Nathaniel Grammel, Ananth Hari, and Luis Santos. 2020.  Parameter sharing is surprisingly useful for multi-agent deep reinforcement learning.  (2020).
  * Wang et al. (2021) Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. 2021.  Recent Advances on Neural Network Pruning at Initialization.  _arXiv e-prints_ (2021), arXiv–2103.
  * Yang et al. (2018) Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. 2018\.  Mean field multi-agent reinforcement learning. In _International conference on machine learning_. PMLR, 5571–5580.

## Appendix A: Experimental Results

|  |
---|---|---
(a) 5m vs 6m (SMAC) | (b) 8m vs 9m (SMAC) | (c) MMM2 (SMAC)
|  |
(d) 27m vs 30m (SMAC) | (f) LBF1 | (f) LBF2
Figure 9. Performance of FuPS (green), FuPS+id (blue), SePS (black), SNP-PS
(red, ours), and SNP-PS+id (red-dotted, ours) on SMAC environments and LBF
environments. More results are provided in Appendix.