# Color-switching in an optical parametric oscillator using a phase-conjugate
mirror

B. E. Anderson  1,* J. Zhao  2 Z. Zhou  3 R. Speirs  3 K. M. Jones  4 and P.
D. Lett3,5  1Department of Physics, The American University, Washington, DC,
USA
2Centre for Quantum Computation and Communication Technology, Department of
Quantum Science and Technology, The Australian National University, Canberra,
ACT, Australia
3Joint Quantum Institute, University of Maryland, College Park, MD, USA
4Department of Physics, Williams College, Williamstown, MA, USA
5National Institute of Standards and Technology, Gaithersburg, MD, USA
<EMAIL_ADDRESS>
††journal: opticajournal††articletype: Research Article

We construct a phase-conjugate resonator which passively produces stable
pulses that alternate between the probe and the conjugate colors. The
requisite phase-conjugate mirror inside the resonator is constructed using
non-degenerate four-wave mixing (4WM) in rubidium vapor. The glancing-angle
phase-conjugate mirror is a 100% output coupler, and therefore this resonator
is unusual in that no light circulates the cavity more than once. Without the
gain of the phase-conjugate mirror, the cavity boundary conditions, and thus
resonant modes, are not defined and therefore can be tuned by the pump. The
output of the optical parametric oscillator that is formed above threshold can
passively mode-lock. The phase-conjugate mirror removes thermal or acoustic
instabilities that are on a MHz or slower timescale. This work provides a new
method for stable pulsing using phase-conjugate optics, and suggests a
platform for producing mode-locked pulses with squeezed light, as the 4WM
process has already demonstrated quantum correlations.

##  1 Introduction

Optical phase conjugation (OPC) allows a field to be generated that is
effectively the time-reversal of an incoming wave [1, 2]. Optical phase
conjugation can be realized by non-degenerate four-wave-mixing (4WM) using a
nonlinear medium with a significant third-order electric susceptibility. The
medium that enables OPC acts as a phase-conjugate mirror (PCM), as shown in
Fig. 1A [3, 4, 5, 6, 7, 8]. A notable application of PCMs is to reverse phase
distortions in optical systems. The phase-healing properties of OPC have long
been viewed as a mechanism that could be used to construct stable resonators
with aberrated beams, and essentially self-healing resonators that can
compensate for changing thermal or acoustic phase distortions [9, 10, 11, 12].
Such resonators have also been shown to be stable in longitudinal mode
operation [7, 13, 14].

When phase conjugate optics were first developed in the late 1970s, a number
of spectacular applications were foreseen, however most have not been realized
or are realized only in demonstration [15]. Phase conjugate materials have
typically been either slow to respond, have low gain, or are simply difficult
to use [16]. Although significant work has been done, several theoretical
predictions for resonators based on PCMs have never been explored because of
this.

Although the emphasis in OPC research has often been on correcting spatial
distortions, here we explore a different application of PCM technology by
constructing a type of optical parametric oscillator (OPO) called a phase-
conjugate resonator (PCR) for stable mode-locking [17]. Although mode-locking
using a PCM has been explored theoretically before [18], and demonstrated
experimentally [19], not much further work has been pursued. Color switching
behavior between frequency modes of a PCR was suggested in [17], but has not
been demonstrated experimentally. Conventionally, a PCR is constructed by
replacing a mirror of an optical cavity with a PCM, as in Fig. 1A, where the
PCM is an optically pumped gas of atoms. Here, as described in detail in the
next section, we construct a PCR (Fig. 1D) from a glancing-angle PCM using
non-degenerate 4WM in a forward phase-matched geometry and a ring cavity built
from a fiber loop. Our aim is to investigate the longitudinal mode properties
of this system.

This resonator is unusual in that reproducing an incoming wavefront requires
two trips around its ring cavity. Therefore, the cavity mode spacing is halved
as compared to an OPO that uses a conventional resonator (CR-OPO), as shown in
Fig. 1E. We find that the PCR builds up optical power from spontaneous
emission and, for a sufficiently long cavity, produces mode-locked pulses in
the two non-degenerate colors supported by the 4WM. Although the gain
bandwidth of our resonator is only tens of MHz, our resonator uses a fiber
loop that can be made long enough to support many longitudinal modes. The
resonator is also unusual in that there are no empty cavity modes in the
absence of the gain in the 4WM medium, and even above threshold no light
circulates more than once through the cavity.

Figure 1: The experimental setup. (A) A non-degenerate phase conjugate
resonator. Two counter-propagating pump beams (green) optically pump an atomic
vapor of 85Rb. An incoming red probe, detuned $-\Delta$ from the pumps,
produces a blue non-degenerate phase conjugate, with detuning $+\Delta$. (B)
The atomic level diagram for 85Rb with our 4WM scheme. The probe lies
$\approx$3 GHz to the red of the pump and the conjugate is the same detuning
to the blue. (C) Conventional single-pass forward 4WM. A single strong pump
(green) beam optically pumps an atomic ensemble in the forward direction and
is discarded at the black beam block. A probe seed is amplified and produces a
non-degenerate blue phase conjugate at a glancing angle determined by phase-
matching conditions. The black dashed line depicts the unused (vacuum) input
port. The probe seed and pump have perpendicular polarizations and are
combined on a polarizing beam splitter (PBS). (D) Our PCR. The output
$B_{out}$ is recycled back to the input $A_{in}$ via a single-mode
polarization maintaining optical fiber (SMPMF) causing the system to lase and
produce (at $A_{out}$) pulses alternating between the red and blue modes. The
SMPMF has a length ranging from several meters to over 100m. An etalon can be
placed in $A_{out}$ to separate the red (probe) and blue (conjugate) pulses
which are separated by 6 GHz in frequency. We can also choose to detect the
light via optical heterodyne on a balanced detector. EOAM, electro-optic
amplitude modulator; EOPM, electro-optic phase modulator; NPBS, non-polarizing
beam splitter; ES, electronic subtraction; det, amplified photodiode detector.
See text for further details. (E) Our CR-OPO. The output $B_{out}$ is recycled
via fiber to $B_{in}$, leaving $A_{in}$ to be vacuum. The output at $A_{out}$
is a single color.

##  2 Results

###  2.1 Experimental set-up

Our basic 4WM scheme is shown in Fig. 1B,C. The gain region can be viewed as a
glancing-angle PCM using non-degenerate 4WM [20]. Our 4WM medium is a vapor of
85Rb at $100\>^{\circ}$C contained in a 12 mm long glass cell. The atoms are
pumped by a continuous-wave Ti:sapphire laser producing a 500 mW, 1 mm
diameter beam tuned blue of the D1 (795 nm, 5S1/2 to 5P1/2) transition by
$\Delta=800$ MHz. This detuning is chosen empirically to optimize the output
power by finding a compromise between absorption in the atomic vapor and non-
linear gain. The 4WM cell acts as an optical amplifier with two input ports
and two output ports. If an independent probe beam detuned $\approx$3 GHz (the
ground state splitting in 85Rb) to the red of the pump crosses the pump beam
at an angle of approximately $0.5^{\circ}$ inside the cell as in Fig 1C, it is
amplified via the 4WM. The 4WM also produces a phase conjugate beam $\approx$3
GHz to the blue of the pump, which exits at the same small angle to the pump
on the opposite side of the pump beam from the probe. The angle between the
pump and incident beam is chosen to facilitate phase-matching in the atomic
medium. Since the output probe and conjugate are slightly different
wavelengths from the pump, they must exit at an angle such that
$2\vec{k}_{\text{pump}}=\vec{k}_{\text{probe}}+\vec{k}_{\text{conj}}$. We note
that this angle is determined by the index of refraction of the atomic vapor,
which is due to the vapor’s significant dispersion. In the experiment we tend
to find this angle empirically by aligning an input probe seed relative to the
pump until we see a conjugate stimulated by the four-wave mixing. A typical
single-pass small-signal gain observed under these circumstances is of order
10. This 4WM configuration is often called ‘nearly nondegenerate’ as the
detuning of the probe or conjugate from the frequency of the pump is small
compared to the pump frequency itself [1].

We can create a resonator in two distinct ways. First, in Fig. 1D, the signal
out at $B_{out}$ is sent back into the input port $A_{in}$ (a PCR). Note that
when probe frequency light is sent into $A_{in}$, conjugate frequency light
emerges from $B_{out}$ and vice versa. In the second resonator construction,
in Fig. 1E, the signal at $B_{out}$ is sent back into the input port $B_{in}$
(a CR-OPO). In this case, whichever frequency light is input at $B_{in}$, the
same frequency light is produced at $B_{out}$. In either case of the PCR or
CR-OPO, there is no input seed in normal operation. Before the cavity is
properly aligned with beam angles commensurate for phase-matching, and without
an independent input probe, the system will not build up optical power.
However, when the output of an optical fiber is aligned to either $A_{in}$ or
$B_{in}$, with an angle dictated by phase matching in the 4WM, optical power
can build up on its own from spontaneous emission.

With the fiber installed as in Fig 1D, the maximum output power of our system
is approximately 30 mW. Stable lasing can be achieved with significantly lower
pump powers, however this conversion efficiency of approximately 6% provided
ample power for the simultaneous measurements shown in Fig. 1D,E. This
conversion efficiency is primarily limited by the insertion loss of the
optical fiber from free space, fiber-to-fiber connectors, and long optical
fiber lengths. The length of optical fiber used is varied between several
meters and 105 m in length. Because the 4WM is sensitive to polarization, only
single-mode polarization-maintaining optical fibers (SMPMF) are used. The PCM
is a $100\%$ output coupler, and therefore any light entering the ring cavity
leaves with gain when it returns to the PCM after one round trip [1]. The
phase-conjugate mirror essentially provides a “copy" of the incoming beam, in
the form of a phase-conjugate, that is then injected back into the cavity to
circulate the cavity once. When the phase-conjugate arrives at the vapor cell
one round trip time later, it immediately exits, but not before stimulating a
phase-conjugate of itself which is then sent into the cavity. This cycle
repeats. In a conventional resonator, the cavity defines resonant modes on its
own. In contrast, this PCR configuration contains no empty cavity modes
without the gain medium to recirculate the light.

We measure the output $A_{out}$ which is separated into three paths via beam
splitters. Etalons are used in the first two paths to separate the probe and
conjugate light before sending them to detectors. The last path sends the
light to optical heterodyne detection which is described below. The etalons
have a bandwidth of approximately 1 GHz and each is centered on either the
probe or conjugate carrier frequencies. Fig. 2A shows separately the probe and
conjugate signals from $A_{out}$ using a 105 m optical fiber length, which
corresponds to approximately a 0.5 $\mu$s round trip time. The output is mode-
locked and consists of nearly square pulses that alternate between the probe
and conjugate modes. Because the probe and conjugate frequencies are only
separated by 6 GHz, if $A_{out}$ is measured on a standard near-infra-red
optical detector without an optical filter we would not be able to
discriminate between the probe and conjugate colors. Typically, the probe and
conjugate pulses are of similar intensity, but either may be larger depending
on the alignment of the cavity. The pulsing repeats itself after two round
trip times.

We can investigate the longitudinal mode structure of the field at $A_{out}$
via optical heterodyne measurements instead of direct measurement on a single
photodiode. $A_{out}$ can be mixed in a non-polarizing beamsplitter with light
from the pump that has $\pm 3$ GHz sidebands applied to it via an electro-
optic phase modulator. Each of the sidebands will be close in frequency to the
probe or conjugate such that the heterodyne beat frequency is small enough to
be measured on an amplified photodiode. An example heterodyne signal is shown
in Fig. 3A. The frequency spacing between comb teeth as a function of fiber
length is shown in Fig. 3B. The data is well fit by the function $\delta
f=\frac{c}{2\left(n_{f}L_{f}+L\right)}$, where $c$ is the speed of light,
$n_{f}$ is the index of refraction of the optical fiber, $L_{f}$ is the length
of the optical fiber, and $L$ is the effective remaining path length of free
space and the atoms. The factor of 2 in the denominator reflects the
assumption that two round trips of the cavity are required for the wavefront
to be reproduced. The good agreement with the data confirms this expectation.
Although the pulsing frequency of the PCR is set by the length of fiber, the
maximum pulsing frequency in our PCR is set by properties of the atomic
medium. The bandwidth of the 4WM process is limited by the natural linewidth
of the excited state in rubidium which is on the order of 10 MHz, as discussed
in Sec. 2.3. This bandwidth could potentially be significantly increased in
other, likely solid-state, 4WM platforms.

Figure 2: Mode-locked pulses from the output of the PCR. The output $A_{out}$
in Fig. 1D is split into two and filtered with etalons to separate the two
colors of light. The probe is shown in solid red and the conjugate is shown in
dotted blue. The optical power is normalized to one for both colors. The
bottoms of the pulses reach zero consistent with what is allowed by the 20 dB
extinction ratio of the etalons. All four frames, (A)-(D), are produced under
nominally the same experimental conditions and selected from a series of runs
in which the laser is restarted each time from noise. Although the most likely
pulse period is the one shown in (A), slight differences in the initial
conditions lead to the system occasionally pulsing at higher frequencies as
shown in B-D. The fiber length used is 105 m, corresponding to a 0.5 $\mu$s
round trip time.

###  2.2 Understanding Pulsing in the PCR

The 4WM process, in the absence of a cavity, is fundamentally spatially
symmetric about the pump beam and permits simultaneous generation of both the
probe and conjugate frequencies into the same spatial modes at low
intensities. At high intensities, mode competition for the gain prevents both
frequencies from being present simultaneously in the same spatial mode in the
gain medium. Without some form of mode coupling an OPO would operate single-
mode (or probe-conjugate mode-pair) above threshold [21]. An additional 4WM
process known as 2-beam coupling [2, 22, 23] provides a coupling between
neighboring frequency modes at the probe color. Thus, a set of frequency modes
can stably oscillate together in our system. The mode competition, however,
forces the output to switch between probe and conjugate colors provided there
is sufficient gain bandwidth to support multiple frequencies at each color,
and thus pulse formation. As a high intensity probe beam exits the cavity at
$A_{out}$, it stimulates the production of a high intensity conjugate into the
resonator mode at $B_{out}$.

Although the resonator most often produces pulses with a comb tooth spacing of
$\delta f=\frac{1}{2T}$, where $T$ is the round-trip time of the cavity, the
resonator will sometimes output $n\,\delta\\!f$ where $n$ is an odd natural
number. In Fig. 2B-D we show tooth spacings $3\,\delta\\!f$, $5\,\delta\\!f$,
and $7\,\delta\\!f$. Surprisingly, this data is taken using identical
experimental conditions as in Fig. 2A. To take the different datasets, we ‘re-
start’ the cavity by rapidly increasing and decreasing the optical loss of the
cavity using a fiber electro-optic amplitude modulator placed in the path of
the SMPMF. Empirically, we find the likelihood of seeing $n\,\delta\\!f$
decreases as $n$ increases, and $7\,\delta\\!f$ is only seen a few percent of
the time.

We can understand how these higher pulse frequencies are physically permitted
by first considering the cavity operating at spacing $\delta\\!f$ and just
starting to output probe light. We consider the cavity to be filled with probe
light which is on its way out of $A_{out}$. The probe then stimulates
conjugate light to enter the cavity, filling in behind the probe light. After
time $T$, the cavity is filled with conjugate light and it begins to exit
$A_{out}$. This stimulates probe light to enter the cavity, filling in behind
the conjugate light and completing the cycle. Another stable pattern occurs if
the cavity is initially filled with three pulses of light: the first $1/3$
probe, the second $1/3$ conjugate, and the final $1/3$ probe. At time $T/3$
later, the first probe light in the cavity will have exited, while stimulating
conjugate light to fill in behind the final probe light. Now the cavity is
filled $1/3$ by conjugate, $1/3$ by probe, and $1/3$ by conjugate. Another
time $T/3$ later will reproduce the initial conditions.

If the resonator could operate at $2\,\delta\\!f$, then the cavity would at
some moment be filled $1/2$ by probe and $1/2$ by conjugate light. As the
probe begins to exit, it stimulates conjugate light to fill the cavity. But
this conjugate light is filling in behind the already existing conjugate light
in the first half of the cavity. At time $T/2$ later, the entire cavity is
filled with conjugate light and the system reverts to operating at mode
spacing $\delta\\!f$. Even multiples of $\delta\\!f$ are not stable in the
cavity, whereas odd multiples are stable. Random fluctuations in the initial
turn-on from spontaneous emission choose from the available stable periods.

Figure 3: Heterodyne spectra and comb frequency spacing of the PCR. (A)
Heterodyne spectra of the probe and conjugate light from $A_{out}$ using a 34
m fiber. The centers of the combs are separated by 6 GHz and the pump is
directly in the middle. Comb teeth of the probe and conjugate that are equally
spaced from the pump are linked via 4WM (red arrows show two examples). Two-
beam coupling links adjacent teeth (blue arrows show three examples) on the
probe beam. This process does not involve the pump beam. The frequency centers
of the spectra have been shifted to zero in post-processing for readability,
and are double-sided spectra. (B) The tooth spacing frequency, measured in
heterodyne data such as Fig. 3A, is the inverse of the pulse period. The fit
shown is $\delta f=\frac{c}{2\left(1.5\times L_{f}+6.4\left[m\right]\right)}$
where $L_{f}$ is the optical fiber length. The factor of two comes from the
PCR reproducing the wavefront after two roundtrips (one probe and one
conjugate successively) around the cavity.

###  2.3 Stability of CR-OPO versus PCR

The PCR is stable against acoustic and thermal noise because the OPC process
has no net phase accumulation after two round trips. This contrasts with the
instability of a CR-OPO. Acoustic and thermal changes in the long fiber
lengths cause drifts in the path lengths which affect the pulse timing. The
stability of the PCR is due to the speed of response of the phase-compensation
of the 4WM, which is dictated by optical pumping times; on the order of 1
$\mu$s. As mentioned above, a homogeneously-broadened gain medium, such as a
CR-OPO, should have strong mode competition due to gain saturation, and with a
single-mode pump this should result in the CR-OPO operating on only a single
mode if there is no coupling between the available modes. Mode coupling
continuously feeds population to cavity longitudinal modes that would
otherwise be suppressed by the mode competition, and therefore prevents a
single mode from winning out. In both our CR-OPO and PCR, a mode coupling
mechanism exists via two-beam coupling which feeds neighboring modes and
allows the system to oscillate stably in many modes. This two-beam coupling is
essentially another 4WM process that mixes adjacent probe-color modes that are
separated by a small enough frequency difference, as in Fig. 3A.

Two-beam coupling is possible in our 4WM system as long as different modes are
on the order of a natural linewidth (6 MHz) apart. Given our 3 dB gain
bandwidth of about 20 MHz, if there are three modes that can compete in this
range, they are already coupled even at low intensities in this system. The
two-beam coupling is self-pumped in that the pump beam driving the high-gain
4WM process indicated in Fig. 1B is not involved; only the neighboring probe-
colored cavity modes mix and exchange energy as in Fig. 3A. The conjugate
frequency is too far from resonance for the coupling to be significant at
these intensities.

As the cavity length is made shorter, the mode-spacing increases and the
number of modes that can lase is reduced. Eventually, with a short enough
cavity, the mode spacing is much larger than the approximately 20 MHz
bandwidth and only a single mode of each probe/conjugate color can lase. Both
colors then lase simultaneously and continuously in the resonator. In this
limit, the color-switching behavior of the PCR system is lost. The lasing is
stable regardless of the cavity length, as the PCM allows lasing at any
arbitrary cavity length. With a slightly longer cavity we see sinusoidal
oscillations on a continuous-wave background due to the beating of the central
mode with weak sidemodes that start to lase one cavity mode spacing away.
These oscillations are approximately out of phase in the probe and conjugate
colors, although on the short time scale set by a fiber of several meters,
group velocity delays in the gain medium become significant, as will be
discussed in a later publication.

Figure 4: Comparison of the outputs of the CR-OPO and PCR. (A-C) Output
optical power of the CR-OPO at different times showing the changing waveform
due to instability in the CR-OPO. (D) A spectrogram of the CR-OPO output
showing regular instability on the order of 100 $\mu$s. The time resolution of
the spectrogram is 20 $\mu$s, as explained in the text. The color scale for
the z-axis shows optical power plotted in dB. (E) A spectrogram of the probe-
color modes of the PCR with the same fiber length. The lowest mode is half the
frequency of the CR-OPO and the lasing modes do not show the instability
present in the CR-OPO, despite the PCR having no active stabilization.

###  2.4 CR-OPO Configuration

We can contrast the behavior of the system operated as a PCR (Fig. 1D) to that
of the system operated as an CR-OPO (Fig. 1E). Figure 4 shows data which
allows us to identify three key differences in the output from the two
configurations. First, the output of the CR-OPO is single color, either probe
or conjugate. Second, the period of the CR-OPO output is half that of the PCR
for a given optical path length. One traversal through the resonator
constitutes a full period. Third, the output is significantly more unstable
than the PCR. In Fig. 4A-C we can see the pulse shape of the CR-OPO evolves on
the timescale of 100 $\mu$s. By taking spectrograms of the PCR and CR-OPO
output, we can more easily compare their longer-timescale behavior. To do
this, as shown in Fig. 4D, we record 1 ms of direct optical detection of
$A_{out}$ from the CR-OPO, which has power only at the probe frequency. In
post-processing, we then plot the power spectrum of the time-domain signals,
with example time-domain data given in Fig 4A-C, using successive 20 $\mu$s
windows as a function of time. These power spectrum windows are assembled
together to make a spectrogram. A similar measurement is done for the PCR in
Fig. 4E using the only the output of the probe etalon since the PCR produces
both probe and conjugate light. Both figures use a 105 m fiber length, and we
can see that the minimum frequency mode spacing of the PCR is half that of the
CR-OPO. Instability in the CR-OPO manifests as blurry wider areas on the
otherwise regular modes that are due to optical path length drift. We can
contrast this to the PCR with its stable output.

Figure 5: The turn-on dynamics of the PCR with a 105 m fiber. (A) The probe
output of the PCR. The PCR is rapidly switched on via a fiber electro-optic
amplitude modulator at time = 1 ms. The dashed line provides a logistic curve
guide of the envelope for the eye. (B) A spectrogram of the probe light
showing the quick convergence to regularly spaced modes. The color scale for
the z-axis shows relative optical power plotted in dB. (C) The conjugate
output of the PCR. (D) In post-processing, the conjugate (black) output is
shifted in time by one round trip time of the resonator and overlaid on the
probe (red) output. The outputs are normalized in amplitude.

###  2.5 Turn-on Dynamics

To understand the dynamics of the PCR when it begins to lase, we inserted an
electro-optic amplitude modulator to rapidly vary the optical loss and thus
bring the system from below the lasing threshold to above. Figure 5A (Fig. 5C)
shows the output at the probe (conjugate) frequency after the loss is lowered
at $t=1$ms. A spectrogram of the probe in Fig. 5B shows the quick convergence
of the system to regularly spaced modes after turning on. While the PCM sets
the same maximum gain phase for each mode at the mirror position, dispersion
inside the cavity would cause the mode spacing to be non-uniform. The only
significant dispersion in this system over the small gain bandwidth is caused
by the gain in the atomic vapor/4WM medium itself. Any non-uniformity in the
mode spacing is quickly removed as typical Kerr mode-locking effects that
balance the dispersion and nonlinearities in the coupled-mode system cause the
formation of regular pulses. The complex dynamics shown in Fig. 5A,C stabilize
after several hundred microseconds to the regular square-like pulses shown in
Fig. 2. During the buildup of optical power in the resonator the probe and
conjugate match each other well in shape. By shifting the data for the
conjugate beam forward in time by one period $T$ in post-processing, one can
see that the two signals match well, as shown in Fig. 5D.

##  3 Conclusion

Aspects of the behavior shown here are reminiscent of behavior seen in Kerr
soliton systems [24, 25]. Our 4WM mixing process provides nonlinearity and
gain, as well as dispersion. In our system dispersion in the fiber is
negligible over our limited bandwidth. Previous work has demonstrated the
dispersion of the non-linear process via a difference in group velocity
between probe and conjugate pulses [26]. Absorption in the atomic medium and
coupling loss into the optical fiber provide dissipation. These are the
ingredients of soliton formation in dissipative Kerr frequency combs. In the
current observations reported above, the pulse duration is primarily set by
the cavity length rather than the properties of the medium, suggesting the
pulses are not solitons. Possibly under different conditions, the PCR could be
modified to produce solitons.

The PCR does not, by itself, define mode positions without gain. The maximum
gain position can be tuned by the detuning of the pump frequency. Thus,
arbitrary mode positions can be obtained enabling the system to have one of
the oscillating modes placed at any frequency within the gain bandwidth. This
freedom to place a lasing line at any arbitrary frequency often requires
physically modifying the resonator in frequency comb devices.

We have established a method for producing mode-locked pulses that exhibit a
novel color-switching behavior. In the PCR configuration, the system is
naturally stable in longitudinal mode due to the phase-conjugate mirror
stabilizing accumulated phase after two round trips of the cavity. The 4WM
process used here has been shown to generate strong quantum correlations
(2-mode squeezing between the probe and conjugate beams) under similar
detuning conditions and with low enough loss [20, 27, 28]. A suitably
optimized PCR system might provide outputs which are quantum correlated.

This proof-of-principle demonstration shows that one can use a phase-conjugate
resonator to create stable, mode-locked, pulses between two frequencies of
light. Although the bandwidth of the system is limited, the physics of this
color-switching process is novel and it could potentially be utilized in
larger bandwidth 4WM systems or parametric down conversion systems. By cavity
dumping the PCR, it is conceivable that one could extract large power quantum-
correlated twin pulses.

Funding Support for KMJ’s contribution was provided by the National Science
Foundation. This work was supported by the Air Force Office of Scientific
Research grant FA9550-16-1-0423.

Acknowledgments We thank Kartik Srinivasan, Gregory Moille, and Jared
Wahlstrand for useful discussions.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not
publicly available at this time but may be obtained from the authors upon
reasonable request.

## References

  * [1] R. Fisher, _Optical Phase Conjugation_ (Academic Press, New York, New York, 1983).
  * [2] R. W. Boyd, _Nonlinear optics_ (Academic Press, an imprint of Elsevier, London, United Kingdom, 2020), fourth edition ed. OCLC: 1164488965.
  * [3] J. Feinberg and R. W. Hellwarth, “Phase-conjugating mirror with continuous-wave gain,” Optics Letters p. 3 (1980).
  * [4] A. Yariv, “Phase conjugate optics and real-time holography,” IEEE Journal of Quantum Electronics 14, 650–660 (1978).
  * [5] P. F. Liao, D. M. Bloom, and N. P. Economou, “cw optical wave-front conjugation by saturated absorption in atomic sodium vapor,” Applied Physics Letters 32, 813–815 (1978).
  * [6] R. W. Hellwarth, “Generation of time-reversed wave fronts by nonlinear refraction*,” Journal of the Optical Society of America 67, 1 (1977).
  * [7] M. Cronin-Golomb, B. Fischer, S.-K. Kwong, _et al._ , “Nondegenerate optical oscillation in a resonator formed by two phase-conjugate mirrors,” Optics Letters 10, 353 (1985).
  * [8] A. Yariv and D. M. Pepper, “Amplified reflection, phase conjugation, and oscillation in degenerate four-wave mixing,” Optics Letters 1, 16 (1977).
  * [9] D. M. Bloom and G. C. Bjorklund, “Conjugate wave-front generation and image reconstruction by four-wave mixing,” Applied Physics Letters 31, 592–594 (1977).
  * [10] I. M. Bel’dyugin, M. G. Galushkin, and E. M. Zemskov, “Properties of resonators with wavefront-reversing mirrors,” Soviet Journal of Quantum Electronics 9, 20–23 (1979).
  * [11] D. M. Pepper and A. Yariv, “Compensation for phase distortions in nonlinear media by phase conjugation,” Optics Letters 5, 59 (1980).
  * [12] R. C. Lind and D. G. Steel, “Demonstration of the longitudinal modes and aberration-correction properties of a continuous-wave dye laser with a phase-conjugate mirror,” Optics Letters 6, 554 (1981).
  * [13] J. F. Lam and W. P. Brown, “Optical resonators with phase-conjugate mirrors,” Optics Letters p. 3 (1979).
  * [14] J. Auyeung, D. Fekete, D. Pepper, and A. Yariv, “A theoretical and experimental investigation of the modes of optical resonators with phase-conjugate mirrors,” IEEE Journal of Quantum Electronics 15, 1180–1188 (1979).
  * [15] M. Y. Lanzerotti and A. L. Gaeta, “Theory of quantum-optical measurements with a phase-conjugate mirror,” Physical Review A 51, 4057–4061 (1995).
  * [16] J. Nilsen, N. S. Gluck, and A. Yariv, “Narrow-band optical filter through phase conjugation by nondegenerate four-wave mixing in sodium vapor,” Optics Letters 6, 380 (1981).
  * [17] P. A. Bélanger, A. Hardy, and A. E. Siegman, “Resonant modes of optical cavities with phase-conjugate mirrors,” Applied Optics 19, 602 (1980).
  * [18] G. R. Gray, G. P. Agrawal, and D. H. DeTienne, “Mode locking in semiconductor lasers by phase-conjugate optical feedback,” Optics Letters 20, 1295 (1995).
  * [19] H. Vanherzeele, J. L. Van Eck, and A. E. Siegman, “Mode-locked laser oscillation using self-pumped phase-conjugate reflection,” Optics Letters 6, 467 (1981).
  * [20] C. F. McCormick, V. Boyer, E. Arimondo, and P. D. Lett, “Strong relative intensity squeezing by four-wave mixing in rubidium vapor,” Optics Letters 32, 178 (2007).
  * [21] C. Fabre, M. Vaupel, N. Treps, _et al._ , “C.w. optical parametric oscillators: single mode or multimode?” Comptes Rendus de Academie des Sciences 1, 553–559 (2000).
  * [22] M.-C. Wu, N. R. Brewer, R. W. Speirs, _et al._ , “Two-beam coupling in the production of quantum correlated images by four-wave mixing,” Optics Express 29, 16665 (2021).
  * [23] M. Kauranen, A. L. Gaeta, R. W. Boyd, and G. S. Agarwal, “Amplification of vacuum fluctuations by two-beam coupling in atomic vapors,” Physical Review A 50, R929–R932 (1994).
  * [24] T. Herr, V. Brasch, J. D. Jost, _et al._ , “Temporal solitons in optical microresonators,” Nature Photonics 8, 145–152 (2014).
  * [25] J. R. Ackerhalt and P. W. Milonni, “Solitons and four-wave mixing,” Physical Review A 33, 3185–3198 (1986).
  * [26] V. Boyer, C. McCormick, E. Arimondo, and P. Lett, “Ultraslow Propagation of Matched Pulses by Four-Wave Mixing in an Atomic Vapor,” Physical Review Letters 99, 143601 (2007).
  * [27] A. L. Gaeta and R. W. Boyd, “Quantum Noise in Phase Conjugation,” Physical Review Letters 60, 2618–2621 (1988).
  * [28] G. S. Agarwal, A. L. Gaeta, and R. W. Boyd, “Quantum statistics of the light generated by phase-conjugate resonators,” Physical Review A 47, 597–601 (1993).


# Multi-Agent Actor-Critic
with Hierarchical Graph Attention Network

Heechang Ryu, Hayong Shin, Jinkyoo Park
Industrial & Systems Engineering, KAIST, Republic of Korea
{rhc93, hyshin<EMAIL_ADDRESS>Corresponding author

###### Abstract

Most previous studies on multi-agent reinforcement learning focus on deriving
decentralized and cooperative policies to maximize a common reward and rarely
consider the transferability of trained policies to new tasks. This prevents
such policies from being applied to more complex multi-agent tasks. To resolve
these limitations, we propose a model that conducts both representation
learning for multiple agents using hierarchical graph attention network and
policy learning using multi-agent actor-critic. The hierarchical graph
attention network is specially designed to model the hierarchical
relationships among multiple agents that either cooperate or compete with each
other to derive more advanced strategic policies. Two attention networks, the
inter-agent and inter-group attention layers, are used to effectively model
individual and group level interactions, respectively. The two attention
networks have been proven to facilitate the transfer of learned policies to
new tasks with different agent compositions and allow one to interpret the
learned strategies. Empirically, we demonstrate that the proposed model
outperforms existing methods in several mixed cooperative and competitive
tasks.

## Introduction

In nature, the battle for dominance rights or desirable territories between
individuals is a typical phenomenon (?). Occasionally, individuals in the same
group cooperate to compete against enemy groups. They can gain stronger
immunity against predators (?) or powerful forces to overpower preys (?). The
cooperation and competition among agents are also important modeling paradigms
in various engineering systems, such as smart grids (?), logistics (?; ?), and
distributed vehicles$/$robots (?; ?; ?). To control such complex systems
composed of many interacting components, researchers have studied multi-agent
reinforcement learning (MARL) for a long time.

Recently, reinforcement learning (RL) combined with deep neural networks has
achieved human-level or higher performances in challenging games (?; ?; ?).
The advances in RL and deep learning have led great interests in MARL, hoping
that it can resolve complex and large scale problems. The majority of MARL
algorithms have focused on deriving decentralized policies for conducting a
collaborative task. For the collection of actions individually determined by
decentralized policies to be coordinated, it is important to impose consensus
when deriving the decentralized policies. To achieve this, under the concept
of centralized training and decentralized execution (CTDE), MARL learns
centralized critics for multiple agents and derives decentralized actors using
partial gradient from the centralized critics. Depending on the information
available in the execution phase, CTDE approach can be further categorized
into “learning-for-consensus approach”, where each agent determines
decentralized action solely based on its local observation (?; ?; ?; ?), and
“learning-to-communicate approach”, where each agent employs communication
schemes (i.e., interchanges messages) during the execution phase (?; ?; ?; ?;
?).

Although a variety of MARL algorithms have been proposed, unresolved issues
still exist in MARL when it is applied to realistic environments. First, in
terms of the generality of the modeling framework, most models focus on
deriving pure cooperation or competition among multiple agents by forcing all
agents to seek a shared common reward rather than modeling the relationships
among heterogeneous agents in a mixed cooperative-competitive task (issue in
modeling flexibility). In addition, the models are limited for modeling a
large number of agents owing to the curse of dimensionality in modeling
centralized critics (issue in scalability). This is connected with a more
fundamental limitation, in which the trained model cannot be transferred to
different tasks with different numbers of agents having different goals (issue
in transferability).

We herein propose a model, called Hierarchical graph Attention-based Multi-
Agent actor-critic (HAMA), that conducts both representation learning for
multi-agent system and policy learning using multi-agent actor-critic in end-
to-end learning. HAMA employs a hierarchical graph neural network to
effectively model the inter-agent relationships in each group of agents and
inter-group relationships among groups. HAMA additionally employs inter-agent
and inter-group attentions to adaptively extract the state-dependent
relationships among multiple agents, which is proven to be effective for
helping policies to adjust their high-level strategies (e.g., cooperate or
compete). The combination of hierarchical graph neural networks with two
distinct attention layers, which we refer to as a Hierarchical Graph Attention
neTwork (HGAT), effectively processes the local observation of each agent into
a single embedding vector, an information-condensed and contextualized state
representation for each individual agent. HAMA sequentially uses the embedding
vector for each agent to compute the individual critic and actor for deriving
decentralized policies.

We empirically demonstrate that HAMA outperforms existing MARL algorithms in
four different game scenarios. Furthermore, we demonstrate that the policies
trained by HAMA in a small-scale game with a small number of agents can be
applied directly to control a large number of agents in a new game. Finally,
we demonstrate that inter-agent and inter-group attentions can be used to
interpret the derived policy and decision-making process.

## Related Work

We categorize existing MARL studies into two categories; learning-for-
consensus approach and learning-to-communicate approach depending on how the
consensus among multiple agents is derived.

#### Learning-for-Consensus Approaches.

Learning-for-consensus approaches in MARL focus on deriving decentralized
policies (actors) for agents, each of which maps a local observation for an
agent to an individual action for it. To make such individually chosen actions
be coordinated to conduct collaborative tasks, these approaches first
construct a centralized critic for either a global reward or individual reward
and use the centralized critic to derive the decentralized actor. MADDPG (?)
has extended DDPG (?) to multi-agent settings for mixed cooperative-
competitive environments. COMA (?) constructs a centralized critic and
computes an agent-specific advantage function to derive a decentralized actor.
FDMARL (?) has proposed a distributed learning approach for each agent to
learn a global critic using its local reward and the transferred critic
parameters from the networked neighboring agents. Because these models
directly use the state or observation in constructing critic or actor
networks, it is difficult to apply such models to a large-scale environment or
transfer them to new environments. As a way to resolve the scalability issue,
the concepts of graph neural network (?; ?; ?) and attention network have been
employed to effectively represent the global state and accordingly centralized
critics. For example, MAAC (?) employs the attention network and graph neural
network to model a centralized critic, from which decentralized actors are
derived using soft actor-critic (?). In addition, DGN (?) applies a graph
convolutional network to model a centralized Q-function for each agent using a
deep Q-network (?).

#### Learning-to-Communicate Approaches.

Another method to achieve consensus among decentralized policies in
cooperative environments is using communication among agents. In this
framework, each agent learns how to transmit messages to other agents and
process the messages received from other agents to determine an individual
action. During the centralized training, such message generating and
processing procedures are learned to induce cooperation among agents. During
the execution phase, agents exchange the messages to determine their actions.
CommNet (?) uses a large single neural network to process all the messages
transmitted by all agents globally, and the processed message is used to guide
all agents to cooperate. BiCNet (?) has proposed a communication channel in
the form of bi-directional recurrent network to accommodate messages from any
number of agents, thus resolving the scalability issue. To effectively specify
the communication structure while considering the relative relationships among
agents, ATOC (?) has proposed a communication channel in a bi-directional LSTM
with an attention layer. The attention layer in ATOC enables each agent to
process the messages from other agents differently depending on their state-
dependent importance. Similar to ATOC, IC3Net (?) has been proposed to
actively select and mask messages from other agents during communication by
applying gating function in the message aggregation step. TarMAC (?) has
proposed a targeted communication protocol to determine whom to communicate
with and what messages to transmit using attention networks. Such
communication-based methods use attention networks to learn the communication
structure$/$protocol effectively. Although the communication helps each agent
to use extensive information in its individual decision making, this approach
requires a separate communication channel and a well-established communication
environment for message exchange. Furthermore, some communication-based
methods can be applied to only cooperative tasks because it does not make
sense to receive messages from competitive agents while playing a game.

#### Novelties of HAMA.

As introduced, the graph neural network and attention network structures have
been widely employed (1) to model a critic for scalable learning in learning-
for-consensus approach, and (2) to model communication structure in learning-
to-communicate approach. HAMA, our proposed model, employs the HGAT to embrace
the merits of employing a graph representation in both learning-for-consensus
and learning-to-communicate approaches. The proposed HGAT capturing enhanced
relative inductive biases (?) enables HAMA to model both centralized critic
and decentralized actor (1) that can be scalable and transferable and (2) that
can effectively utilize the contextualized state representation. In
particular, because it learns how to represent the partial observation through
graph embedding, it is different from other communication approaches that
utilize the messages processed by other agents. Therefore, our approach is
considered to be a kind of learning-for-consensus approaches and can be
applied for mixed cooperative and competitive environments easily.

Figure 1: Overview of HAMA.

## Background

#### Partially Observable Markov Game (POMG).

A POMG is an extension of partially observable Markov decision process to a
game with multiple agents. A POMG for $\mathnormal{N}$ agents is defined as
follows: $s\in\mathcal{S}$ denotes the global state of the game;
${o_{i}}\in\mathcal{O}_{i}$ denotes a local observation that agent $i$ can
acquire; $a_{i}\in\mathcal{A}_{i}$ is an action for agent $i$. The reward for
agent $i$ is computed as a function of state $s$ and joint action $\mathbf{a}$
as
${r}_{i}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\mapsto\mathbb{R}$.
The state evolves to the next state according to the state transition model
$\mathcal{T}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\mapsto\mathcal{S}$.
The initial state is determined by the initial state distribution
$\rho:\mathcal{S}\mapsto[0,1]$. The agent $i$ aims to maximize its discounted
return $R_{i}=\sum_{t=0}^{T}\gamma^{t}r^{t}_{i}$, where $\gamma\in[0,1]$ is a
discount factor.

#### Multi-Agent Deep Deterministic Policy Gradient (MADDPG).

Deterministic policy gradient (DPG) (?) aims to directly derive a
deterministic policy, $a=\mu(s;\theta)$, that maximizes the expected return
$\mathcal{J(\theta)}=\mathbb{E}_{s\sim\rho^{\mu},a\sim\mu_{\theta}}[R]\approx\mathbb{E}_{s\sim\rho^{\mu},a\sim\mu_{\theta}}[Q^{\mu}(s,a;\phi)]$,
where
$Q^{\mu}(s,a;\phi)=\mathbb{E}_{s^{\prime}}[r(s,a)+\gamma\mathbb{E}_{a^{\prime}\sim\mu}[Q^{\mu}(s^{\prime},a^{\prime})]$.
The parameter $\theta$ of $\mu(s;\theta)$ is subsequently optimized by the
gradient of $\mathcal{J(\theta)}$:
$\nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}_{s\sim\mathcal{D}}[\nabla_{\theta}\mu(s;\theta){\nabla_{a}}Q^{\mu}(s,a;\phi)\arrowvert_{a=\mu(s;\theta)}]$.
The $\mathcal{D}$ is an experience replay buffer that stores
$(s,a,r,{s^{\prime}})$ samples. Deep deterministic policy gradient (DDPG), an
actor-critic model based on DPG, uses deep neural networks to approximate the
critic and actor of each agent.

MADDPG is a multi-agent extension of DDPG for deriving decentralized policies
for the POMG. MADDPG comprises the individual $Q$-network and policy network
for each agent. The $Q$-network for agent $i$ is learned by minimizing the
loss:
$\mathcal{L}(\phi_{i})={\mathbb{E}_{\mathbf{o},\mathbf{a},r,{\mathbf{o}^{\prime}}\sim\mathcal{D}}}[(Q_{i}^{\mu}(\mathbf{o},\mathbf{a};\phi_{i})-y_{i})^{2}],$
where $\mathbf{o}=(o_{1},\dots,o_{N})$ and $\mathbf{a}=(a_{1},\dots,a_{N})$
are, respectively, the observations and actions of all agents, and
$y_{i}=r_{i}(s,\mathbf{a})+\gamma{Q_{i}^{\mu^{\prime}}}({\mathbf{o}^{\prime}},\mathbf{a}^{\prime};{\phi_{i}}^{\prime})\arrowvert_{a_{j}^{\prime}=\mu^{\prime}(o^{\prime}_{j});\theta^{\prime})}$.
The policy network $\mu_{i}(o_{i};\theta_{i})$ of agent $i$ is optimized using
the gradient:
$\nabla_{\theta_{i}}\mathcal{J}(\theta_{i})=\mathbb{E}_{\mathbf{o},\mathbf{a}\sim\mathcal{D}}[\nabla_{\theta_{i}}\mu_{i}(o_{i};\theta_{i})\nabla_{a_{i}}Q_{i}^{\mu}(\mathbf{o},\mathbf{a};\phi_{i})\arrowvert_{a_{i}=\mu_{i}(o_{i};\theta_{i})}]$.

#### Graph Attention Network (GAT).

The GAT (?) is an effective model to process structured data that is
represented as a graph. The GAT has proposed a way to compute the node-
embedding vector of graph nodes by aggregating node embeddings $h_{j}$ from
neighboring nodes $\\{j\in\mathcal{N}_{i}\\}$ that are connected to the target
node $i$ as
$h^{\prime}_{i}=\sigma(\sum_{j\in\mathcal{N}_{i}}\alpha_{ij}\mathbf{W}h_{j})$.
The attention weight $\alpha_{ij}=\text{softmax}_{j}(e_{ij})$, where
$e_{ij}=a(\mathbf{W}h_{i},\mathbf{W}h_{j})$, quantifies the importance of node
$j$ to node $i$ in computing node-embedding value $h^{\prime}_{i}$.

## Methods

HAMA comprises a representation learning framework for processing the state
represented as a graph and a multi-agent actor-critic network for deriving
decentralized policies for the agents. As shown in Figure 1, HAMA represents
the game state as a graph and computes for each agent the node-embedding
vector that compactly summarizes each agent’s status in relation with other
groups of agents and environment. The computed node-embedding vector for each
agent is subsequently used to compute the Q-value and action in an actor-
critic framework.

### State Representation Using HGAT

We propose HGAT, a network stacking multiple GATs hierarchically, that
processes each agent’s local observation into a high-dimensional node-
embedding vector to represent the hierarchical inter-agent and inter-group
relationships of each agent.

#### Agent Clustering.

The first step in representation learning is to cluster all the agents into
distinct groups $C^{k}$ using prior knowledge or data. For pure cooperative
tasks, all the agents can be categorized into a single group. If the target
task involves competition between two groups, we can cluster the agents into
two groups. In addition, we can cluster into a group the agents that do not
execute any actions but participate in the game (i.e., terrain components or
obstacles). In this study, we assume that the agents can be easily clustered
into $K$ groups using prior knowledge on the agents, which implies that HAMA
utilizes enhanced relative inductive biases regarding the group relationships.

#### Node-Embedding Using GAT in Each Cluster.

Agent $i$ has the local observation $o_{i}=\\{s_{j}\arrowvert\ j\in V(i)\\}$
where $s_{j}$ is the local state of agent $j$, and $V(i)$ specifies the visual
range of agent $i$. The visual range can be specified depending on environment
settings so that agent $i$ can observe the agents within a certain distance.
Thus, our agent can observe nearby agents as a partial observation. Agent $i$
computes the different node-embedding vectors $\bar{h}_{i}^{k}$ for different
groups $k=1,...,K$ to summarize the individual relationships between agent $i$
and agents from different groups. To compute $\bar{h}_{i}^{k}$, agent $i$
first computes embedding $h^{k}_{ij}=f^{k}_{M}(s_{i},s_{j};w^{k}_{M})$ between
itself and agents in ${j\in C^{k}\cap V(i)}$ and computes the aggregated
embedding $\bar{h}^{k}_{i}=\sum_{j\in{C^{k}\cap
V(i)}}\alpha^{k}_{ij}h^{k}_{ij}$. The inter-agent attention weight
$\alpha_{ij}^{k}$ quantifies the importance of the embedding $h^{k}_{ij}$ from
agent $j$ to agent $i$. The inter-agent attention weight is computed as
softmax $\alpha^{k}_{i,\cdot}\propto\text{exp}(e^{k}_{i,\cdot})$ where
$e^{k}_{ij}=f^{k}_{\alpha}(s_{i},s_{j};w^{k}_{\alpha})$. The attention can be
extended to multiple attention heads (?), but the current study employs only
plain and classical attention networks. It is noteworthy that agent $i$
computes embedding $h^{k}_{ij}$ by processing its own observation on other
agents; therefore, the other agents are not required to send messages to agent
$i$, unlike other learning-to-communicate approaches that require agents to
exchange their hidden vectors.

#### Hierarchical State Representation Using Multi-Graph Attention.

This step aggregates the group-level node-embedding vectors
$\bar{h}_{i}^{1},...,\bar{h}_{i}^{K}$ of agent $i$ for the information-
condensed and contextualized state representation of agent $i$ as
$h_{i}=\sum_{k=1}^{K}{\beta^{k}_{i}\bar{h}^{k}_{i}}$ while considering the
relationships between agent $i$ and the groups of other agents. The inter-
group attention weight $\beta^{k}_{i}$ guides which group agent $i$ should
focus more on to achieve its objective. For example, if $\beta^{k}_{i}$ is
large for the same group which agent $i$ belongs to, it implies that agent $i$
focuses on cooperating with the agents in the same group. Otherwise, agent $i$
would focus more on competing with agents from different groups. The inter-
group attention weight is computed as softmax
$\beta_{i}=(\beta^{1}_{i},...,\beta^{K}_{i})\propto\text{exp}(q_{i})$ where
$q_{i}=[q^{1}_{i},\dots,q^{K}_{i}]=f_{\beta}([\bar{h}^{1}_{i},\dots,\bar{h}^{K}_{i}];w_{\beta})$.
The hierarchical state representation is particularly useful when considering
mixed cooperative-competitive games where each agent or group possesses their
own objectives, which will be empirically shown by various experimental
results in this study. The embedding and attention functions in this study
comprise a two-layered MLP with 256 units and ReLUs.

### Multi-Agent Actor-Critic

The proposed method uses the embedding vectors $h_{i}^{C}$ and $h_{i}^{A}$ of
agent $i$ to compute, respectively, the individual Q-value
$Q_{c(i)}(\mathbf{o},\mathbf{a})\approx Q_{c(i)}(h_{i}^{C};\phi_{c(i)})$ and
determine the action
$a_{i}=\mu_{c(i)}(o_{i})\approx\mu_{c(i)}(h_{i}^{A};\theta_{c(i)})$, where
$c(i)$ is the group to which agent $i$ belongs. Note that the embedding
vectors $h_{i}^{C}$ and $h_{i}^{A}$ are computed separately using two
different HGATs; computing $h_{i}^{C}$ requires a joint action $\mathbf{a}$ in
the training phase under CTDE. Additionally, agents in the same group share
the actor and critic networks for generalization.

Compared to using raw observation as an input for the critic and actor network
(?), using node-embedding vectors computed from HGAT as inputs offers the
following advantages: (1) a node-embedding vector can be computed by
considering the hierarchical relationships among agents, i.e., relative
inductive biases, thus providing contextualized state representation; (2) it
is scalable to a large number of agents as the dimension of a node-embedding
vector does not change with the number of agents; and (3) HGAT enables the
learned policy to be used in environments of any agent or group size, i.e.,
the property that transfer learning aims to achieve.

The training of HAMA is similar to that of MADDPG. The shared critic $Q_{k}$
for agent $i$ in group $k$ is trained to minimize the loss $\mathcal{L}$:

$\mathcal{L}(\phi_{k})={\mathbb{E}_{{\mathbf{o}},\mathbf{a},r_{i},{\mathbf{o}^{\prime}}\sim\mathcal{D}}}[(Q_{k}^{\mu}({\mathbf{o}},\mathbf{a};\phi_{k})-y_{i})^{2}],\\\
y_{i}=r_{i}+{\gamma}Q_{k}^{\mu^{\prime}}(\mathbf{o}^{\prime},\mathbf{a}^{\prime};\phi^{\prime}_{k})\arrowvert_{a_{i}^{\prime}=\mu^{\prime}(o_{i}^{\prime};\theta^{\prime})}$
(1)

where $Q^{\mu^{\prime}}$ and $\mu^{\prime}$ are, respectively, the target
critic and actor networks for stable learning with delayed parameters (?). In
CTDE framework, the joint observation and action are assumed to be available
for training. The shared actor $\mu_{k}$ for agent $i$ in group $k$ is then
trained using gradient ascent algorithm with the gradient computed as:

$\nabla_{\theta_{k}}\mathcal{J}(\theta_{k})=\\\
\mathbb{E}_{\mathbf{o},\mathbf{a}\sim\mathcal{D}}[\nabla_{\theta_{k}}\mu_{k}(o_{i};\theta_{k})\nabla_{a_{i}}Q_{k}^{\mu}(\mathbf{o},\mathbf{a};\phi_{k})\arrowvert_{a_{i}=\mu_{k}(o_{i};\theta_{k})}]$
(2)

where $a_{i}$ is the action of agent $i$ in $\mathbf{a}$. During the training,
the joint observation $\mathbf{o}$ and joint action $\mathbf{a}$ are used,
whereas, during the execution, only the learned policy
$\mu_{k}(o_{i};\theta_{k})\approx\mu_{c(i)}(h_{i}^{A};\theta_{c(i)})$ is used
with the embedding vector $h_{i}^{A}$ computed using only local observation
$o_{i}$ of agent $i$.

Figure 2: Illustrations of the experimental environments, including a)
_Cooperative Navigation_ , b) _3 vs. 1 Predator-Prey_ , c) _3 vs. 3 Predator-
Prey_ , and d) _The More-The Stronger_.

## Experiments

Figure 2 shows the environments we use to evaluate the performances of the
proposed and baseline MARL algorithms. It includes cooperative environments
that have been widely used in existing studies (?; ?) as well as mixed
cooperative-competitive environments extended from well-known environments. As
baseline algorithms for comparing the performances, we consider MADDPG and
MAAC because they belong to learning-for-consensus approaches and are designed
to process only local observation during the execution phase as HAMA does.
These algorithms are more general for mixed cooperative-competitive games
where communication is not always possible. For the cooperative navigation, we
additionally consider ATOC, one of learning-to-communicate approaches, as a
baseline because this game is fully cooperative; thus, the communication-based
method can be naturally considered. All the performance measures are obtained
by executing the trained policies with 3 different random seeds on 200
episodes. Regarding the visual range of the agent, we assume that each agent
observes up to three nearest neighboring agents per each group with relative
positions and velocities in all the experiment settings.

### Cooperative Navigation

First, the proposed model is validated in the cooperative navigation, where
only cooperation among agents exists. In the game, all the agents, which are
homogeneous, are required to reach one of the landmarks without colliding with
each other. Each episode starts with $n$ randomly generated agents and
landmarks and ends after 25 timesteps. During an episode, each agent receives
$-d$, the distance to the nearest landmark, as a reward. In addition, each
agent receives an additional reward, $-1$, whenever it collides with other
agents during navigation. It is an optimal strategy that each agent occupies
its distinct landmark.

Figure 3: Penalties during training on cooperative navigation with 3 agents. Table 1: The mean and standard deviation of penalties in cooperative navigation. $n$ | MADDPG | MAAC | ATOC | HAMA
---|---|---|---|---
3 | 0.22$\pm$0.010 | 0.19$\pm$0.012 | 0.17$\pm$0.007 |  0.15$\pm$0.008
Transfer learning using the policy trained with $n=3$
50 | - | - | 0.09 (70) |  0.05 (90)
100 | - | - | 0.07 (81) |  0.04 (96)

Figure 3 compares the normalized mean penalties of four different MARL
algorithms (instead of representing results with rewards that are negative, we
present the results in terms of penalty as a negative reward since it is more
intuitive for this game). A smaller mean penalty implies closer to the nearest
landmark and fewer collisions with other agents. The penalty is averaged over
every 10,000 steps until reaching 3 million steps.

In training, as shown in Figure 3, HAMA converges fast to the lowest value.
This is possible because HAMA effectively represents the state of each agent
by considering their relative positions and velocities through HGAT. Table 1
compares the normalized mean penalties that the agents obtain during testing
with 200 episodes with the trained models. ATOC achieves smaller mean penalty
than MADDPG and MAAC. Because ATOC employs an active communication scheme
based on attention network, it can effectively derive the cooperative behavior
among agents. Our model has a lower mean penalty than ATOC. This indicates
that the cooperative strategies trained by HAMA can effectively induce
coordination among agents even without having active communication among
agents.

Due to the use of a shared actor with efficient state representation, the
trained policies by HAMA and ATOC can be applied to the cooperative game with
any number of agents$/$landmarks, whereas the policies trained by MADDPG and
MAAC cannot be transferred. The performance of transfer learning is also
summarized in Table 1. When the policies trained by 3 agents are used to play
the game with 50 and 100 agents, HAMA has the lower average penalty and higher
percentages of landmark occupation (provided in the parenthesis in the table)
by the participating agents. Note that when we conduct the transfer learning
experiments, we reduce the size of agents (25 times smaller) to have a large
number of agents in the same environment, where each agent can observe three
nearest agents and three landmarks.

Table 2: The mean and standard deviation of scores for predators in 3 vs. 1 predator-prey game. predator | prey $(n=1)$
---|---
$(n=3)$ | MADDPG | MAAC | HAMA
MADDPG | 0.30$\pm$0.02 | 0.23$\pm$0.02 | 0.07$\pm$0.01
MAAC | 0.35$\pm$0.02 |  0.39$\pm$0.03 | 0.07$\pm$0.01
HAMA |  0.45$\pm$0.03 |  0.39$\pm$0.03 |  0.16$\pm$0.01

### 3 vs. 1 Predator-Prey

A predator-prey game consists of two groups of agents competing with each
other, along with obstacles that participate in the game but do not take an
action. The goal of three homogeneous predators is to capture one prey, while
the goal of the prey is to escape from the predators. For the predators to
capture the prey, they need to cooperate with each other because of their
slower speed and acceleration compared with those of the prey. Each predator
gets a positive reward, $+10$, when it catches the prey, and the prey receives
a negative reward, $-10$, when it is caught by a predator. When the prey
leaves a certain zone, the prey receives a negative reward to prevent it from
leaving farther. It is noteworthy that each agent seeks to maximize their
accumulated rewards, which results in the competition between predators and
prey.

We compare the performance of HAMA with two other models as baselines: MADDPG
and MAAC. Each model is trained while self-playing (i.e., predators and prey
are trained with the same model), and the trained policies are validated while
having the trained policies compete with other policies trained by different
models. Table 2 summarizes the average scores that a predator can obtain per
step in an episode. The results indicate that the predators trained by HAMA
have higher or similar scores than MADDPG and MAAC when competing with the
prey trained by other models. Similarly, the prey trained by HAMA results in
the lowest score when competing with the predators trained by different
models. Note that when HAMA plays the role of a single prey where no
cooperation is required, it still performs best in defending itself from the
predators because it effectively configures the relationships with the
predators and obstacles by using HGAT.

### 3 vs. 3 Predator-Prey

The next game we consider is 3 vs. 3 predator-prey game, a variant of the
original 3 vs. 1 predator-prey game. The game rules are identical to those of
3 vs. 1 predator-prey game. In this game, if a predator recaptures a prey that
has already been captured, neither reward occurs. Instead, each predator
receives an additional reward, $+10*t_{r}$, when the predators capture all
preys, where $t_{r}$ is the number of remaining timesteps in the episode, and
the game ends. Although the game is similar to that of the original predator-
prey game, the optimal strategy of the agents is no longer clear because more
diverse and complex strategic interactions occur among the two groups of
agents. For example, a predator can choose to either cooperate with other
predators to chase a prey or to capture a prey individually if the prey is
nearby. In addition to MADDPG and MAAC, we consider two heuristic strategies
for the predators. In Heuristic 1, all the predators chase the same prey that
has not been captured yet. In Heuristic 2, each predator chases the prey
closest to the predator.

Table 3: The mean and standard deviation of scores for predators in 3 vs. 3 predator-prey game. predator | prey $(n=3)$
---|---
$(n=3)$ | MADDPG | MAAC | HAMA
Heuristic1 | 0.35$\pm$0.07 | 0.15$\pm$0.10 | 0.005$\pm$0.001
Heuristic2 | 0.72$\pm$0.10 | 0.30$\pm$0.14 | 0.01$\pm$0.001
MADDPG | 1.18$\pm$0.13 | 1.05 $\pm$0.22 | 0.02$\pm$0.01
MAAC | 0.65$\pm$0.20 | 0.33$\pm$0.13 | 0.07$\pm$0.04
HAMA |  6.33$\pm$0.10 |  3.36$\pm$0.34 |  1.19$\pm$0.09
Table 4: The mean and standard deviation of scores for different architectures in 3 vs. 3 predator-prey game. predator | prey $(n=3)$
---|---
$(n=3)$ | HG-IAGA (HAMA)
SG-IAA (MAAC) | 0.07$\pm$0.04
HG-NA | 0.57$\pm$0.07
HG-IAA | 1.03$\pm$0.08
HG-IGA | 0.37$\pm$0.06
HG-IAGA (HAMA) |  1.19$\pm$0.09

Table 3 compares the results of the game when the predators and preys, each of
which is trained by self-playing, compete against each other. As shown in the
table, the predators trained by HAMA achieve the highest scores against the
preys trained by all other algorithms. Similarly, the preys trained by HAMA
defend the best against the predators trained by all different algorithms,
including the two heuristic strategies. The performance of HAMA is
incomparably superior to those of other methods in both roles of predator and
prey. This is remarkable in that HAMA and MAAC achieve a similar performance
in the 3 vs. 1 predator-prey game where the only strategy of the predators is
to cooperate to capture a unique prey. Meanwhile, in the 3 vs. 3 predator-prey
game, each predator can choose from various strategies, such as cooperating
with other predators or chasing prey individually. When chasing prey, a
predator can also choose which prey to chase. The superior performance of HAMA
is possible because it learns to represent better the hierarchical
relationships among agents in the dynamic game owing to the relative inductive
biases imposed by HGAT.

We validate our hypothesis on the success of HAMA’s strategy by conducting an
ablation study. Table 4 summarizes the performances of the following variant
models:

  * •

SG-IAA: Single-Graph & Inter-Agent Attention

  * •

HG-NA: Hierarchical-Graph & No Attention

  * •

HG-IAA: Hierarchical-Graph & Inter-Agent Attention

  * •

HG-IGA: Hierarchical-Graph & Inter-Group Attention

  * •

HG-IAGA (HAMA): Hierarchical-Graph & Inter-Agent & Inter-Group Attention

Note that the SG-IAA has a similar architecture with MAAC. Compared to the SG-
IAA, a hierarchical graph attention architecture always scores higher
regardless of whether attention is used. We assume that this effectiveness is
due to the use of enhanced relative inductive biases regarding both the agent-
level interactions and the group-level interactions. In comparing the role of
attention, when both attentions are considered, the HG-IAGA outperforms the
others. The combination of hierarchical graph structure and specially designed
attentions is a key factor that induces the superior performance of HAMA.

#### Transfer Learning.

Figure 4: Results of transfer learning with the trained policies by HAMA in 3
vs. 3 predator-prey game. The success rate of predators is the rate of
episodes in which predators capture all preys.

In general, when the number of predators is large and the number of preys is
small, the predators have a higher chance to win the game (i.e., capture all
the preys within a single episode). As shown in Figure 4, this general trend
is well realized when the predator and prey policies trained by HAMA in the 3
vs. 3 predator-prey game are transferred to play an $m$ vs. $n$ predator-prey
game. It shows that the success rate of the predators is close to 1 when $m$
(number of predators) $>n$ (number of preys).

#### Interpreting Strategies.

Figure 5: Reasoning on the strategy of HAMA.

We explain why a certain action of the agent is induced at a certain state by
analyzing and interpreting the inter-agent and inter-group attention weights
in HAMA. In Figure 5, the blue, red, and gray circles represent the predators,
preys, and obstacles, respectively. The plots in each row show how each
predator agent, which is represented by the blue circle with a black outline,
attends other agents in the same and different groups over time. The width of
the arrow indicates the magnitude of the attention weight $\alpha_{ij}^{k}$ on
the agent the arrow is pointing out in each group. The blue and red bars at
the top of each figure indicate the magnitudes of inter-group attention
weights $\beta_{i}^{k}$ to the predator ($k=1$) and the prey groups ($k=2$),
respectively. The black arrow indicates the agent’s action (i.e., direction
and speed). From the predator’s perspective, the attention to predators and
preys can be interpreted as the attention to cooperation and competition.
Figure 5 depicts the situation where predator $1$ and $3$ increase the
cooperative attention (i.e., attention to the same group) over time to jointly
chase the prey into a corner of the box. Meanwhile, predator $2$ attempts to
catch one prey, whose strategy is indicated by the attention to the
competition (i.e., attention to the different group).

### The More-The Stronger

The more-the stronger game keeps the framework of the 3 vs. 3 predator-prey
game. The additional game rule in this game is that when the preys are
clustered together, only a group of predators whose size is equal to or larger
than that of the clustered preys can capture the preys. For example, one
predator can capture one prey by itself, but three-gathered predators are
required to capture three-gathered preys. HAMA outperforms other models in
this game.

## Conclusions

We herein proposed a multi-agent actor-critic model based on state
representation by a hierarchical graph attention network. Empirically, we
demonstrated that the learned model outperformed other MARL models on a
variety of cooperative and competitive multi-agent environments. In addition,
the proposed model has been proven to facilitate the transfer of learned
policies to new tasks with different agent compositions and allow one to
interpret the learned strategies.

## Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (No. 2017R1A2B4006290) and by the
Technology Innovation Program (or Industrial Strategic Technology Development
Program, 10067705, Development of AI-based Smart Construction System for 20%
Cost Saving) funded by the Ministry of Trade, Industry & Energy (MI, Korea).

## References

  * [Battaglia et al. 2018] Battaglia, P. W.; Hamrick, J. B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Tacchetti, A.; Raposo, D.; Santoro, A.; Faulkner, R.; et al.  2018\.  Relational inductive biases, deep learning, and graph networks.  arXiv preprint arXiv:1806.01261.
  * [Cao et al. 2013] Cao, Y.; Yu, W.; Ren, W.; and Chen, G.  2013\.  An overview of recent progress in the study of distributed multi-agent coordination.  IEEE Transactions on Industrial informatics 9(1):427–438.
  * [Corke, Peterson, and Rus 2005] Corke, P.; Peterson, R.; and Rus, D.  2005\.  Networked robots: Flying robot navigation using a sensor net.  In Robotics research. The eleventh international symposium, 234–243.  Springer.
  * [Dall’Anese, Zhu, and Giannakis 2013] Dall’Anese, E.; Zhu, H.; and Giannakis, G. B.  2013\.  Distributed optimal power flow for smart microgrids.  IEEE Trans. Smart Grid 4(3):1464–1475.
  * [Das et al. 2019] Das, A.; Gervet, T.; Romoff, J.; Batra, D.; Parikh, D.; Rabbat, M.; and Pineau, J.  2019\.  Tarmac: Targeted multi-agent communication.  In International Conference on Machine Learning, 1538–1546.
  * [Fax and Murray 2004] Fax, J. A., and Murray, R. M.  2004\.  Information flow and cooperative control of vehicle formations.  IEEE transactions on automatic control 49(9):1465–1476.
  * [Foerster et al. 2018] Foerster, J. N.; Farquhar, G.; Afouras, T.; Nardelli, N.; and Whiteson, S.  2018\.  Counterfactual multi-agent policy gradients.  In Thirty-Second AAAI Conference on Artificial Intelligence.
  * [Gori, Monfardini, and Scarselli 2005] Gori, M.; Monfardini, G.; and Scarselli, F.  2005\.  A new model for learning in graph domains.  In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, 729–734.  IEEE.
  * [Haarnoja et al. 2018] Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S.  2018\.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International Conference on Machine Learning, 1856–1865.
  * [Iqbal and Sha 2019] Iqbal, S., and Sha, F.  2019\.  Actor-attention-critic for multi-agent reinforcement learning.  In International Conference on Machine Learning, 2961–2970.
  * [Jiang and Lu 2018] Jiang, J., and Lu, Z.  2018\.  Learning attentional communication for multi-agent cooperation.  In Advances in Neural Information Processing Systems, 7254–7264.
  * [Jiang, Dun, and Lu 2018] Jiang, J.; Dun, C.; and Lu, Z.  2018\.  Graph convolutional reinforcement learning for multi-agent cooperation.  arXiv preprint arXiv:1810.09202.
  * [Lillicrap et al. 2015] Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D.  2015\.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971.
  * [Lowe et al. 2017] Lowe, R.; Wu, Y.; Tamar, A.; Harb, J.; Abbeel, O. P.; and Mordatch, I.  2017\.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Advances in Neural Information Processing Systems, 6379–6390.
  * [Matignon et al. 2012] Matignon, L.; Jeanpierre, L.; Mouaddib, A.-I.; et al.  2012\.  Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes.  In AAAI.
  * [Mnih et al. 2015] Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al.  2015\.  Human-level control through deep reinforcement learning.  Nature 518(7540):529.
  * [Peng et al. 2017] Peng, P.; Wen, Y.; Yang, Y.; Yuan, Q.; Tang, Z.; Long, H.; and Wang, J.  2017\.  Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games.  arXiv preprint arXiv:1703.10069.
  * [Powell and Clark 2004] Powell, S., and Clark, E.  2004\.  Combat between large derived societies: a subterranean army ant established as a predator of mature leaf-cutting ant colonies.  Insectes Sociaux 51(4):342–351.
  * [Scarselli et al. 2009] Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagenbuchner, M.; and Monfardini, G.  2009\.  The graph neural network model.  IEEE Transactions on Neural Networks 20(1):61–80.
  * [Silver et al. 2014] Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; and Riedmiller, M.  2014\.  Deterministic policy gradient algorithms.  In ICML.
  * [Silver et al. 2016] Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al.  2016\.  Mastering the game of go with deep neural networks and tree search.  Nature 529(7587):484.
  * [Silver et al. 2017] Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; et al.  2017\.  Mastering the game of go without human knowledge.  Nature 550(7676):354.
  * [Singh, Jain, and Sukhbaatar 2019] Singh, A.; Jain, T.; and Sukhbaatar, S.  2019\.  Learning when to communicate at scale in multiagent cooperative and competitive tasks.  In International Conference on Learning Representations.
  * [Smith and Price 1973] Smith, J. M., and Price, G. R.  1973\.  The logic of animal conflict.  Nature 246(5427):15.
  * [Sukhbaatar, Fergus, and others 2016] Sukhbaatar, S.; Fergus, R.; et al.  2016\.  Learning multiagent communication with backpropagation.  In Advances in Neural Information Processing Systems, 2244–2252.
  * [Ugelvig and Cremer 2007] Ugelvig, L. V., and Cremer, S.  2007\.  Social prophylaxis: group interaction promotes collective immunity in ant colonies.  Current Biology 17(22):1967–1971.
  * [Vaswani et al. 2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I.  2017\.  Attention is all you need.  In Advances in neural information processing systems, 5998–6008.
  * [Veličković et al. 2017] Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y.  2017\.  Graph attention networks.  arXiv preprint arXiv:1710.10903.
  * [Ying and Dayong 2005] Ying, W., and Dayong, S.  2005\.  Multi-agent framework for third party logistics in e-commerce.  Expert Systems with Applications 29(2):431–436.
  * [Zhang et al. 2018] Zhang, K.; Yang, Z.; Liu, H.; Zhang, T.; and Başar, T.  2018\.  Fully decentralized multi-agent reinforcement learning with networked agents.  arXiv preprint arXiv:1802.08757.

Figure 6: Reasoning on the strategy of HAMA in another episode.

## Supplementary Material

### Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
Algorithm

For completeness, we provide the HAMA algorithm below.

Algorithm 1 Multi-Agent Actor-Critic with HGAT Algorithm for $N$ agents

1:Initialize actor networks $\mu$, critic networks $Q$, target networks
$\mu^{\prime}$ and $Q^{\prime}$, replay buffer $\mathcal{D}$

2:for episode $=1$ to $M$ do

3: Initialize a random process $\mathcal{N}$ for action exploration

4: Receive initial observation $\mathbf{o}$

5: for $t=1$ to $T$ do

6: for each agent $i$ in agent group $k$, select action

7: $a_{i}=\mu_{k}(o_{i};\theta_{k})+\mathcal{N}_{t}$

8: Execute actions $\mathbf{a}$ and receive reward

9: $\mathbf{r}=(r_{1},\dots,r_{N})$ and new observation $\mathbf{o}^{\prime}$

10: Store $(\mathbf{o},\mathbf{a},\mathbf{r},\mathbf{o}^{\prime})$ in replay
buffer $\mathcal{D}$

11: $\mathbf{o}\leftarrow\mathbf{o}^{\prime}$

12: for $k=1$ to $K$ for agent group $C^{k}$ do

13: for agent $i=1$ to $|C^{k}|$ in group $C^{k}$ do

14: Sample a minibatch of $\mathcal{B}$ samples

15: $(\mathbf{o}^{j},\mathbf{a}^{j},\mathbf{r}^{j},\mathbf{o}^{\prime j})$
from $\mathcal{D}$

16: Set $y^{j}_{i}=r^{j}_{i}+{\gamma}Q_{k}^{\mu^{\prime}}({\mathbf{o}^{\prime
j}},\mathbf{a}^{\prime};\phi^{\prime}_{k})\arrowvert_{a^{\prime}=\mu^{\prime}(o^{\prime
j};\theta^{\prime})}$

17: Update critic by minimizing the loss

18:
$\mathcal{L}(\phi_{k})=\frac{1}{\mathcal{B}}\sum_{j}(Q_{k}^{\mu}({\mathbf{o}^{j}},\mathbf{a}^{j};\phi_{k})-y^{j}_{i})^{2}$

19: Update actor using the sampled policy

20: gradient:

21: $\nabla_{\theta_{k}}\mathcal{J}(\theta_{k})\approx$

22:$\frac{1}{\mathcal{B}}\sum_{j}\nabla_{\theta_{k}}\mu_{k}(o^{j}_{i};\theta_{k})\nabla_{a_{i}}Q_{k}^{\mu}({\mathbf{o}^{j}},\mathbf{a}^{j};\phi_{k})\arrowvert_{a_{i}=\mu_{k}(o^{j}_{i};\theta_{k})}$

23: end for

24: end for

25: Update target network parameters for each agent

26: group $C^{k}$:
$\displaystyle\phi_{k}^{\prime}\leftarrow\tau\phi_{k}+(1-\tau)\phi_{k}^{\prime}$
$\displaystyle\theta_{k}^{\prime}\leftarrow\tau\theta_{k}+(1-\tau)\theta_{k}^{\prime}$

27: end for

28:end for

### Hyperparameters for Experiments

For the experiments in the environments, all the functions including embedding
and attention functions in the actor and critic networks comprise a two-
layered MLP with 256 units and a ReLU for nonlinear activation. The learning
rate of Adam optimizer for updating the actor and critic network parameters is
$1e-4$ and $1e-3$, respectively. The discount factor is 0.95. The target
networks are updated with $\tau=1e-3$ as a soft target. The network parameters
that are initialized from Xavier uniform initialization are updated with a
mini-batch of 1024. The size of experience replay buffer is $10^{6}$, and the
parameters of the networks are updated every 100 samples added to the buffer
after 2.5% of the replay buffer is filled. The output layer of actor network
provides the action as a five-sized tensor for hold, right, left, up, and
down. We used three different random seeds and 120, 000 episodes with 25 steps
(total 3 million steps) for training the networks in the environments.
Subsequently, the trained models are validated in 200 episodes.

Table 5: The mean and standard deviation of scores for predators in the more-the stronger game. predator | prey $(n=3)$
---|---
$(n=3)$ | MADDPG | MAAC | HAMA
Heuristic1 | 0.19$\pm$0.05 | 0.17$\pm$0.09 | 0.003$\pm$0.001
Heuristic2 | 0.73$\pm$0.10 | 0.49$\pm$0.18 | 0.005$\pm$0.001
MADDPG | 1.64$\pm$0.13 | 1.81$\pm$0.29 | 0.02$\pm$0.001
MAAC | 0.77$\pm$0.22 | 0.71$\pm$0.23 | 0.64$\pm$0.20
HAMA |  5.45$\pm$0.12 |  3.45$\pm$0.30 |  2.08$\pm$0.13

### Results for The More-The Stronger

The more-the stronger game keeps the framework of the 3 vs. 3 predator-prey
game. The additional game rule in this game is that when the preys are
clustered together, only a group of predators whose size is equal to or larger
than that of the clustered preys can capture the preys. For example, one
predator can capture one prey by itself, but three-gathered predators are
required to capture three-gathered preys. The more predators are gathered, the
stronger is the power; the more preys are gathered, the stronger is the
shield. When fewer predators approach the gathered preys, the predators cannot
capture the preys. Therefore, the agents in the game require more complex
strategies to achieve their objectives. The rewards of the predators and preys
in the game are the same as those in the 3 vs. 3 predator-prey game. The
results are summarized in Table 5. As shown in the table, HAMA outperforms
other models in this game as well.

### Interpreting Strategies for 3 vs. 3 Predator-Prey

We present the inter-agent and inter-group attentions of HAMA in another
episode involving more steps in Figure 6. In (c), (j), and (o) in the figure,
the preys are captured by the predators. When the chased prey is captured by
the predators, the predators immediately move their attention to other preys
that have not been captured yet. The already caught preys interrupt the
predators, thus delaying the capture of other preys.

### Learning Curves on 3 vs. 3 Predator-Prey

Figure 7: Rewards during training on 3 vs. 3 predator-prey.

In Figure 7, the rewards over 3 million steps are averaged every 10,000 steps
and normalized. The positive and negative rewards of each model represent the
rewards of the predators and preys, respectively. The rewards of the predators
and preys are symmetric but not completely symmetric owing to the penalty
obtained by the preys when they are outside of a particular zone. In the
figure, the predators of MADDPG exhibit a higher score than the predators of
HAMA during training. When HAMA is considered to outperform MADDPG from the
primary results in the main paper, we assume that the more intelligent preys
have learned to counter the intelligent predators in HAMA. Furthermore, in the
early stages of training, the preys leave a certain zone to escape from the
predators, which appears to be extremely negative in the early rewards for the
preys. HAMA exhibits this phenomenon much earlier than MADDPG. We assume that
this is because HAMA trains the given samples more efficiently than MADDPG.