# Harnessing excitons at the nanoscale - photoelectrical platform for
quantitative sensing and imaging

Zhurun Ji1,2,3, Mark E. Barber1,2,3, Ziyan Zhu4, Carlos R. Kometter1,3,4,
Jiachen Yu2,3,4,Kenji Watanabe5, Takashi Taniguchi6, Mengkun Liu7,8,
Thomas P. Devereaux3,4,9, Ben Feldman1,3,4, Zhixun Shen1,2,3,4

1Department of Physics, Stanford University, Stanford, CA 94305, USA
2Department of Applied Physics, Stanford University, Stanford, CA 94305, USA
3Geballe Laboratory for Advanced Materials, Stanford University, Stanford, CA
94305, USA
4Stanford Institute for Materials and Energy Sciences, SLAC National
Accelerator Laboratory,
Menlo Park, CA 94025, USA
5Research Center for Electronic and Optical Materials, National Institute for
Materials Science,
1-1 Namiki, Tsukuba 305-0044, Japan
6Research Center for Materials Nanoarchitectonics, National Institute for
Materials Science,
1-1 Namiki, Tsukuba 305-0044, Japan
7Department of Physics and Astronomy, Stony Brook University, Stony Brook, NY
11794, USA
8National Synchrotron Light Source II, Brookhaven National Laboratory, Upton,
NY 11973, USA
9Department of Materials Science and Engineering, Stanford University, CA
94305, USA
∗To whom correspondence should be addressed; E-mail<EMAIL_ADDRESS>
> Excitons — quasiparticles formed by the binding of an electron and a hole
> through electrostatic attraction — hold promise in the fields of quantum
> light confinement and optoelectronic sensing. Atomically thin transition
> metal dichalcogenides (TMDs) provide a versatile platform for hosting and
> manipulating excitons, given their robust Coulomb interactions and
> exceptional sensitivity to dielectric environments. In this study, we
> introduce a cryogenic photoelectrical nanosensing platform, termed exciton-
> resonant microwave impedance microscopy (ER-MIM). ER-MIM enables ultra-
> sensitive probing of exciton polarons and their Rydberg states at the
> nanoscale. Utilizing this technique, we explore the interplay between
> excitons and material properties at the nanoscale, including carrier
> density, in-plane electric field, and dielectric screening. Furthermore, we
> employ a neural network algorithm for automated data analysis and
> quantitative extraction of nanoscale electrical information, unveiling the
> potential of exciton-assisted nano-electrometry. Our findings establish an
> invaluable sensing platform and readout mechanism, advancing our
> understanding of exciton excitations and their applications in the quantum
> realm.

## Introduction

One of the fundamental tenets of quantum mechanics is the existence of
discrete energy levels, which underpins various applications in the field of
quantum technology. For instance, qubits, the building blocks of quantum
computing and communication, leverage this concept to perform complex
calculations (?). In the nanoscale sensing domain, techniques that make use of
discrete energy levels, such as nitrogen-vacancy and Rydberg atom sensing,
enable precise detection (?, ?, ?). The aspiration to sense physical
quantities at the nanoscale motivates the development of novel discrete energy
systems and sensing mechanisms.

Quasiparticle excitations in quantum materials form various types of
discretized energy systems, including excitons, polarons, and 1D topological
edge modes. Particularly, excitons are pairs of bound electrons and holes,
attracted to each other by the electrostatic Coulomb force. Their energy
structure, coherence, and interactions with photons make excitons an ideal
candidate for sensing devices (?, ?, ?). Atomically thin transition metal
dichalcogenides (TMDs) are particularly attractive for hosting robust
excitonic effects, stemming from their large binding energies, direct bandgap,
and pronounced light-matter interactions (?, ?). The intrinsic Coulomb
interactions within excitons give rise to strong exciton-exciton and exciton-
phonon coupling, culminating in intriguing phenomena such as exciton
condensation and the formation of, e,g., trions (?) and exciton-polarons (?).
Under an electric field, these interactions can substantially alter exciton
properties, impacting quantities like binding energies, radiative lifetimes,
and emission spectra (?). The rich response of excitons to external stimuli
suggests the potential for effective control and sensing. Importantly, in
compact two-dimensional (2D) heterostructures, excitons provide the capability
for high sensing accuracy, as they can be positioned in close proximity to the
samples (?, ?).

To successfully harness excitons as nanoscale sensors, the development of an
excitation and imaging method is imperative. Detecting excitons locally is a
critical first step, allowing one to gather precise data without disturbing
the quantum mechanical properties. At room temperature, tip-enhanced
photoluminescence (TEPL) techniques have been used to study nanoscale excited
state recombination in TMDs (?, ?, ?, ?, ?). Scattering-type scanning near-
field optical microscope (s-SNOM) studies have also provided insights into the
complex dielectric function of excitons (?, ?) and their waveguide modes (?,
?, ?). However, for accurate sensing, it is imperative to attain a fine
linewidth in exciton absorption, which often necessitates lower temperatures
and direct optical excitation. Near-field optical microscopies are challenging
at extremely low temperatures (e.g., $<$4 K). Emission experiments, on the
other hand, do not provide direct access to the discrete energy levels of
excitons. Furthermore, investigating how excitons interact with their
immediate environment demands exceptional spatial resolution and a
comprehensive understanding of environmental parameters, rendering these
endeavors highly challenging. These multifaceted challenges underscore the
pressing need for an innovative hyperspectral imaging technique that can fully
capture the fine spectroscopic details of excitons at the nanoscale in a
cryogenic environment.

In this study, we develop an advanced technique, exciton resonant microwave
impedance microscopy (ER-MIM), to visualize excitons at the nanoscale. By
combining the sensitivity of optical spectroscopy with the high spatial
resolution of MIM, we gain deep insights into exciton dynamics and their
interactions with the local environment. We also employ neural network
algorithms to reconstruct the electrical and dielectric environment, our
research demonstrates a methodology of using excitons to holistically quantify
all three important electrical quantities — conductivity, electric field and
surrounding permittivity. This “all-in-one” capability for local electrical
sensing paves the way for advances in quantum material-based sensing.

##  1 Results

###  1.1 Local optoelectronic detection of discrete exciton Rydberg states

We first introduce the experimental technique that enables us to sense
excitons using photoelectrical effects at the nanoscale: exciton resonant
microwave impedance microscopy (ER-MIM). As shown in Fig. 1A, our setup
consists of two parts: a continuously wavelength-tunable laser that is fiber-
coupled into a Helium-3 cryostat and illuminates the tip, and a microwave
transmission line impedance-matched to a metallic scanning probe (tip) (?).
The real and imaginary parts of the reflected microwave signals are recorded,
which are in- and out-of-phase with the reference microwave excitation line,
respectively. Our measurements were conducted at $\sim$1.5 K and 3 GHz. In
Fig. 1B, we present two calculated response curves that illustrate the
relationship between the real (in-phase) and imaginary (out-of-phase) parts of
the reflected microwave signals and the local complex conductivity of the
sample.

The material we study is a prototypical 2D TMD device — a monolayer (ML) of
MoSe2 encapsulated within hBN layers with graphite as a back gate. The optical
spectrum of monolayer MoSe2 possesses a series of pronounced resonances,
categorized as A- and B-excitons (see Fig. 1C). These excitons arise from two
optical transitions involving states in the upper and lower energy spin
valence bands. These excitons possess Rydberg atom-like energy levels, each
designated by its principal quantum number, $n$.

By tracking the evolution of the microwave response as a function of
excitation wavelength and back gate voltage, we construct a 2D ER-MIM signal
plot (imaginary part shown in Fig. 1D). From the response curve in Fig. 1B,
the blue and red colored regions correspond to low and high conductivity of
MoSe2, respectively. The overall dependence of conductivity on back gate
voltage agrees with the transfer curve of a semiconductor, i.e., it decreases
when crossing from the conduction band into the bandgap, and increases when
entering the valence band. On top of this overall trend, at specific photon
energies, the conductivity has an extra increase at smaller $\rm|V_{bg}|$, and
another decrease at large $\rm|V_{bg}|$. We identify prominent features
corresponding to neutral $A_{1s}$ and $B_{1s}$ excitons, and the excited
Rydberg states including $A_{2s}$, $A_{3s}$, and $B_{2s}$, as well as weaker
modulations in intensity features corresponding to their charged counterparts
like $AT_{1s}$ and $AT_{2s}$ (which are usually referred to as trion or
exciton polaron, and their distinction will be discussed below).

The changes in ER-MIM-Im signal near excitonic resonances imply the
observation of photoconductivity associated with exciton formation. It has a
sign flip from positive (red) at small $\rm|V_{bg}|$ to negative (blue) at
large $\rm|V_{bg}|$. The origin of these features can be attributed to two
major processes (a discussion on ruling out other possible optical processes
is in supplementary note 3). One is the photoconductive effect (?), where the
exposure of TMD-based devices to light could generate neutral electron-hole
pairs. Then, the trapping of one carrier and the release of the other free
carrier from an exciton could result in an enhanced conductivity (?). This
process explains our observation of positive conductivity near excitonic
resonances. The other is the Auger assisted tunneling effect (?, ?). When the
photon energy is in resonance with the binding energy of excitons, a generated
exciton can excite a free hole when it recombines during the Auger process.
The hole can subsequently tunnel through the hBN barrier to reach the bottom
gate, which effectively reduces the MoSe2 conductivity. Since the energy
barrier in the hole side between MoSe2 and hBN is much smaller than the
electron side, the negative photoconductivity is only observed when MoSe2 is
hole-doped (see supplementary note 3). Therefore, the conductivity decrease at
large $\rm|V_{bg}|$ is attributed to the dominance of the Auger effect.

We further compare two linecuts of ER-MIM-Im spectrum with previously reported
far-field spectra of excitons in monolayer MoSe2 (Fig. 1E). The red and blue
curves show excitonic peaks and dips, highlighting the signal from the
photoconductivity and Auger-assisted tunneling effects, respectively. The
comparison reveals excellent agreement between the identified exciton energies
in ER-MIM-Im and the results obtained from optical reflectance (?),
photoluminescence (?), and photocurrent (?) measurements. It is noteworthy
that our spectral resolution stands on par with, or surpasses, those of the
above-mentioned area-averaged methods. This compelling correspondence
demonstrates that our ER-MIM technique enables accurate photoelectric
measurements of exciton spectra with the added benefit of exceptional spatial
resolution and access to a temperature regime that is not accessible with
near-field optical techniques.

###  1.2 The interplay between excitons and the charge environment

To untangle the exciton-electron interactions at the nanoscale, we
systemically investigate local excitonic behaviors by collecting ER-MIM
spectra in three distinct spatial regions of the sample (Fig. 2A): (I) a
region that extends beyond the graphite back gate (BG), (II) the region
directly above the BG, and (III) the depletion region between I and II.

Fig. 2B depicts two scenarios of exciton-electron interactions. The first
involves trions, charged and weakly coupled three-particle complexes formed by
binding two electrons (or holes) to one hole (or electron). Trion and exciton
eigenenergies differ by the trion binding energy, denoted as $\rm E_{T}$ (left
schematic). The second scenario features charge-neutral bosons resulting from
excitons interacting with the degenerate Fermi sea of excess charge carriers
(right schematic). In this context, excitons become dressed by Fermi sea
excitations, giving rise to attractive and repulsive exciton-polaron (AP and
RP) quasiparticles, akin to Fermi-polarons in the context of cold atoms. The
energy difference between AP and RP branches, or the exciton polaron binding
energy is approximately $\rm E_{T}+\epsilon_{F}$, with $\epsilon_{F}$
representing the Fermi energy (?, ?). To examine these two physical pictures
locally, we take advantage of the sensitivity of ER-MIM spectra to
$\epsilon_{F}$ (gate)-tunable local excitonic energy levels.

In Fig. 2C, we present the ER-MIM spectrum obtained in region I, corresponding
to the intrinsically $n$-doped (grounded and ungated) area. The spectrum
reveals two distinct resonance features: one centered at $\sim$1.64 eV,
corresponding to exciton $A_{1s}$ (green dashed line), and the other centered
at $\sim$1.59 eV, corresponding to trion $T_{1s}$ (purple dashed line).
Notably, at 1.5 K, the linewidth of the exciton $A_{1s}$ dip is approximately
$\rm\sim$2 meV, closely approaching the theoretically calculated intrinsic
linewidth of excitons (?).

On the other hand, we conducted spatially-resolved ER-MIM measurements at four
distinct locations spaced 50 nm apart in region II (Fig. 2A, spots 1-4) at a
constant $\rm V_{bg}=-2.6\,V$. The line spectra zoomed in near the trion
resonance energy are shown in Fig. 2D. The formerly designated $T_{1s}$
feature divides into two separate dips (marked by the red and blue dashed
lines). Moreover, the intensity of the two dips exhibits significant
variations across different spots.

To further understand the spatial dependence, we measured the back gate
dependence of the ER-MIM spectra sitting at spot 4. The red and blue circles
in Fig. 2D are extracted exciton eigenenergies at different $\rm V_{bg}$ near
formerly designated $T_{1s}$, and the green circles in Fig. 2E are another
eigenenergy near formerly designated $A_{1s}$. The red and blue $T_{1s}$
branches display linear redshifts while the $A_{1s}$ exciton energy
demonstrates an almost linear blueshift with $\rm V_{bg}$. This trend remains
consistent throughout the range of $\rm|V_{bg}|$ in our experiments. The
increasing energy separation between the branches in Fig. 2E and the other one
in Fig. 2F is a signature of exciton-polaron formation at elevated carrier
doping levels. In this framework, the formerly assigned $A_{1s}$ becomes
repulsive exciton-polaron (RP). The $T_{1s}$ branch turn into attractive
exciton-polarons (AP1 and AP2), where the splitting between AP1 and AP2 can be
understood as excitons coupling with the Fermi seas in two different valleys
(Fig. 2B).

The observed gate dependent shifts in AP and RP eigenenergies indicate
interactions between excitons and free carriers. These energy shifts exhibit
an almost linear dependence on $\rm V_{bg}$, reflecting a collective effect of
exciton binding energy, quasiparticle bandgap renormalization, and the binding
energy of the exciton polaron. To quantify the shifts, we introduce a
coefficient $\beta$, where $\delta E=\beta\rm n$, with $\delta E$ being the
eigenenergy of an exciton polaron state which has a linear dependence on
carrier density $\rm n$. The observed nanoscale spatial variation of $\delta
E$ suggests that the eigenenergy changes can be used as a sensitive local
indicator of the charge environment.

###  1.3 The interplay between excitons and the electric environment

To understand the intricate interplay between excitons and the local
electrical fluctuations, we introduce a negative $\rm V_{bg}$ in region II
while keeping region I grounded and ungated. In this way, a controlled $p$-$n$
junction ($n$-doped on the region I side and $p$-doped on the region II side)
is formed, as shown in the schematic in Fig. 3A. Region III is then the
depletion region where an in-plane electric field drives electrons and holes
towards regions I and II, respectively.

To illustrate the excitonic excitation under the in-plane field, a systematic
ER-MIM scan across the $p$-$n$ junction was carried out as a function of $\rm
V_{bg}$ and excitation energy. We mark the locations in Fig. 3B and plot the
results in Fig. 3C. It is evident that a continuous variation in the ER-MIM-Im
signal near the $\rm A_{1s}$ AP and RP resonances is observed across the
$p$-$n$ junction. Especially, when on resonance, there is a continuous change
from positive to negative photoconductivity from the left-hand side of the
nanojunction to the right-hand side. This marks the conversion from a
photoconductive effect-dominated mechanism in the ungated region of MoSe2 to
Auger tunneling-dominated mechanism in the back-gated region of MoSe2.

The field effect can be studied in greater detail via performaing high-
resolution images in region III. (Fig. 3D and 3E, with locations marked by
stars in Fig. 3B). First of all, the gate dependent spectra in region III are
very different from the ones obtained from region II (Fig. 1B). They feature
mostly positive photoconductivity at excitonic resonances (red colored
regions), and the gating appears to be less effective. Within region III,
measurement in the right part (purple star in Fig. 3B, Fig. 3E) shows a more
effective gating effect, where the restoration of some negative
photoconductivity can be identified. This observation further elucidates the
$p-n$ junction behavior in region III.

To study the excitonic response to electric field, we extracted the RP
eigenenergy from Fig. 3E, and plot it in Fig. 3F. There are two distinct
features that differ from region II (Fig. 2F): the RP energy exhibits a
redshift behavior with doping level, instead of a blueshift; the RP energy has
an abrupt blueshift at $V_{bg}=-2.2$ V. The first feature agrees with the dc
Stark effect of excitons: $\delta E=-\alpha F^{2}/2$, where $\alpha$ is the
exciton polarizability (?), $F$ represents the electric field strength and
$\delta E$ signifies the energy shift. The second feature underscores the
nonlocal effect of the dielectric constant on exciton energies. Since a
carrier density gradient exists in this region, when the right part of the
sample goes across the band edge, it leads to a sharp increase in carrier
compressibility $\frac{1}{n^{2}}(\frac{\partial n}{\partial\mu})$, which then
changes the exciton screening in the left part. This is consistent with the
Coulomb potential screening through a free electron gas, which leads to
another describable dependency of excitons on the environmental permittivity:
$E=E(\epsilon)$. These observations underscore how excitons can serve as
imaging tools for a $p$-$n$ junction and reveal intricate nanoscale parameters
such as electric field and dielectric constant.

###  1.4 Quantitative photoelectrical imaging and exciton-facilitated
electrometry

To quantitatively analyze the ER-MIM data and utilize it for predicting
material properties, we developed a neural network (NN) (?, ?, ?) based on a
dataset of MIM maps. These maps consist of 256$\times$100 pixel images,
obtained with variations in optical excitation energies and back gate voltages
within a 1 $\mathrm{\mu}$m $\times$ 1 $\mathrm{\mu}$m region at the graphite
back gate boundary. Due to the complex, multidimensional nature of the
datasets, traditional fitting approaches prove to be inadequate. In Fig. 4A,
we present three representative sets of MIM-Im and MIM-Re data collected at
$\rm V_{bg}=-4~{}V$, under different conditions: with the light off, under
1.651 eV light excitation, and 1.655 eV light excitation (complete dataset in
SI note 3). The 1.651 eV excitation is off-resonance with excitons, while the
1.655 eV excitation is on resonance with the RP branch of the $A_{1s}$
exciton.

In the following, we show how to use this neural network to predict the local
environment, ($\sigma$, $\kappa$, $h$), i.e., sheet conductivity of MoSe2, the
relative permittivity of the hBN-MoSe2-hBN stack, and relative tip-sample
distance $h$, for a given MIM data set. We construct a feedforward network
(Fig. 4B), with the inputs being MIM-Re and MIM-Im, either from the experiment
or finite element simulations (see SI note 3), and predict the probability at
each set of discretized $(\sigma,\kappa,h)$ values. We train the network with
a total of 64,000 sets of simulation data, which is divided into bins along
the $\sigma$, $\kappa$, and $h$ directions. Each data point is encoded into a
3840-dimensional vector with the elements being its bin membership – 1 for a
member and 0 for a non-member. Due to the lack of one-to-one correspondence
between MIM-Re and MIM-Im, and $(\sigma,\kappa,h)$, we adopt a Binary Cross
Entropy loss. In this way, we transform the prediction into a classification
problem, where the network output represents the probability for each bin. A
good agreement is found between the neural network outputs and the ground
truth.

We then apply the trained network to predict the probability distribution of
$(\sigma,\kappa)$ for a given set of MIM maps in Fig. 4A. The results of
$(\sigma,\kappa)$ are shown in Fig. 4C. The obtained $\kappa$ maps indicate
the presence of local dielectric disorder in the environmental permittivity,
as well as permittivity change of MoSe2 upon back gate doping and optical
excitation. The obtained $\sigma$ maps present the spatially dispersive
negative photoconductivity from the RP exciton polaron. These quantified
$(\sigma,\kappa)$ maps hence provide a direct measure of the exciton-induced
photoconductivity and the impact of the dielectric environment.

Following the quantification of $(\sigma,\kappa)$ maps and our understanding
of the interactions among various variables, such as $F$, $\sigma$, $\kappa$,
and exciton eigenenergies discussed in the last two sections, we formulate the
subsequent expression to describe the local exciton eigenenergy $E$:

$E=E_{0}(\kappa)+\beta n-\frac{1}{2}\alpha(\kappa)F^{2}.$ (1)

The first term of the expression, $E_{0}$, represents the exciton eigenenergy
for monolayer MoSe2 under the influence of environmental permittivity, without
further considering exciton-electron interactions or electric field effects.
The variable $n$ corresponds to carrier density, and $\beta$ is a fitting
coefficient. The second term represents the interaction between the exciton
and free carriers. The linear relationship is extracted from experimental
results, such as Fig. 2C, and the coefficient $\beta$ is derived from the back
gate sweeping data. The third term accounts for the Stark effect resulting
from in-plane electric fields with strength $|F|$. The contribution from out-
of-plane electric field is negligible since the in-plane exciton
polarizability $\alpha$ is approximately two orders of magnitude larger than
the out-of-plane value due to hBN dielectric screening (?). It is important to
note that the parameter $\beta$ results from a combination of both bandgap
renormalizations and the reduction in exciton binding energies, and these
effects largely cancel each other out. Therefore, the third term would
dominate the spatial variation of $E$.

Subsequently, we determine the exciton eigenenergies at each grid point
through fitting, interpolate them, and present the results in Fig. 4D.
Following this, by using Eqn. 1 with NN-predicted $(\sigma,\kappa)$ and $E$ as
inputs, we generate the predicted electric field distribution, which is
illustrated in Fig. 4E. Through finite element simulation based on
$(\sigma,\kappa)$, we independently derive the electric field map, as shown in
Fig. 4F. It is evident that the exciton-assisted prediction of the electric
field map effectively captures the hotspots in the nanojunction at the
graphite boundary and the surrounding regions with changes in conductivity.
This underscores the potential of exciton-assisted electrometry.

##  2 Conclusions

Our work represents a significant advancement in the field of nano-imaging for
exciton dynamics. The ER-MIM introduces distinct advantages over existing
methods including low photon count, cryogenic compatibility, high spatial and
spectral resolution, and superior sensitivity. The application of ER-MIM to a
gate tunable monolayer MoSe2 device uncovered the intricate interplay between
excitons, charge carriers, and external perturbations in a confined dielectric
environment at cryogenic temperatures. Employing a neural network, we
developed an algorithm for determining the local electric field, conductivity,
and environmental permittivity. These insights lay the foundations for
employing excitons as an all-in-one sensor for local electrical information
and marks a leap in advancing sensing modalities for quantum materials and
devices.

##  3 Acknowledgements

Funding: This work was supported by the QSQM, an Energy Frontier Research
Center funded by the U.S. Department of Energy (DOE), Office of Science, Basic
Energy Sciences (BES), under Award #DE-SC0021238. B. F. acknowledges support
from NSF-DMR-2103910 for sample fabrication. K.W. and T.T. acknowledge support
from the JSPS KAKENHI (Grant Numbers 20H00354 and 23H02052) and World Premier
International Research Center Initiative (WPI), MEXT, Japan. Part of this work
was performed at the Stanford Nano Shared Facilities (SNSF), supported by the
National Science Foundation under award ECCS-2026822. Z. J. and Z. Z.
acknowledge support from the Stanford Science fellowship. M.K.L. acknowledges
support from the NSF Faculty Early Career Development Program under Grant No.
DMR - 2045425. The research is funded in part by a QuantEmX grant from ICAM
and the Gordon and Betty Moore Foundation through Grant GBMF9616 to M.K.L..
M.K.L. also acknowledges helpful discussion with Xinzhong Chen.

Figure 1: Sensing exciton Rydberg states at the nanoscale. A. The experimental
setup of the Exciton resonant - microwave impedance microscopy (ER-MIM). B.
The response curves of MIM-Im and MIM-Re as a function of the sheet
conductivity of MoSe2. The darker blue color indicates a conductivity decrease
relative to neighboring regions, and a deeper red a conductivity increase. C.
A schematic showing relevant energy levels and optical transitions forming 1s,
2s, and 3s Rydberg states of A and B excitons in monolayer MoSe2. D. 2D plot
of MIM-Im signal as a function of back gate voltage $\rm V_{bg}$ and optical
excitation energy. The blue and red dashed lines correspond to two
representative doping levels. E. The comparison of ER-MIM results (at the
doping levels marked in D) with spectra from three far-field optical
techniques: optical reflectance (?), photoluminescence (?), and photocurrent
(?) measurements.

Figure 2:  The interplay between exciton behaviors with local charge
environment. A. A MIM map showing the MoSe2 sample (flake boundary outlined by
a white dashed line) partially on top of a graphite back gate (boundary
outlined by a black dashed line). The device can be divided into three
regions, I (outside of graphite), II (above graphite),s and III (depletion
region, with boundaries outlined by two white dotted lines). B. Schematics of
the exciton and trion spectra (left), and the exciton-polaron spectra (right).
C. The ER-MIM-Im spectrum measured in region I. The grey dashed and dotted
lines are fittings of $A_{1s}$ exciton and $T_{1s}$ trion. D. ER-MIM-Im
spectra measured at four locations in region II. The spots are marked by
orange triangles in A. E. The evolution of two eigenenergies of the A exciton:
the red and blue dotted lines are the two attractive branches $\rm AP_{1}$ and
$\rm AP_{2}$. F. The evolution of the repulsive polaron branch, as shown by
the green dotted line. e-ex. int stands for exciton-electron interactions.

Figure 3: The interplay between exciton behaviors and electric environment. A.
A schematic of the $p$-$n$ junction formed in region III. B. MIM-Im image
showing the sample geometry, with colored circles and stars marking different
locations of further measurements. (The same region as Fig. 2A but at a larger
scale, with the MoSe2 flake boundary outlined by white dashed lines, and the
graphite boundary outlined by a black dashed line.) C. A series of ER-MIM-Im
measurements at the locations specified by the colored circles in B. The color
of the boxes are in correspondence with the color of the circles. D. ER-MIM-Im
measured at the yellow star in B. E. ER-MIM-Im measured at the purple star in
B. F. The extracted repulsive polaron eigenenergy as a function of back gate
voltage. The red dashed lines in E and F mark the $V_{bg}$ at which the RP
exciton energy has a sudden blueshift.

Figure 4:  Holistic electrical sensing with excitons. A. MIM-Re and MIM-Im
images measured with light off, off resonance with exciton (1.625 eV), and on
resonance with exciton (1.647 eV), in a 1 $\mathrm{\mu m}\times 1$
$\mathrm{\mu m}$ region. The black dashed line denotes the graphite boundary.
Scale bar: 200 nm. B. Neural network architecture. C. The neural network
training result of permittivity $\kappa$, conductivity $\sigma$ for the three
columns of MIM data in A. Scale bar: 200 nm. D. Left: The ER-MIM hyperspectra
collected on a 9$\times$9 grid. Right: The interpolated exciton eigenenergy
extracted from the hyperspectra. Scale bar: 200 nm. E. The interpolated in-
plane electric field strength prediction based on Eq. (1), and the input from
C and D, as indicated by the green flow arrows. Scale bar: 200 nm. F. The
interpolated in-plane electric field strength prediction by finite element
simulation, with the input from C, as indicated by the black flow arrows.
Scale bar: 200 nm.

## References and Notes

  * 1. M. Saffman, T. G. Walker, K. Mølmer, Reviews of modern physics 82, 2313 (2010).
  * 2. F. Casola, T. Van Der Sar, A. Yacoby, Nature Reviews Materials 3, 1 (2018).
  * 3. R. Schirhagl, K. Chang, M. Loretz, C. L. Degen, Annual review of physical chemistry 65, 83 (2014).
  * 4. C. L. Degen, F. Reinhard, P. Cappellaro, Reviews of modern physics 89, 035002 (2017).
  * 5. A. Popert, et al., Nano Letters 22, 7363 (2022).
  * 6. S. B. Chand, et al., Nano Letters 22, 3087 (2022).
  * 7. A. Raja, et al., Nature communications 8, 15251 (2017).
  * 8. S. Manzeli, D. Ovchinnikov, D. Pasquier, O. V. Yazyev, A. Kis, Nature Reviews Materials 2, 1 (2017).
  * 9. G. Wang, et al., Reviews of Modern Physics 90, 021001 (2018).
  * 10. C. Lui, et al., Physical review letters 113, 166801 (2014).
  * 11. D. K. Efimkin, E. K. Laird, J. Levinsen, M. M. Parish, A. H. MacDonald, Physical Review B 103, 075417 (2021).
  * 12. J. Klein, et al., Nano letters 16, 1554 (2016).
  * 13. K. F. Mak, J. Shan, Nature Nanotechnology 17, 686 (2022).
  * 14. E. C. Regan, et al., Nature Reviews Materials 7, 778 (2022).
  * 15. W. Bao, et al., Nature communications 6, 7993 (2015).
  * 16. T. P. Darlington, et al., Nature Nanotechnology 15, 854 (2020).
  * 17. J. Zhou, et al., Nano letters (2023).
  * 18. K. Hasz, Z. Hu, K.-D. Park, M. B. Raschke, Nano Letters 23, 198 (2022).
  * 19. M. Plankl, et al., Nature Photonics 15, 594 (2021).
  * 20. S. Zhang, et al., Nature communications 13, 542 (2022).
  * 21. F. Hu, et al., Nature Photonics 11, 356 (2017).
  * 22. Y. Luan, et al., Nano Letters 22, 1497 (2022).
  * 23. R. B. Iyer, Y. Luan, R. Shinar, J. Shinar, Z. Fei, Nanoscale 14, 15663 (2022).
  * 24. Z. Chu, et al., Proceedings of the National Academy of Sciences 117, 13908 (2020).
  * 25. T. Handa, et al., arXiv preprint arXiv:2306.10814 (2023).
  * 26. C. M. E. Chow, et al., Nano Letters 20, 5538 (2020).
  * 27. A. Sushko, et al., Nanophotonics 10, 105 (2020).
  * 28. A. Arora, K. Nogajewski, M. Molas, M. Koperski, M. Potemski, Nanoscale 7, 20769 (2015).
  * 29. E. Liu, et al., Nature communications 12, 6131 (2021).
  * 30. D. Vaquero, J. Salvador-Sánchez, V. Clericò, E. Diez, J. Quereda, Nanomaterials 12, 322 (2022).
  * 31. M. Selig, et al., Nature communications 7, 13279 (2016).
  * 32. S. V. Kalinin, B. G. Sumpter, R. K. Archibald, Nature materials 14, 973 (2015).
  * 33. A. Krull, P. Hirsch, C. Rother, A. Schiffrin, C. Krull, Communications Physics 3, 54 (2020).
  * 34. S. V. Kalinin, et al., ACS nano 15, 12604 (2021).
  * 35. T. G. Pedersen, Physical Review B 94, 125424 (2016).


#  QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms
to Cooperative Multi-Agent Reinforcement Learning

Pascal Leroy,1 Damien Ernst,1 Pierre Geurts,1
Gilles Louppe,1 Jonathan Pisane,2 Matthia Sabatelli1

###### Abstract

This paper introduces four new algorithms that can be used for tackling multi-
agent reinforcement learning (MARL) problems occurring in cooperative
settings. All algorithms are based on the Deep Quality-Value (DQV) family of
algorithms, a set of techniques that have proven to be successful when dealing
with single-agent reinforcement learning problems (SARL). The key idea of DQV
algorithms is to jointly learn an approximation of the state-value function
$V$, alongside an approximation of the state-action value function $Q$. We
follow this principle and generalise these algorithms by introducing two fully
decentralised MARL algorithms (IQV and IQV-Max) and two algorithms that are
based on the centralised training with decentralised execution training
paradigm (QVMix and QVMix-Max). We compare our algorithms with state-of-the-
art MARL techniques on the popular StarCraft Multi-Agent Challenge (SMAC)
environment. We show competitive results when QVMix and QVMix-Max are compared
to well-known MARL techniques such as QMIX and MAVEN and show that QVMix can
even outperform them on some of the tested environments, being the algorithm
which performs best overall. We hypothesise that this is due to the fact that
QVMix suffers less from the overestimation bias of the $Q$ function.

## Introduction

Reinforcement Learning (RL) is a machine learning paradigm where an agent
needs to learn how to interact with an environment (Sutton and Barto 2018).
While interacting, the agent performs certain actions, which are all then
associated with a certain reward. This reward signal $r$ gives the agent some
feedback about the quality of the actions it has performed over time, and the
central goal of training an RL agent consists of learning a policy $\pi$,
which allows the agent to perform actions which will maximise its expected
cumulative reward. Over the years, several progress has been made by combining
traditional RL techniques with deep neural networks (LeCun, Bengio, and Hinton
2015; Schmidhuber 2015). The successful marriage between RL and deep learning,
which comes with the name of deep reinforcement learning (DRL) (François-Lavet
et al. 2018), has led to many algorithms that have managed to outperform each
other over the years (Mnih et al. 2015; Hasselt, Guez, and Silver 2016; Hessel
et al. 2017; Sabatelli et al. 2018). With the rise of always more successful
and powerful algorithms, part of the DRL community has started to shift its
attention from single-agent RL (SARL) problems to multi-agent RL (MARL), an RL
scenario which is characterised by the presence of multiple agents that
interact together. In the special case of cooperative MARL, agents cooperate
in order to achieve a common goal. Cooperation in multi-agent systems is of
large practical interest since many real-world problems can be formulated in a
cooperative MARL setting. Some examples of these are found in robotics (Aşık
and Akın 2012), sports (Zhao et al. 2019), and, as we will be approaching in
this paper, games.

Sometimes, when compared to more traditional SARL, cooperative MARL can be
characterised by the additional challenge of the environment being partially
observable. Formally, this corresponds to agents interacting in a
decentralised partially observable Markov decision process (Dec-POMDP)
(Oliehoek, Amato et al. 2016), which results in agents that only perceive a
subset of the state space and that select their actions independently (more
details are given in the background section). It is well known that partial
observability can be particularly challenging and prevents a centralised
controller to be used to solve these kinds of MARL problems. As a result, the
MARL community first considered SARL approaches to independently control each
agent with a fully decentralised (FD) controller. Later it exploited a method
called centralised training with decentralised execution (CTDE), which allows
exploiting additional information of the environment.

This paper focuses on these approaches by considering the popular StarCraft
Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) as a testbed. We do this
by first adapting and then improving the DRL algorithms of the Deep Quality-
Value (DQV) family of techniques (Sabatelli et al. 2020). These algorithms are
characterised by jointly learning an approximation of the state-value function
and the state-action value function. They have proven to significantly
outperform popular algorithms such as DQN (Mnih et al. 2015) and DDQN
(Hasselt, Guez, and Silver 2016) in SARL. Specifically, this paper has three
main contributions:

  1. 1.

We generalise the DQV family of algorithms to cooperative MARL problems and
test their performance in an FD training setting.

  2. 2.

Based on their overall performance, we identify their fundamental limitations
and introduce two novel MARL algorithms, called QVMix and QVMix-Max. Both
algorithms combine the original benefits of the DQV algorithms with CTDE,
resulting in a better performance than state-of-the-art techniques.

  3. 3.

We link the performance of the best-performing algorithm, QVMix, to the
overestimation bias of the $Q$ function that characterises model-free RL
algorithms. Our experiments suggest that QVMix suffers less from this
phenomenon than other algorithms.

The rest of this paper is structured as follows: we first provide further
details about the cooperative MARL set-up considered in this paper and briefly
present some of the algorithms that have been introduced over the years. We
then present our novel algorithm. Then, we define the experimental setup and
describe our results. The paper ends with a critical discussion and highlights
possible directions for future work.

## Background

This section presents some preliminary knowledge that will be used throughout
this paper. We start by formally introducing Dec-POMDPs, since all the
experiments that will be reported fall into this specific MARL setting. We
then introduce some of the methods that have been introduced over the years
for dealing with cooperative MARL and which will serve us as a baseline when
investigating the performance of the novel algorithms presented in this paper.

### Dec-POMDPs

In MARL, we can consider Markov Games (Littman 1994), a framework that is
characterised by the fact that each agent has its own reward function.
However, when agents cooperate to achieve a common goal and share the reward
function, another possible framework is Dec-POMDP (Oliehoek, Amato et al.
2016). In this paper we formally define a Dec-POMDP by a tuple
$[n,\mathcal{S},O,\mathcal{Z},\mathcal{U},R,P,\gamma]$, where $n$ agents
cooperate by choosing an action at every timestep $t$. An agent is denoted by
$a\in\\{1,...,n\\}$. The agent action space is defined by $\mathcal{U}_{a}$
such that $\mathcal{U}=\mathcal{U}_{1}\times..\times\mathcal{U}_{n}$ is the
combined set of $n$ action spaces, one per agent. The state of the environment
at timestep $t$ is $s_{t}\in S$ while $o_{t}^{a}\in\mathcal{Z}$ is the
observation of this state perceived by the agent $a$ coming from the
observation function $O:\mathcal{S}\times\\{1,...,n\\}\rightarrow\mathcal{Z}$.
At each timestep $t$, each agent simultaneously executes an action
$u_{t}^{a}\in\mathcal{U}_{a}$ such that the state $s_{t}$ transits to a new
state $s_{t+1}$ with a probability defined by the transition function
$P(s_{t+1}|s_{t},\mathbf{u_{t}}):\mathcal{S}^{2}\times\mathcal{U}\rightarrow\mathbb{R^{+}}$
where $\mathbf{u_{t}}=\bigcup_{a\in\\{1,..,n\\}}u_{t}^{a}$. After all agents’
actions are performed, a common team reward
$r_{t}=R(s_{t+1},s_{t},\mathbf{u_{t}}):\mathcal{S}^{2}\times\mathcal{U}\rightarrow\mathbb{R}$
is assigned to the agents. Agent $a$ chooses its action based on its current
observation $o_{t}^{a}\in\mathcal{Z}$ and its history
$h_{t}^{a}\in(\mathcal{Z}\times\mathcal{U}_{a})^{t-1}$. Its policy is the
function
$\pi^{a}(u_{t}^{a}|h_{t}^{a},o_{t}^{a}):(\mathcal{Z}\times\mathcal{U}_{a})^{t}\rightarrow\mathbb{R^{+}}$
that maps, from its history and the current observation at timestep $t$, the
probability of taking action $u_{t}^{a}$ at timestep $t$. During an episode,
the cumulative discounted reward that is obtained from timestep $t$ over the
next $T$ timesteps is defined by $R_{t}=\sum_{i=0}^{T-1}\gamma^{i}r_{t+i}$,
where $\gamma\in[0,1)$ is the discount factor. In this paper, the goal of each
agent is to maximise its expected $R_{0}$ over a number of timesteps $T$
defined by the episode time limit. $R_{0}$ depends on the team’s policy
defined from the individual agents policies:
$\mathbf{\pi}={\pi^{1},...,\pi^{n}}$. The optimal team’s policy $\pi^{*}$
maximises the expected cumulative discounted reward:
$\pi^{*}=\operatorname*{argmax}_{\pi}R_{0}$. To evaluate the team’s policy, we
consider the state-value function
$V^{\pi}(s)=\mathds{E}[R_{t}|s_{t}=s,\mathbf{\pi}]$ also called the $V$
function. Because rewards are shared among agents, the $V$ function has the
same value for all agents: $V^{\pi}(s)=V^{\pi^{1}}(s)=V^{\pi^{n}}(s)$. We also
define the state-joint-actions value function
$Q^{\mathbf{\pi}}(s,\mathbf{u})=\mathds{E}[R_{t}|s_{t}=s,\mathbf{u_{t}}=\mathbf{u},\mathbf{\pi}]$,
also called $Q$ function, which characterises the expected return of the team
when playing action $\mathbf{u}$ at timestep $t$ and following policy $\pi$
afterwards. The corresponding individual state-action value function of an
agent policy is defined by
$Q^{\pi^{a}}(s,u)=\mathds{E}[R_{t}|s_{t}=s,u^{a}_{t}=u,\mathbf{\pi}]=Q^{\mathbf{\pi}}(s,\mathbf{u})$,
and denoted later as $Q_{a}$ for the sake of conciseness.

### Value-Based Methods

While several techniques have been introduced over the years, ranging from
value-based RL algorithms (Son et al. 2019; Yang et al. 2020; Wang et al.
2020), to policy gradient methods (Foerster et al. 2018; Du et al. 2019), in
this paper we mostly restrict our analysis to the first family of methods,
while keeping a more thorough potential comparison to policy gradient methods
as future work.

In value-based RL, the main goal consists of learning value functions that
indicate how good or bad it is for an agent to select a certain action in a
particular state. Formally, this corresponds to learning the optimal $Q$
function which, in a single-agent setting, realises the optimal expected
return $Q^{\pi^{*}}(s,u)=\underset{\pi}{\max}\>Q^{\pi}(s,u)\ \text{for all}\
s\in\mathcal{S}\ \text{and}\ u\in\mathcal{U}.$ Once the optimal $Q$ function
has been learned, it becomes a straightforward task to derive an optimal
policy by greedy-selection $u^{*}=\operatorname*{argmax}_{u}Q(s_{t},u)$. It is
well-known that when the complexity of the environment increases, it is not
possible to learn $Q^{\pi^{*}}(s,u)$ with tabular RL techniques such as
Q-Learning (Watkins and Dayan 1992). To overcome this problem, a function
approximator, such as a convolutional neural network parametrised by $\theta$,
can be used for modeling $Q(s,u;\theta)\approx Q^{\pi^{*}}(s,u)$. As first
introduced by Mnih et al. (2015) DQN’s algorithm, the network can be trained
with an objective function that resembles Q-Learning coming in the following
form:

$\begin{split}\mathcal{L}(\theta)=\mathds{E}_{\langle.\rangle\sim
B}\bigg{[}\big{(}r_{t}+\gamma\max_{u\in\mathcal{U}}Q(s_{t+1},u;\theta^{\prime})\\\
-Q(s_{t},u_{t};\theta)\big{)}^{2}\bigg{]}\end{split}$ (1)

where $B$ is the experience replay memory buffer used for storing and
uniformly sampling RL trajectories $\langle.\rangle$ in the form of $\langle
s_{t},a_{t},r_{t},s_{t+1}\rangle$, and $\theta^{\prime}$ are the parameters of
the target-network. While Mnih et al. (2015) consider $\theta$ to be the
parameters of a convolutional neural network, when partially observable
environments are considered $\theta$ usually parametrises a recurrent neural
network (RNN) such as an LSTM (Hochreiter and Schmidhuber 1997) or a GRU
(Chung et al. 2014) as in (Hausknecht and Stone 2015). Since the agent does
not perceive the state $s_{t}$, the input of such network is the observation
$o_{t}$. Although Q-Learning (Watkins and Dayan 1992) has inspired the Mnih et
al. (2015) DQN algorithm, in what follows we consider its MARL extension
instead: Independent Q-Learning (IQL) Tan (1993).

### IQL

IQL consists of a fully decentralised algorithm where each agent learns its
respective $Q$ function independently. This is done in a SARL-like fashion
with all agents ignoring any type of information other than the one that is
provided through their own observations. Since agents deal with partial
observability, it is a common practice to approximate each individual $Q$
function with RNNs.

The problem with IQL is that agents must select actions that maximise
$Q(s_{t},\mathbf{u_{t}})$ and not their own $Q$ function. As they cannot know
which actions are taken by others, it is usually not possible to approximate
this function in a decentralised way. However, it is possible to approximate
$Q(s_{t},\mathbf{u_{t}})$ as a function of individual $Q$ functions such that
the taken actions maximise both the joint and the individual $Q$ functions. If
individual $Q$ functions satisfy the Individual-Global-Max condition (IGM)
(Son et al. 2019):
$\operatorname*{argmax}_{\mathbf{u_{t}}}Q(s_{t},\mathbf{u_{t}})=\bigcup_{a}\operatorname*{argmax}_{u_{t}^{a}}Q_{a}(s_{t},u_{t}^{a})$,
it is possible to factorise $Q(s_{t},\mathbf{u_{t}})$ with these individual
$Q$ functions. This is the core idea defining centralised training in value
based methods.

### QMIX

QMIX (Rashid et al. 2018) is a CTDE method where the factorisation of the
state-joint-actions value function, denoted as
$Q_{mix}(s_{t},\mathbf{u_{t}})$, is performed as a monotonic function of the
individual $Q$ functions
$Q_{mix}=\text{Mixer}\left(Q_{a_{1}}(s_{t},u_{t}^{a_{1}}),..,Q_{a_{n}}(s_{t},u_{t}^{a_{n}}),s_{t}\right)$
where Mixer is a function that ensures IGM through the monotonic
$\frac{\partial Q_{mix}(s_{t},\mathbf{u_{t}})}{\partial
Q_{a}(s_{t},u_{t}^{a})}\geq 0\text{ }\forall a\in\\{a_{1},..,a_{n}\\}$. This
is satisfied by using a hypernetwork (Ha, Dai, and Le 2016) defined by a
parameter network $h_{p}(.):\mathcal{S}\rightarrow\mathbb{R}^{|\phi|+}$ which
takes the state $s_{t}$ as input and predicts the strictly positive
parameters111To be exact, the offset defined by $h_{p}()$ is not constrained
to be positive. $\phi$ of the main network
$h_{o}(.):\mathbb{R}^{n}\times\phi\rightarrow\mathbb{R}$, called the mixer
network, therefore,
$Q_{mix}(s_{t},\mathbf{u_{t}})=h_{o}\left(Q_{a_{1}}(),..,Q_{a_{n}}(),h_{p}(s_{t})\right)$.
The monotonicity of $Q_{mix}$ w.r.t the individual $Q$ functions is satisfied
because a neural network made of monotonic functions and strictly positive
weights is monotonic w.r.t to its inputs. In addition, the outputs of the
$Q$-Mixer depend on the state thanks to the hypernetwork. The optimisation
procedure follows the same principles used by the DQN algorithm but applied to
$Q_{mix}(s_{t},\mathbf{u_{t}})$.

### MAVEN

Mahajan et al. (2019) demonstrated that in certain circumstances, the
exploration capabilities of QMIX are limited. To tackle this problem, they
added a latent space to influence the behaviour of the agents. It does so by
being the input of a parameter network that computes parameters of the fully
connected layer linking recurrent cells to the output in the individual $Q$
network. This latent variable $z$ is generated once an episode begins by a
hierarchical policy network, taking as input the initial state of the
environment together with a random variable which is typically discrete and
sampled from a uniform distribution. The objective that is used for training
MAVEN’s networks is composed of three parts. First, once the latent variable
network is fixed the two hypernetworks (the mixer and the latent space
parameter network) and the recurrent networks are trained using the same loss
that is also used by QMIX. Second, when these three networks are fixed, the
latent variable network can be trained using any policy optimisation such as
the policy gradient method (Sutton et al. 1999), computed with the total sum
of rewards per episode. Finally, the influence of the latent variable on the
agents behaviour is enforced via a mutual information loss between the latent
variable and consecutive transitions. For more details on how MAVEN works we
refer the reader to (Mahajan et al. 2019).

## Generalising the DQV family of algorithms to MARL

Our contributions build on top of the work revolving around the Deep Quality-
Value (DQV) family of DRL algorithms (Sabatelli et al. 2018, 2020). The core
idea of these SARL techniques is to learn an approximation of the state-value
function
$V^{\pi}(s)=\mathds{E}[\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}|s_{t}=s,\pi]$
alongside an approximation of the state-action value function
$Q^{\pi}(s,u)=\mathds{E}[\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}|s_{t}=s,u_{t}=u,\pi]$.
The DQV algorithms generalise the tabular RL algorithms presented in (Wiering
2005) and (Wiering and Van Hasselt 2009) and use deep neural networks serving
as value function approximators. Learning a joint approximation of the $V$
function and the $Q$ function can be done in two different ways, both of them
requiring two different neural networks for learning $Q(s,u;\theta)\approx
Q^{\pi^{*}}(s,u)$ and $V(s;\phi)\approx V^{\pi^{*}}(s)$.

  * •

DQV-Learning learns an approximation of the state-value function by minimising
the following objective

$\begin{split}\mathcal{L}(\phi)=\mathds{E}_{\langle.\rangle\sim
B}\bigg{[}\big{(}r_{t}+\gamma V(s_{t+1};\phi^{\prime})\\\
-V(s_{t};\phi)\big{)}^{2}\bigg{]},\end{split}$ (2)

while the following loss is minimised for learning the $Q$ function:

$\begin{split}\mathcal{L}(\theta)=\mathds{E}_{\langle.\rangle\sim
B}\bigg{[}\big{(}r_{t}+\gamma V(s_{t+1};\phi^{\prime})\\\
-Q(s_{t},u_{t};\theta)\big{)}^{2}\bigg{]}.\end{split}$ (3)

  * •

DQV-Max Learning uses the same equation presented in Eq. (3) for learning the
$Q$ function and the following loss for learning the $V$ function:

$\begin{split}\mathcal{L}(\phi)=\mathds{E}_{\langle.\rangle\sim
B}\bigg{[}\big{(}r_{t}+\gamma\>\underset{u\in\mathcal{U}}{\max}\>Q(s_{t+1},u;\theta^{\prime})\\\
-V(s_{t};\phi)\big{)}^{2}\bigg{]}.\end{split}$ (4)

We test the update rules of DQV and DQV-Max in a FD training setting and in a
CTDE setting. In the first case DQV and DQV-Max follow the same principles as
the ones of Independent Q-Learning and we therefore refer to these algorithms
as IQV and IQV-Max. In the second case, both algorithms are used in
combination with hypernetworks in a QMIX like fashion. We refer to these
algorithms as QVMix and QVMix-Max.

## Experimental setup

(a) 3s5z

(b) so_many_baneling

Figure 1: Two pictures of different SMAC scenarios:3s5z at the top and
so_many_baneling at the bottom.

We test a total of seven algorithms: the four DQV extensions introduced in the
previous section are compared to the three algorithms introduced in the
background section: IQL, QMIX and MAVEN. As a test-bed, we use the popular
SMAC (Samvelyan et al. 2019) environment and evaluate the methods on eight
different scenarios (also called maps): 3m, 8m, so_many_baneling, 2m_vs_1z,
MMM, 2s3z, 3s5z and 3s_vs_3z. We refer the reader to Samvelyan et al. (2019)
for a more in-depth explanation of the differences between map configurations.
It is worth noting that the maps chosen for our experiments differ in terms of
complexity. Two examples of different scenarios are visually represented in
Fig. 1. No matter which map is used, the SMAC environment provides the
different agents with the following information: agents can observe their
health, type, last action performed and a list of possible directions they can
move to. If other agents are within their sight range, they can also observe
which unit type these agents are part of, how distant they are, and their
health. More importantly, if other agents are an agent’s ally, their last
performed action is visible, if not, the only information provided corresponds
to whether such an agent is within an attack range. All this information is
encoded as a Boolean or a value $\in[0,1]$. Among the different possible
actions, agents can decide whether to move in one of the four cardinal
directions, not to perform any action at all, or attack an opponent if the
latter one is within their attack range. Each attack has the effect of
removing a certain amount of health from the opponent. At each time step, the
agent receives a zero or positive reward, shared among the team’s agents. The
goal is to maximise the reward, achieved by reducing each opponent team unit’s
health to zero, which is then called a win.

Each algorithm is trained on every map ten times, where each neural network is
trained from scratch. We train the network for either $5m$ or $10m$ timesteps
depending on the considered map and the time that is required to achieve
convergence. Every $20000$ timesteps, the parameters of the networks are saved
and we perform $24$ testing episodes.

To keep the comparison among the different tested algorithms as fair as
possible, we ensure that each algorithm uses the same type of hyper-parameters
ranging. Specifically, we referred to the authors of QMIX, MAVEN and IQL to
determine the set of hyper-parameters and have simply kept the same values for
QVMix, QVMix-Max, IQV and IQV-Max. For a more thorough presentation of all
used hyper-parameters we refer the reader to the open-sourced
code222https://github.com/PaLeroy/QVMix. As is common practice within the
literature, to improve the learning speed of the algorithms, the parameters of
the individual networks are shared among agents.

## Results

Training steps | Map | QMIX | MAVEN | QVMix | QVMix-Max | IQL | IQV | IQV-Max
---|---|---|---|---|---|---|---|---
$5m$ | 3m | 100 | 98.7 | 100 | 100 | 93.3 | 93.3 | 96.6
8m | 96.6 | 98.3 | 100 | 96.6 | 83.3 | 93.3 | 90
so_many_baneling | 100 | 97 | 100 | 100 | 50 | 40 | 40
2m_vs_1z | 100 | 100 | 100 | 96.6 | 100 | 100 | 100
$10m$ | MMM | 100 | 97.0 | 93.3 | 96.6 | 61.6 | 83.3 | 50
2s3z | 96.6 | 97.5 | 96.6 | 100 | 59.9 | 56.6 | 40
3s5z | 40 | 40.8 | 86.6 | 43.3 | 16.6 | 13.3 | 0
3s_vs_3z | 100 | 97.9 | 100 | 100 | 83.3 | 76.6 | 63.3
Table 1:  Means of win-rates achieved at the end of training by QMIX, MAVEN,
QVMix, QVMix-Max, IQL, IQV, and IQVMax in eight scenarios. In the first four
scenarios, 3m, 8m, so_many_baneling and 2m_vs_1z, it is measured after $5$
millions training timesteps. In the last four, MMM, 2s3z, 3s5z and 3s_vs_3z it
is measured after $10$ millions training timesteps. In the green and yellow
cells, we report the best and second-best means respectively. When results are
equivalent, the cells report the fastest and second-fastest method that
reaches a win-rate of $100\%$ as it can be seen in Figure 3.

We report the results of our experiments in two different ways. We start by
first analysing the win-rate of each tested method and then also investigate
the quality of the value functions that are learned by all algorithms.

### Overall performance

In Table 1, we present for each map and algorithm the respective win-rate by
reporting the means that are measured at the end of training. If the
algorithms perform equally in terms of overall performance, meaning that the
average win-rate is the same, we consider the one which significantly
converges the fastest to be the best performing algorithm. Please note that
reporting the win-rate of an episode is a good indicator of the quality of an
agent’s learned policy, since as introduced in the previous section, a win
directly corresponds to the best achievable sum of rewards an agent can
receive. We start by observing the differences in terms of performance between
the FD methods (IQL, IQV and IQV-Max) and their respective CTDE extensions
(QMIX, QVMix and QVMix-Max) and MAVEN. As one might expect, we can see from
the results reported in Fig. 2 that FD methods converge more slowly when
compared to their CTDE counterparts on the considered 2_vs_1z map. This is
particularly interesting since it already shows that the DQV family of
algorithms can be successfully adapted when it comes to MARL, both in an FD
training set-up and in a CTDE one. However, these results are challenged once
the number of agents in the maps increases: examples of such maps are
so_many_baneling, MMM or 3s5z. As is clearly reported both in Table 1 and in
the plots represented in Fig. 3 the performance of FD methods starts to drop,
highlighting that CTDE methods are required once the complexity of the
training scenario increases.

Figure 2: Mean of win-rates achieved in the 2m_vs_1z scenario by QVMix, QVMix-
Max, QMIX, MAVEN, IQV, IQVMax and IQL. The error band is proportional to the
variance of the measure. We can observe that all methods, although CTDE
methods result in faster training than FD methods. We can also see that all
four of the novel algorithms based on the DQV family of algorithms can be
successfully used in cooperative MARL. Figure 3: Means of win-rates achieved
by QVMix, QVMix-Max, QMIX, MAVEN, IQV, IQVMax and IQL in eight scenarios. Top
to bottom, left to right, the scenarios are 3m, 8m, so_many_baneling,
2m_vs_1z, MMM, 2s3z, 3s5z and 3s_vs_3z. The error band is proportional to the
variance of win-rates.

Therefore, we direct our attention to CTDE methods only and we can then start
to see that both QVMix and QVMix-Max in most of the eight maps perform as well
as QMIX and MAVEN. We again report the evolution of the win-rate of each
algorithm on every tested map in Fig. 3. When we consider the MMM, 3m,
2m_vs_1z and the so_many_baneling maps we can observe that there is no
significant difference between the performance that is obtained by our
algorithms and that of QMIX and MAVEN. All methods converge towards the best
possible winning-rate and in terms of convergence speed perform closely.

However, when considering the 2s3z, 3s5z and 8m maps we can now observe that
the performance of QVMix results in even faster learning. Of even greater
interest, when looking at the results obtained on the 3s5z map, QVMix is the
only algorithm which approaches the best possible win-rate. It is also worth
noting that the performance of QVMix-Max is always competitive with the one
obtained by QVMix, QMIX and MAVEN. These results do not come as a surprise
since similar performance has been observed when DQV-Max was tested in an SARL
set-up (Sabatelli et al. 2020).

### Overestimation bias

Figure 4: $Q$ values obtained and estimed when training QVMix, QVMix-Max,
MAVEN and QMIX. Dash-dotted lines represent the obtained $Q$ values while
solid lines represent the estimated ones.

To understand the reasons behind why QVMix is the best performing algorithm
overall, we analyse how well each method estimates the state-joint-action
value function $Q(s_{t},\mathbf{}{u_{t}})$. Since, in most maps, FD methods do
not perform as well as CTDE methods, we restrict our analysis to CTDE
algorithms only where their respective mixer networks give the estimated
$Q(s_{t},\mathbf{u_{t}})$. Since the state space of the maps provided by the
SMAC environment is not finite, it is unfortunately not possible to compute
the real $Q(s_{t},\mathbf{u_{t}})$ of each state. To overcome this problem, we
instead compute the discounted sum of rewards obtained with the current policy
in each visited state during an episode and compare the results with the value
function inferred from the $Q$ values estimated by the mixer network for these
states. The closer the estimates are to the real
$Q(s_{t},\operatorname*{argmax}_{\mathbf{u}}(Q(s_{t},\mathbf{u})))$, the more
accurate the learned value function is.

For this experiment, we selected two different maps: the 2m_vs_1z map, which
corresponds to the map on which the best results have been achieved by all
methods at the same time, and the 3s5z map, which on the other hand, is the
map on which QVMix performed less well. We report in Fig. 4 the averaged
estimated $Q$ values, represented by the solid lines, and the actual
discounted sum of rewards, represented by the dash-dotted lines. All are
computed for each visited state at testing time. In both scenarios, we can
observe that the $Q$ values that are estimated by QMIX and MAVEN suffer from
the overestimation bias of the $Q$ function, while this is not the case for
QVMix and QVMix-Max. We, therefore, justify the better quality of QVMix and
QVMix-Max policies by a better approximation of the $Q$ functions, although
further work will be required to understand this phenomenon more in detail.

## Discussion and Conclusion

We introduced four new value-based methods that can be used to train a team of
agents in a cooperative MARL framework. Two of our methods are designed for a
fully decentralised execution training scheme (IQV and IQVMax) while two are
dedicated to centralised-training with decentralised execution (QVMix and
QVMix-Max). We compared our algorithms with three methods taken from the
literature and used the StarCraft Multi-Agent Challenge as benchmark. We have
shown that QVMix and QVMix-Max achieve the same results as popular state-of-
the-art techniques (QMIX and MAVEN) and that on some of the maps, QVMix can
result in faster and better learning. We suggest that this better performance
can be related to the fact that QVMix seems to suffer less from the
overestimation bias of the $Q$ function. As future work, it would be
interesting to analyze each agent’s behaviour and study the impact of the
value function in the optimisation procedure. Furthermore we also aim at
finding the best possible set of hyper-parameters of each algorithm so that
their performance can be exploited even further and more fairly.

## Acknowledgments

PL acknowledges the financial support of the Walloon Region in the context of
IRIS, a MecaTech Cluster project.

MS acknowledges the financial support of BELSPO, Federal Public Planning
Service Science Policy, Belgium, in the context of the BRAIN-be project.

## References

  * Aşık and Akın (2012) Aşık, O.; and Akın, H. L. 2012.  Solving multi-agent decision problems modeled as dec-pomdp: A robot soccer case study.  In _Robot Soccer World Cup_ , 130–140. Springer.
  * Chung et al. (2014) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In _NIPS Workshop on Deep Learning_.
  * Du et al. (2019) Du, Y.; Han, L.; Fang, M.; Liu, J.; Dai, T.; and Tao, D. 2019.  LIIR: Learning individual intrinsic reward in multi-agent reinforcement learning.  In _Advances in Neural Information Processing Systems_ , volume 32, 4403–4414.
  * Foerster et al. (2018) Foerster, J. N.; Farquhar, G.; Afouras, T.; Nardelli, N.; and Whiteson, S. 2018\.  Counterfactual multi-agent policy gradients.  _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_ 2974–2982.
  * François-Lavet et al. (2018) François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M. G.; Pineau, J.; et al. 2018.  An introduction to deep reinforcement learning.  _Foundations and Trends® in Machine Learning_ 11(3-4): 219–354.
  * Ha, Dai, and Le (2016) Ha, D.; Dai, A.; and Le, Q. V. 2016.  HyperNetworks.  _5th International Conference on Learning Representations, Toulon_ .
  * Hasselt, Guez, and Silver (2016) Hasselt, H. v.; Guez, A.; and Silver, D. 2016.  Deep reinforcement learning with double Q-Learning.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , 2094–2100.
  * Hausknecht and Stone (2015) Hausknecht, M.; and Stone, P. 2015.  Deep recurrent q-learning for partially observable MDPs.  _AAAI Fall Symposium Series_ .
  * Hessel et al. (2017) Hessel, M.; Modayil, J.; Van Hasselt, H.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M.; and Silver, D. 2017.  Rainbow: Combining improvements in deep reinforcement learning.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long short-term memory.  _Neural computation_ 9(8): 1735–1780.
  * LeCun, Bengio, and Hinton (2015) LeCun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep learning.  _Nature_ 521(7553): 436–444.
  * Littman (1994) Littman, M. L. 1994.  Markov games as a framework for multi-agent reinforcement learning.  In _Proceedings of the Eleventh International Conference on Machine Learning_ , 157–163.
  * Mahajan et al. (2019) Mahajan, A.; Rashid, T.; Samvelyan, M.; and Whiteson, S. 2019.  Maven: Multi-agent variational exploration.  In _Advances in Neural Information Processing Systems_ , 7613–7624.
  * Mnih et al. (2015) Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015.  Human-level control through deep reinforcement learning.  _Nature_ 518(7540): 529–533.
  * Oliehoek, Amato et al. (2016) Oliehoek, F. A.; Amato, C.; et al. 2016.  _A concise introduction to decentralized POMDPs_ , volume 1.  Springer.
  * Rashid et al. (2018) Rashid, T.; Samvelyan, M.; de Witt, C. S.; Farquhar, G.; Foerster, J.; and Whiteson, S. 2018.  QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning.  _Proceedings of the 35th International Conference on Machine Learning, Stockholm_ .
  * Sabatelli et al. (2020) Sabatelli, M.; Louppe, G.; Geurts, P.; and Wiering, M. 2020.  The deep quality-value family of deep reinforcement learning algorithms.  In _International Joint Conference on Neural Networks (IJCNN)_.
  * Sabatelli et al. (2018) Sabatelli, M.; Louppe, G.; Geurts, P.; and Wiering, M. A. 2018.  Deep quality-value (DQV) learning.  _Advances in Neural Information Processing Systems (NeurIPS), Deep Reinforcement Learning Workshop. Montreal_ .
  * Samvelyan et al. (2019) Samvelyan, M.; Rashid, T.; Schroeder de Witt, C.; Farquhar, G.; Nardelli, N.; Rudner, T. G.; Hung, C.-M.; Torr, P. H.; Foerster, J.; and Whiteson, S. 2019.  The starCraft multi-agent challenge.  In _Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems_ , 2186–2188.
  * Schmidhuber (2015) Schmidhuber, J. 2015.  Deep learning in neural networks: An overview.  _Neural networks_ 61: 85–117.
  * Son et al. (2019) Son, K.; Kim, D.; Kang, W. J.; Hostallero, D. E.; and Yi, Y. 2019.  QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.  _Proceedings of the 36th International Conference on Machine Learning, Long Beach, California_ .
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Sutton et al. (1999) Sutton, R. S.; McAllester, D.; Singh, S.; and Mansour, Y. 1999.  Policy gradient methods for reinforcement learning with function approximation.  _Advances in neural information processing systems_ 12: 1057–1063.
  * Tan (1993) Tan, M. 1993.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  In _Proceedings of the tenth international conference on machine learning_ , 330–337.
  * Wang et al. (2020) Wang, J.; Ren, Z.; Liu, T.; Yu, Y.; and Zhang, C. 2020.  Qplex: Duplex dueling multi-agent q-learning.  _arXiv preprint arXiv:2008.01062_ .
  * Watkins and Dayan (1992) Watkins, C. J.; and Dayan, P. 1992.  Q-learning.  _Machine learning_ 8(3-4): 279–292.
  * Wiering (2005) Wiering, M. A. 2005.  QV (lambda)-learning: A new on-policy reinforcement learning algrithm.  In _Proceedings of the 7th European Workshop on Reinforcement Learning_ , 17–18.
  * Wiering and Van Hasselt (2009) Wiering, M. A.; and Van Hasselt, H. 2009.  The QV family compared to other reinforcement learning algorithms.  In _2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning_ , 101–108. IEEE.
  * Yang et al. (2020) Yang, Y.; Hao, J.; Liao, B.; Shao, K.; Chen, G.; Liu, W.; and Tang, H. 2020.  Qatten: A general framework for cooperative multiagent reinforcement learning.  _arXiv preprint arXiv:2002.03939_ .
  * Zhao et al. (2019) Zhao, Y.; Borovikov, I.; Rupert, J.; Somers, C.; and Beirami, A. 2019.  On multi-agent learning in team sports games.  _Proceedings of the 36th International Conference on Machine Learning, Long Beach, California_ .