Present Address : ]Department of Physics and Astronomy, University of Calgary,
Calgary T2N1N4, Alberta, Canada.

# Evolution of two-mode quantum states under a dissipative environment: which
quantum resource survives better, squeezing or entanglement?

Rishabh<EMAIL_ADDRESS>[  Department of Physical Sciences, Indian
Institute of Science Education and Research (IISER), Mohali, Sector 81 SAS
Nagar, Manauli PO 140306, Punjab, India  Chandan Kumar
<EMAIL_ADDRESS>Department of Physical Sciences, Indian Institute of
Science Education and Research (IISER), Mohali, Sector 81 SAS Nagar, Manauli
PO 140306, Punjab, India  Geetu Narang<EMAIL_ADDRESS>Department of Applied
Sciences, U.I.E.T, Panjab University, Chandigarh 160040, India  Arvind
<EMAIL_ADDRESS>Department of Physical Sciences, Indian Institute of
Science Education and Research (IISER), Mohali, Sector 81 SAS Nagar, Manauli
PO 140306, Punjab, India  Vice Chancellor, Punjabi University Patiala, Punjab
147002, India

###### Abstract

We explore the relative robustness of squeezing and entanglement (which are
quantum resources interconvertible via passive optics) for two-mode Gaussian
states under different dissipative environments. When the individual modes
interact with identical local baths, entanglement and squeezing decay at the
same rate. However, when only one of the modes interacts with a local bath,
the comparative robustness of entanglement and squeezing depends on the
initial squeezing of the state. Similarly, when the system interacts with a
global bath, the robustness of entanglement and squeezing depends on the
initial squeezing. Thus depending on the nature of dissipative environments
and the initial squeezing of the state, one can select the more robust form of
resource out of squeezing and entanglement to store quantumness. This can be
used to effectively enhance the performance of various quantum information
processing protocols based on continuous variable Gaussian states.

##  I Introduction

Continuous variable (CV) quantum information processing (QIP) based on quantum
optical sources has been gaining attention over time Weedbrook  _et al._
(2012); Adesso  _et al._ (2014). Developments in continuous variable quantum
key distribution(CV-QKD) protocols Pirandola  _et al._ (2020) and photonic
computing Bourassa  _et al._ (2021) are particularly noteworthy. Nonclassical
Gaussian states play a key role in this context, as they can be easily
produced, manipulated, and measured in the laboratory Vahlbruch  _et al._
(2008, 2016). Nonclassicality or genuine quantumness in quantum states is a
resource that needs to be defined, identified, and preserved for use in QIP
protocols Simon (2000); Duan  _et al._ (2000); Paz and Roncaglia (2008); Yadin
_et al._ (2018).

In the quantum optical sense, if the Glauber-Sudarshan $P$ function Glauber
(1963); Sudarshan (1963) behaves like a regular probability distribution, the
corresponding state can be simulated by ensembles of solutions of Maxwell
equations and the state cannot exhibit any nonclassical features Mandel and
Wolf (1995). On the other hand, nonpositivity of the Glauber-Sudarshan $P$
function indicates nonclassicality Miranowicz  _et al._ (2010); Bartkowiak
_et al._ (2011). Squeezing, when quadrature noise drops below the shot noise
limit, is one specific form of quantumness based on P representation Gerry and
Knight (2004). Gaussian squeezed states are thus an important class of
nonclassical states which are extremely useful for QIP. A different notion of
quantumness arises from information theory viewpoint, where correlations in
composite quantum systems can go beyond classically allowed values leading to
nonclassical situations. Quantum entanglement is one such resource that can
lead to situations that violate local realism Horodecki  _et al._ (2009).
While single mode squeezing and intermode entanglement are very different
notions of nonclassicality, they can be inter-converted into each other via
passive optical elements such as beam splitters, phase shifters, wave-plates,
and mirrors Scheel and Welsch (2001); Xiang-bin (2002); Wolf  _et al._ (2003);
Bowen  _et al._ (2003).

Environmental interactions can cause disturbances which invariably lead to the
diminishing of quantum resources which may be present in the form of squeezing
and entanglement. The environmental interactions are detrimental to the
performance of various QIP tasks. Therefore, it is of significant importance
to analyze the evolution of quantum systems under different dissipative
environments and find ways to protect resources against environmental effects.
A lot of work has already been done in this regard Zhao and Chen (2002);
Prauzner-Bechcicki (2004); Serafini  _et al._ (2005); Paz and Roncaglia
(2008); Xiang  _et al._ (2008, 2009); Goyal and Ghosh (2010); Xiang  _et al._
(2011). Entanglement dynamics has been studied for local as well as global
dissipative environments under both Markovian and non-Markovian assumptions.
Many interesting phenomena have been observed, for instance, sudden death of
entanglement in local as well as global dissipative environments Paz and
Roncaglia (2008); Goyal and Ghosh (2010). Further, researchers have also shown
that entanglement can be produced in two-mode separable squeezed states and
can even be enhanced in two-mode squeezed vacuum (TMSV) states evolving under
the presence of a global thermal bath Benatti and Floreanini (2006); Xiang
_et al._ (2008). The effect of decoherence on nonlocality, discord, and
steering has also been studied He  _et al._ (2011); Freitas and Paz (2012);
Daffer  _et al._ (2003); Wang  _et al._ (2016). Moreover, much research has
been conducted to study decoherence in non-Gaussian states, particularly
photon subtracted states Biswas and Agarwal (2007); Hu  _et al._ (2010).

In this article, we strive to find which of the two resources, squeezing or
entanglement, is more robust to environmental noise Arvind and Narang (2006);
Yadsan-Appleby and Serafini (2011); Garai and Ivan (2019). We consider two
different cases that provide an insight into the relative sensitivity of
squeezing and entanglement resources to dissipative environments under the
Markovian assumption. The first case considers squeezing of the individual
modes followed by an evolution under a dissipative environment, and finally,
the two modes are entangled using passive optics (beam splitter). The second
case considers squeezing of the individual modes, which then are entangled
using passive optics, and finally, we let them evolve in a dissipative
environment. We have considered our system to be interacting with local and
global thermal baths, which are Gaussian channels, i.e., Gaussian states
remain Gaussian under such interactions. We provide the time-dependent
covariance matrix, which is used for entanglement analysis. Although a full
characterization of entanglement for CV systems is not possible, for Gaussian
states, necessary and sufficient criteria for the detection of entanglement
exist Simon (2000); Duan  _et al._ (2000). Further, we can quantify
entanglement using logarithmic negativity in two-mode Gaussian states which we
use in our work Życzkowski  _et al._ (1998); Vidal and Werner (2002); Plenio
(2005).

It is natural to expect that entanglement in the TMSV state, where intermodal
correlations are present, will be more fragile compared to squeezing in a two-
mode separable squeezed state. However, our analysis reveals that the relative
robustness of entanglement and squeezing depends on the dissipative
environment that the system is interacting with. When the two modes interact
with identical baths, the squeezing and entanglement decay in exactly the same
way. On the other hand, when only one of the modes interacts with a local
bath, the results depend on the initial squeezing of the state. There exists a
threshold of the initial squeezing of the state, below which entanglement is
more robust than squeezing. Otherwise, squeezing is more robust than
entanglement. Similarly, such a threshold also exists when the system
interacts with a global bath, below which squeezing is more robust than
entanglement. Otherwise, entanglement is more robust than squeezing. We
provide analytical expressions for these thresholds of the squeezing
parameter, which will enable experimentalists to identify more robust
resources against a dissipative environment.

The paper is organized as follows. In Sec. II, we review the formalism for the
CV systems and describe the notions squeezing and entanglement for two-mode
Gaussian states. In Sec. III.1, we set out to study the relative robustness of
squeezing and entanglement against a disturbance caused by a noisy dissipative
environment. We consider the system evolution under local thermal baths in
Sec. III.2, while Sec. III.3 deals with the evolution of the system under a
global bath. Finally, in Sec. IV, we provide some concluding remarks and
future directions.

##  II Two mode systems, entanglement and squeezing

We describe the formalism of two-mode CV systems and discuss the two
nonclassical notions namely squeezing and entanglement which we intend to
study under different noisy dissipative environments.

###  II.1 Two mode CV systems and symplectic transformations

We consider a two-mode CV quantum system described by the quadrature operators
$\hat{q}_{1}$, $\hat{p}_{1}$, $\hat{q}_{2}$, and $\hat{p}_{2}$ Arvind  _et
al._ (1995); Braunstein and van Loock (2005); Weedbrook  _et al._ (2012);
Adesso  _et al._ (2014); Arvind  _et al._ (1995). To handle the analysis of
the two-mode system compactly, we introduce the column vector

$\bm{\hat{\xi}}=(\hat{\xi}_{j})=(\hat{q}_{1},\,\hat{p}_{1},\,\hat{q}_{2},\,\hat{p}_{2})^{T}.$
(1)

The canonical commutation relations can be written as

$\begin{array}[]{c}[\hat{\xi}_{j},\hat{\xi}_{k}]=i(\omega^{\oplus 2})_{jk}\\\
{\scriptstyle j,k=1,2,3,4}\\\ {\scriptstyle\hbar=1}\end{array}\quad{\rm
with}\quad\omega=\begin{pmatrix}0&1\\\ -1&0\end{pmatrix}.$ (2)

The annihilation and creation operators
$\hat{a}_{j}\,\text{and}\,{\hat{a}_{j}}^{\dagger}$ ($j=1,2$) can be expressed
in terms of quadrature operators as follows:

$\hat{a}_{j}=\frac{1}{\sqrt{2}}(\hat{q}_{j}+i\hat{p}_{j}),\quad\hat{a}^{\dagger}_{j}=\frac{1}{\sqrt{2}}(\hat{q}_{j}-i\hat{p}_{j}).$
(3)

The Hilbert space of the two-mode system has an orthogonal basis in the Fock
representation $|n_{1},n_{2}\rangle$ with
$\\{n_{1},\,n_{2}=0,\,1,\dots,\infty\\}$, which are simultaneous eigenvectors
of the number operators $\hat{a}_{1}^{\dagger}\hat{a}_{1}$ and
$\hat{a}_{2}^{\dagger}\hat{a}_{2}$.

The linear homogeneous transformations $S$ specified by real $4\times 4$
matrices acting on the quadrature operators (1) and preserving the canonical
commutation relation (2) form the symplectic group $Sp(4,\,\mathcal{R})$. The
quadrature operators transform as
$\hat{\xi}_{i}\rightarrow\hat{\xi}_{i}^{\prime}=S_{ij}\hat{\xi}_{j}$. Further,
the symplectic condition for matrix $S$ is

$S\omega^{\oplus 2}S^{T}=\omega^{\oplus 2}\implies S\in Sp(4,\,\mathcal{R}).$
(4)

The symplectic group can be decomposed as $S=PK(X,Y)$, where $P$ is the
noncompact part and $K(X,Y)$ is the maximally compact subgroup of
$Sp(4,\mathcal{R})$. The elements of the set $P$ act on the states through
their infinite-dimensional unitary representation (metaplectic
representation), change the total number of photons and are called active
operations. Active operations can transform a classical state into a
nonclassical state and vice versa. On the other hand, the elements of $K(X,Y)$
which form the maximally compact subgroup while acting on the states via their
metaplectic representation conserve the total number of photons and are called
passive operations. Passive operations cannot create or destroy the
nonclassicality of a state and can be implemented using passive optical
elements like beam splitters, phase shifters, waveplates, and mirrors. We
discuss two basic symplectic operations which are important for our work
Arvind  _et al._ (1995); Braunstein and van Loock (2005); Weedbrook  _et al._
(2012); Adesso  _et al._ (2014).

Single mode squeezing operation : The transformation matrix corresponding to
the single-mode squeezing operator, which acts on the quadrature operators
$\hat{q}_{i}$ and $\hat{p}_{i}$, is given by

$S_{i}(r)=\begin{pmatrix}e^{-r}&0\\\ 0&e^{r}\end{pmatrix}.$ (5)

The corresponding $Sp(4,\mathcal{R})$ element will be
$S_{1}(r)\oplus\mathbb{1}_{2}$ or $\mathbb{1}_{2}\oplus S_{2}(r)$ depending
upon which of the two modes the squeezing operator acts upon. Here
$\mathbb{1}_{2}$ represents a $2\times 2$ identity matrix. Single-mode
squeezing operators are active operations which can transform a classical
state into a nonclassical state.

Beam splitter operation : The transformation matrix corresponding to a beam
splitter acting on the two-mode quadrature operators
$\hat{\xi}=(\hat{q}_{1},\,\hat{p}_{1},\,\hat{q}_{2},\,\hat{p}_{2})^{T}$ is
given by

$B_{12}(\theta)=\begin{pmatrix}\cos\theta\,\mathbb{1}_{2}&\sin\theta\,\mathbb{1}_{2}\\\
-\sin\theta\,\mathbb{1}_{2}&\cos\theta\,\mathbb{1}_{2}\end{pmatrix},$ (6)

where $\theta$ is related to the transmittance of the beam splitter according
to $\tau=\cos^{2}\theta$. Angle $\theta=\pi/4$ corresponds to a balanced
(50:50) beam splitter. Beam splitter transformation is a passive operation
belonging to the $K(X,Y)$ subgroup of $Sp(4,\mathcal{R})$ which cannot
transform the classical or nonclassical status of a state.

###  II.2 Squeezing and entanglement in Gaussian states

Gaussian states can be described by Gaussian-Wigner functions in phase space.
They can be completely specified by the mean values and covariances of the
quadrature operators. However, without any loss of generality, we can consider
a zero-centered state as any general Gaussian state can be made zero-centered
by application of the displacement operator without affecting the entanglement
content of the state. The Wigner function for zero-centered Gaussian states
can be written as Weedbrook  _et al._ (2012)

$W(\bm{\xi})=\frac{\exp[-(1/2)\bm{\xi}^{T}V^{-1}\bm{\xi}]}{(2\pi)^{2}\sqrt{\text{det}V}},$
(7)

where $V$ is the covariance matrix whose elements are given by

$V=(V_{ij})=\frac{1}{2}\langle\\{\Delta\hat{\xi}_{i},\Delta\hat{\xi}_{j}\\}\rangle,$
(8)

where $\Delta\hat{\xi}_{i}=\hat{\xi}_{i}-\langle\hat{\xi}_{i}\rangle$, and
$\\{\,,\,\\}$ denotes the anti-commutator. The uncertainty principle can be
expressed in terms of the covariance matrix as

$V+\frac{i}{2}\Omega\geq 0.$ (9)

The covariance matrix for the zero-centered two-mode Gaussian state is given
by

$V=\left(\begin{array}[]{cccc}\langle
q_{1}^{2}\rangle&\frac{1}{2}\langle\\{q_{1},p_{1}\\}\rangle&\langle
q_{1}q_{2}\rangle&\langle q_{1}p_{2}\rangle\\\
\frac{1}{2}\langle\\{q_{1},p_{1}\\}\rangle&\langle p_{1}^{2}\rangle&\langle
q_{2}p_{1}\rangle&\langle p_{1}p_{2}\rangle\\\ \langle
q_{1}q_{2}\rangle&\langle q_{2}p_{1}\rangle&\langle
q_{2}^{2}\rangle&\frac{1}{2}\langle\\{q_{2},p_{2}\\}\rangle\\\ \langle
q_{1}p_{2}\rangle&\langle
p_{1}p_{2}\rangle&\frac{1}{2}\langle\\{q_{2},p_{2}\\}\rangle&\langle
p_{2}^{2}\rangle\end{array}\right).$ (10)

A quantum state is said to be squeezed in quadrature $\hat{\xi}_{i}$ if the
fluctuations in the corresponding quadrature reduce below the coherent state
value, i.e., $(\Delta\hat{\xi}_{i})^{2}<1/2$. The single mode squeezing
operator $S(r)$ (5) acting on a state through its metaplectic representation
alters fluctuations in the quadratures, and hence, can transform a non-
squeezed state into a squeezed state. Specifically, as an example, if $S(r)$
acts on the first mode, the fluctuations in the $\hat{\xi}_{1}$ quadrature
transform as $(\Delta\hat{\xi}_{1})^{2}\rightarrow
e^{-2r}(\Delta\hat{\xi}_{1})^{2}$. Since one is allowed to redefine
quadratures by mixing them via passive operations, a state is squeezed even if
the noise in one of the transformed quadratures falls below the coherent state
value. We will consider squeezing caused by single mode squeezing
transformations $S(r)$ as described above.

While the detection of entanglement in general states of CV systems remains an
open problem, for Gaussian states, the Simon criterion provides necessary and
sufficient conditions for the detection of entanglement Duan  _et al._ (2000);
Simon (2000). By Simon’s criterion,

$\displaystyle\text{det}\,A\,\text{det}\,B+\Big{(}\frac{1}{4}-|\text{det}\,C|\Big{)}^{2}-\text{Tr}[A\omega
C\omega B\omega C^{T}\omega]$ (11)
$\displaystyle-\frac{1}{4}(\text{det}\,A+\text{det}\,B)\geq 0,$

is necessary and sufficient for a state to be separable. Here $A$, $B$, and
$C$ are $2\times 2$ matrices and are related to the covariance matrix of a
two-mode system as

$V=\left(\begin{array}[]{cccc}A&C\\\ C^{T}&B\end{array}\right).$ (12)

Furthermore for Gaussian states, the logarithmic negativity can be used as a
measure of entanglement between the two modes. The logarithmic negativity for
a two-mode Gaussian state is defined as

$E_{N}=\text{max}\\{0,-\text{log}_{2}(2n_{-})\\},$ (13)

where $n_{-}$ is the smallest symplectic eigenvalue of the partially
transposed covariance matrix, which can be compactly written using the block
matrices form of the covariance matrix (12) as follows:

$\displaystyle n_{-}^{2}=$
$\displaystyle\frac{1}{2}\bigg{[}\Sigma-\sqrt{\Sigma^{2}-4\,\text{det}\,{V}}\bigg{]},$
(14)

where $\Sigma=\text{det}\,A+\text{det}\,B-2\,\text{det}\,C$. We employ this
measure to quantify entanglement throughout this paper.

##  III Effect of environment on quantum resources of squeezing and
entanglement

In this section, we set out to explore the effect of environmental coupling on
diminishing the quantum resources of squeezing and entanglement. We couple the
two-mode system with thermal baths in several different ways to study the
relative performance of squeezing and entanglement in terms of their ability
to withstand the effects of environmental effects.

###  III.1 Environment coupled to the two-mode optical system

We consider a two-mode system interacting with a thermal bath. The bath is
considered to be comprised of a large set of harmonic oscillators. The
corresponding total Hamiltonian can be written as

$\hat{H}=\hat{H}_{S}+\hat{H}_{B}+\hat{H}_{SB},$ (15)

where $\hat{H}_{S}$ and $\hat{H}_{B}$ are the system and the bath
Hamiltonians, respectively, and $\hat{H}_{SB}$ is the interaction Hamiltonian.
We consider two distinct cases:

Case 1: In this case a two-mode separable squeezed state interacting with a
thermal bath is considered. We start with a two-mode system initialized to the
vacuum state $|\textbf{0}\rangle$, which can be represented by the following
covariance matrix:

$V_{|\textbf{0}\rangle}=\frac{1}{2}\left(\begin{array}[]{cccc}1&0&0&0\\\
0&1&0&0\\\ 0&0&1&0\\\ 0&0&0&1\end{array}\right).$ (16)

To generate a two-mode separable squeezed state, the two-mode vacuum state is
squeezed by equal and opposite amounts by the squeezing operation
$S_{1}(r)\oplus S_{2}(-r)$. Thereafter, the covariance matrix of the two-mode
separable squeezed state can be written as

$V_{1}(t=0)=\frac{1}{2}\left(\begin{array}[]{cccc}e^{-2r}&0&0&0\\\
0&e^{2r}&0&0\\\ 0&0&e^{2r}&0\\\ 0&0&0&e^{-2r}\end{array}\right).$ (17)

The system is then allowed to interact with a thermal bath. We set the time to
$t=0$, when the interaction with the bath is switched on. Thus, the covariance
matrix of the state at time $t=0$ is given by Eq. (17). The interaction with
the bath thermal bath is then switched on which can lead to decay of
squeezing. After a time $t=\tau$ , the interaction is switched off, and the
modes are mixed using a 50:50 beam splitter with a view to convert the
remaining nonclassicality into entanglement.

Case 2: In this case, we consider a two-mode squeezed vacuum state (TMSV), an
entangled state, interacting with a thermal bath. The two-mode entangled state
is generated by first squeezing the two-mode system in the vacuum state by an
equal and opposite amount by the squeezing operation $S_{1}(r)\oplus
S_{2}(-r)$. The modes are then mixed via a 50:50 beam splitter in order to
convert the squeezing into entanglement, and subsequently the thermal bath is
switched on. The covariance matrix at time $t=0$ for this case is given by

$V_{2}(t=0)=\frac{1}{2}\begin{pmatrix}\cosh(2r)\,\mathbb{1}_{2}&\sinh(2r)\,\mathbb{Z}\\\
\sinh(2r)\,\mathbb{Z}&\cosh(2r)\,\mathbb{1}_{2}\end{pmatrix},$ (18)

where $\mathbb{Z}$ is $\text{diag}(1,-1)$. It should be noted here that for
the two-mode separable squeezed state, the quantum resource is in the form of
squeezing, while for the TMSV state, squeezing has been converted into
entanglement Tahira  _et al._ (2009); Ge  _et al._ (2015). Thus, for a two-
mode mode separable squeezed state, squeezing decays due to environmental
interactions. On the other hand, for TMSV state, both squeezing and
entanglement decay due to environmental interactions. The two cases differ
from each other only in the fact that quantumness has been completely stored
in the form of squeezing in the former case, while in the latter case,
quantumness has been converted into the form of entanglement. The settings
above have been constructed to address our main question: which form of
quantumness, squeezing or entanglement, is more resilient to a dissipative
environment? To this end, we consider the evolution of the two aforementioned
cases in the following two dissipative environments:

  * 1.

The two modes of the system interact with two local thermal baths.

  * 2.

The two modes of the system interact with a global thermal bath

These studies will enable us to answer whether we should store the quantum
resource as squeezing or as entanglement in a given situation.

###  III.2 Evolution under local thermal baths

In this subsection, we consider that our two-mode system interacts with two
local thermal baths. The interaction Hamiltonian is given by

$\hat{H}_{SB}=g_{1}\sum^{\infty}_{k=1}\Big{(}\hat{a}_{1}\hat{b}^{\dagger}_{k}+\hat{a}^{\dagger}_{1}\hat{b}_{k}\Big{)}+g_{2}\sum^{\infty}_{l=1}\Big{(}\hat{a}_{2}\hat{c}^{\dagger}_{l}+\hat{a}^{\dagger}_{2}\hat{c}_{l}\Big{)},$
(19)

where $g_{1}$ and $g_{2}$ are the coupling constants, and $\hat{b}_{k}$ and
$\hat{b}^{\dagger}_{k}$ are annihilation and creation operators of the
$k^{th}$ mode of the reservoir interacting with the first mode of the system.
Similarly, $\hat{c}_{l}$ and $\hat{c}^{\dagger}_{l}$ are annihilation and
creation operators of the $l^{th}$ mode of the reservoir interacting with the
second mode of the system.

Under Markovian assumption, we can write the master equation for the evolution
of system density operator $\rho$ as

$\displaystyle\frac{\partial}{\partial t}\rho=$
$\displaystyle\bigg{\\{}\sum_{i=1,2}\frac{\gamma_{i}}{2}(N_{i}+1)(2\hat{a}_{i}\rho\hat{a}_{i}^{\dagger}-\hat{a}_{i}^{\dagger}\hat{a}_{i}\rho-\rho\hat{a}_{i}^{\dagger}\hat{a}_{i})$
(20)
$\displaystyle\quad+\frac{\gamma_{i}}{2}N_{i}(2\hat{a}_{i}^{\dagger}\rho\hat{a}_{i}-\hat{a}_{i}\hat{a}_{i}^{\dagger}\rho-\rho\hat{a}_{i}\hat{a}_{i}^{\dagger})\bigg{\\}},$

where $\gamma_{i}$’s are the decay constants, and $N_{i}$’s represent the mean
photon number of the individual baths. Equation (20) can be used to find the
time evolution of variances of quadrature operators, and hence, the evolution
of the covariance matrix.

Figure 1:  Schematic representation of the dissipation under local thermal
baths. (a) Two-mode separable squeezed state is let to evolve under local
thermal baths and after a time $t$, the two modes are mixed using a $50:50$
beam splitter. (b) Two-mode separable squeezed state is first mixed using beam
splitter and then it is let to evolve under local thermal baths.  Figure 2:
Logarithmic negativity $E_{N}$ of two-mode Gaussian state as a function of
dimensionless time $\tau(=1-e^{-2\gamma t})$. Only one of the two modes is let
to evolve under the presence of a local bath. The mean number of photons in
the bath is taken to be $N=4$. (a) For values of the initial squeezing
parameter such that $|r|<r_{t}^{L}<r_{c}^{L}$ [(25) and (28)], entanglement is
more robust than squeezing. (b) For $r_{t}^{L}<|r|<r_{c}^{L}$, squeezing is
more robust than entanglement. (c) For $|r|>r_{c}^{L}>r_{t}^{L}$, squeezing is
more robust than entanglement and the two-mode separable case always remains
entangled.

Case 1: We consider the case where each mode of the two-mode separable
squeezed state interacts with distinct local thermal baths. The schematic
diagram is shown in Fig. 1(a). The covariance matrix of the initial state at
time $t=0$ is given by $V_{1}(0)$ (17). Using the master equation (20), we
obtain the covariance matrix after an interaction with the bath for time $t$
as Olivares (2012)

$V_{1}(t)=X(t)V_{1}(0)X(t)^{T}+\frac{1}{2}Y(t),$ (21)

where $X(t)$ and $Y(t)$ are $4\times 4$ diagonal matrices given by

$\displaystyle X(t)$
$\displaystyle=\begin{pmatrix}(1-\tau_{1})^{1/4}\,\mathbb{1}_{2}&0\\\
0&(1-\tau_{2})^{1/4}\,\mathbb{1}_{2}\end{pmatrix},$ (22) $\displaystyle Y(t)$
$\displaystyle=\begin{pmatrix}(\frac{N_{1}}{2}+1)\tau_{1}\,\mathbb{1}_{2}&0\\\
0&(\frac{N_{2}}{2}+1)\tau_{2}\,\mathbb{1}_{2}\end{pmatrix},$

where $\tau_{1}=1-e^{-2\gamma_{1}t}$ and $\tau_{2}=1-e^{-2\gamma_{2}t}$ are
dimensionless time parameters. We note that while $t$ goes from $0$ to
$\infty$, $\tau_{i}$ goes from $0$ to $1$. The final covariance matrix after
passing both the modes through a 50:50 beam splitter (6) is given by

$V^{L}_{1}(t)=B_{12}\left(\frac{\pi}{4}\right)\Big{[}X(t)V_{1}(0)X(t)^{T}+\frac{1}{2}Y(t)\Big{]}B_{12}\left(\frac{-\pi}{4}\right),$
(23)

where the superscript L over $V_{1}(t)$ stands for the local bath, and $t$
represents the time duration of the system-bath interaction.

Case 2:  We consider the case where each mode of the TMSV state interacts with
a distinct local thermal bath. The schematic is depicted in Fig. 1(b). The
initial covariance matrix of the TMSV state is given by Eq. (18). Using the
master equation (20), the covariance matrix at time $t$ is evaluated as

$V^{L}_{2}(t)=X(t)V_{2}(0)X(t)^{T}+\frac{1}{2}Y(t).$ (24)

From Eqs. (23) and (24), various conclusion can be drawn. We consider the
following two special cases of symmetric and asymmetric interaction of the
local baths with the system:

Symmetric interaction: Consider the case where $g_{1}=g_{2}=g$ and both baths
are at same temperature so that $\gamma_{1}=\gamma_{2}=\gamma$ and
$N_{1}=N_{2}=N$. Thus the two local baths are identical. This leads to the
same final covariance matrix for two-mode separable squeezed state (23) and
TMSV state (24). Therefore both the resources, squeezing and entanglement, are
equally sensitive to decoherence when two identical local thermal baths act on
each mode of the system.

Extreme asymmetric interaction: Consider $g_{1}=g$ and $g_{2}=0$, and
therefore $\gamma_{1}=\gamma,\,\gamma_{2}=0$, and $N_{1}=N,\,N_{2}=0$. Thus,
only the first mode of the system interacts with the thermal bath. The
covariance matrix for two-mode separable squeezed state (23) and TMSV state
(24) are not the same in this situation, and hence we expect different rates
of decay for logarithmic negativity of the final state.

Using Eq. (11), the condition on the initial squeezing parameter $r$, such
that, the two-mode separable squeezed state never becomes disentangled is
given by

$|r|>r_{c}^{L}=\frac{1}{2}\left[\text{ln}\left(1+\frac{N}{2}\right)\right].$
(25)

However, for values of $|r|\leq r_{c}^{L}$, entanglement dies out for
interaction times longer than

$\tau_{a}=\frac{8\,e^{4|r|}[2+N-2\,\cosh(2|r|)]\,\sinh(2|r|)}{[(2+N)e^{2|r|}-2]^{2}}.$
(26)

On the other hand, the time of disentanglement for TMSV state evaluates to

$\tau_{b}=\frac{8(2+N)}{(4+N)^{2}},$ (27)

which is independent of the initial squeezing parameter $r$. Further,
$\lim\limits_{N\to 0}\tau_{b}$=1, which means the entanglement of TMSV state
survives indefinitely for a zero temperature bath. In general, Eqs. (26) and
(27) imply the existence of finite disentanglement time under the
aforementioned conditions for two-mode separable as well as TMSV states. This
corresponds to the phenomenon of entanglement sudden death.

Further, if the initial squeezing parameter $r$ is such that, $|r|$ is less
than a certain value $r_{t}^{L}$, then it is a better strategy to store the
resource in the form entanglement, otherwise it is better to store resources
in form of squeezing. The expression for $r_{t}^{L}$ can be evaluated by
equating $\tau_{a}$ and $\tau_{b}$ and is given by

$r_{t}^{L}=\frac{1}{2}\Big{[}\text{ln}\Big{(}\frac{2+N+\sqrt{2(2+4N+N^{2})}}{4+N}\Big{)}\Big{]}.$
(28)

We plot the logarithmic negativity for different values of initial squeezing
parameter in Fig. 2. For $N=4$, the numerical values of $r_{c}^{L}$ and
$r_{t}^{L}$ turn out to be $r_{c}^{L}=0.55$ and $r_{t}^{L}=0.29$. Therefore,
for $r=0.20<r_{t}^{L}<r_{c}^{L}$, we observe that entanglement is better
resource than squeezing. For $r_{t}^{L}<r=0.40<r_{c}^{L}$, we see that
squeezing is better resource than entanglement. Finally, for
$r=0.60>r_{c}^{L}>r_{t}^{L}$, squeezing is better resource than entanglement
and the two-mode separable case always stays entangled.

###  III.3 Evolution under global Bath

In this section, we consider the scenario where the two modes are coupled to a
common thermal reservoir, which we refer to as the global bath. The
interaction Hamiltonian $\hat{H}_{SB}$ is given by

$\hat{H}_{SB}=g\sum_{i=1,2}\Big{(}\hat{a}_{i}\sum^{\infty}_{k=1}\hat{b}^{\dagger}_{k}+\hat{a}^{\dagger}_{i}\sum^{\infty}_{k=1}\hat{b}_{k}\Big{)},$
(29)

where $\hat{b}_{k}$ and $\hat{b}^{\dagger}_{k}$ are annihilation and creation
operators of the $k^{th}$ mode of the reservoir, and $g$ is the coupling
constant between the system and the environment. Under Markovian assumption,
we can write the master equation for the evolution of system density operator
$\rho$ as

$\displaystyle\frac{\partial}{\partial t}\rho=$
$\displaystyle\frac{\gamma}{2}\bigg{\\{}\sum_{i=1,2}(N+1)(2\hat{a}_{i}\rho\hat{a}_{i}^{\dagger}-\hat{a}_{i}^{\dagger}\hat{a}_{i}\rho-\rho\hat{a}_{i}^{\dagger}\hat{a}_{i})$
(30)
$\displaystyle+N(2\hat{a}_{i}^{\dagger}\rho\hat{a}_{i}-\hat{a}_{i}\hat{a}_{i}^{\dagger}\rho-\rho\hat{a}_{i}\hat{a}_{i}^{\dagger})$
$\displaystyle+\sum_{j\neq
i=1,2}(N+1)(2\hat{a}_{i}\rho\hat{a}_{j}^{\dagger}-\hat{a}_{j}^{\dagger}\hat{a}_{i}\rho-\rho\hat{a}_{j}^{\dagger}\hat{a}_{i})$
$\displaystyle+N(2\hat{a}_{j}^{\dagger}\rho\hat{a}_{i}-\hat{a}_{i}\hat{a}_{j}^{\dagger}\rho-\rho\hat{a}_{i}\hat{a}_{j}^{\dagger})\bigg{\\}},$

where $\gamma$ is the decay constant, and $N$ represents the mean photon
number of the bath. Equation (30) can be used to find the time evolution of
the variances of quadrature operators, and hence the time evolution of the
covariance matrix.

Case 1:  We consider the case where the two-mode separable squeezed state
interacts with a global bath. The schematic diagram is shown in Fig. 3(a). The
initial covariance matrix of the two-mode separable squeezed state is given by
Eq. (17). Using Eq. (30), the covariance matrix after an interaction for time
$t$ with the bath is evaluated to be

$\frac{1}{4}\left(\begin{array}[]{cccc}\sigma_{1}(t)&0&\sigma_{3}(t)&0\\\
0&\sigma_{2}(t)&0&\sigma_{3}(t)\\\ \sigma_{3}(t)&0&\sigma_{2}(t)&0\\\
0&\sigma_{3}(t)&0&\sigma_{1}(t)\end{array}\right),$ (31)

where

$\begin{split}\sigma_{1}(t)&=(2N+1)\tau-\cosh(2r)(\tau-2)-2\,\sinh(2r)\sqrt{1-\tau},\\\
\sigma_{2}(t)&=(2N+1)\tau-\cosh(2r)(\tau-2)+2\,\sinh(2r)\sqrt{1-\tau},\\\
\sigma_{3}(t)&=(2N+1-\cosh(2r))\tau,\\\ \end{split}$ (32)

where $\tau=1-e^{-2\gamma t}$. We now pass both the modes through a 50:50 beam
splitter, and the resultant covariance matrix is given by

$V^{G}_{\text{1}}(t)=\frac{1}{2}\left(\begin{array}[]{cccc}\sigma^{\prime}_{1}(t)&0&\sigma^{\prime}_{3}(t)&0\\\
0&\sigma^{\prime}_{1}(t)&0&-\sigma^{\prime}_{3}(t)\\\
\sigma^{\prime}_{3}(t)&0&\sigma^{\prime}_{2}(t)&0\\\
0&-\sigma^{\prime}_{3}(t)&0&\sigma^{\prime}_{2}(t)\end{array}\right),$ (33)

where

$\begin{split}\sigma^{\prime}_{1}(t)&=(2N+1)\tau-\cosh(2r)(\tau-1),\\\
\sigma^{\prime}_{2}(t)&=\cosh(2r),\\\
\sigma^{\prime}_{3}(t)&=\sinh(2r)\sqrt{1-\tau}.\end{split}$ (34)

Figure 3:  Schematic representation of the dissipation under a global bath.
(a) The two-mode separable squeezed state is let to evolve under the global
thermal bath, and after a time $t$, the two modes are mixed using a $50:50$
beam splitter. (b) The two-mode separable squeezed state is first mixed using
a $50:50$ beam splitter, and then it is let to evolve under the global thermal
bath. Figure 4:  Logarithmic negativity $E_{N}$ as a function of dimensionless
time $\tau(=1-e^{-2\gamma t})$. The system is let to evolve under the presence
of a global thermal bath. The mean number of photons in the bath has been
taken to be $N=4$. (a) For values of the initial squeezing parameter such that
$|r|<r_{t}^{G}<r_{c}^{G}$ [(37) and (40)], squeezing is more robust than
entanglement. (b) For $r_{t}^{G}<|r|<r_{c}^{G}$, entanglement is more robust
than squeezing. (c) For $|r|>r_{c}^{G}>r_{t}^{G}$, entanglement is more robust
than squeezing and the TMSV state always remains entangled.

Case 2:  We consider the case where the TMSV state interacts with a global
bath. The schematic is presented in Fig. 3(b). The initial covariance matrix
of the TMSV state is given by Eq. (18). The covariance matrix after an
interaction for time $t$ with the bath is evaluated to be

$V^{G}_{2}(t)=\frac{1}{2}\left(\begin{array}[]{cccc}\delta_{1}(t)&0&\delta_{3}(t)&0\\\
0&\delta_{2}(t)&0&\delta_{4}(t)\\\ \delta_{3}(t)&0&\delta_{1}(t)&0\\\
0&\delta_{4}(t)&0&\delta_{2}(t)\end{array}\right),$ (35)

where

$\begin{split}\delta_{1}(t)&=\frac{1}{2}(2N+1-e^{2r})\tau+\cosh(2r),\\\
\delta_{2}(t)&=\frac{1}{2}(2N+1-e^{-2r})\tau+\cosh(2r),\\\
\delta_{3}(t)&=\frac{1}{2}(2N+1-e^{2r})\tau+\sinh(2r),\\\
\delta_{4}(t)&=\frac{1}{2}(2N+1-e^{-2r})\tau-\sinh(2r).\end{split}$ (36)

The evolved covariance matrix for the two-mode separable squeezed state (33)
and the TMSV state (35) are not the same in this situation, and hence we
expect different rates of decay for logarithmic negativity. For the TMSV
state, there exists a critical value of initial squeezing parameter
$r_{c}^{G}$, above which the entanglement never becomes zero Prauzner-
Bechcicki (2004):

$|r|>r_{c}^{G}=\frac{1}{2}[\text{ln}(2N+1)].$ (37)

For values of $|r|$ less than $r_{c}^{G}$, entanglement sudden death occurs
after a time

$\tau_{c}=\frac{2\,\sinh(2|r|)}{1-e^{-2|r|}+2N}.$ (38)

It is also observed that for two-mode separable squeezed state, entanglement
always becomes zero at a particular value of time irrespective of the initial
squeezing of the state:

$\tau_{d}=\frac{1}{1+N}.$ (39)

Thus entanglement sudden death always occurs for two-mode separable squeezed
state except for zero temperature bath.

If the initial squeezing parameter $r$ is such that, $|r|$ is less than a
certain value $r_{t}^{G}$, then it is a better strategy to store the quantum
resource in the form squeezing, otherwise it is better to store the
quantumness in the form of entanglement. The expression for $r_{t}^{G}$ can be
evaluated by equating $\tau_{c}$ and $\tau_{d}$ and is given by

$r_{t}^{G}=\frac{1}{2}\Big{[}\text{ln}\Big{(}\frac{1+2N+\sqrt{1+8N+8N^{2}}}{2(1+N)}\Big{)}\Big{]}.$
(40)

We have shown the plots of logarithmic negativity for different values of
initial squeezing parameter in Fig. 4. For $N=4$, the numerical values of
$r_{c}^{G}$ and $r_{t}^{G}$ turn out to be $r_{c}^{G}=1.10$ and
$r_{t}^{G}=0.39$. Therefore, in corroboration with analytical results, we
observe that for $r=0.20<r_{t}^{G}<r_{c}^{G}$, squeezing is better resource
than entanglement. Further, for $r_{t}^{G}<r=0.60<r_{c}^{G}$, entanglement is
better resource than squeezing. Finally, for $r=1.60>r_{c}^{G}>r_{t}^{G}$,
entanglement is better resource than squeezing; however, the TMSV state always
remains entangled. We have summarized the results of this section in Table 1.

Table 1:  Better resource for storage of quantumness: squeezing or entanglement Environment Nature |  $r$ Range | Result
---|---|---
Identical local baths |  | Squeezing $\equiv$ Entanglement
Single local bath | $|r|<r_{t}^{L}$ | Squeezing $<$ Entanglement
| $|r|>r_{t}^{L}$ | Squeezing $>$ Entanglement
Global bath | $|r|<r_{t}^{G}$ | Squeezing $>$ Entanglement
| $|r|>r_{t}^{G}$ | Squeezing $<$ Entanglement

##  IV Conclusion

In this paper, we compared the robustness of squeezing and entanglement
resources in two-mode Gaussian states evolving in different noisy dissipative
environments. To this end, we considered two different cases where in the
first case, two mode separable squeezed states were allowed to evolve in the
dissipative environment, and then were entangled using passive operations. In
the second case, the squeezed modes were first entangled using passive
operations and then were evolved in the dissipative environment. The results
show that the robustness of squeezing and entanglement depends on the initial
squeezing of the state and the nature of the dissipative environment. We also
observe entanglement sudden death in specific cases. The fact that
interconversion of squeezing and entanglement can be made by using passive
optical elements like beam splitters, wave plates, phase shifters and mirrors,
makes it convenient to save the resource in one or other form depending on the
situation. One of the directions that we are pursuing is to generalise this
work by considering more general environment models and non-Gaussian states
which will have implications for recent CV based key distribution Hu  _et al._
(2020) and quantum teleportation Lee and Nha (2013); Hu  _et al._ (2017)
protocols.

## Acknowledgement

A and C.K. acknowledge the financial support from
DST/ICPS/QuST/Theme-1/2019/General Project number Q-68.

## References

  * Weedbrook  _et al._ (2012) C. Weedbrook, S. Pirandola, R. García-Patrón, N. J. Cerf, T. C. Ralph, J. H. Shapiro, and S. Lloyd, Rev. Mod. Phys. 84, 621 (2012).
  * Adesso  _et al._ (2014) G. Adesso, S. Ragy, and A. R. Lee, Open Syst. Inf. Dyn. 21, 1440001, 47 (2014).
  * Pirandola  _et al._ (2020) S. Pirandola, U. L. Andersen, L. Banchi, M. Berta, D. Bunandar, R. Colbeck, D. Englund, T. Gehring, C. Lupo, C. Ottaviani, J. L. Pereira, M. Razavi, J. S. Shaari, M. Tomamichel, V. C. Usenko, G. Vallone, P. Villoresi, and P. Wallden, Adv. Opt. Photon. 12, 1012 (2020).
  * Bourassa  _et al._ (2021) J. E. Bourassa, R. N. Alexander, M. Vasmer, A. Patil, I. Tzitrin, T. Matsuura, D. Su, B. Q. Baragiola, S. Guha, G. Dauphinais, K. K. Sabapathy, N. C. Menicucci, and I. Dhand, Quantum 5, 392 (2021).
  * Vahlbruch  _et al._ (2008) H. Vahlbruch, M. Mehmet, S. Chelkowski, B. Hage, A. Franzen, N. Lastzka, S. Goßler, K. Danzmann, and R. Schnabel, Phys. Rev. Lett. 100, 033602 (2008).
  * Vahlbruch  _et al._ (2016) H. Vahlbruch, M. Mehmet, K. Danzmann, and R. Schnabel, Phys. Rev. Lett. 117, 110801 (2016).
  * Simon (2000) R. Simon, Phys. Rev. Lett. 84, 2726 (2000).
  * Duan  _et al._ (2000) L.-M. Duan, G. Giedke, J. I. Cirac, and P. Zoller, Phys. Rev. Lett. 84, 2722 (2000).
  * Paz and Roncaglia (2008) J. P. Paz and A. J. Roncaglia, Phys. Rev. Lett. 100, 220401 (2008).
  * Yadin  _et al._ (2018) B. Yadin, F. C. Binder, J. Thompson, V. Narasimhachar, M. Gu, and M. S. Kim, Phys. Rev. X 8, 041038 (2018).
  * Glauber (1963) R. J. Glauber, Phys. Rev. 131, 2766 (1963).
  * Sudarshan (1963) E. C. G. Sudarshan, Phys. Rev. Lett. 10, 277 (1963).
  * Mandel and Wolf (1995) L. Mandel and E. Wolf,  _Optical Coherence and Quantum Optics_ (Cambridge University Press, Cambridge, 1995).
  * Miranowicz  _et al._ (2010) A. Miranowicz, M. Bartkowiak, X. Wang, Y.-x. Liu, and F. Nori, Phys. Rev. A 82, 013824 (2010).
  * Bartkowiak  _et al._ (2011) M. Bartkowiak, A. Miranowicz, X. Wang, Y.-x. Liu, W. Leoński, and F. Nori, Phys. Rev. A 83, 053814 (2011).
  * Gerry and Knight (2004) C. Gerry and P. Knight,  _Introductory Quantum Optics_ (Cambridge University Press, Cambridge, 2004).
  * Horodecki  _et al._ (2009) R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Rev. Mod. Phys. 81, 865 (2009).
  * Scheel and Welsch (2001) S. Scheel and D.-G. Welsch, Phys. Rev. A 64, 063811 (2001).
  * Xiang-bin (2002) W. Xiang-bin, Phys. Rev. A 66, 064304 (2002).
  * Wolf  _et al._ (2003) M. M. Wolf, J. Eisert, and M. B. Plenio, Phys. Rev. Lett. 90, 047904 (2003).
  * Bowen  _et al._ (2003) W. P. Bowen, N. Treps, R. Schnabel, T. C. Ralph, and P. K. Lam, Journal of Optics B: Quantum and Semiclassical Optics 5, S467 (2003).
  * Zhao and Chen (2002) Y. Zhao and G. H. Chen, Phys. Rev. E 65, 056120 (2002).
  * Prauzner-Bechcicki (2004) J. S. Prauzner-Bechcicki, Journal of Physics A: Mathematical and General 37, L173 (2004).
  * Serafini  _et al._ (2005) A. Serafini, M. G. A. Paris, F. Illuminati, and S. D. Siena, Journal of Optics B: Quantum and Semiclassical Optics 7, R19 (2005).
  * Xiang  _et al._ (2008) S.-H. Xiang, B. Shao, and K.-H. Song, Phys. Rev. A 78, 052313 (2008).
  * Xiang  _et al._ (2009) S.-H. Xiang, B. Shao, K.-H. Song, and J. Zou, Phys. Rev. A 79, 032333 (2009).
  * Goyal and Ghosh (2010) S. K. Goyal and S. Ghosh, Phys. Rev. A 82, 042337 (2010).
  * Xiang  _et al._ (2011) S.-H. Xiang, K.-H. Song, W. Wen, and Z.-G. Shi, The European Physical Journal D 62, 289 (2011).
  * Benatti and Floreanini (2006) F. Benatti and R. Floreanini, Journal of Physics A: Mathematical and General 39, 2689 (2006).
  * He  _et al._ (2011) Q.-L. He, J.-B. Xu, D.-X. Yao, and Y.-Q. Zhang, Phys. Rev. A 84, 022312 (2011).
  * Freitas and Paz (2012) J. N. Freitas and J. P. Paz, Phys. Rev. A 85, 032118 (2012).
  * Daffer  _et al._ (2003) S. Daffer, K. Wódkiewicz, and J. K. McIver, Phys. Rev. A 68, 012104 (2003).
  * Wang  _et al._ (2016) Z.-X. Wang, S. Wang, Q. Li, T.-J. Wang, and C. Wang, Quantum Information Processing 15, 2441 (2016).
  * Biswas and Agarwal (2007) A. Biswas and G. S. Agarwal, Phys. Rev. A 75, 032104 (2007).
  * Hu  _et al._ (2010) L.-y. Hu, X.-x. Xu, Z.-s. Wang, and X.-f. Xu, Phys. Rev. A 82, 043842 (2010).
  * Arvind and Narang (2006) Arvind and G. Narang, AIP Conference Proceedings 864, 171 (2006).
  * Yadsan-Appleby and Serafini (2011) H. Yadsan-Appleby and A. Serafini, Physics Letters A 375, 1864 (2011).
  * Garai and Ivan (2019) S. Garai and J. S. Ivan, Phys. Rev. A 100, 013842 (2019).
  * Życzkowski  _et al._ (1998) K. Życzkowski, P. Horodecki, A. Sanpera, and M. Lewenstein, Phys. Rev. A 58, 883 (1998).
  * Vidal and Werner (2002) G. Vidal and R. F. Werner, Phys. Rev. A 65, 032314 (2002).
  * Plenio (2005) M. B. Plenio, Phys. Rev. Lett. 95, 090503 (2005).
  * Arvind  _et al._ (1995) Arvind, B. Dutta, N. Mukunda, and R. Simon, Pramana 45, 471 (1995).
  * Braunstein and van Loock (2005) S. L. Braunstein and P. van Loock, Rev. Mod. Phys. 77, 513 (2005).
  * Tahira  _et al._ (2009) R. Tahira, M. Ikram, H. Nha, and M. S. Zubairy, Phys. Rev. A 79, 023816 (2009).
  * Ge  _et al._ (2015) W. Ge, M. E. Tasgin, and M. S. Zubairy, Phys. Rev. A 92, 052328 (2015).
  * Olivares (2012) S. Olivares, The European Physical Journal Special Topics 203, 3 (2012).
  * Hu  _et al._ (2020) L. Hu, M. Al-amri, Z. Liao, and M. S. Zubairy, Phys. Rev. A 102, 012608 (2020).
  * Lee and Nha (2013) J. Lee and H. Nha, Phys. Rev. A 87, 032307 (2013).
  * Hu  _et al._ (2017) L. Hu, Z. Liao, and M. S. Zubairy, Phys. Rev. A 95, 012310 (2017).


contents=  ,color=blue,scale=1,angle=0

# BGC: Multi-Agent Group Belief with Graph Clustering

Tianze Zhou,1 Fubiao Zhang, 1 Pan Tang 1 Chenfei Wang 2

###### Abstract

Recent advances have witnessed that value decomposed-based multi-agent
reinforcement learning methods make an efficient performance in coordination
tasks. Most current methods assume that agents can make communication to
assist decisions, which is impractical in some situations. In this paper, we
propose a semi-communication method to enable agents can exchange information
without communication. Specifically, we introduce a group concept to help
agents learning a belief which is a type of consensus. With this consensus,
adjacent agents tend to accomplish similar sub-tasks to achieve cooperation.
We design a novel agent structure named Belief in Graph Clustering(BGC),
composed of an agent characteristic module, a belief module, and a fusion
module. To represent each agent characteristic, we use an MLP-based
characteristic module to generate agent unique features. Inspired by the
neighborhood cognitive consistency, we propose a group-based module to divide
adjacent agents into a small group and minimize in-group agents’ beliefs to
accomplish similar sub-tasks. Finally, we use a hyper-network to merge these
features and produce agent actions. To overcome the agent consistent problem
brought by GAT, a split loss is introduced to distinguish different agents.
Results reveal that the proposed method achieves a significant improvement in
the SMAC benchmark. Because of the group concept, our approach maintains
excellent performance with an increase in the number of agents.

## Introduction

The multi-agent system is widely applied in many real-life applications,
including sensor networks, aircrafts formation flight, multi-robot cooperative
control (Hüttenrauch, Sosic, and Neumann 2017), and networked autonomous
vehicles. In these systems, cooperative multi-agents aim to complete a
specific task via cooperating (Busoniu, Babuska, and De Schutter 2008). Most
methods assume that agents can communicate in cooperation, which is
impractical in some situations. Even though agents can communicate, the
connection is unreliable due to interferences in environments. In this
situation, the solution will be sub-optimal.

Figure 1: The process in SMAC via the Graph Clustering method. In this
8m_vs_9m map, agents are divided into three groups to complete the
corresponding sub-tasks.

Many animal populations tend to work in groups to execute tasks, such as ants
and geese. This hints that the agents in the same group may achieve better
performance with less communication or even no communication. The reason is
that agents in the same group tend to achieve similar tasks. Moreover, by
dividing all agents into specific groups, the relationship between all agents
is decomposed into the connection between groups, significantly reducing the
complexity. Inspired by these, we propose a group concept for multi-agent
reinforcement learning (MARL) to handle cooperative tasks without
communication. Specifically, we assume that each agent has a belief to assist
in decision making, and agents in the same group tend to generate similar
beliefs.

In the large-scale multi-agents scenario, it is challenging to model all
agents in a limited time efficiently. A simple way is to use a bidirectional
recurrent neural network to represent all agents as a group explicitly.
Nevertheless, it ignores the second-order relationship information between
agents and is limited by the sequence of agents. In this paper, we propose a
straightforward approach to map agents. Inspired by the neighborhood cognitive
consistency (Mao et al. 2019), we propose to use the k-Nearest Neighbor (kNN)
method to map agent via agent-relative position. The idea is that agents in
adjacent positions will have similar observation features, which leads to a
similar decision.

Our contribution has four parts. (1) We design a novel modular agent structure
that can achieve the Centralised Training and Decentralised Execution (CTDE),
compared to the communication-based method. (2) We propose to use the agent
belief to represent the approximate group feature which is optimized via the
GAT module and achieve approximate performance without communication. (3) We
propose an explicit method to mapping agents to achieve similar beliefs in
adjacent agents. (4) To overcome the consistent problem brought by GAT, we
introduce a split loss to distinguish agents.

The proposed group-based method is evaluated on several unit micromanagement
tasks based on StarCraft II (Vinyals et al. 2017). The results indicate that
our algorithm performs better than traditional ones in the literature. The
performance of the method scales well with the number of agents. To observe
the correlation between group features and agent’s topology information, the
t-SNE method (Laurens and Hinton 2008) is used. The experimental result
reveals that the feature of agents in adjacent positions are also adjacent.

## Related Work

Over the past years, deep multi-agent reinforcement learning has made a
considerable breakthrough, widely used in games, traffic control, and other
fields. This research concentrates on cooperative MARL with a value function-
based method. In this setting, all teams receive a global team reward, so it
is essential to divide the global team reward into individual rewards. Sunehag
et al. (2017) decomposes the joint Q value into the individual Q value of each
agent. Rashid et al. (2018) presents the constraint in which the joint Q value
and the individual Q value have a monotonous setting based on the VDN
algorithm. The suboptimal problem and decentralization in multi-agents are
balanced by Son et al. (2019) via an L2-penalty term.

Among current multi-agent algorithms, communication is a crucial point. The
communication-based approaches assume that multiple agents can make essential
information interactions to assist in deciding. Foerster et al. (2016) applies
a deep feedforward neural network to generate communication vectors for
agents’ communication. Peng et al. (2017) utilizes a bidirectional recurrent
neural network for communication between multiple agents. Ding, Huang, and Lu
(2020) proposes a selectable point-to-point communication method adopted to
determine whether agents communicate with each other by constructing a belief
vector. Unlike these methods, Jiang, Dun, and Lu (2018) employs graph
convolution network and multi-head dot-product attention to aggregate agent
features. Because agents generate their actions based on other agents’ beliefs
or relative features, these methods always need centralized execution that
follows the CTCE framework. However, when the actual scenario restricts some
agents from being difficult to communicate, the agent may obtain inaccurate
beliefs or wrong beliefs, leading to sub-optimal actions.

Current researches on agent mapping always focus on the attention mechanism.
(Liu et al. 2020) utilizes two-stage attention to build an adjacent matrix.
Hard attention is devoted to delete related weak edges, and soft attention
generates the weight coefficients of the retained graph structure. EPC-MADDPG
(long2020evolutionary) uses the scalar dot product attention to fuse the
variable entity features to achieve a global mapping. However, simply using
the attention mechanism to capture the association between agents for mapping
will cause the algorithm to fall into a locally optimal solution. If the
initial attention mechanism considers that two far apart agents are similar
and tend to similar tasks, all agents will behave the same. All agents will
consistent and lose the meaning of grouping.

(a) The schematics of our approach

(b) The schematics of our approach base on the distributed framework

Figure 2: (a) The Encoder generates individual features and Gaussian
distribution’s group features. Then group features are sampled via
reparameterization trick and are fed into the GAT network to make feature
aggregation. The hypernetwork merges the group features and the individual
features into the individual Q value to generate the total Q value via QMIX.
The black line indicates the route of gradient propagation of the split loss.
(b) Our algorithm can be decomposed into modules. Specifically, we use a new
group feature network instead of the original group network to achieve the
purpose of distributed exection. The new group feature network is independent
of other agents. This new network is trained through minimizing the KL
divergence between the group feature network’s output and the original group
network’s output.

## Background

In the present research, the problem is regarded as a fully cooperative multi-
agent task viewed as a Dec-POMDP (Oliehoek, Amato et al. 2016) comprising of a
tuple $\left<I,S,U,Z,P,R,O,n,\gamma\right>$, where $s\in S$ depicts the global
state of the environment. At any time, each agent $i\in I\equiv\\{1,...,n\\}$
interacts with the environment by generating corresponding actions
$u_{i}\in{U}$ through the observation vector $z_{i}\in Z$ according to the
observation function $O(s,i)$. Agents learn to maximize the reward $R$ for
environmental feedback. This process is based on a state transition function
$P\left(s^{\prime}\mid s,a\right)$. Moreover, $n$ signifies the number of
agents, and $\gamma$ represents a discount factor.

Interaction-based MARL algorithms are generally implemented via the
centralized training with centralized execution (CTCE) framework. Agent’s
strategy $\pi_{i}(u_{i}|\tau_{i})$ is generated based on the observation
sequence $\tau$, the global state $s$, and the interactive features of other
agents. Via introducing the modular mechanism, our framework is designed as
the distributed and applying a regularization tool to generate representative
group features without interaction.

### Graph Attention Network

Graph Attention Network (GAT) can correlate similar features between agents
using masked self-attentional layers. To better correlate similar features
between agents, GAT uses masked self-attentional layers. By introducing the
attention mechanism, the neural network can focus on the most relevant parts
of the input, which helps learn the correlation features adaptively between
nodes (Veličković et al. 2017). Besides, GAT can capture relative features
between disconnect nodes by stacking GAT layers. GAT has two attention
mechanism types, the Global Graph Attention and Masked Graph Attention. The
former builds the attention operation between each one node and all the other,
while the latter only performs the same operation on neighboring nodes.

Distant agents may bring a negative effect on the current agent. Also, it is
very computationally expensive to calculate the relationship between all
agents, which causes the Global Graph Attention-based method unreasonable in
large-scale scenarios. While Masked Graph Attention-based method only captures
the relationship between the adjacent agents, which leads neighbor agents to
hold similar actions. Besides, agents can capture the unconnected agent
features via stacking the Masked Graph Attention layers and broaden the
receptive field of agents.

## Method

In this section, we introduce the group concept into the multi-agent algorithm
and propose the Belief in Graph Clustering (BGC) methods. The overall
framework of the algorithm is shown in Figure 2. To obtain a distributed
framework, a modular approach is applied to build an algorithm network and
generate groups and individual features of agents separately. Due to the idea
of modular construction, we utilize a new group feature network to represent
agent group features via minimizing the difference of original group features
without agent intersection. We use the group features to approximate the
effect of communication to implement the distributed execution.

In the overall pipeline, each agent generates an individual Q value via its
local observation and then passes it into a mixed network (such as QMIX) for
generating the global Q value. Specifically, each agent encodes its local
observation sequence via a GRU network to produce agent group features and
individual features. The group features are obtained via exercising the
multivariate Gaussian distribution condition on the observation sequence. The
reparameterization trick is practiced for sampling to ensure the continuity of
the gradient. Then, to cluster group features and obtain relationship features
from adjacency agents, a GAT network is adopted. Group and individual features
are feeds into a hyper network (Ha, Dai, and Le 2016) to generate individual Q
Value. Besides, to prevent all group features from converging together, we
introduce a split loss to distinguish non-adjacent agents.

### Adjacent Matrix via kNN

In this section, we introduce a k-Nearest Neighbor (kNN) method to represent
agent relationships on topology and construct the adjacency matrix for the
masked attention-based GAT network. Due to the centralized training setting,
it is easy to get all agents’ position information from agent observation.
Agent adjacency graph is constructed by defining the agent with the k nearest
agents as the same type. Then we take the Laplace transform and regularization
of adjacent graphs to produce the agent adjacency matrix.

In our setting, the hyperparameter k in kNN is two. In this setting, all
agents will be divided into several groups, as shown in Figure 5(d). The non-
information exchange between the group level leading different groups to
achieve different sub-tasks to realize group diversity. Besides, when all
agents are adjacent, it leads the group-level information transfer, which
helps accomplish the finished task via group coordination.

### Belief in Graph Clustering

In this part, we introduce the details on how to generate agent belief via
graph clustering. First, we embed agent observation via an MLP network. To
capture the relevant features on time series, we introduce the GRU module. To
add the uncertainty to agent group features and enhance the exploration
ability, we use a Gaussian Sampling module, which takes the condition on the
embedding features of the GRU module. This module uses MLP networks to
generate the mean and variance (take the exponential function on the
logarithmic variance term) to construct the independent Gaussian distribution.
Besides, we introduce the reparameterization trick to sample the actual group
features for the gradient’s continuity.

$\displaystyle\left(\mu_{g_{i}},\sigma_{g_{i}},s_{i}\right)=f\left(\tau_{i};\theta_{i}\right)$
(1a) $\displaystyle
g_{i}=\mu_{g_{i}}+\sigma_{g_{i}}\odot\varepsilon_{i}\quad\varepsilon_{i}\sim
N(0,1)$ (1b)

where $\theta_{i}$ represents parameters of network, $\mu_{g_{i}}$ and
$\sigma_{g_{i}}$ signify the mean and standard deviation of group features,
$s_{i}$ represents individual features, $g_{i}$ signifies the actual group
feature, and $\varepsilon_{i}$ is noise.

In centralized training, agents can use some adjacent agent features to assist
in deciding. After getting the adjacency matrix, we utilize the GAT module to
cluster the relevant features into group features. Under this setup, the agent
interacts with agents in the adjacent position via the attention mechanism and
generates the correlation weight between agents. This weight is used to
calculate the agent relationship features for the current agent, and the agent
fuses these features to produce final group features.

$\displaystyle
g_{i}^{\prime}=\sigma\left(\sum_{j\in\mathcal{N}_{i}}\alpha_{ij}g_{j}\right)$
(2)

where $\sigma(\cdot)$ is the ReLu activation function, and $\alpha$ is the
clustering coefficient calculated via the function.

$\displaystyle
e_{ij}=a\left(\left[Wg_{i}\|Wg_{j}\right]\right),j\in\mathcal{N}_{i}$ (3a)
$\displaystyle\alpha_{ij}=\frac{\exp\left(\text{LeakyReLU}\left(e_{ij}\right)\right)}{\sum_{k\in\mathcal{N}_{i}}\exp\left(\text{LeakyReLU}\left(e_{ik}\right)\right)}$
(3b)

Finally, we use a hyper network module to fuse group features and individual
features to generation agent action. In this module, the group features are
utilized to generate the MLP network weight, multiplying individual features
to get the final action.

### Split Loss

Although mask attention-based GAT network can prevent all agents from forming
a single group, current agents’ features will still extend to all other agents
due to the time perspective. Therefore, we propose a split loss to alleviate
this issue. We use Kullback-Leibler (KL) divergence to measure differences
between agents’ group features and keep the non-adjacent agents at a fixed
distance.

$\displaystyle\max_{\pi\in\Pi}\quad$
$\displaystyle\mathbb{E}_{\tau\sim\rho_{\pi}}\left[\sum_{t=0}^{T}r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\right]$
(4a) $\displaystyle s.t.\quad$
$\displaystyle\mathbf{KL}(g_{i}||g_{j})\geq\delta,\forall\ edge(i,j)=\phi$
(4b)

where $g_{i}$ represents agent group features and $edge(i,j)$ determines
whether there is a connection between agent $i$ and agent $j$.

Therefore, a split loss is introduced to separate the agent:

$\displaystyle
L_{split}=-\sum_{i}^{N}\sum_{j}^{N}min(\mathbf{KL}(g_{i}||g_{j})-\delta,0),$
$\displaystyle\forall\ edge(i,j)=\phi,i\neq j$ (5)

The hyperparameter $\delta$ is employed to keep agents at the $\delta$
distance, changing with the scenario.

### Decentralization Execution

To realize the complete decentralized execution without agent interaction, we
apply a new sub-network to learn the new group features and use the KL
divergence to minimize the distance between the new group features and the
original group features learned by the GAT network. Then, the gradient will
flow through the new sub-network and train the new sub-network. This frame can
be view as a teacher-student structure.

Figure 3: Result of starcraft II. The comparison between the group-based
method and other algorithms with the number of agents increasing from 5 to 15.
Empirical results show that as the number of agents increases, our algorithm
show more superiority comparing with the baseline algorithm. Due to the
limitation of graphics card memory(RTX2080Ti), we did not test the actual
effect of G2ANet in the 12m_vs_14m map and 15m_vs_17m map.

## Experiment

### Starcraft II

In the experiment part, we test our algorithm in the SMAC benchmark. The
difference between SMAC and the traditional Starcraft II environment is that
it focuses on unit micromanagement. It leverages the natural multi-agent
microstructure by proposing a modified version of the problem designed
specifically for decentralized control. To test the algorithm robustness, we
run our algorithm in 6 to 10 million steps in 4 random seeds. Besides, to
estimate the performance of the method, we experiment in five scenarios, three
standard maps of 5m_vs_6m (very hard), 8m_vs_9m (very hard), and 10m_vs_11m
(very hard), and two custom maps of 12m_vs_14m (very hard) and 15m_vs_17m
(very hard).

Figure 3 illustrates the performance difference between the proposed algorithm
and other benchmark algorithms in five scenarios. We first compare our method
with the origin QMIX algorithm. The result shows that BGC-QMIX is
outperforming the origin algorithm on the final test win rate and the
performance variance, as shown in Figure 3 (BGC+QMIX vs. QMIX).

Except for the scenario with a small number of $5m\\_vs\\_6m$(very hard), our
algorithm demonstrates a significant improvement compared to the baseline
methods. In the $5m\\_vs\\_6m$ scenario, the proposed algorithm has excellent
advantages over other ones in terms of convergence speed. In this scenario
set, the small number of agents causes agents’ group features to be
consistent, leading to less than ideal performance. To check the influence of
the number of agents, we further perform our method in other scenarios. 3
(bottom right) shows that it can maintain an excellent performance with the
number of agents increases while QMIX does not. The results show that the mask
attention-based GAT method and the split loss performance.

Furthermore, our algorithm is compared with G2ANet, which is also based on the
attention mechanism. G2ANet practices the hard attention to determine the
communication target and soft attention to complete the communication. G2ANet
overlooks agent with topology information’s association using a bi-LSTM
network to construct the hard attention operation. Further, the G2ANet is
tested based on the QMIX framework, and the empirical result shows that our
graph clustering-based algorithm performs better. (As the tests were under the
hardware condition of RTX 2080Ti, $12m\\_vs\\_14m$ and $15m\\_vs\\_17m$
scenarios can cause cuda memory exhausted. Thus, only three standard scenes
are tested.)

Figure 4: Ablation test. we test the our graph clustering-based method base on
the VDN framework. The result shows its superiority.

(a) 8m_vs_9m(a)

(b) 8m_vs_9m(b)

(c) 10m_vs_11m(a)

(d) 10m_vs_11m(b)

(e) 12m_vs_14m(a)

(f) 12m_vs_14m(b)

(g) 15m_vs_17m(a)

(h) 15m_vs_17m(b)

Figure 5: Group feature representation base on t-SNE.

### Representation

To demonstrate the superiority of our proposed algorithm, the t-SNE algorithm
is used to reduce the group features’ dimension for visually illustrating the
effect of agent grouping. Pictures show the representation after compressing
the group feature of the agent to two dimensions in multiple map scenarios.
Pictures include various scenarios from the beginning of the game, the middle
of the game, and the agent’s death in the later stage. Figure 5(c) shows that
BGC-QMIX first divides the agent group features into three groups at the
beginning. Agents’ group features are divided into two groups without any
information interaction between groups with the game running. Figure 5(h)
indicates that group features of the dead agents can be divided into a
specific group that demonstrates the rationality of the algorithm. Moreover,
we find that the agent mapping becomes more straightforward and more
representative as the number of agents increases.

### Ablation

To check the robustness of our proposed algorithm, an ablation experiment is
conducted. The VDN network is used instead of the QMIX network to aggregate
the independent Q values of the agent. Experimental results in 8m_vs_9m map,
10m_vs_11m map, and 12m_vs_14m map are compared. In two scenarios, where the
number of agents is small, the group clustering-based method has a certain
degree of improvement compared with the VDN and even the QMIX algorithm. In
the more significant number 12m_vs_14m scenario, the VDN algorithm is better
than that of QMIX; the VDN method based on graph clustering is still better
than these two.

### Distributed Execution

The introduced algorithm is tested within the distributed framework. The KL
divergence method is used for the agents trained based on the CTCE framework
to minimize the new group feature and the group feature after graph
clustering. The overall algorithm training is about 0.5M steps, and the
original network error on the test_battle_win_mean index is within $\pm 1\%$.

## Conclusion

This paper introduces the group concept into multi-agent reinforcement
learning to achieve excellent performance in the non-communication setup. We
design a novel agent structure to realize this non-communication goal via an
individual agent module to generate individual features and an agent belief
module to produce group features. Apply a fusion module to fuse these features
to generate agent actions. In the agent belief module, we take the GAT network
to merge the adjacent agent features and use a split loss to prevent all
agents from being consistent.

Empirical results show that the proposed algorithm’s effect is vastly improved
compared to the current baseline algorithms, and this improvement increases
with the increase in the number of agents. The t-SNE method is applied to
verify that the group features of adjacent agents are more similar, indicating
the agent group features representability under the graph attention network
and the split loss. Additionally, the overall algorithm is integrated into the
VDN network, and results show the performance improved compared to the
original VDN network, even the QMIX method.

In future work, we will consider apply intrinsic rewards to group agents more
efficiently and feasibly.

## Acknowledgments

The work has been supported by grant number: Z181100003218013 from Beijing
Municipal Science&Technology Commission.

## References

  * Agarwal, Kumar, and Sycara (2019) Agarwal, A.; Kumar, S.; and Sycara, K. P. 2019.  Learning Transferable Cooperative Behavior in Multi-Agent Teams.  _CoRR_ abs/1906.01202.  URL http://arxiv.org/abs/1906.01202.
  * Busoniu, Babuska, and De Schutter (2008) Busoniu, L.; Babuska, R.; and De Schutter, B. 2008.  A comprehensive survey of multiagent reinforcement learning.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 38(2): 156–172.
  * Chung et al. (2014) Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ .
  * Ding, Huang, and Lu (2020) Ding, Z.; Huang, T.; and Lu, Z. 2020.  Learning Individually Inferred Communication for Multi-Agent Cooperation.  _arXiv preprint arXiv:2006.06455_ .
  * Du et al. (2019) Du, Y.; Han, L.; Fang, M.; Liu, J.; Dai, T.; and Tao, D. 2019.  LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 32_ , 4403–4414. Curran Associates, Inc.  URL http://papers.nips.cc/paper/8691-liir-learning-individual-intrinsic-reward-in-multi-agent-reinforcement-learning.pdf.
  * Dudani (1976) Dudani, S. A. 1976.  The distance-weighted k-nearest-neighbor rule.  _IEEE Transactions on Systems, Man, and Cybernetics_ (4): 325–327.
  * Foerster et al. (2016) Foerster, J. N.; Assael, Y. M.; de Freitas, N.; and Whiteson, S. 2016.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning.  _CoRR_ abs/1605.06676.  URL http://arxiv.org/abs/1605.06676.
  * Ha, Dai, and Le (2016) Ha, D.; Dai, A.; and Le, Q. V. 2016.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ .
  * Hausknecht and Stone (2015) Hausknecht, M.; and Stone, P. 2015.  Deep recurrent q-learning for partially observable mdps.  In _2015 AAAI Fall Symposium Series_.
  * Hüttenrauch, Sosic, and Neumann (2017) Hüttenrauch, M.; Sosic, A.; and Neumann, G. 2017.  Guided Deep Reinforcement Learning for Swarm Systems.  _CoRR_ abs/1709.06011.  URL http://arxiv.org/abs/1709.06011.
  * Jiang, Dun, and Lu (2018) Jiang, J.; Dun, C.; and Lu, Z. 2018.  Graph Convolutional Reinforcement Learning for Multi-Agent Cooperation.  _CoRR_ abs/1810.09202.  URL http://arxiv.org/abs/1810.09202.
  * Kraemer and Banerjee (2016) Kraemer, L.; and Banerjee, B. 2016.  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  _Neurocomputing_ 190(may 19): 82–94.
  * Kusner, Paige, and Hernández-Lobato (2017) Kusner, M. J.; Paige, B.; and Hernández-Lobato, J. M. 2017.  Grammar variational autoencoder.  _arXiv preprint arXiv:1703.01925_ .
  * Laurens and Hinton (2008) Laurens, V. D. M.; and Hinton, G. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ 9(2605): 2579–2605.
  * Liu et al. (2020) Liu, Y.; Wang, W.; Hu, Y.; Hao, J.; Chen, X.; and Gao, Y. 2020.  Multi-Agent Game Abstraction via Graph Attention Neural Network.  In _AAAI_ , 7211–7218.
  * Mao et al. (2019) Mao, H.; Liu, W.; Hao, J.; Luo, J.; Li, D.; Zhang, Z.; Wang, J.; and Xiao, Z. 2019\.  Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning.  _arXiv preprint arXiv:1912.01160_ .
  * Oliehoek, Amato et al. (2016) Oliehoek, F. A.; Amato, C.; et al. 2016.  _A concise introduction to decentralized POMDPs_ , volume 1.  Springer.
  * Peng et al. (2017) Peng, P.; Yuan, Q.; Wen, Y.; Yang, Y.; Tang, Z.; Long, H.; and Wang, J. 2017.  Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games.  _CoRR_ abs/1703.10069.  URL http://arxiv.org/abs/1703.10069.
  * Rashid et al. (2018) Rashid, T.; Samvelyan, M.; de Witt, C. S.; Farquhar, G.; Foerster, J. N.; and Whiteson, S. 2018.  QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning.  _CoRR_ abs/1803.11485.  URL http://arxiv.org/abs/1803.11485.
  * Son et al. (2019) Son, K.; Kim, D.; Kang, W. J.; Hostallero, D.; and Yi, Y. 2019.  QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning.  _CoRR_ abs/1905.05408.  URL http://arxiv.org/abs/1905.05408.
  * Sunehag et al. (2017) Sunehag, P.; Lever, G.; Gruslys, A.; Czarnecki, W. M.; Zambaldi, V. F.; Jaderberg, M.; Lanctot, M.; Sonnerat, N.; Leibo, J. Z.; Tuyls, K.; and Graepel, T. 2017.  Value-Decomposition Networks For Cooperative Multi-Agent Learning.  _CoRR_ abs/1706.05296.  URL http://arxiv.org/abs/1706.05296.
  * Van Erven and Harremos (2014) Van Erven, T.; and Harremos, P. 2014.  Rényi divergence and Kullback-Leibler divergence.  _IEEE Transactions on Information Theory_ 60(7): 3797–3820.
  * Veličković et al. (2017) Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ .
  * Vinyals et al. (2017) Vinyals, O.; Ewalds, T.; Bartunov, S.; Georgiev, P.; Vezhnevets, A. S.; Yeo, M.; Makhzani, A.; Küttler, H.; Agapiou, J.; Schrittwieser, J.; Quan, J.; Gaffney, S.; Petersen, S.; Simonyan, K.; Schaul, T.; van Hasselt, H.; Silver, D.; Lillicrap, T. P.; Calderone, K.; Keet, P.; Brunasso, A.; Lawrence, D.; Ekermo, A.; Repp, J.; and Tsing, R. 2017.  StarCraft II: A New Challenge for Reinforcement Learning.  _CoRR_ abs/1708.04782.  URL http://arxiv.org/abs/1708.04782.
  * Wang et al. (2019) Wang, T.; Wang, J.; Wu, Y.; and Zhang, C. 2019.  Influence-based multi-agent exploration.  _arXiv preprint arXiv:1910.05512_ .
  * Wang et al. (2020) Wang, W.; Yang, T.; Liu, Y.; Hao, J.; Hao, X.; Hu, Y.; Chen, Y.; Fan, C.; and Gao, Y. 2020.  From Few to More: Large-Scale Dynamic Multiagent Curriculum Learning.  In _AAAI_ , 7293–7300.