# Effect of induced transition on the quantum entanglement and coherence in
two-coupled double quantum dots system

Zakaria Dahbi<EMAIL_ADDRESS>Lab of High Energy Physics-Modeling
and Simulation, Faculty of Sciences,
Mohammed V University in Rabat, 4 Avenue Ibn Battouta B.P. 1014 RP, Rabat,
Morocco  Maron F. Anka<EMAIL_ADDRESS>Instituto de Física, Universidade
Federal Fluminense, Av. Gal. Milton Tavares de Souza s/n, 24210-346 Niterói,
Rio de Janeiro, Brazil  Mostafa Mansour<EMAIL_ADDRESS>Laboratory of High Energy Physics and Condensed Matter, Department of Physics,
Faculty of Sciences of Ain Chock, Hassan II University, P.O. Box 5366 Maarif,
Casablanca 20100, Morocco  Moisés Rojas<EMAIL_ADDRESS>Departamento de
Física, Universidade Federal de Lavras, 37200-900, Lavras-MG, Brazil  Clebson
Cruz<EMAIL_ADDRESS>Grupo de Informação Quântica e Física
Estatística, Centro de Ciências Exatas e das Tecnologias, Universidade Federal
do Oeste da Bahia—Campus Reitor Edgard Santos. Rua Bertioga, 892, Morada Nobre
I, 47810-059 Barreiras, Bahia, Brazil

###### Abstract

Studying quantum properties in solid-state systems is a significant avenue for
research. In this scenario, double quantum dots (DQDs) appear as a versatile
platform for technological breakthroughs in quantum computation and
nanotechnology. This work inspects the thermal entanglement and quantum
coherence in two-coupled DODs, where the system is exposed to an external
stimulus that induces an electronic transition within each subsystem. The
results show that the introduction of external stimulus induces a quantum
level crossing that relies upon the Coulomb potential changing the degree of
quantum entanglement and coherence of the system. Thus, the quantum properties
of the system can be tuned by changing the transition frequency, leading to
the enhancement of its quantum properties.

##  I Introduction

Studying theoretical quantum information quantifiers in condensed matter
physics systems has paramount importance as a key resource for several
information processing protocols [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. In this
scenario, double quantum dots (DQD) appear as a protean platform for condensed
matter physics due to its role in the development of emerging quantum
technologies [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. Entanglement and
quantum dynamics of two electrons within coupled DQDs were examined, and
aspects of quantum coherence were addressed in Refs. [22, 23]. In general,
quantum entanglement has attracted the attention of the scientific community
in the past few years due to its pivotal role in developing new quantum-based
technologies [1, 2, 4, 5, 6, 7, 8, 9].

Despite its enormous promise, measuring and characterizing entanglement in
multipartite quantum systems is a rather complicated task [24, 25]. Numerous
research for characterizing the degree of entanglement in multipartite quantum
systems have been conducted [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36].
However, entanglement does not encompass all quantumness in a system [37, 38].
On the other hand, quantum coherence is derived from the quantum state
superposition principle [39]. Similar to quantum correlations, it is a
fundamental idea in quantum theory. It is also a crucial physical resource for
quantum computations [40, 41, 42]. Quantum coherence is defined as the
generalized quantum uncertainty in reference 43 and is regarded as a useful
and necessary resource in quantum information tasks [40, 44, 45, 3]. In order
to investigate the relationship between quantum coherence and quantum
correlations, a new measure has been introduced known as correlated coherence
[46, 47]. Correlated coherence measures the total amount of coherence stored
in quantum correlations. The link between quantum coherence and correlations
is discussed in references 40, 48, 49. In the scenario of DQDs, these
connections between quantum coherence and other quantum resources have
received considerable attention [50, 51, 23, 52, 53].

In this regard, this work studies the effect of an external stimulus on the
quantum aspects of thermal coherence and entanglement in a two-coupled DQDs
system. The external stimulus is characterized by the transition frequency of
the electrons between the left and right dots. The physical model is
introduced in order to provide the literature with a comparison between the
total, local, and correlated $l_{1}$-norm quantum coherence with the amount of
entanglement in the system. The influence of the transition frequency on these
quantum resources is studied, typifying the degree of local and correlated
quantum features in coupled DQDs systems and exploring the influence of the
temperature, Coulomb coupling, and tunneling strengths. The results show that
the competition between the Coulomb Coupling and the energetic contribution of
the external stimulus leads to a quantum level crossing between a separable
and an entangled configuration of the system, which leads to the enhancement
of its quantum properties.

##  II Two coupled double quantum dots model

The proposed physical model comprises two sets of double quantum dots (DQDs),
where each quantum dot can only contain a single electron. Each electron has
two degrees of freedom, so it can either be in the left dot ($\ket{l_{i}}$) or
in the right dot ($\ket{r_{i}}$). A sketched setup of the model is shown in
Fig. 1. An external stimulus is conveyed by an electromagnetic wave ($\gamma$)
emitted from an external source. In reaction to the external stimulus, the
DQDs system display variations in its quantum aspects concerning the energy
absorbed by each electron, resulting in electron transitions between left and
right dots.

Figure 1: A scheme of the two coupled DQDs model with an external stimulus
(left) and its equivalent circuit (right). $C_{lr}$, $C_{rl}$, $C_{ll}$ and
$C_{rr}$ are the capacitors connecting the DQDs.

The Hamiltonian of the two coupled double quantum dots can be written on its
explicit form in the local basis $\\{\ket{i_{A},j_{B}}\\}_{i,j=l,r}$ as

$H=\left(\begin{array}[]{cccc}\mathcal{V}+2\hbar\omega&\delta_{B}&\delta_{A}&0\\\
\delta_{B}&-\mathcal{V}&0&\delta_{A}\\\
\delta_{A}&0&-\mathcal{V}&\delta_{B}\\\
0&\delta_{A}&\delta_{B}&\mathcal{V}-2\hbar\omega\\\ \end{array}\right).$ (1)

where $\omega$ stands for a frequency of an external stimulus that induces a
local transition in each subsystem, $\delta_{i}$ is the tunneling coupling
strength in each coupled quantum dot, and the $\mathcal{V}$ represents the
Coulomb interaction between the electrons. In such a scenario, the density
matrix provides a simple approach to extend the investigation to finite
temperatures when the system is in equilibrium with a thermal bath. In the
canonical ensemble at temperature $T$, the thermal density matrix of the
system can be written in the Gibbs form

$\displaystyle\varrho_{T}=\frac{1}{{Z}}\exp(-\beta H),$ (2)

where $\beta=1/k_{B}T$ and $Z$ being the canonical partition function
$Z=tr(e^{-\beta H})$. In this study, Boltzmann’s constant $k_{B}$ and Planck’s
constant $\hbar$ are set to $1$. The thermal state, Eq. (2), can be
parameterize in terms of the identity $\mathbb{I}$ and Pauli matrices
$\\{\sigma^{A(B)}_{i}\\}$ as follows

$\displaystyle\varrho_{T}$ $\displaystyle=$
$\displaystyle\frac{1}{4}\big{(}\mathbb{I}_{AB}+a_{1}(T)\sigma^{A}_{x}\otimes\mathbb{I}_{B}+a_{2}(T)\sigma^{A}_{z}\otimes\mathbb{I}_{B}$
$\displaystyle+$ $\displaystyle
b_{1}(T)\mathbb{I}_{A}\otimes\sigma^{B}_{x}+b_{2}(T)\mathbb{I}_{A}\otimes\sigma^{B}_{z}+\sum_{i,j=1}^{3}t_{ij}(T)\sigma^{A}_{i}\otimes\sigma^{B}_{i}\big{)},$

where the non-vanishing thermal coefficients $a_{i}(T)$, $b_{i}(T)$ and
$t_{ij}(T)$ can be written as

$\displaystyle a_{1}(T)=2\varrho_{13}+2\varrho_{24},\quad
a_{2}(T)=\varrho_{11}+\varrho_{22}-\varrho_{33}-\varrho_{44},$ (4)
$\displaystyle b_{1}(T)=2\varrho_{12}+2\varrho_{34},\quad
b_{2}(T)=\varrho_{11}-\varrho_{22}+\varrho_{33}-\varrho_{44},$ (5)
$\displaystyle t_{11}(T)=2\varrho_{14}+2\varrho_{23},\quad
t_{22}(T)=2\varrho_{23}-2\varrho_{14},\quad
t_{33}(T)=\varrho_{11}-\varrho_{22}-\varrho_{33}+\varrho_{44},$ (6)
$\displaystyle t_{13}(T)=2\varrho_{13}-2\varrho_{24},\quad
t_{31}(T)=2\varrho_{12}-2\varrho_{34},$ (7)

with $\varrho_{mn}$ being the matrix elements of the thermal state in the
local basis ($\\{\ket{i_{A},j_{B}}\\}_{i,j=l,r}$). The populations and the
eigenvectors of the two coupled DQD system can be obtained numerically 111The
numerical results were obtained using functional programming (FP) paradigms
using Julia programming language. FP produces numerically precise results with
improved computational-time complexity due to the benefit of backward
functions. by diagonalizing the above density matrix. Figure 2 depicts the
behavior of the system’s populations as a function of temperature and Coulomb
coupling in two scenarios: when $\omega<\mathcal{V}$ and when
$\omega>\mathcal{V}$. In each scenario, a concrete comparison with the absence
of the stimulus effect is presented.

Figure 2: Comparison of the system’s populations with and without the external
stimulus effect ($\omega=0,15$, and $45$). 2: versus temperature. 2: versus
Coulomb coupling.  As can be seen, in the absence of the external stimulus,
there is no energy level crossing, thus, no population inversion. In this
scenario, the effect of the Coulomb coupling leads the system to the
configuration described by the maximally entangled state
$\left(\ket{r_{A}l_{B}}+\ket{l_{A}r_{B}}\right)/\sqrt{2}$. However, when the
external stimulus is turned on, it splits the energy levels in such a way that
both electrons are found in the state $\ket{r_{A}r_{B}}$ for
$\omega>\mathcal{V}$. Thus, the introduction of the external stimulus creates
a dispute between its energetic contribution and the Coulomb coupling, which
leads to a quantum level crossing between the separable state
$\ket{r_{A}r_{B}}$ and the maximally entangled one
$\left(\ket{r_{A}l_{B}}+\ket{l_{A}r_{B}}\right)/\sqrt{2}$, characterized by
the threshold point $\mathcal{V}_{c}=\omega$. As can be seen, increasing the
strength of the Coulomb coupling will suppress the external stimulus, leading
the system to the state
$\left(\ket{r_{A}l_{B}}+\ket{l_{A}r_{B}}\right)/\sqrt{2}$. This quantum level
crossing between a separable and an entangled state plays an important role in
the behavior of the quantum features of the system, as shown in section IV.

##  III Quantum information quantifiers

In the following, we introduce the quantum information quantifiers used to
characterize the quantum coherence and entanglement in the thermal state of
two coupled DQDs in the presence of an external stimulus.

###  III.1 $l_{1}$-norm coherence

Quantum coherence is a basis-dependent feature of a quantum state which
emerges from the superposition of the quantum states of the system,
represented by the off-diagonal elements of the density matrix $\varrho$, Eq.
(II), in a certain basis [3, 44]. Baumgratz $et\leavevmode\nobreak\ al.$ [44]
has proved that $l_{1}$-norm is a reliable coherence measurement and is
effortlessly computed for a given density matrix $\varrho$ as

$\mathcal{C}_{l_{1}}(\varrho)=\sum_{i\neq j}|\varrho_{ij}|.$ (8)

It is worth noting that quantum coherence, distinct from other quantum
correlation quantifiers, gives additional information about a quantum system’s
essential nature. Furthermore, while quantum correlations needs the definition
of at least two parties, quantum coherence may be established for just a sole
qubit system. On the other hand, for any given multipartite state $\varrho$,
the correlated contribution in the quantum coherence can computed by taking
away the local coherence of subsystems $\varrho_{A}$ and $\varrho_{B}$ from
the total coherence [48, 23, 46]. The reduced density matrix of subsystem $A$
is given by

$\displaystyle\varrho_{T}^{A}$ $\displaystyle=$
$\displaystyle(\varrho_{11}+\varrho_{33})\ket{l_{A}}\bra{l_{A}}+(\varrho_{22}+\varrho_{44})\ket{r_{A}}\bra{r_{A}}$
$\displaystyle+$
$\displaystyle(\varrho_{13}+\varrho_{24})(\ket{l_{A}}\bra{r_{A}}+\ket{r_{A}}\bra{l_{A}}).$

Similarly, for subsystem $B$, we obtain

$\displaystyle\displaystyle\varrho_{T}^{B}$ $\displaystyle=$
$\displaystyle(\varrho_{11}+\varrho_{33})\ket{l_{B}}\bra{l_{B}}+(\varrho_{22}+\varrho_{44})\ket{r_{B}}\bra{r_{B}}$
$\displaystyle+$
$\displaystyle(\varrho_{12}+\varrho_{34})(\ket{l_{B}}\bra{r_{B}}+\ket{r_{B}}\bra{l_{B}}).$

Thus, the definition of correlated coherence based on the $l_{1}$-norm of
coherence is defined as

$\mathcal{C}_{cc}(\varrho):=\mathcal{C}_{l_{1}}(\varrho)-\mathcal{C}_{l_{1}}(\varrho_{A})-\mathcal{C}_{l_{1}}(\varrho_{B}),$
(9)

where the density matrices of local subsystems are $\varrho_{A}=tr_{B}\varrho$
and $\varrho_{B}=tr_{A}\varrho$. In the case of two coupled DQDs, the
inclusion of the external stimulus contribution renders the analytical
formulation for the eigenvalues of the Hamiltonian and the density matrix
entries too complicated to be fully stated within this text. For the sake of
simplicity, using the implicit form of the density matrix in Eq. (II), one can
straightforwardly compute the total ($\mathcal{C}_{l_{1}}^{T}$), local
($\mathcal{C}_{l_{1}}^{L}$) and correlated ($\mathcal{C}_{l_{1}}^{C}$)
${l_{1}}$-norm based quantum coherence:

$\displaystyle\mathcal{C}_{l_{1}}^{T}(\varrho_{T})$ $\displaystyle=$
$\displaystyle
2\Big{(}\left|\varrho_{12}\right|+\left|\varrho_{13}\right|+\left|\varrho_{14}\right|$
(10) $\displaystyle+$
$\displaystyle\left|\varrho_{23}\right|+\left|\varrho_{24}\right|+\left|\varrho_{34}\right|\Big{)},$
$\displaystyle\mathcal{C}_{l_{1}}^{L}(\varrho_{T})$ $\displaystyle=$
$\displaystyle\left|a_{1}(T)\right|+\left|b_{1}(T)\right|$ (11)
$\displaystyle=$ $\displaystyle
2\Big{(}\left|\varrho_{13}+\varrho_{24}\right|+\left|\varrho_{12}+\varrho_{34}\right|\Big{)},$
$\displaystyle\mathcal{C}_{l_{1}}^{C}(\varrho_{T})$ $\displaystyle=$
$\displaystyle
2\Big{(}\left|\varrho_{12}\right|+\left|\varrho_{13}\right|+\left|\varrho_{14}\right|+\left|\varrho_{23}\right|+\left|\varrho_{24}\right|$
(12) $\displaystyle+$
$\displaystyle\left|\varrho_{34}\right|-(\left|\varrho_{13}+\varrho_{24}\right|+\left|\varrho_{12}+\varrho_{34}\right|)\Big{)}.$

In Sec. IV, we give and discuss the main findings concerning quantum
correlations and quantum coherence simulation in terms of temperature and the
different parameters appearing in the given Hamiltonian of the system.

###  III.2 Thermal entanglement

In order to make a comparison between the amount of entanglement and the
coherence in the system, we adopt the measurement of concurrence, which is
typically used to quantify the entanglement in bipartite systems [54, 55]. The
thermal concurrence explores the similarity between the state of the system
under consideration in thermal equilibrium, Eq. (II), and its bit-flipped
density matrix, defined as:

$R=\varrho_{T}(\sigma^{y}\otimes\sigma^{y})\varrho_{T}^{*}(\sigma^{y}\otimes\sigma^{y}),$
(13)

with $\sigma^{y}$ being a Pauli matrix. Thus, Concurrence is then defined as

$\mathcal{C}(\varrho_{T}):=\rm{max}\\{0,\sqrt{\lambda_{1}}-\sqrt{\lambda_{2}}-\sqrt{\lambda_{3}}-\sqrt{\lambda_{4}}\\},$
(14)

where $\lambda_{i}$’s are sorted in decreasing order,
$\lambda_{1}>\lambda_{2}\geq\lambda_{3}\geq\lambda_{4}$, to ensure concurrence
positivity. In the following Sec. IV, concurrence is applied to to analyze the
effect of an induced transition on the quantum entanglement of a two coupled
DQDs system.

##  IV Results and discussion

In order to understand the effect of an induced transition on the quantum
coherence and entanglement of our system, we need to analyze how the quantum
information quantifiers change as a function of each parameter of the
Hamiltonian. Thus, we dedicate this section to presenting and discussing the
behavior of the quantum features presented in Section III, associated with the
change of the transition frequency, Coulomb potential, and tunneling
strengths. Then, in light of the findings, we present a comprehensive analysis
of outcomes and assess the impact of a transition induced by an external
stimulus on the proposed model.

As our first analysis, we keep the Coulomb coupling and the tunneling coupling
strength fixed as we allow the temperature to increase for selected
frequencies. Fig. 3 shows the quantum information quantifiers as a function of
the temperature for different values of transition frequency with fixed
Coulomb coupling and tunneling strengths.

The effect of the external stimulus is responsible for the transition between
the right and left dots, i.e., the higher its frequency higher is the
probability of transition from one dot to another. On the other hand, this
effect also depends on the values of the other parameters, i.e., there is a
competition on the effect of the Coulomb, tunneling coupling strength, and the
transition frequency. This competition leads to a change in the occupation of
the energy levels, as presented in Fig. 2. As a consequence, increasing the
transition frequency will change the degree of correlation in the system, as
depicted in Fig. 3.

As it can be seen, increasing the frequency above the quantum level crossing
$\omega>\mathcal{V}$ (2) will populate many incoherent separable states,
decreasing the degree of coherence and entanglement of the system. On the
other hand, for frequencies lower than the Coulomb coupling
($\omega<\mathcal{V}$), at low temperatures, the increasing in the transition
frequency may lead the system to some excited states with a larger degree of
coherence and entanglement, yielding a enhancement on the quantum features.

As expected, the effect will become indistinguishable as we increase the
temperature. Thus, for higher values of temperature, the quantum features of
the system tend to fade, and the correlation between both electrons is
destroyed. In particular, for a given critical temperature $T_{c}$, where
there is a sudden death to the concurrence (entanglement). However, from the
correlated coherence, one can verify that this system remains correlated
despite the absence of entanglement. This result strengths the fact that
correlated coherence in DQD system is more robust than the thermal
entanglement [23]. On the other hand, as correlated coherence goes to zero for
higher temperatures, the local coherence is still greater and becomes the main
contribution to the quantum coherence of the system. Thus, in high-temperature
regimes, one can still observe some coherence from the local contribution,
even in the absence of correlations between the electrons.

Figure 3: Total coherence 3, local coherence 3, correlated coherence 3 and
concurrence 3 as a function of the temperature for different values of
transition frequency $\omega<\mathcal{V}$ and $\omega>\mathcal{V}$ (inset),
with $\delta_{A}=\delta_{B}=2$ and $\mathcal{V}=30$.

In the following, we keep the system at the low temperature of $T=0.1$ while
the Coulomb coupling is allowed to change. At this low value of temperature,
we find our system with a high degree of coherence and concurrence, according
to Fig. 3. On the other hand, Fig. 4 shows the dependence of the quantum
information quantifiers as a function of the Coulomb coupling. As can be seen
from total and local coherence, for non-zero transition frequency, a first
increase in the Coulomb coupling strength enhances quantum resources within
the system. However, after it exceeds a threshold value
$\mathcal{V}_{c}=\omega$, these quantifiers decrease as we increase the
Coulomb coupling. This behavior is explained by the same mechanism presented
in Fig. 2. For $\mathcal{V}_{c}>\omega$ occurs a population inversion due to
the energy levels crossing due to the competition between the Coulomb coupling
strength and the transition frequency, as observed in Fig.2 and 3.

Figure 4: Total coherence 4, local coherence 4, correlated coherence 4 and
concurrence 4 as a function of the Coulomb coupling $\mathcal{V}$ for
different values of transition frequency, with $\delta_{A}=\delta_{B}=2$ and
$T=0.1$.

It is worth noting that the concurrence is zero for $\mathcal{V}=0$. This
effect is expected since the Coulomb coupling is responsible for the
interaction between both electrons. On the other hand, the correlated
coherence goes to a finite non-zero value, reaching its maximum for
$\omega=0$. Thus, the correlation contribution to the coherence is not all due
to the presence of entanglement.

Furthermore, as we increase the Coulomb coupling, we change the predominant
contribution for the total coherence from the local to the correlated
coherence. This effect relies on the fact that the Coulomb coupling increases
the interaction between both electrons in the different dots while decreasing
the superposition generated by the transition frequencies.

In Fig. 5, we plot the quantum information quantifiers as a function of the
tunneling strengths $\delta_{A}=\delta_{B}$ for different values of transition
frequency. As can be seen, different from Fig. 3, we find that the behavior of
the correlated coherence and the concurrence does not match. We observed a
sudden increase in the entanglement for weak values of the tunneling
parameter. After reaching a maximum value, the entanglement gradually
decreases to zero as we increase the tunneling strengths. On the other hand,
as we increase the tunneling strength, the coherence emerging from the
superposition reaches its maximum value, which indicates the presence of pure
states in this regime. In addition, it can be seen that higher transition
frequencies rapidly drive the state of the system from being a mixed entangled
state to a pure state with a lower degree of entanglement.

Figure 5: Total coherence 5, local coherence 5, correlated coherence 5 and
concurrence 5 as a function of the tunneling strengths $\delta_{A}=\delta_{B}$
for different values of transition frequency, with $\mathcal{V}=30$ and
$T=0.1$. Inset shows the enhancement of the quantum coherence and entanglement
due to the increase of the transition frequency for $0<\omega<\mathcal{V}$

##  V Conclusions

In this work, we analyzed the effect of an induced transition, yielded by an
external stimulus, on the quantum features of a two-coupled double quantum
dots system by quantifying the thermal quantum coherence and entanglement. The
quantifiers were defined based on the formulated resource theory of coherence
and the measurement of concurrence. The physical model is numerically solved
in order to obtain the thermal density operator, which allows the evaluation
of the quantum information quantifiers. The introduction of an external
stimulus induces a quantum-level crossing on the populations of the system due
to the competition between the energetic contribution of the stimulus and the
Coulomb coupling of the system. The results have shown that this competition
plays a pivotal role in the quantum properties of the system, leading to the
enhancement of the entanglement and the coherence for $0<\omega<\mathcal{V}$.
In this regard, the transition frequency can be tuned in order to handle the
quantum properties of the two coupled DQDs systems. These results provide
insight into the quantum properties of DQDs and contribute to a better
understanding of these properties leading to promising applications in
semiconductor nanostructures such as double quantum dots.

## Acknowledgement

ZD would like to thank the Abdus Salam International Centre for Theoretical
Physics (ICTP) for their hospitality and access to their research facilities
during his stay, which assisted in part of this work. Maron F. Anka thanks
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) -
Finance Code 001, and the Carlos Chagas Filho Foundation for Research Support
of Rio de Janeiro (FAPERJ) for financial support. MR acknowledges support from
National Council for Scientific and Technological Development (CNPq) - Grant
No. 317324/2021-7.

## References

  * [1] C. H. Bennett and G. Brassard, “Quantum cryptography: Public key distribution and coin tossing,” arXiv preprint arXiv:2003.06557, 2020.
  * [2] A. K. Ekert, “Quantum Cryptography and Bell’s Theorem,” in Quantum Measurements in Optics, pp. 413–418, Springer, 1992.
  * [3] C. Cruz and M. Anka, “Quantifying quantum coherence in a metal-silicate framework,” EPL (Europhysics Letters), vol. 130, no. 3, p. 30006, 2020\.
  * [4] C. Cruz, M. F. Anka, M. S. Reis, R. Bachelard, and A. C. Santos, “Quantum battery based on quantum discord at room temperature,” Quantum Science and Technology, vol. 7, no. 2, p. 025020, 2022.
  * [5] Y.-Y. Zhang, T.-R. Yang, L. Fu, and X. Wang, “Powerful harmonic charging in a quantum battery,” Physical Review E, vol. 99, no. 5, p. 052106, 2019.
  * [6] C. H. Bennett, G. Brassard, C. Crépeau, R. Jozsa, A. Peres, and W. K. Wootters, “Teleporting an unknown quantum state via dual classical and Einstein-Podolsky-Rosen channels,” Physical Review Letters, vol. 70, no. 13, p. 1895, 1993.
  * [7] D. Bouwmeester, J.-W. Pan, K. Mattle, M. Eibl, H. Weinfurter, and A. Zeilinger, “Experimental quantum teleportation,” Nature, vol. 390, no. 6660, pp. 575–579, 1997.
  * [8] A. Keet, B. Fortescue, D. Markham, and B. C. Sanders, “Quantum secret sharing with qudit graph states,” Physical Review A, vol. 82, no. 6, p. 062315, 2010.
  * [9] M. Mansour and Z. Dahbi, “Quantum secret sharing protocol using maximally entangled multi-qudit states,” International Journal of Theoretical Physics, vol. 59, no. 12, pp. 3876–3887, 2020.
  * [10] C. Cruz, D. O. Soares-Pinto, P. B. ao, A. M. dos Santos, and M. S. Reis, “Carboxylate-based molecular magnet: One path toward achieving stable quantum correlations at room temperature,” EPL (Europhysics Letters), vol. 113, no. 4, p. 40004, 2016.
  * [11] S. Donsa, S. Andergassen, and K. Held, “Double quantum dot as a minimal thermoelectric generator,” Physical Review B, vol. 89, no. 12, p. 125103, 2014.
  * [12] J. L. D. de Oliveira, M. Rojas, and C. Filgueiras, “Two coupled double quantum-dot systems as a working substance for heat machines,” Physical Review E, vol. 104, no. 1, p. 014149, 2021.
  * [13] S. Elghaayda, Z. Dahbi, and M. Mansour, “Local quantum uncertainty and local quantum Fisher information in two-coupled double quantum dots,” Optical and Quantum Electronics, vol. 54, no. 7, pp. 1–15, 2022.
  * [14] M. Ferreira, O. Rojas, and M. Rojas, “Thermal entanglement and quantum coherence of a single electron in a double quantum dot with Rashba Interaction,” arXiv preprint arXiv:2203.06301, 2022.
  * [15] P. Majek and I. Weymann, “Majorana mode leaking into a spin-charge entangled double quantum dot,” Physical Review B, vol. 104, no. 8, p. 085416, 2021\.
  * [16] T. Zhang, H. Liu, F. Gao, G. Xu, K. Wang, X. Zhang, G. Cao, T. Wang, J. Zhang, X. Hu, et al., “Anisotropic g-factor and spin–orbit field in a germanium hut wire double quantum dot,” Nano Letters, vol. 21, no. 9, pp. 3835–3842, 2021.
  * [17] F. Ginzel, A. R. Mills, J. R. Petta, and G. Burkard, “Spin shuttling in a silicon double quantum dot,” Physical Review B, vol. 102, no. 19, p. 195418, 2020.
  * [18] X. Dai, Y. Deng, X. Peng, and Y. Jin, “Quantum-dot light-emitting diodes for large-area displays: towards the dawn of commercialization,” Advanced materials, vol. 29, no. 14, p. 1607022, 2017.
  * [19] R. C. Flagan, H. A. Atwater, and K. J. Vahala, “Fabrication of semiconductor quantum dots,” Journal of Aerosol Science, vol. 22, no. S1, pp. S31–S33, 1991.
  * [20] F. Troiani, U. Hohenester, and E. Molinari, “Exploiting exciton-exciton interactions in semiconductor quantum dots for quantum-information processing,” Physical Review B, vol. 62, no. 4, p. R2263, 2000.
  * [21] J. H. Reina, L. Quiroga, and N. F. Johnson, “Quantum entanglement and information processing via excitons in optically driven quantum dots,” Physical Review A, vol. 62, no. 1, p. 012305, 2000.
  * [22] F. Souza, P. Oliveira, and L. Sanz, “Quantum entanglement driven by electron-vibrational mode coupling,” Physical Review A, vol. 100, no. 4, p. 042309, 2019.
  * [23] C. Filgueiras, O. Rojas, and M. Rojas, “Thermal entanglement and correlated coherence in two coupled double quantum dots systems,” Annalen der Physik, vol. 532, no. 8, p. 2000207, 2020.
  * [24] B. C. Hiesmayr, “Free versus bound entanglement, a NP-hard problem tackled by machine learning,” Scientific Reports, vol. 11, no. 1, pp. 1–13, 2021\.
  * [25] J. Roik, K. Bartkiewicz, A. Černoch, and K. Lemr, “Entanglement quantification from collective measurements processed by machine learning,” arXiv preprint arXiv:2203.01607, 2022.
  * [26] C. Cruz, Á. Alves, R. dos Santos, D. Soares-Pinto, J. de Jesus, J. de Almeida, and M. Reis, “Influence of the external pressure on the quantum correlations of molecular magnets,” EPL (Europhysics Letters), vol. 117, no. 2, p. 20004, 2017.
  * [27] C. Castro, M. R. Araújo, and C. Cruz, “Entanglement dynamics of a dc SQUID interacting with a single-mode radiation field,” Physica Scripta, vol. 96, no. 10, p. 105101, 2021.
  * [28] A. R. Carvalho, F. Mintert, and A. Buchleitner, “Decoherence and multipartite entanglement,” Physical Review Letters, vol. 93, no. 23, p. 230501, 2004\.
  * [29] F. Mintert, M. Kuś, and A. Buchleitner, “Concurrence of mixed bipartite quantum states in arbitrary dimensions,” Physical Review Letters, vol. 92, no. 16, p. 167902, 2004.
  * [30] P. J. Love, A. M. Van Den Brink, A. Y. Smirnov, M. H. Amin, M. Grajcar, E. Il’ichev, A. Izmalkov, and A. M. Zagoskin, “A characterization of global entanglement,” Quantum Information Processing, vol. 6, no. 3, pp. 187–195, 2007.
  * [31] D. A. Meyer and N. R. Wallach, “Global entanglement in multiparticle systems,” Journal of Mathematical Physics, vol. 43, no. 9, pp. 4273–4278, 2002.
  * [32] G. K. Brennen, “An observable measure of entanglement for pure states of multi-qubit systems,” arXiv preprint quant-ph/0305094, 2003.
  * [33] Q.-Q. Guo, X.-Y. Chen, and Y.-Y. Wang, “Measures of genuine multipartite entanglement for graph states,” Chinese Physics B, vol. 23, no. 5, p. 050309, 2014.
  * [34] A. J. Scott, “Multipartite entanglement, quantum-error-correcting codes, and entangling power of quantum evolutions,” Physical Review A, vol. 69, no. 5, p. 052330, 2004.
  * [35] M. Mansour, Z. Dahbi, M. Essakhi, and A. Salah, “Quantum correlations through spin coherent states,” International Journal of Theoretical Physics, vol. 60, no. 6, pp. 2156–2174, 2021.
  * [36] M. Mansour, M. Daoud, and Z. Dahbi, “Randomized entangled mixed states from phase states,” International Journal of Theoretical Physics, vol. 59, no. 3, pp. 895–907, 2020.
  * [37] H. Ollivier and W. H. Zurek, “Quantum discord: a measure of the quantumness of correlations,” Physical Review Letters, vol. 88, no. 1, p. 017901, 2001\.
  * [38] L. Henderson and V. Vedral, “Classical, quantum and total correlations,” Journal of Physics A: Mathematical and General, vol. 34, no. 35, p. 6899, 2001.
  * [39] Z. Xi, Y. Li, and H. Fan, “Quantum coherence and correlations in quantum system,” Scientific reports, vol. 5, no. 1, pp. 1–9, 2015.
  * [40] A. Streltsov, U. Singh, H. S. Dhar, M. N. Bera, and G. Adesso, “Measuring quantum coherence with entanglement,” Physical Review Letters, vol. 115, no. 2, p. 020403, 2015.
  * [41] X. Yuan, G. Bai, T. Peng, and X. Ma, “Quantum uncertainty relation using coherence,” Physical Review A, vol. 96, no. 3, p. 032313, 2017.
  * [42] M.-L. Hu, X. Hu, J. Wang, Y. Peng, Y.-R. Zhang, and H. Fan, “Quantum coherence and geometric quantum discord,” Physics Reports, vol. 762, pp. 1–100, 2018.
  * [43] X. Yuan, G. Bai, T. Peng, and X. Ma, “Quantum uncertainty relation using coherence,” Physical Review A, vol. 96, no. 3, p. 032313, 2017.
  * [44] T. Baumgratz, M. Cramer, and M. B. Plenio, “Quantifying coherence,” Physical Review Letters, vol. 113, no. 14, p. 140401, 2014.
  * [45] D. Girolami, “Observable measure of quantum coherence in finite dimensional systems,” Physical Review Letters, vol. 113, no. 17, p. 170401, 2014.
  * [46] T. Kraft and M. Piani, “Genuine correlated coherence,” Journal of Physics A: Mathematical and Theoretical, vol. 51, no. 41, p. 414013, 2018.
  * [47] K. C. Tan, H. Kwon, C.-Y. Park, and H. Jeong, “Unified view of quantum correlations and quantum coherence,” Physical Review A, vol. 94, no. 2, p. 022329, 2016.
  * [48] K. C. Tan and H. Jeong, “Entanglement as the symmetric portion of correlated coherence,” Physical Review Letters, vol. 121, no. 22, p. 220401, 2018\.
  * [49] J. Ma, B. Yadin, D. Girolami, V. Vedral, and M. Gu, “Converting coherence to quantum correlations,” Physical Review Letters, vol. 116, no. 16, p. 160407, 2016.
  * [50] G. Shinkai, T. Hayashi, T. Ota, and T. Fujisawa, “Correlated coherent oscillations in coupled semiconductor charge qubits,” Physical Review Letters, vol. 103, no. 5, p. 056802, 2009.
  * [51] A. Ablimit, D. Hitjan, and A. Abliz, “Non-Markovian Dynamics of Geometric Quantum Discord in a Double Quantum Dot System,” Journal of Low Temperature Physics, vol. 205, no. 3, pp. 126–134, 2021.
  * [52] J. Taylor, J. Petta, A. Johnson, A. Yacoby, C. Marcus, and M. Lukin, “Relaxation, dephasing, and quantum control of electron spins in double quantum dots,” Physical Review B, vol. 76, no. 3, p. 035315, 2007.
  * [53] T. Fujisawa, T. Hayashi, and S. Sasaki, “Time-dependent single-electron transport through quantum dots,” Reports on Progress in Physics, vol. 69, no. 3, p. 759, 2006.
  * [54] S. A. Hill and W. K. Wootters, “Entanglement of a pair of quantum bits,” Physical Review Letters, vol. 78, no. 26, p. 5022, 1997.
  * [55] W. K. Wootters, “Entanglement of formation and concurrence.,” Quantum Inf. Comput., vol. 1, no. 1, pp. 27–44, 2001.


# Effective Multi-Agent Deep Reinforcement Learning Control with
Relative Entropy Regularization

Chenyang Miao1,2, Yunduan Cui2,∗, Huiyun Li2 and Xinyu Wu2 This research is
supported in part by the National Natural Science Foundation of China under
Grants 62103403; in part by Guangdong Basic and Applied Basic Research
Foundation under Grant 2020B515130004.1 University of Chinese Academy of
Sciences, China2 Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine
Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, China.∗ Corresponding author: Yunduan
Cui (e-mail<EMAIL_ADDRESS>
###### Abstract

In this paper, a novel Multi-agent Reinforcement Learning (MARL) approach,
Multi-Agent Continuous Dynamic Policy Gradient (MACDPP) was proposed to tackle
the issues of limited capability and sample efficiency in various scenarios
controlled by multiple agents. It alleviates the inconsistency of multiple
agents’ policy updates by introducing the relative entropy regularization to
the Centralized Training with Decentralized Execution (CTDE) framework with
the Actor-Critic (AC) structure. Evaluated by multi-agent cooperation and
competition tasks and traditional control tasks including OpenAI benchmarks
and robot arm manipulation, MACDPP demonstrates significant superiority in
learning capability and sample efficiency compared with both related multi-
agent and widely implemented signal-agent baselines and therefore expands the
potential of MARL in effectively learning challenging control scenarios. The
open source code of MACDPP is available at
https://github.com/AdrienLin1/MACDPP.

##  I Introduction

Guided by task-related reward functions, Reinforcement Learning (RL) provides
an effective solution to autonomously explore and gradually learn optimal or
near-optimal control strategies by iteratively interacting with the
environment in the absence of task-specific prior knowledge [1, 2]. Utilizing
the power of deep neural networks [3] to adapt abstract features from high-
dimensional input states, RL has demonstrated superior performances than
humans in various complex scenarios including board games [4], video games
[5], and robot control [6]. Based on the successful implementations of single-
agent RL approaches, people naturally attempt to develop Multi-agent
Reinforcement Learning (MARL) to effectively explore optimal control policies
of large-scale systems and achieve promising results in a wide range of tasks
[7, 8, 9, 10, 11, 12, 13, 14]. On the other hand, transferring RL from single-
agent environments to multi-agent environments raises a new challenge: the
environments affected by the joint actions from multiple agents become non-
stationary, and each agent faces a moving-target problem while its optimal
strategy strongly depends on the frequently changing policies of other agents.
This characteristic not only breaks the Markov property of the environment but
also greatly compromises the learning capability and converge velocity of
traditional RL approaches designed for single agent [15].

Compared with the approaches like Independent Q-Learning (IQL) [16] that
directly implemented single-agent RL approaches in multi-agent scenarios to
separately explore independent polices [17, 18], MARL methods based on the
Centralized Training with Decentralized Execution (CTDE) framework provides an
appealing prospect for addressing the issue of non-stationary [19]. It enables
the multiple agents to learn decentralized policies in a centralized end-to-
end fashion: all agents are accessible to the global information during the
training stage while the decision-making of each agent is independent during
the interaction with the environment. From the perspective of the value
function, Value-Decomposition Networks (VDN) [20] decomposed the value
function for multiple agents under a CTDE framework and achieved better
cooperation behaviors in a simulation maze environment. QMIX [21] further
proposed a network-based mixture strategy for multiple agents’ value function
based on VDN and enjoyed astonishing results performances in StarCraft Multi-
Agent Challenge (SMAC) [22]. Based on the policy-based RL approaches, Multi-
agent Proximal Policy Optimization (MAPPO) [23] demonstrated better
performances than traditional MARL methods in both Multi-Agent Particle
Environment (MPE) and SMAC. Employing an Actor-Critic (AC) structure, Multi-
Agent Deep Deterministic Policy Gradient (MADDPG) [24] combines the strengths
of both value function-based and policy-based approaches and has achieved good
learning performances in both cooperative and competitive tasks. Based on this
approach, Multi-agent TD3 (MATD3) [25] tackled the issue of overestimated
value function with additional critic networks. The minimax algorithm was
further introduced by MiniMax Multi-agent Deep Deterministic Policy Gradient
(M3DDPG) [26] for enhanced learning capability in both cooperative and
competitive tasks. As demonstrated above, CTDE-based MARL approaches have
improved the learning capability of agents from the perspective of structure.
However, at the algorithmic level, the inherent inconsistency of multiple
agents’ policy updates and the resulting deterioration of learning performance
has not been sufficiently addressed.

By incorporating the relative entropy between the current and previous
policies as a regularization term into the value function, Dynamic Policy
Programming (DPP) [27] effectively constrains excessive policy updates in
single-agent environments. Theoretically, DPP significantly reduces the
estimated error of the value function with superior error bounds [28, 29]. In
engineering applications of single-agent scenarios, DPP has demonstrated
superior sample efficiency and robustness in several robot control tasks [30,
31]. As one of the pioneering works applying relative entropy regularized RL
to multi-agent scenarios, Factorial Kernel Dynamic Policy Programming (FKDPP)
[32] was proposed to control large-scale chemical plants with multiple DPP
agents. It outperformed the control strategy designed by human experts in
production rate, profit, and safety on a simulated Vinyl Acetate Monomer (VAM)
plant [33] and has been successfully implemented in a real-world chemical
plant for 35 days 111This implementation of FKDPP was conducted by Yokogawa
Electric Corporation and JSR Corporation. For more details, please see:
https://www.yokogawa.com/news/press-releases/2022/2022-03-22/.. Although this
work fully indicates the great potential of relative entropy regularized MARL
in real-world systems, FKDPP was developed based on neither deep neural
networks nor CTDE framework and was limited in discrete action space without
supporting the AC structure. These characteristics restrict its application
scope in more challenging and flexible control scenarios.

TABLE I: Comparison of MACDPP and the related MARL baselines

Approach |  Relative Entropy |  AC |  CTDE |  Cooperative & Competitive
---|---|---|---|---
IQL [16] | $\times$ | $\times$ | $\times$ | $\bigcirc$
QMIX [21] | $\times$ | $\times$ | $\times$ | $\times$
MAPPO [23] | $\times$ | $\times$ | $\bigcirc$ | $\times$
MADDPG [24] | $\times$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
M3DDPG [26] | $\times$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
MATD3 [25] | $\times$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
FKDPP [32, 33] | $\bigcirc$ | $\times$ | $\times$ | $\times$
MACDPP (ours) | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$

This paper focuses on integrating the relative entropy regularization to the
modern MARL under the CTDE framework in order to alleviate the inconsistency
of multiple agents’ policy updates in various control scenarios. According to
the characteristics compared with MARL baselines in Table I, our proposed
approach, Multi-Agent Continuous Dynamic Policy Gradient (MACDPP)222Code
available https://github.com/AdrienLin1/MACDPPnaturally extends the power of
FKDPP from kernel-based value function approximation and discrete action space
to the CTDE framework with AC structure with superior learning capability and
sample-efficiency. MACDPP reduces the intractable computational burden of
FKDPP in the actor network with continuous actions by Monte Carlo sampling and
naturally obtains a superior exploration strategy based on the Boltzmann
softmax operator. Evaluated by both cooperation and competition tasks in MPE
environment and OpenAI benchmark control task where multiple agents
collaborate to control one high-dimensional system, the proposed method
successfully demonstrated superiority in both learning capability and sample-
efficiency compared with various multi-agent and signal-agent RL baselines.
The contributions of this paper can be summarized as:

  1. 1.

Our work first attempts to integrate relative entropy regularization into the
CTDE framework-based MARL to address the inherent inconsistency of policy
updates for multiple agents at an algorithmic level. We propose a novel MARL
approach that is compatible with both cooperative and competitive tasks in a
multi-agent scenario, as well as single systems collaboratively controlled by
multiple agents.

  2. 2.

As one natural extension of previous works [32, 33] with successful
engineering applications to not only the deep neural networks function
approximator but also the AC structure with continuous action space, the
proposed MACDPP can be seen as a comprehensive upgrade to FKDPP, targeting
enhanced learning capability and control flexibility.

  3. 3.

The proposed method was evaluated by several benchmarks from MPE to
traditional control tasks in terms of the learning capability and sample
efficiency compared to both related CTDE framework-based MARL and widely-
applied single-agent RL approaches. We further analyzed the impact of relative
entropy regularization in the CTDE framework-based MARL on convergence and
control behaviors.

The remainder of this paper is organized as follows. Section II introduces the
preliminaries of Markov games, MARL, and CTDE framework. Section III details
the proposed approach MACDPP. The experimental results are presented in
Section IV. Finally, Section V concludes this paper.

##  II Preliminaries

###  II-A Markov Games

Markov games are widely utilized to model a multi-agent environment satisfying
partially observable Markov processes (POMDP). It is generally defined by a
sextuple $(N,\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$. $N$
represents the number of agents in the target environment, $\mathcal{S}$
defines the general state space. The locally observed state of the $i$-th
agent is denoted as $\bm{s}_{i}$ which is a subset of the global observed
state $\bm{s}\in\mathcal{S}$. The joint action is made up of the local actions
from all agents $\bm{a}=\bm{a}_{1}\times\cdots\times\bm{a}_{N}\in\mathcal{A}$.
The subspace of each agent’s action is presented as $\mathcal{A}_{i}$. The
state transition probability over all agents is presented as $\mathcal{P}$.
$\mathcal{R}$ is a set of reward functions for specific tasks, and each agent
has its own reward function
$R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})\in\mathcal{R}$ based on its
local state, action and the next step state. The discount factor
$\gamma\in[0,1)$ is utilized to gradually ignore the accumulative rewards in a
long-term horizon.

Based on the Markov games, MARL introduces the value function $V_{i,\pi_{i}}$
and the state-action value function $Q_{i,\pi_{i}}$ to measure the long-term
accumulative rewards obtained by the $i$-th agent under its policy:

$\displaystyle\begin{split}V_{i,\pi_{i}}(\bm{s}_{i})\\!=\\!\mathbb{E}_{\begin{subarray}{c}\bm{s}_{t+1}\sim\mathcal{P},\\\
\bm{a}_{i,t}\sim\pi_{i}\end{subarray}}\\!\\!\left[\sum_{t=0}^{\infty}\gamma^{t}R_{i}(\bm{s}_{i,t},\bm{a}_{i,t},\bm{s}_{i,t+1})\mid\bm{s}_{i,0}=\bm{s}_{i}\right],\end{split}$
(1)
$\displaystyle\begin{split}Q_{i,\pi_{i}}(\bm{s}_{i},\bm{a}_{i})\\!=\\!\mathbb{E}_{\begin{subarray}{c}\bm{s}_{t+1}\sim\mathcal{P},\\\
\bm{a}_{i,t}\sim\pi_{i}\end{subarray}}\\!\\!\left[\sum_{t=0}^{\infty}\bm{\gamma}^{t}R_{i}(\bm{s}_{i,t},\bm{a}_{i,t},\bm{s}_{i,t+1})\mid\begin{subarray}{c}\bm{s}_{i,0}=\bm{s}_{i},\\\
\bm{a}_{i,0}=\bm{a}_{i}\end{subarray}\right]\end{split}$ (2)

where the global state in the next time step $\bm{s}_{t+1}$ is determined by
the current global state $\bm{s}_{t}$ and action $\bm{a}_{t}$ under
$\mathcal{P}$. Define $\mathcal{P}_{\bm{s}\bm{s}^{\prime}}^{\bm{a}}$ as the
probability of translating from state $\bm{s}$ to state $\bm{s}^{\prime}$
under action $\bm{a}$ in a global perspective, the goal of each agent in MARL
is to learn an optimal control policy to maximize its optimal value function
following a Bellman equation:

$\displaystyle
V^{*}_{i}(\bm{s}_{i})=\max_{{\pi}_{i}}\sum_{\bm{a}\in\mathcal{A}\atop\bm{s}^{\prime}\in\mathcal{S}}\pi_{i}(\bm{a}_{i}|\bm{s}_{i})\mathcal{P}_{\bm{s}\bm{s}^{\prime}}^{\bm{a}}\left(R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})+\gamma{V^{*}_{i}}(\bm{s}^{\prime}_{i})\right).$
(3)

###  II-B Multi-agent Reinforcement Learning in CTDE framework

In the CTDE-framework based MARL like MADDPG [24], the AC structure is
implemented to separately estimate the state-action value function and model
control policy by critic network $\hat{Q}_{i}(\cdot,\bm{\theta}_{i})$ and
actor network $\hat{\pi}_{i}(\cdot,\bm{\phi}_{i})$ for each agent where
$\bm{\theta}_{i}$ and $\bm{\phi}_{i}$ are the corresponding parameters. In the
centralized training process, All critic networks are globally updated with
the shared observation information. Define one global training sample from
sample set $\mathcal{D}$ as $(\bm{s},\bm{a},\bm{s}^{\prime},\bm{r})$ where
$\bm{r}=[R_{1}(\bm{s}_{1},\bm{a}_{1},\bm{s}_{1}^{\prime}),...,R_{N}(\bm{s}_{N},\bm{a}_{N},\bm{s}_{N}^{\prime})]$
is the vector of reward signal for all agents, the $i$-th agent’s critic
networks receive the states and actions from all agents
$\hat{Q}_{i}(\bm{s},\bm{a},\bm{\theta}_{i})$ and measure its own long-term
reward of $R_{i}(\bm{s}_{i},\bm{a}_{i})$. Determining the global action of the
next step by all agents according to their local states
$\bm{a}_{i}^{\prime}=\hat{\pi}_{i}(\bm{s}_{i}^{\prime},\bm{\phi}_{i}),\forall
i=1,...,N$, the corresponding Temporal-Difference (TD) errors that guide the
update of critic networks in gradient descent optimization is calculated
following:

$\displaystyle\begin{split}\mathcal{L}(\bm{\theta}_{i})=\left(\hat{Q}_{i}(\bm{s},\bm{a},\bm{\theta}_{i})-R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})-\gamma\hat{Q}_{i}(\bm{s}^{\prime},\bm{a}^{\prime},\bm{\theta}_{i})\right)^{2}.\end{split}$
(4)

The actor networks are updated to maximize the returns of the current critic
networks based on the local observation of each agent. The corresponding
gradient of error is defined as:

$\displaystyle\begin{split}\nabla_{\bm{\phi_{i}}}\leftarrow\nabla_{\bm{a}^{*}_{i}=\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})}\hat{Q}_{i}(\bm{s},\bm{a}^{*},\bm{\theta}_{i})\nabla_{\bm{\phi}_{i}}\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})\end{split}$
(5)

where $\bm{a}^{*}$ is selected by all agents with shared information. In the
decentralized execution process, on the other hand, the control actions of
each agent are determined by the actor only without the consideration of other
agents.

##  III Approach

In this section, the proposed method MACDPP was detailed. It naturally
extended the relative entropy regularization term from DPP [27] and FKDPP [32]
to the modern MARL under the CTDE framework and AC structure. The multi-agent
critic networks regularized by relative entropy were introduced in Section
III-A, and the corresponding actor that supports continuous actions was
introduced in Section III-B. The factorization strategy of MACDPP for
cooperative and competitive tasks was discussed in Section III-C with a
summary of MACDPP’s learning procedure.

###  III-A Relative Entropy Regularized Critics

Following the existing relative entropy regularized RL approaches [27, 28],
the difference between the current policy $\pi_{i}$ and previous policy
$\bar{\pi}_{i}$ of the $i$-th agent on state $\bm{s}_{i}$ was defined as:

$\displaystyle\mathbb{D}_{\mathrm{KL}}(\bm{s}_{i})=\sum_{\bm{a}\in\mathcal{A}}\pi_{i}(\bm{a}_{i}|\bm{s}_{i})\log\left(\frac{\pi_{i}(\bm{a}_{i}|\bm{s}_{i})}{\bar{\pi}_{i}(\bm{a}_{i}|\bm{s}_{i})}\right).$
(6)

This term was then incorporated into the value function as a regularization
term controlled by a parameter $\eta$:

$\displaystyle
V_{i}^{\prime}(\bm{s}_{i})=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(R_{i}(\bm{s}_{i,t},\bm{a}_{i,t},\bm{s}_{i,t+1})-\frac{1}{\eta}\mathbb{D}_{\mathrm{KL}}(\bm{s}_{i,t})\right)\right].$
(7)

Combining Eqs.(3) and (7), the resulted optimal value function was still a
Bellman equation with an additional term
$-\frac{1}{\eta}\mathbb{D}_{\mathrm{KL}}(\bm{s}_{i,t})$ in Eq. (3). Assume the
action of each agent $\bm{a}_{i}\in\mathcal{A}_{i}$ is discrete, an iterative
update form of both value function and policy can be found based on DPP [27]:

$\displaystyle
V_{i,t+1}^{\prime}(\bm{s}_{i})=\frac{1}{\eta}\log\sum_{\bm{a}_{i}\in\mathcal{A}_{i}}\exp\left(\eta\cdot\Psi_{i,t}\left(\bm{s}_{i},\bm{a}_{i}\right)\right),$
(8)
$\displaystyle\bar{\pi}_{i,t+1}(\bm{a}_{i}|\bm{s}_{i})=\frac{\exp\left(\eta\cdot\Psi_{i,t}\left(\bm{s}_{i},\bm{a}_{i}\right)\right)}{\sum_{\bm{a}_{i}^{\prime}\in\mathcal{A}_{i}}\exp\left(\eta\cdot\Psi_{i,t}\left(\bm{s}_{i},\bm{a}_{i}^{\prime}\right)\right)}$
(9)

where $t$ is the iteration of update, $\Psi_{i,t}(\cdot)$ is the action
preferences function [1] which can be treat as a regularized Q function:

$\displaystyle\begin{aligned}
\Psi_{i,t}(\bm{s}_{i},\bm{a}_{i})=&\sum_{\bm{s}^{\prime}\in\mathcal{S}}\mathcal{P}_{\bm{s}\bm{s}^{\prime}}^{\bm{a}}\left(R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})+\gamma
V_{i,t}^{\prime}\left(\bm{s}^{\prime}_{i}\right)\right)\\\
&+\frac{1}{\eta}\log\bar{\pi}_{i,t}(\bm{a}_{i}|\bm{s}_{i}).\end{aligned}$ (10)

With a discrete action space, once the critic network accurately estimates the
action preferences function $\Psi_{i}(\cdot)$, both the value function and
policy can be directly calculated. In practice, the transition probability
matrix $\mathcal{P}$ is usually too large and inaccessible. DPP proposed an
update rule of $\Psi_{i}(\cdot)$ based on sampling by inserting Eqs (8) and
(9) into Eq. (10). Given a sample $(\bm{s},\bm{a},\bm{s}^{\prime},\bm{r})$,
the action preferences function of the $i$-th agent was updated following:

$\displaystyle\begin{aligned}
\Psi_{i,t+1}(\bm{s}_{i},\bm{a}_{i})=&\Psi_{i,t}(\bm{s}_{i},\bm{a}_{i})-\mathcal{B}_{\eta}\Psi_{i,t}(\bm{s}_{i})\\\
&+R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})+\mathcal{B}_{\eta}\Psi_{i,t}(\bm{s}_{i}^{\prime})\end{aligned},$
(11)
$\displaystyle\mathcal{B}_{\eta}\Psi_{i,t}(\bm{s}_{i})=\sum_{\bm{a}_{i}\in\mathcal{A}_{i}}\frac{\exp\left(\eta\cdot\Psi_{i,t}(\bm{s}_{i},\bm{a}_{i})\right)\Psi_{i,t}(\bm{s}_{i},\bm{a}_{i})}{\sum_{\bm{a}^{\prime}_{i}\in\mathcal{A}_{i}}\exp\left(\eta\cdot\Psi_{i,t}\left(\bm{s}_{i},\bm{a}^{\prime}_{i}\right)\right)}.$
(12)

$\mathcal{B}_{\eta}(\cdot)$ is a Boltzmann softmax operator.

It is straightforward to estimate the action preferences function instead of
the Q function by the critic neural networks under CTDE framework when the
action is discrete. Let the critic networks receive the global information of
all agents, the loss function of the critic networks
$\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}_{i})$ is calculated by integrating
Eq. (11) into Eq. (4):

$\displaystyle\begin{split}\mathcal{L}(\bm{\theta}_{i})=\left(\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}_{i})-y(\bm{s},\bm{a},\bm{s}_{i}^{\prime},\bm{\theta}_{i}^{-})\right)^{2},\end{split}$
(13)
$\displaystyle\begin{split}y(\bm{s},\bm{a},\bm{s}^{\prime},\bm{\theta}_{i}^{-})&=R_{i}(\bm{s}_{i},\bm{a}_{i},\bm{s}_{i}^{\prime})+\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}_{i}^{-})\\\
&-\mathcal{B}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})+\gamma\mathcal{B}_{\eta}\hat{\Psi}_{i}(\bm{s}^{\prime},\bm{\theta}_{i}^{-})\end{split}$
(14)

Where $\bm{\theta}_{i}^{-}$ is the parameters of the corresponding target
networks. The Boltzmann softmax operator is conducted over the global action
of all agents:

$\displaystyle\mathcal{B}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})=\sum_{\bm{a}\in\mathcal{A}}\frac{\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}_{i}^{-})\right)\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}_{i}^{-})}{\sum_{\bm{a}^{\prime}\in\mathcal{A}}\exp\left(\eta\cdot\hat{\Psi}_{i}\left(\bm{s},\bm{a}^{\prime},\bm{\theta}_{i}^{-}\right)\right)}.$
(15)

However, the application of the relative entropy regularization in the AC
structure remains limited due to the intractable calculation over the whole
continuous action space in $\mathcal{B}_{\eta}(\cdot)$. We detailed our
solution in the next subsection.

###  III-B Actors with Boltzmann Softmax Sampling

To effectively calculate Eq. (15) in continuous action space, we estimated it
within a local range of the input action in MACDPP. The global action $\bm{a}$
was extended to a vector with $M+1$ Monte Carlo samples:

$\displaystyle\begin{split}\mathcal{A}^{\text{MC}}=[\bm{a},\bm{a}+\bm{e}^{1},\bm{a}+\bm{e}^{2},...,\bm{a}+\bm{e}^{M}]\end{split}$
(16)

where $\bm{e}^{j}\sim\mathcal{N}(\bm{0},\bm{\zeta}^{\text{MC}})$ for
$j=1,...,M$ is the Monte Carlo sampling noise controlled by
$\bm{\zeta}^{\text{MC}}$. The locally estimation of Eq. (12) therefore was
calculated as:

$\displaystyle\mathcal{B}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})\approx\sum_{\bm{a}\in\mathcal{A}^{\text{MC}}}\frac{\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}^{-}_{i})\right)\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}^{-}_{i})}{\sum_{\bm{a}^{\prime}\in\mathcal{A}^{\text{MC}}}\exp\left(\eta\cdot\hat{\Psi}_{i}\left(\bm{s},\bm{a}^{\prime},\bm{\theta}_{i}^{-}\right)\right)}$
(17)

According to [34], one critical issue of the Boltzmann softmax operator in RL
is its multiple fixed points without non-expansion property which guarantees
the convergence of ”Q-learning like” algorithms to a unique fixed point in
theory. One effective solution is to replace it with the Mellowmax operator
with a unique fix point and non-expansion property:

$\displaystyle\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})\approx\frac{1}{\eta}\log\left(\frac{\sum_{\bm{a}\in\mathcal{A}^{\text{MC}}}\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}^{-}_{i})\right)}{M+1}\right).$
(18)

Algorithm 1 summarized the calculation of
$\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})$. In practice,
numerical issues usually arise in Eq. (18) with a large $\eta$. We
alternatively calculated it following:

$\displaystyle\begin{split}&\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})\approx\\\
&\frac{1}{\eta}\log\left(\frac{\sum_{\bm{a}\in\mathcal{A}^{\text{MC}}}\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}^{-}_{i})-C\right)}{M+1}\right)+C\end{split}$
(19)

where
$C=\eta\cdot\max_{\bm{a}^{\prime}\in\mathcal{A}^{\text{MC}}}[\hat{\Psi}_{i}(\bm{s},\bm{a}^{\prime},\bm{\theta}^{-}_{i})]$.

Employing the Monte Carlo sampling to estimate
$\mathcal{M}_{\eta}\hat{\Psi}_{i}(\cdot)$, any policy network maps the local
states to the local actions can be used as an actor. In the decentralized
execution process, the trained agent made decisions based on its own actor
with local observation
$\bm{a}_{i}^{*}=\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})$ The actor was updated
in the centralized training following the gradient below:

$\displaystyle\begin{split}\nabla_{\bm{\phi_{i}}}\leftarrow\nabla_{\bm{a}^{*}_{i}=\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})}\hat{\Psi}_{i}(\bm{s},\bm{a}^{*},\bm{\theta}_{i})\nabla_{\bm{\phi}_{i}}\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})\end{split}$
(20)

where the $\bm{a}^{*}$ was jointly calculated by all agents’ policies.

Function  _MC_Estimate(_$\bm{s},\bm{a},\bm{\theta}_{i}$_)_:

# Build Monte Carlo sample set

for  _$m=1$ to $M$_ do

$\bm{e}^{m}\sim\mathcal{N}(\bm{0},\bm{\zeta})$

$\mathcal{A}^{\text{MC}}=[\bm{a},\bm{a}+\bm{e}^{1},\bm{a}+\bm{e}^{2},...,\bm{a}+\bm{e}^{M}]$

# Locally estimate Mellowmax operation

$\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s},\bm{\theta}_{i}^{-})\approx\frac{1}{\eta}\log\left(\frac{\sum_{\bm{a}\in\mathcal{A}^{\text{MC}}}\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a},\bm{\theta}^{-}_{i})-C\right)}{M+1}\right)+C$

Return $\mathcal{M}_{\eta}\hat{\Psi}_{\text{target}}(\bm{s},\bm{\theta}^{-})$

Algorithm 1 Estimation by Monte Carlo Sampling

Function  _BS_Sampling(_$\bm{s},\bm{\theta}_{i}$_)_:

for  _$j=1$ to $N$_ do

$\bm{a}_{j}^{*}=\hat{\pi}_{j}(\bm{s},\bm{\phi}_{j})$

$\bm{a}^{*}=[\bm{a}_{1}^{*},...,\bm{a}_{N}^{*}]$

for  _$m=1$ to $M$_ do

$\bm{e}^{m}_{i}\sim\mathcal{N}(\bm{0},\bm{\zeta}^{\text{Sampling}}),\bm{e}^{m}_{j\neq
i}=\bm{0}$

$\bm{e}^{m}=[\bm{e}^{m}_{1},...,\bm{e}^{m}_{N}]$

$\mathcal{A}^{\text{explore}}=[\bm{a}^{*},\bm{a}^{*}+\bm{e}^{1},\bm{a}^{*}+\bm{e}^{2},...,\bm{a}^{*}+\bm{e}^{M}]$

Select $\bm{a}^{\text{explore}}\in\mathcal{A}^{\text{explore}}$ following:

$p(\bm{a}^{\text{explore}})=\frac{\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a}^{\text{explore}},\bm{\theta}^{-}_{i})\right)}{\sum_{\bm{a}^{\prime}\in\mathcal{A}^{\text{explore}}}\exp\left(\eta\cdot\hat{\Psi}_{i}\left(\bm{s},\bm{a}^{\prime},\bm{\theta}_{i}^{-}\right)\right)}$

return _$\bm{a}^{\text{explore}}_{i}$_

Algorithm 2 Exploration using Boltzmann Softmax

Unlike MADDPG which explores by directly adding noises to its actions, MACDPP
proposed an effective exploitation that naturally related to the relative
entropy regularized value function based on the shared information in the
centralized training process. Define the global action as
$\bm{a}^{*}=[\bm{a}_{1}^{*},...,\bm{a}_{N}^{*}]$ where
$\bm{a}_{i}^{*}=\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i}),i=1,...N$. An
exploration set for the $i$-th agent with $M+1$ candidates was built following
Algorithm 2:
$\mathcal{A}^{\text{explore}}=[\bm{a}^{*},\bm{a}^{*}+\bm{e}^{1},\bm{a}^{*}+\bm{e}^{2},...,\bm{a}^{*}+\bm{e}^{M}]$
where exploration noises $\bm{e}^{m}$ added Gaussian noises only to the local
actions related to the $i$-th agent. Please note that the variance of sampling
$\bm{\zeta}^{\text{Sampling}}$ which affected the decision-making of the agent
was independent of $\bm{\zeta}^{\text{MC}}$ in Algorithm. 1 which locally
estimated the Mellowmax operation. An effective exploration action was
randomly selected following the probability below:

$\displaystyle
p(\bm{a}^{\text{explore}})=\frac{\exp\left(\eta\cdot\hat{\Psi}_{i}(\bm{s},\bm{a}^{\text{explore}},\bm{\theta}^{-}_{i})\right)}{\sum_{\bm{a}^{\prime}\in\mathcal{A}^{\text{explore}}}\exp\left(\eta\cdot\hat{\Psi}_{i}\left(\bm{s},\bm{a}^{\prime},\bm{\theta}_{i}^{-}\right)\right)}.$
(21)

The $i$-th agent utilized the corresponding local action
$\bm{a}^{\text{explore}}_{i}$ in $\bm{a}^{\text{explore}}$ to interact with
the environment. Although Eq. (21) required the global actions of all agents
for an effective exploration, the execution of all agents can be decentralized
in evaluation since the deterministic local action
$\bm{a}_{i}^{*}=\hat{\pi}_{i}(\bm{s}_{i},\bm{\phi}_{i})$ was related to only
the local observation $\bm{s}_{i}$.

for  _$i=1$ to $N$_ do

Initialize buffer $D_{i}$, networks weights $\bm{\theta}_{i}$,
$\bm{\phi}_{i}$.

Copy the target networks with parameters $\bm{\theta}_{i}^{-}$,
$\bm{\phi}_{i}^{-}$.

for  _$e=1$ to $E$_ do

for  _$t=1$ to $T$_ do

# Interaction phase

Observe state $\bm{s}$

for  _$i=1$ to $N$_ do

$\bm{a}_{i}=\text{BS\\_Sampling}(\bm{s},\bm{\theta}^{-}_{i})$

Execute $\bm{a}=[\bm{a}_{1},...,\bm{a}_{N}]$

Observe next state $\bm{s}^{\prime}$ and reward $\bm{r}$

Separately store sample to $\mathcal{D}_{i},i=1,...,N$

# Centralized training phase

for  _$i=1$ to $N$_ do

Sample mini-batch of $J$ samples from $\mathcal{D}_{i}$

for  _$j=1$ to $J$_ do

# Restore information from $\mathcal{D}_{k},k\neq i$

$\bm{s}_{j}=[\bm{s}_{1,j},...,\bm{s}_{N,j}]$

$\bm{a}_{j}=[\bm{a}_{1,j},...,\bm{a}_{N,j}]$

$\bm{s}_{j}^{\prime}=[\bm{s}_{1,j}^{\prime},...,\bm{s}_{N,j}^{\prime}]$

# Calculate the next action of all agents

for  _$k=1$ to $N$_ do

$\bm{a}_{k,j}^{\prime}=\hat{\pi}_{k}(\bm{s}_{k}^{\prime},\bm{\phi}^{-}_{k})$

$\bm{a}^{\prime}_{j}=[\bm{a}_{1,j}^{\prime},...,\bm{a}_{N,j}^{\prime}]$

# Monte Carlo Estimation

$\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s}_{j})=\text{MC\\_Estimate}(\bm{s}_{j},\bm{a}_{j},\bm{\theta}_{i}^{-})$

$\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s}_{j}^{\prime})=\text{MC\\_Estimate}(\bm{s}_{j}^{\prime},\bm{a}_{j}^{\prime},\bm{\theta}_{i}^{-})$

# Calculate TD error following Eq. (14)

$y_{j}=R_{i}(\bm{s}_{i,j},\bm{a}_{i,j},\bm{s}_{i,j}^{\prime})+\gamma\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s}_{j}^{\prime})+\hat{\Psi}_{i}(\bm{s}_{j},\bm{a}_{j},\bm{\theta}^{-}_{i})-\mathcal{M}_{\eta}\hat{\Psi}_{i}(\bm{s}_{j})$

# Update critic $\hat{\Psi}_{i}(\cdot,\bm{\theta}_{i})$ and actor
$\hat{\pi}_{i}(\cdot,\bm{\phi}_{i})$

$\bm{\theta}_{i}\leftarrow\operatorname*{arg\,min}\limits_{\bm{\theta}_{i}}\frac{1}{J}\sum_{j=1}^{J}(y_{j}-\hat{\Psi}(\bm{s}_{j},\bm{a}_{i,j},\bm{\theta}_{i}))^{2}$

# Update actor $\hat{\pi}_{i}(\cdot,\bm{\phi}_{i})$

$\nabla_{\bm{\phi}_{i}}\\!\\!\leftarrow\\!\\!\frac{\sum_{j=1}^{J}\\!\\!\\!\nabla_{\bm{a}^{\prime}_{i,j}}\\!\\!\\!\\!\hat{\Psi}(\bm{s}_{j},\bm{a}^{\prime}_{j},\bm{\theta})\\!\nabla_{\bm{\phi}_{i}}\\!\\!\hat{\pi}_{i}(\bm{s}_{i,j},\bm{\phi}_{i})}{J}$

# Update target networks

$\bm{\theta}^{-}_{i}\leftarrow\tau\bm{\theta}_{i}+(1-\tau)\bm{\theta}^{-}_{i}$

$\bm{\phi}^{-}_{i}\leftarrow\tau\bm{\phi}_{i}+(1-\tau)\bm{\phi}^{-}_{i}$

return _$\hat{\Psi}_{i},\hat{\pi}_{i},i=1,...,N$_

Algorithm 3 Learning Process of MACDPP

###  III-C Factorization of Multi-agents in Different Tasks

In this subsection, we detailed the training procedure of MACDPP in both
multi-agent cooperative/competitive environments and single systems that are
collaboratively controlled by multiple agents. The learning process of the
proposed method in a multi-agent cooperative/competitive scenario was
summarized in Algorithm 3. Given the length of episode $E$ and the length of
one rollout $T$, at the beginning, the parameters of both critic and actor
networks were randomly initialized as
$\bm{\theta}_{i},\bm{\phi}_{i},i=1,...,N$. Those weights were copied to the
target networks as $\bm{\theta}_{i}^{-},\bm{\phi}_{i}^{-}$. At each step, the
global state $\bm{s}$ was first observed. The control action of each agent
$\bm{a}_{i}$ was determined by $\text{BS\\_Sampling}(\cdot)$ following
Algorithm 2. Conducted global action $\bm{a}=[\bm{a}_{1},...,\bm{a}_{N}]$ by
all agents, the global state in the next step and the vector of $N$ reward
functions
$\bm{r}=[R_{1}(\bm{s}_{1},\bm{a}_{1},\bm{s}_{1}^{\prime}),...,R_{N}(\bm{s}_{N},\bm{a}_{N},\bm{s}_{N}^{\prime})]$
were then observed and stored to the separate replay buffer. During the
centralized training buffer, the update of each agent was separately conducted
with its own $J$ mini-batch samples while the samples from other agents’
buffers were used to restore the global information. The TD error was
calculated following Eq. (13) and Algorithm 1 to updated actor and critic
networks. The target networks were then smoothly updated with a smooth
parameter $\tau$ according to $\bm{\theta}_{i},\bm{\phi}_{i},i=1,...,N$.

Figure 1: The differences between the learning process of MACDPP in
cooperative/competitive (top) and joint control (bottom) scenarios.

When implementing the proposed MACDPP to jointly control one complex system by
multiple agents following our previous work [32, 33], only one global replay
buffer $\mathcal{D}$ was built. At each step, the global observed state
$\bm{s}$ was sent in parallel to all agents. The control actions of all actors
were then integrated as $\bm{a}$ and conducted to the target system. The
resulting next step state $\bm{s}^{\prime}$ and the corresponding reward were
received and stored in $\mathcal{D}$. Please note that all agents shared one
reward function $\bm{r}=[R(\bm{s},\bm{a},\bm{s}^{\prime})]$ designed for the
whole system. Unlike the case in charge of multi-agent cooperative/competitive
environments, MACDPP did not separately conduct $J$ mini-batch sampling for
each agent when jointly controlling one system but rather shared $J$ samples
during the update. In addition, the actor network of the $i$-th agent
$\hat{\pi}_{i}(\bm{s},\bm{\phi}_{i})$ directly received the global observed
state in the decision-making process. The difference between the learning
process of MACDPP in cooperative/competitive and joint control scenarios was
illustrated in Fig. 1.

Figure 2: Eight Benchmark tasks for evaluation: (a) Physical Deception from
MPE (mixed cooperative competitive); (b) Covert Communication from MPE (mixed
cooperative competitive); (c) Keep Away from MPE (mixed cooperative
competitive); (d) Cooperative Communication from MPE (pure cooperative); (e)
HalfCheetah from Mujoco (joint control by two agents); (f) HalfCheetah from
Mujoco (joint control by six agents); (g) Hopper from Mujoco (joint control by
three agents); (h) UR5 End Effector Positioning from robo-gym (joint control
by five agents)

##  IV Experimental Results

###  IV-A Experimental Settings

TABLE II: Common hyperparameter settings of compared approaches

| Physical Deception | Covert Communication | Keep Away | Cooperative Communication | HalfCheetah | Hopper | UR5
---|---|---|---|---|---|---|---
Critic Learning Rate | $0.01$ | $0.01$ | $0.1$ | $0.1$ | $10^{-3}$ | $5\times 10^{-4}$ | $5\times 10^{-4}$
Actor Learning Rate | $0.01$ | $0.01$ | $0.01$ | $0.01$ | $10^{-3}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$
Actor and Critic Structure | $(64,64)$ | $(64,64)$ | $(64,64)$ | $(64,64)$ | $(400,300)$ | $(400,300)$ | $(400,300)$
Target Update Rate ($\tau$) | $1\times 10^{-3}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $5\times 10^{-3}$ | $5\times 10^{-3}$ | $5\times 10^{-3}$
Batch Size ($J$) | $1024$ | $1024$ | $1024$ | $1024$ | $100$ | $100$ | $100$
Discount Factor ($\gamma$) | 0.95 | 0.95 | 0.95 | 0.95 | 0.99 | 0.99 | 0.99
Memory Size | $10^{5}$ | $10^{5}$ | $10^{5}$ | $10^{5}$ | $10^{6}$ | $10^{6}$ | $10^{6}$
Warmup Steps | $0$ | $0$ | $0$ | $0$ | $10^{4}$ | $10^{4}$ | $10^{4}$
Steps per Update | $100$ | $50$ | $25$ | $25$ | $1$ | $1$ | $1$

TABLE III: Specific Hyperparameters of MACDPP

| $\eta$ | $M_{1}$ | $M_{2}$ | $\bm{\zeta}^{\text{MC}}$ | $\bm{\zeta}^{\text{Sampling}}$
---|---|---|---|---|---
Physical Deception | $20$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.2}$
Covert Communication | $20$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.2}$
Keep Away | $0.1$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.1}$
Cooperative Communication | $0.1$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.1}$
HalfCheetah | $20$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.1}$
Hopper | $5$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.1}$
UR5 | $0.05$ | $30$ | $50$ | $\bm{0.1}$ | $\bm{0.1}$

In this section, we evaluated MACDPP in multi-agent and traditional control
tasks in terms of learning capability and sample efficiency. For the multi-
agent scenario, we selected the physical deception, Covert Communication, keep
away and cooperative communication tasks from the Multi-Agent Particle
Environment (MPE) [24]333https://github.com/openai/multiagent-particle-envs.
The first three are mixed cooperative-competitive tasks and the last one is a
pure cooperative task. MADDPG [24], MATD3 [25] and M3DDPG [26] were selected
as the compared MARL baselines. For the traditional control scenario, we
selected the Ant, HalfCheetah from Mujoco simulation [35] and the UR5 robot
arm simulation task ur_ee_position developed in robo-gym [36]. For each MARL
approach, the HalfCheetah was two scenarios: 2 agents separately controlled
the front and back body, and six agents controlled six joints. The Hopper and
UR5 were jointly controlled by three and five agents for each controllable
joint. We not only compared the proposed method with MARL approaches MADPPG
[24] but also the widely implemented single-agent RL approaches including Deep
Deterministic Policy Gradient (DDPG) [37], Twin Delayed Deep Deterministic
Policy Gradient (TD3) [38] and Soft Actor-Critic (SAC) [39]. All benchmark
control tasks were illustrated in Fig. 2. The hyperparameters of all compared
methods for each task were summarized in Table II. All actors and critics
shared the same network structures. The tunable hyperparameters of the
proposed MACDPP including $\eta$, the Monte Carlo sampling numbers in
Algorithms 1 and 2 $M_{1},M_{2}$ and the sampling noise
$\bm{\zeta}^{\text{MC}},\bm{\zeta}^{\text{Sampling}}$ were listed in Table
III. The proposed MACDPP was developed by PaddlePaddle [40] under its RL
toolkit PARL 444https://github.com/PaddlePaddle/PARL.. All experiments were
conducted on a workstation with Intel Xeon W2265 CPU, NVIDIA GeForce RTX 3080
GPU, 64GB memory and Ubuntu 20.04 OS. The experimental results were summarized
over five independent trials with different random seeds for statistical
evidence.

###  IV-B Cooperation and Competition in MPE Benchmarks

####  IV-B1 Evaluation of the Learning Capability

TABLE IV: Maximum Average Returns over four MPE benchmark tasks

| MACDPP | MADDPG | MATD3 | M3DDPG
---|---|---|---|---
Physical Deception | $\bm{47.02\pm 14.28}$ | $-0.36\pm 2.20$ | $0.67\pm 2.63$ | $3.77\pm 0.23$
Covert Communication | $\bm{31.68\pm 19.07}$ | $0.72\pm 2.71$ | $1.76\pm 8.78$ | $13.48\pm 10.52$
Keep Away | $\bm{-5.27\pm 1.27}$ | $-6.27\pm 1.01$ | $-5.55\pm 1.33$ | $-8.53\pm 0.10$
Cooperative Communication | $-13.10\pm 6.79$ | $-15.88\pm 10.78$ | $\bm{-11.90\pm 6.95}$ | $-40.07\pm 11.90$

Figure 3: Learning curves of MACDPP and other MARL baselines in MPE benchmark
tasks. The shaded region represents the corresponding standard deviation over
five trials.

We first compared the proposed methods with the related MARL approaches in
four benchmark tasks from the MPE environment. The learning curves of all
compared approaches were shown in Fig. 3 while the maximum average returns of
each method in the evaluation phase during the learning were listed in Table
IV (the number in red indicates the best result in the corresponding term).
The trials of all four tasks were conducted by $25$k episodes, each episode
had $25$ steps.

In the Physical Deception task, MADDPG, MATD3 and M3DDPG converged to close
performances near $0$ average return after $25$k episodes. As a comparison,
our method quickly suppressed other baselines in the first $2000$ episodes and
converged to $40$ average return after $25$k episodes. In the evaluation task,
MACDPP outperformed MATD3 and M3DDPG with over $7000\%$ and $1200\%$ maximum
average return while MADDPG achieved a negative average return. In the Covert
Communication task, M3DDPG outperformed MADDPG and MATD3 in both average
return and converge velocity thanks to its Minimax operator. On the other
hand, MACDPP converged to over $100\%$ more average return during learning and
achieved $135\%$ maximum average in evaluation compared with the suboptimal
method M3DDPG. In the Keep Away task, MADDPG converged to a relatively low
average return. MATD3 converged to near $-10$ average return within $10$k
episodes but could not maintain its performance. Although M3DDPG converged
quickly in the first $10$k episodes, MACDPP outperformed it with the best
maximum average return and the lowest standard deviation in the learning curve
which indicated a more stable training procedure. In the Cooperative
Communication task which requires pure cooperation over all agents, it is
observed that M3DDPG which is good at competition failed to learn good
cooperation policies. Although MACDPP, MADDPG and MATD3 all learned to a close
performance, the proposed method enjoyed the fastest convergence velocity.
Overall, the proposed method demonstrated significantly superior learning
capability than related MARL baselines in various MPE benchmark tasks.

####  IV-B2 Evaluation of the Sample Efficiency

Figure 4: Number of interactions utilized by all compared approaches in MPE
benchmark tasks to reach the lower boundary of maximum average return. TABLE
V: Average calculation time of $1000$ episodes over four MPE benchmark tasks

| MACDPP | MADDPG | MATD3 | M3DDPG
---|---|---|---|---
Physical Deception |  $124.77\pm 1.77$ s |  $32.84\pm 0.20$ s |  $33.31\pm 0.20$ s |  $105.86\pm 2.41$ s
Covert Communication |  $161.73\pm 1.40$ s |  $41.23\pm 0.35$ s |  $41.27\pm 0.45$ s |  $125.83\pm 2.38$ s
Keep Away |  $149.58\pm 1.33$ s |  $39.30\pm 0.32$ s |  $42.94\pm 0.39$ s |  $115.28\pm 1.86$ s
Cooperative Communication |  $144.11\pm 0.55$ s |  $37.10\pm 0.23$ s |  $39.19\pm 0.41$ s |  $108.56\pm 2.44$ s

TABLE VI: Maximum Average Returns over four benchmark control tasks

| MACDPP | MADDPG | DDPG | TD3 | SAC
---|---|---|---|---|---
HalfCheetah (2 agents) | $\bm{10590.80\pm 443.92}$ | $9404.49\pm 1252.08$ | $9991.71\pm 1533.64$ | $9754.40\pm 955.57$ | $10426.10\pm 1577.89$
HalfCheetah (6 agents) | $9767.99\pm 419.40$ | $6482.54\pm 1296.47$ | $9821.45\pm 1393.96$ | $9730.61\pm 962.49$ | $\bm{9828.28\pm 803.50}$
Hopper (3 agents) | $\bm{3452.87\pm 60.85}$ | $2741.40\pm 935.69$ | $2212.34\pm 1139.88$ | $3384.24\pm 230.43$ | $3325.33\pm 270.24$
UR5 (5 agents) | $\bm{1.86\pm 0.03}$ | $1.66\pm 0.73$ | $0.60\pm 1.04$ | $1.60\pm 0.30$ | $-1.13\pm 0.19$

The sample efficiency which is important to the implementation of RL in real-
world systems was evaluated in Fig. 4. We define the measure of sample
efficiency in this subsection as the number of interactions used by each
approach to reach the lower boundary of the maximum average returns in Table
IV over the four benchmark tasks from the MPE environment. It is clearly
observed that the proposed method achieved the overall superior sample
efficiency among all compared MARL approaches, it reduced $70\%$, $54.6\%$ and
$48.3\%$ usage of samples than M3DDPG, MADDPG and MATD3 to reach a certain
level of control performances. At the same time, we found that MACDPP has
overall effectiveness in improving sample efficiency, whether in cooperative
or competitive tasks. This result demonstrated the great potential of the
proposed method in quickly learning proper multi-agent control policies in
complex scenarios with few sampling costs.

####  IV-B3 Evaluation of the Computational Efficiency

In this subsection, we investigated the impact of additional Monte Carlo
sampling and Boltzmann softmax operator in MACDPP on computational efficiency.
We measured the average calculation time of $1000$ episodes over all compared
MARL baselines in Table V. It is observed that MACDPP brought additional
computational burdens in all four tasks. It required $285.54\%$, $270.19\%$
and $27.36\%$ more calculation time compared with MADDPG, MATD3 and M3DDPG. On
the other hand, considering the obvious advantages of our approach in learning
ability, convergence velocity, and sample efficiency, we believed that these
increased computational complexities in training and decision-making were
acceptable.

Figure 5: Learning curves of MACDPP with different values of parameter $\eta$
compared with MADDPG in the Keep Away task. The shaded region represents the
standard deviation of the average evaluation over five trials.

####  IV-B4 Impact of the Specific Parameter

In this subsection, we explored the impact of the special parameter $\eta$
that controls the strength of the relative entropy term in MACDPP. The
learning curves of the Keep Away task with different values of $\eta$ using
MACDPP are shown in Fig. 5. With a wide range of $\eta$ from $0.001$ to $0.5$,
MACDPP consistently outperformed the baseline method MADDPG in both the mean
and standard deviation of the average returns. It is also observed that a
proper selection of $\eta$ could significantly improve MACDPP’s learning
performance. The most superior learning curve was obtained when $\eta=0.1$. An
over-small parameter $\eta=0.001$ resulted in extremely slow convergence and a
very large standard deviation of return at the beginning. As a comparison, the
large one has less effect on smooth policy updates and may fail in learning
more optimal control policies within a limited number of interactions.

###  IV-C Cooperation in Traditional Control Task

Figure 6: Learning curves of MACDPP and all baselines in Mujoco and robo-gym
benchmark tasks. The shaded region represents the corresponding standard
deviation over five trials. Figure 7: Number of interactions utilized by all
compared approaches in Mujoco and robo-gym benchmark tasks to reach the lower
boundary of maximum average return.

####  IV-C1 Evaluation of the Learning Capability

In this section, we moved to the traditional control scenarios where the MARL
approaches were employed to jointly control one system. In this section, we
compared two MARL approaches MACDPP and MADDPG which were treated as the
proposed method without using relative entropy regularization. Three widely
implemented single-agent RL approaches DDPG, TD3 and SAC were also compared.
The learning curves of all compared approaches in 2-agent HalfCheetah (the
system was jointly controlled by MARL with two agents), 6-agent HalfCheetah,
3-agent Hopper and 5-agent UR5 robot arm were illustrated in Fig. 6. The
average maximum returns in the evaluation phase are listed in Table. VI.
Please note that the results of single-agent methods in two HalfCheetah were
slightly different since we used difficult random seeds for each task.

In the 2-agent HalfCheetah task, MADPPG learned the worst policy with the
lowest average return. In this case, the joint control strategy failed to
suppress the single-agent approach while introducing an additional
computational burden. As a comparison, our method converged to the best
average return overall compared approaches with a significant advantage in
convergence velocity. Regarding the maximum average return in evaluation,
MACDPP outperformed $12.61\%$, $8.57\%$ and $6.0\%$ compared with MADDPG, TD3
and DDPG while achieving slightly better results than SAC with significantly
superior converge velocity. In the 6-agent HalfCheetah task, the learning
capability of MADPPG hugely deteriorated so that the six joints could not
effectively cooperate. As a comparison, the proposed method successfully
learned the task as well as other single-agent baselines with not only a
superior converge velocity but also less standard deviation in the average
returns. It enjoyed over $50\%$ more maximum average returns than MADDPG which
did not employ the relative entropy regularization. In the 3-agent Hopper
task, MADDPG outperformed DDPG in average return and converge velocity while
the proposed MACDPP achieved overall superior performances than all compared
baselines. It quickly converged to the best maximum average return which was
$56.07\%$, $25.95\%$, $3.84\%$ and $2.03\%$ than DDPG, MADDPG, SAC and TD3,
respectively. In the more practical UR5 control scenarios where five
independent joints were jointly controlled by five agents in MARL methods,
both MACDPP and MADDPG outperformed single-agent RL approaches. Within $200$k
steps, SAC and DDPG could not learn the task (SAC could not converge at all,
and DDPG learned extremely slowly at the first $70$k steps) while only TD3
achieved a close average return to MADDPG. As a comparison, MACDPP
consistently demonstrated superiority in both learning capability and converge
velocity, it quickly reached an average return over $0$ within $70$k steps and
finally obtained $210\%$, $12.05\%$, $264.6\%$ and $16.3\%$ higher maximum
average return than DDPG, MADDPG, SAC and TD3.

####  IV-C2 Evaluation of the Sample Efficiency

TABLE VII: Average calculation time of $1000$ steps over four benchmark
control tasks

| MACDPP | MADDPG | DDPG | TD3 | SAC
---|---|---|---|---|---
HalfCheetah (2 agents) |  $13.84\pm 0.29$ s |  $9.44\pm 0.30$ s |  $4.51\pm 0.20$ s |  $4.18\pm 0.07$ s |  $8.36\pm 0.10$ s
HalfCheetah (6 agents) |  $54.84\pm 0.15$ s |  $38.35\pm 0.29$ s |  $4.66\pm 0.24$ s |  $4.34\pm 0.29$ s |  $8.63\pm 0.41$ s
Hopper (3 agents) |  $22.61\pm 0.89$ s |  $15.83\pm 0.72$ s |  $4.86\pm 0.36$ s |  $4.69\pm 0.32$ s |  $8.84\pm 0.64$ s
UR5 (5 agents) |  $284.58\pm 28.80$ s |  $219.32\pm 29.31$ s |  $162.03\pm 16.44$ s |  $177.65\pm 22.73$ s |  $144.68\pm 0.64$ s

Figure 8: Trajectories of control actions using MADDPG and MACDPP in one test
rollout of 6-agent HalfCheetah task. The actions of MADDPG and MACDPP were
drawn in purple and blue respectively. Figure 9: Trajectories of control
actions using MADDPG and MACDPP in one test rollout of 5-agent UR5 control
task ur_ee_position . The actions of MADDPG and MACDPP were drawn in purple
and blue respectively. The last dimension Wrist3 joint was not included in the
controllable action as it was fixed to $0$ in the ur_ee_position task.

The sample efficiency of MACDPP was evaluated in Fig. 7. Compared with MADDPG
which converged slower to a certain level of control performances than the
signal-agent baselines, the proposed method demonstrated great advantage in
sample efficiency with the regularization of relative entropy. It only spent
$36.6\%$ samples to reach the same performance. This result indicated the
importance of properly restricting large policy updates in MARL for superior
effectiveness. MCDPP successfully reduced $44.91\%$, $37.94\%$ and $27.98\%$
usage of interactions than DDPG, SAC and TD3.

####  IV-C3 Evaluation of the Computational Efficiency

The computational burden was evaluated in Table VII by summarizing the average
calculation time of the first $1000$ steps in four control scenarios. Although
the proposed method required $132.85\%$, $213.5\%\%$, $196.92\%$ and
$220.43\%$ more computational times compared with MADDPG, DDPG, TD3 and SAC.
It was observed that the computational burden of MACDPP was alleviated in
traditional control scenarios where the system operation took more time. The
proposed method additionally consumed $32.85\%$ time than MADDPG. Compared
with the faster single-agent methods, our method increased the computation
time from $96\%$ to $120\%$ while employing about four times more agents.
Furthermore, with the increasing system operation and communication times
(i.e., from Mujoco to robo-gym based on ROS toolkit), the phenomenon above
became more and more noticeable. It demonstrated the potential and
effectiveness of MACDPP in jointly controlling large-scale systems.

####  IV-C4 Case Study

In this subsection, we investigated the superior control behaviors of MACDPP
through the rollouts of learned policies in both 6-agent HalfCheetah and
5-agent UR5 control scenarios. In the first case study, we explored the
learned policies of MADDPG and MACDPP under the same setting of parameter and
random seed. The test rollouts with $20$ steps and the corresponding
trajectories of each joint were analyzed in Fig. 8. It is clearly observed
that the learned control behavior of MACDPP is more effective than the one of
MADDPG. Effectively coordinating six joints by six agents, the proposed method
learned a superior control strategy. Each joint timely conducted proper torque
according to the current system states, resulting in a faster movement. In
comparison, MADDPG had significant disadvantages in terms of coordinating six
joints by separate agents. Although the agent in charge of dimension 2
successfully learned a similar policy to the one of MACDPP, the whole multi-
agent system struggled to generate proper torques from other agents: the agent
of dimension 5 only produced effective torque near the $15$-th step while the
agent of dimension 1 was fixed with $-1$ torque during the how rollout. Due to
the lack of relative entropy regularization from the algorithmic perspective,
the multiple agents in MADDPG were unable to learn effective and cooperative
control strategies.

In the next case study, we studied the test rollouts using MADDPG and MACDPP
in the ur_ee_position task which aims to control the end-effector of the UR5
robot arm to reach the randomly generated targets. It is observed that MACDPP
quickly drove the robot to finish the task within $60$ steps. The action
trajectories showed proper cooperation between each joint. The base and elbow
joints continuously output $-180^{\circ}$ and $60^{\circ}$ throughout the
task. At step $25$, the shoulder joint and wrist 1 joint were coordinated to
guide the end-effector to move forward to the target position. Around step
$60$, the shoulder and wrist 2 joints worked together to quickly reach the
target. The action Trajectories of all dimensions were effective with minimal
jitter. Compared with our method, MADDPG could not sufficiently learn the
cooperative strategy over five joints. The base joint could not continuously
output a certain degree, it had strong trembling between steps $40$ to $80$.
The shoulder and wrist 1 joints failed to cooperate effectively at the
beginning, resulting in redundant movements of the end-effector. After step
$40$, the shoulder and wrist 2 joints could not achieve seamless coordination.
Both of them experienced sustained tremors which ultimately resulted in highly
degraded control performance.

The experimental results above revealed the advantages of MACDPP in the joint
control of robot systems. The multiple agents reduced the exploration
complexity in one system and resulted in faster policy convergence compared to
single-agent approaches with the same amount of interactions. At the same
time, the relative entropy regularization significantly avoided the mismatch
between the update of multi-agent policies during the learning, promoting the
effectiveness in the learning of cooperative control strategies.

##  V Conclusions

This article proposed a novel MARL approach MACDPP to improve the learning
capability and sample effectively in a wide range of control scenarios
including multiple agents cooperative/competitive tasks and joint control of a
single complicated system. It naturally alleviated the inherent inconsistency
over multiple agents policy updates by integrating the relative entropy
regularization to the AC structure and CTDE framework. MACDPP successfully
extended FKDPP which has been successfully implemented in the real-world
chemical plant by Yokogawa [32, 33] towards a modern approach that supports
deep neural networks, AC structure and CTDE framework in order to fit a wider
range of control scenarios. Through evaluation of different benchmark tasks,
ranging from multi-agent cooperation/competition to Mujoco simulator and robot
arm manipulation, our proposed method consistently demonstrated significant
superiority in both learning capability and sample efficiency compared with
related multi-agent and single-agent RL baselines. All these results indicated
the potential of relative entropy regularized MARL in effectively learning
complex systems divided into multiple agents with lower sampling costs and
better control performance.

## References

  * [1] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.  MIT press, 2018.
  * [2] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013.
  * [3] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015.
  * [4] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al., “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” Science, vol. 362, no. 6419, pp. 1140–1144, 2018.
  * [5] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
  * [6] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: lessons we have learned,” The International Journal of Robotics Research, vol. 40, no. 4-5, pp. 698–721, 2021.
  * [7] Y. Jin, S. Wei, J. Yuan, and X. Zhang, “Hierarchical and stable multiagent reinforcement learning for cooperative navigation control,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 1, pp. 90–103, 2023.
  * [8] C. Sun, W. Liu, and L. Dong, “Reinforcement learning with task decomposition for cooperative multiagent systems,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 5, pp. 2054–2065, 2021.
  * [9] A. Oroojlooy and D. Hajinezhad, “A review of cooperative multi-agent deep reinforcement learning,” Applied Intelligence, vol. 53, no. 11, pp. 13677–13722, 2023.
  * [10] A. Wong, T. Bäck, A. V. Kononova, and A. Plaat, “Deep multiagent reinforcement learning: Challenges and directions,” Artificial Intelligence Review, vol. 56, no. 6, pp. 5023–5056, 2023.
  * [11] X. Yao, C. Wen, Y. Wang, and X. Tan, “Smix($\lambda$): Enhancing centralized value functions for cooperative multiagent reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 1, pp. 52–63, 2023.
  * [12] J. Chai, W. Li, Y. Zhu, D. Zhao, Z. Ma, K. Sun, and J. Ding, “Unmas: Multiagent reinforcement learning for unshaped cooperative scenarios,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 4, pp. 2093–2104, 2023.
  * [13] M. Brittain and P. Wei, “Scalable autonomous separation assurance with heterogeneous multi-agent reinforcement learning,” IEEE Transactions on Automation Science and Engineering, vol. 19, no. 4, pp. 2837–2848, 2022.
  * [14] Z. Yan, A. R. Kreidieh, E. Vinitsky, A. M. Bayen, and C. Wu, “Unified automatic control of vehicular systems with reinforcement learning,” IEEE Transactions on Automation Science and Engineering, vol. 20, no. 2, pp. 789–804, 2023.
  * [15] L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 2, pp. 156–172, 2008.
  * [16] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” PloS one, vol. 12, no. 4, p. e0172395, 2017.
  * [17] L. Matignon, G. J. Laurent, and N. Le Fort-Piat, “Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems,” The Knowledge Engineering Review, vol. 27, no. 1, pp. 1–31, 2012.
  * [18] J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” in Autonomous Agents and Multiagent Systems (AAMS), pp. 66–83, Springer, 2017.
  * [19] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” Advances in neural information processing systems (NIPS), vol. 29, 2016.
  * [20] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al., “Value-decomposition networks for cooperative multi-agent learning based on team reward,” in Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085–2087, 2018.
  * [21] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 7234–7284, 2020.
  * [22] M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” in Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186–2188, 2019\.
  * [23] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” Advances in Neural Information Processing Systems (NIPS, vol. 35, pp. 24611–24624, 2022.
  * [24] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in neural information processing systems (NIPS), vol. 30, 2017\.
  * [25] J. Ackermann, V. Gabler, T. Osa, and M. Sugiyama, “Reducing overestimation bias in multi-agent domains using double centralized critics,” arXiv preprint arXiv:1910.01465, 2019.
  * [26] S. Li, Y. Wu, X. Cui, H. Dong, F. Fang, and S. Russell, “Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, pp. 4213–4220, 2019.
  * [27] M. G. Azar, V. Gómez, and H. J. Kappen, “Dynamic policy programming,” The Journal of Machine Learning Research, vol. 13, no. 1, pp. 3207–3245, 2012.
  * [28] T. Kozuno, E. Uchibe, and K. Doya, “Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning,” in The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2995–3003, PMLR, 2019.
  * [29] N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist, “Leverage the average: an analysis of kl regularization in reinforcement learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 12163–12174, 2020.
  * [30] Y. Cui, T. Matsubara, and K. Sugimoto, “Kernel dynamic policy programming: Applicable reinforcement learning to robot systems with high dimensional states,” Neural networks, vol. 94, pp. 13–23, 2017.
  * [31] Y. Tsurumine, Y. Cui, E. Uchibe, and T. Matsubara, “Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation,” Robotics and Autonomous Systems, vol. 112, pp. 72–83, 2019\.
  * [32] Y. Cui, L. Zhu, M. Fujisaki, H. Kanokogi, and T. Matsubara, “Factorial kernel dynamic policy programming for vinyl acetate monomer plant model control,” in 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE), pp. 304–309, 2018.
  * [33] L. Zhu, Y. Cui, G. Takami, H. Kanokogi, and T. Matsubara, “Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process,” Control Engineering Practice, vol. 97, p. 104331, 2020.
  * [34] K. Asadi and M. L. Littman, “An alternative softmax operator for reinforcement learning,” in International Conference on Machine Learning, pp. 243–252, PMLR, 2017.
  * [35] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026–5033, IEEE, 2012.
  * [36] M. Lucchi, F. Zindler, S. Mühlbacher-Karrer, and H. Pichler, “robo-gym–an open source toolkit for distributed deep reinforcement learning on real and simulated robots,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5364–5371, IEEE, 2020.
  * [37] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning.,” in International Conference on Learning Representations (ICLR), 2016.
  * [38] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in International conference on machine learning, pp. 1587–1596, PMLR, 2018.
  * [39] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning, pp. 1861–1870, PMLR, 2018\.
  * [40] Y. Ma, D. Yu, T. Wu, and H. Wang, “Paddlepaddle: An open-source deep learning platform from industrial practice,” Frontiers of Data and Domputing, vol. 1, no. 1, pp. 105–115, 2019.