# Mava: a research framework for distributed multi-agent reinforcement
learning

Arnu Pretorius1,∗, Kale-ab Tessera1,∗, Andries P. Smit2,†,∗, Claude
Formanek3,†, St John Grimbly3,†, Kevin Eloff2,†, Siphelele Danisa3,†, Lawrence
Francis1, Jonathan Shock3, Herman Kamper2, Willie Brink2, Herman Engelbrecht2,
Alexandre Laterre1, Karim Beguir1
1InstaDeep
2Stellenbosch University
3University of Cape Town
†Work done during a research internship at InstaDeep
∗Equal contribution

###### Abstract

Breakthrough advances in reinforcement learning (RL) research have led to a
surge in the development and application of RL. To support the field and its
rapid growth, several frameworks have emerged that aim to help the community
more easily build effective and scalable agents. However, very few of these
frameworks exclusively support multi-agent RL (MARL), an increasingly active
field in itself, concerned with decentralised decision making problems. In
this work, we attempt to fill this gap by presenting Mava: a research
framework specifically designed for building scalable MARL systems. Mava
provides useful components, abstractions, utilities and tools for MARL and
allows for simple scaling for multi-process system training and execution,
while providing a high level of flexibility and composability. Mava is built
on top of DeepMind’s Acme (Hoffman et al., 2020), and therefore integrates
with, and greatly benefits from, a wide range of already existing single-agent
RL components made available in Acme. Several MARL baseline systems have
already been implemented in Mava. These implementations serve as examples
showcasing Mava’s reusable features, such as interchangeable system
architectures, communication and mixing modules. Furthermore, these
implementations allow existing MARL algorithms to be easily reproduced and
extended. We provide experimental results for these implementations on a wide
range of multi-agent environments and highlight the benefits of distributed
system training. Mava’s source code is available here:
https://github.com/instadeepai/Mava.

##  1 Introduction

Reinforcement learning (RL) has proven to be a useful computational framework
for building effective decision making systems (Sutton and Barto, 2018). This
has been shown to be true, not only in simulated environments and games (Mnih
et al., 2015; Silver et al., 2017; Berner et al., 2019; Vinyals et al., 2019),
but also in real-world applications (Mao et al., 2016; Gregurić et al., 2020).
Advances in deep learning (Goodfellow et al., 2016) combined with modern
frameworks for distributed environment interaction and data generation (Liang
et al., 2018; Hoffman et al., 2020) have allowed RL to to deliver research
breakthroughs that were previously out of reach. However, as the complexity of
a given problem increases, it might become impossible to solve by only
increasing representational capacity and agent training data. This is the case
when a problem poses an exponential expansion of the problem representation as
it scales, for example, in distributed decision making problems such as fleet
management (Lin et al., 2018) where the action space grows exponentially with
the size of the fleet.

Multi-agent reinforcement learning (MARL) extends the decision-making
capabilities of single-agent RL to include distributed decision making
problems. In MARL, multiple agents are trained to act as individual decision
makers of some larger system, typically with the goal of achieving a scalable
and effective decentralised execution strategy when deployed (Shoham and
Leyton-Brown, 2008). MARL systems can be complex and time consuming to build,
while at the same time, be difficult to design in a modular way. This
modularity helps to be able to quickly iterate and test new research ideas.
Furthermore, despite the added complexity of MARL, it is crucial that these
systems maintain the advantages of distributed training. Often the core system
implementation and the distributed version of these systems pose two
completely different challenges, which makes scaling single process prototypes
towards deployable multi-process solutions a nontrivial undertaking.

In this work, we present Mava, a research framework for distributed multi-
agent reinforcement learning. Mava aims to provide a library for building MARL
systems in a modular, reuseable and flexible manner using components and
system architectures specifically designed for the multi-agent setting. A key
criterion for Mava is to provide enough flexibility for conducting research.
At its core, Mava leverages the single-agent distributed RL research framework
Acme (Hoffman et al., 2020) and closely follows Acme’s design principles. This
means Mava immediately benefits from a large repository of single-agent RL
components and abstractions, such as commonly used networks, loss functions,
utilities and other tools that will keep growing alongside Mava as Acme is
further developed. Furthermore, we inherit the same RL ecosystem built around
Acme. Most notably, Mava interfaces with Reverb (Cassirer et al., 2021) for
data flow management and supports simple scaling using Launchpad (Yang et al.,
2021). As a consequence, Mava is able to run systems at various level of scale
without changing a single line of the system code. Finally, Mava integrates
with popular environment suites for MARL, including PettingZoo (Terry et al.,
2020), multi-agent Starcraft (SMAC) (Samvelyan et al., 2019), Flatland
(Mohanty et al., 2020) and OpenSpiel (Lanctot et al., 2019).

##  2 Related work

To support the rapid growth in research and development in RL, several
frameworks have been created to assist the community in building, as well as
scaling, modern deep RL agents. As is the case with Mava, most frameworks aim
to adhere to a set of criteria that center around flexibility, reusability,
reproducibility and ease of experimentation.

Single-Agent RL.  Among the earliest software libraries for single-agent deep
RL are RLlab (Duan et al., 2016), baselines (Dhariwal et al., 2017) and
Spinning up (Achiam, 2018). Other similar libraries include Dopamine (Castro
et al., 2018), TF agents (Guadarrama et al., 2018), Surreal (Fan et al.,
2018), SEED RL (Espeholt et al., 2019) and RLax (Budden et al., 2020). These
libraries all aim to provide either useful abstractions for building agents,
or specific well-tested agent implementations, or both. RLLib (Liang et al.,
2018) and Arena (Song et al., 2020) are libraries that support both single and
multi-agent RL and is built on top of Ray (Moritz et al., 2017) for
distributed training. Finally, Acme (Hoffman et al., 2020), a recently
released research framework, provides several agent implementations as well as
abstractions and tools for building novel agents. Acme is built around an
ecosystem of RL related libraries including Reverb (Cassirer et al., 2021) for
controlling data flow between actors, learners and replay tables and Launchpad
(Yang et al., 2021) for distributed computation.

Multi-Agent RL (MARL).  Software libraries for MARL are fewer in number
compared to single-agent RL. As mentioned above, RLlib and Arena (which
directly leverages RLlib) both support MARL. RLlib is able to provide options
such as parameter sharing between independent agents for some of its single-
agent implementations, as well as shared critic networks for actor-critic
algorithms. But it has not been specifically designed for MARL: RLlib provides
only a few MARL specific algorithm implementations such as MADDPG (Lowe et
al., 2017) and QMIX (Rashid et al., 2018). PyMARL (Samvelyan et al., 2019), is
a MARL library centered around research on value decomposition methods, such
as VDN (Sunehag et al., 2017) and QMIX. Recent work on networked MARL (Chu et
al., 2020), provide several open-source implementations of MARL algorithms, in
particular those capable of using communication across a network, such as DIAL
(Foerster et al., 2016) and CommNet (Sukhbaatar et al., 2016) as well as the
author’s own proposal, NeurComm. OpenSpiel (Lanctot et al., 2019) is a
collection of environments and algorithm implementations that focus on
research in games. Most of the algorithms in OpenSpiel are inspired by
research in game theory as well as RL, and algorithms are provided that are
suitable for either single-agent or multi-agent RL.

To the best of our knowledge, Mava is the first software library that is
solely dedicated to providing a comprehensive research framework for large-
scale distributed multi-agent reinforcement learning. The abstractions, tools
and components in Mava serve a single purpose: to make building scalable MARL
systems easier, faster and more accessible to the wider community.

In the remainder of the paper, we will discuss the design philosophy behind
Mava, showcase some existing system implementations and demonstrate the
effectiveness of distributed training. However, for the sake of making the
paper self-contained, in the following we first provide some background on the
topic of RL and MARL before moving on to discuss the framework itself.

##  3 Background

Reinforcement learning is concerned with optimal sequential decision making
within an environment. In single agent RL, the problem is often modeled as a
Markov decision process (MDP) defined by the following tuple
$(\mathcal{S},\mathcal{A},r,p,\rho_{0},\gamma)$ (Andreae, 1969; Watkins,
1989). At time step $t$, in a state $s^{t}\in\mathcal{S}$, the agent selects
an action $a^{t}$ from a set of actions $\mathcal{A}$. The environment state
transition function $p(s^{t+1}|s^{t},a^{t})$ provides a distribution over next
states $s^{t+1}$, and a reward function $r(s^{t},a^{t},s^{t+1})$ returns a
scalar reward, given the current state, action and next state. The initial
state distribution is given by $\rho_{0}$, with $s^{0}\sim\rho_{0}$, and
$\gamma\in(0,1]$ is a discount factor controlling the influence of future
rewards. The goal of RL is to find an optimal policy $\pi^{*}$, where the
policy is a mapping from states to a distribution over actions, that maximises
long-term discounted future reward such that
$\pi^{*}=\text{argmax}_{\pi}\mathbb{E}_{\tau\sim\pi}[\sum^{\infty}_{t=0}\gamma^{t}r(s^{t},a^{t},s^{t+1})]$.
If the environment state is partially observed by the agent, an observation
function $o(s^{t})$ is assumed and the agent has access only to the
observation $o^{t}=o(s^{t})$ at each time step, with the full observation
space defined as $\mathcal{O}=\\{o(s)|s\in\mathcal{S}\\}$.

Deep RL.  Popular algorithms for solving classic (tabular) RL problems include
value-based methods such as $Q$-learning (Watkins and Dayan, 1992) and policy
gradient methods such as the REINFORCE algorithm (Williams, 1992). Q-learning
learns a value function $Q(s,a)$ for state-action pairs and obtains a policy
by selecting actions according to these learned values using a specific action
selector, e.g. $\epsilon$-greedy (Watkins, 1989) or UCB (Auer et al., 2002).
In contrast, policy gradient methods learn a parameterised policy
$\pi_{\theta}$, with parameters $\theta$, directly by following a performance
gradient signal with respect to $\theta$. The above approaches are combined in
actor-critic methods (Sutton et al., 2000), where the actor refers to the
policy being learned and the critic to a value function. In deep RL, the
policy and value functions are parameterised using deep neural networks as
high-capacity function approximators. These networks are capable of learning
abstract representations from raw input signals capable of generalising across
states, making it possible to scale RL beyond the tabular setting. Recent
state-of-the-art deep RL methods include Deep Q-Networks (DQN) (Mnih et al.,
2013) and related variants (Hessel et al., 2017), as well as advanced actor-
critic methods such as DDPG (Lillicrap et al., 2015), PPO (Schulman et al.,
2017) and SAC (Haarnoja et al., 2018). For a more in-depth review of deep RL,
we recommend the following resources (Arulkumaran et al., 2017; Li, 2017).

Distributed RL.  Distributed algorithms such as A3C (Mnih et al., 2016),
Impala (Espeholt et al., 2018), R2D2 (Kapturowski et al., 2018) and D4PG
(Barth-Maron et al., 2018) are examples of algorithms specifically designed to
train value-based and actor-critic methods at scale. Different paradigms exist
for distributing the training of RL agents. For example, one way to distribute
training is to batch the observations of distributed actors and use a
centralised inference server (Espeholt et al., 2019). A different approach is
to launch a large number of workers that are responsible for both data
collection and gradient updates (the worker here is both the actor and the
learner) and to periodically synchronize their gradients across different
processes (Berner et al., 2019). In Acme (Hoffman et al., 2020), distributed
RL is achieved by distributing the actor processes that interact with the
environment. The generated data is collected by a central or distributed store
such as a replay table, where learner processes are able to periodically
sample from the table and update the parameters of the actor networks using
some form of gradient based learning. Finally, actors periodically synchronize
their parameters with the latest version of the trainer and the process
continues. In general, distributed training often leads to significant speed-
ups in the time to convergence compared to standard single process RL.

Deep MARL.  In multi-agent RL, we can model the problem as a partially
observable Markov game (Littman, 1994) between $N$ agents, defined as the
tuple given above for the single-agent case, but with observation and action
spaces given by the following cartesian products:
$\mathcal{O}=\prod^{N}_{i=1}\mathcal{O}_{i}\subseteq\mathcal{S}$ and
$\mathcal{A}=\prod^{N}_{i=1}\mathcal{A}_{i}$, for agents $i=1,...,N$. In a
typical cooperative setting, the goal is to find an optimal joint policy
$\pi^{*}(a_{1},...,a_{N}|o_{1},...,o_{N})$ that maximises a shared long-term
discounted future reward for all agents as
$\pi^{*}=\text{argmax}_{\pi}\mathbb{E}[\sum^{N}_{i=1}\sum^{\infty}_{t=0}\gamma^{t}r(o^{t}_{i},a^{t}_{i},o^{t+1}_{i})]$.
However, the framework can easily be extended beyond pure cooperation to more
complicated settings. Figure 1 shows the canonical environment-system
interaction loop in MARL where multiple agents each take an action in the
environment and subsequently receive a new observation and reward from taking
that particular action.

Early work in MARL investigated the effectiveness of simply training multiple
independent $Q$-learning agents (Tan, 1993). This work has since been extended
to include deep neural networks, or more specifically independent DQNs (Tampuu
et al., 2017).

Figure 1:  Multi-agent reinforcement learning system-environment interaction
loop.

However, from the perspective of an individual agent, these approaches treat
all other learning agents as part of the environment, resulting in the optimal
policy distribution to become non-stationary. Furthermore, if the environment
is only partially observable, the learning task can become even more
difficult, where agents may struggle with credit assignment due to spurious
rewards received from unobserved actions of other agents (Claus and Boutilier,
1998). To mitigate the issue of non-stationarity, MARL systems are often
designed within the paradigm of centralised training with decentralised
execution (CTDE) (Lowe et al., 2017; Foerster et al., 2017a). In CTDE, a
centralised information processing unit (most often a critic network in actor-
critic methods) is conditioned on the global state, or joint observations, and
all of the actions of the agents during training. This makes the learning
problem stationary. The central unit is then later removed once the individual
agents’ policies have been learned, making it possible to use each policy
independently during system execution.

Research in MARL has grown in popularity over the past few years in tandem
with single-agent deep RL and has seen several key innovations such as novel
centralised training, learned communication, replay stabilisation, as well as
value and policy function decomposition strategies, all developed only fairly
recently (Foerster et al., 2016, 2017b; Lowe et al., 2017; Foerster et al.,
2017a; Rashid et al., 2018; Das et al., 2019; de Witt et al., 2020; Wang et
al., 2020a). For an overview of MARL, past and present, we recommend the
following surveys (Powers et al., 2003; Busoniu et al., 2008; Hernandez-Leal
et al., 2019a, b; OroojlooyJadid and Hajinezhad, 2019; Nguyen et al., 2019;
Zhang et al., 2019; Yang and Wang, 2020). We discuss distributed training of
MARL systems in the next section where we give an overview of Mava.

##  4 Mava

In this section, we provide a detailed overview of Mava: its design,
components, abstractions and implementations. We will discuss the Executor-
Trainer paradigm and how Mava systems interact with the environment, inspired
by the canonical Actor-Learner paradigm in distributed single-agent RL. We
proceed by giving an overview of distributed system training. We will mention
some of the the available components and architectures in Mava for building
systems and provide a brief description of the currently supported baseline
system implementations. Furthermore, we will demonstrate Mava’s functionality,
for example, how to change the system architecture in only a few lines of
code. Finally, we will show how Launchpad is used to launch a distributed MARL
system in Mava.

Systems and the Executor-Trainer paradigm.  At the core of Mava is the concept
of a system. A system refers to a full MARL algorithm specification consisting
of the following components: an executor, a trainer and a dataset. The
executor is a collection of single-agent actors and is the part of the system
that interacts with the environment, i.e. performs an action for each agent
and observes the reward and next observation for each agent. The dataset
stores all of the information generated by the executor in the form of a
collection of dictionaries with keys corresponding to the individual agent
ids. All data transfer and storage is handled by Reverb, which provides highly
efficient experience replay as well as data management support for other data
structures such as first-in-first-out (FIFO), last-in-first-out (LIFO) and
priority queues. The trainer is a collection of single-agent learners,
responsible for sampling data from the dataset and updating the parameters for
every agent in the system. The left panel in Figure 2 gives a visual
illustration of the building blocks of a system. Essentially, executors are
the multi-agent version of the actor class (that use policy networks to act
and generate training data) and trainers are the multi-agent version of the
typical learner class (responsible for updating policy parameters for the
actors).

Figure 2:  Multi-agent reinforcement learning systems and the executor-trainer
paradigm. Left: Executor-Trainer system design. Right: Distributed systems.

The executor-environment interaction loop.  Environments in Mava adhere to the
DeepMind environment API (Muldal et al., 2019) and support multi-agent
versions of the API’s TimeStep and specs classes in addition to the core
environment interface. The timestep represents a container for environment
transitions, which in Mava holds a set of dictionaries of observations and
rewards indexed by agent ids, along with a discount factor shared among agents
and the environment step type (first, middle or last). Each environment also
has a multi-agent spec specifying environment details such as the
environment’s action and observation spaces (which could potentially be
different for each agent in the system).

⬇

while True:

# make initial observation for each agent

step = environment.reset()

executor.observe_first(step)

while not step.last():

# take agent actions and step the environment

obs = step.observation

actions = executor.select_actions(obs)

step = environment.step(actions)

# make an observation for each agent

executor.observe(actions, next_timestep=step)

# update the executor policy networks

executor.update()

Block 1 The executor-environment interaction loop in Mava

Block 1 shows a code example of how an executor interacts with an environment.
The loop closely follows the canonical single-agent environment loop. When an
executor calls observe and related functions, an internal adder class
interfaces with a reverb replay table and adds the transition data to the
table. Mava already supports environment wrappers for well-known MARL
environments, including PettingZoo (Terry et al., 2020), (a library similar to
OpenAI’s Gym (Brockman et al., 2016) but for multi-agent environments), multi-
agent Starcraft (SMAC) (Samvelyan et al., 2019), Flatland (Mohanty et al.,
2020) and OpenSpiel (Lanctot et al., 2019).

Distributed systems.  As mentioned before, Mava shares much of its design
philosophy with Acme for the following reasons: to provide a high level of
composability suitable for novel research (i.e. building new systems) as well
as making it possible to easily run systems distributed at various levels of
scale, using the same underlying MARL system code. For example, the system
executor may be distributed across multiple processes, each with a copy of the
environment. Each process collects and stores data which the trainer uses to
update the parameters of the actor networks used within each executor. This
basic design is illustrated in the right panel of Figure 2. The distribution
of processes is defined through constructing a multi-node graph program using
Launchpad. In Mava, we use Launchpad’s CourierNode to connect executor and
trainer processes. Furthermore, we use the specifically designed ReverbNode to
expose reverb table function calls to the rest of the system. For each node in
the graph, we can specify the local resources (CPU and GPU usage) that will be
made available to that node by passing a dictionary of node keys with their
corresponding environment variables as values.

Figure 3:  Multi-agent reinforcement learning system architectures. Left:
Decentralised. Middle: Centralised. Right: Networked.

Components.  Mava aims to provide several useful components for building MARL
systems. These include custom MARL specific networks, loss functions,
communication and mixing modules (for value and policy factorisation methods).
Perhaps the most fundamental component is the system architecture. The
architecture of a system defines the information flow between agents in the
system. For example, in the case of centralised training with decentralised
execution using a centralised critic, the architecture would define how
observation and action information is shared in the critic during centralised
training and restrict information being shared between individual agent
policies for decentralised execution. In other cases, such as inter-agent
communication, the architecture might define a networked structure specifying
how messages are being shared between agents. An illustration of popular
architectures for MARL is given in Figure 3. Decentralised architectures
create systems with fully independent agents, whereas centralised
architectures allow for information to be shared with some centralised unit,
such as a critic network. Networked architectures define an information
topology where only certain agents are allowed to share information, for
example with their direct neighbours.

System implementations.  Mava has a set of already implemented baseline
systems for MARL. These systems serve as easy-to-use and out-of-the-box
approaches for MARL, but perhaps more importantly, also as building blocks and
examples for designing custom systems. We provide the following overview of
implemented systems as well as related approaches that could potentially be
implemented in future versions of Mava using its built-in components.

spaMulti-Agent Deep Q-Networks (MADQN): Tampuu et al. (2017) studied
independent multi-agent DQN, whereas Foerster et al. (2017c), proposed further
stabilisation techniques such as policy fingerprints to deal with non-
stationarity. In Mava, we provide implementations of MADQN using either feed-
forward or recurrent actors in the executor and allow for the addition of
replay stabilisation using policy fingerprints. Our systems are also
interchangeable with respect to the different variants of DQN that are
available in Acme.
spaLearned Communication (DIAL): A common component of MARL systems is agent
communication. Mava supports general purpose components specifically for
implementing systems with communication. In terms of baseline systems, Mava
has an implementation of differentiable inter-agent learning (DIAL) (Foerster
et al., 2016). However, other communication protocols such as RIAL (Foerster
et al., 2016), CommNet (Sukhbaatar et al., 2016), TarMAC (Das et al., 2019),
and NeurComm (Chu et al., 2020), should be accessible given the general
purpose communication modules provided inside Mava.
spaValue Decomposition (VDN/QMIX): Value function decomposition in MARL has
become a very active area of research, starting with VDN (Sunehag et al.,
2017) and QMIX (Rashid et al., 2018), and subsequently spawning several
extensions and related ideas such as MAVEN (Mahajan et al., 2019), Weighted
QMIX (Rashid et al., 2020), QTRAN (Son et al., 2019), QTRAN++ (Son et al.,
2020), QPLEX (Wang et al., 2020b), AI-QMIX (Iqbal et al., 2020) and UneVEn
(Gupta et al., 2020). Mava has implementations of both VDN and QMIX and
provides general purpose modules for value function decomposition, as well as
associated hypernetworks.
spaMulti-Agent Deep Deterministic Policy Gradient (MADDPG/MAD4PG): The
deterministic policy gradient (DPG) method (Silver et al., 2014) has in recent
years been extended to deep RL, in DDPG (Lillicrap et al., 2015) as well as to
the multi-agent setting using a centralised critic, in MADDPG (Lowe et al.,
2017). More recently, distributional RL (Bellemare et al., 2017) approaches
have been incorporated into a scaled version of the algorithm, in D4PG (Barth-
Maron et al., 2018). In Mava, we provide an implementation of the original
MADDPG, as well as a multi-agent version of D4PG.

##  5 Examples

To illustrate the flexibility of Mava in terms of system design and scaling,
we step through several examples of different systems and use cases. We will
begin by providing a system template using MADQN which we will keep updating
as we go through the different examples. The code for the template is
presented in Block 2, where the distributed system program is built, starting
on line 10. The first two arguments to the program are environment and network
factory functions. These helper functions are responsible for creating the
networks for the system, initialising their parameters on the different
compute nodes and providing a copy of the environment for each executor. The
next argument num_executors sets the number of executor processes to be run.
The inset on the right shows the program graph consisting of two executor
courier nodes, a trainer courier node and the replay table node. After
building the program we feed it to Launchpad’s launch function and specify the
launch type to perform local multi-processing, i.e. running the distributed
program on a single machine. Scaling up or down is simply a matter of
adjusting the number of executor processes.

⬇

1# Mava imports

2from mava.systems.tf import madqn

3from mava.components.tf.architectures import DecentralisedPolicyActor

4from mava.systems.tf.system.helpers import environment_factory,
network_factory

5

6# Launchpad imports

7import launchpad

8

9# Distributed program

10program = madqn.MADQN(

11 environment_factory=environment_factory,

12 network_factory=network_factory,

13 architecture=DecentralisedPolicyActor,

14 num_executors=2,

15 ).build()

16

17# Launch

18launchpad.launch(

19 program,

20 launchpad.LaunchType.LOCAL_MULTI_PROCESSING,

21 )

Block 2 Launching a distributed Mava system: MADQN

⬇

# Recurrent MADQN

program = madqn.MADQN(

...

executor_fn=madqn.MADQNRecurrentExecutor,

trainer_fn=madqn.MADQNRecurrentTrainer,

...

).build()

\--------------------------------------------------------

⬇ # Inside MADQN system code from mava.components.tf.modules import
communication # Wrap architecture in communication module
communication.BroadcastedCommunication( architecture=architecture,
shared=True, channel_size=1, channel_noise=0, )

Block 3 Code changes for communication

Figure 4:  MADQN system code and experiments. Top Left: Code changes to MADQN.
Top Right: Experiments on switch game and Starcraft 2, showing the benefits of
communication.

Modules.  Our first use case is communication. We will add a communication
module to MADQN to have agent’s be able to learn how to communicate across a
differentiable communication channel. However, before adding the communication
module, we first show how to change our MADQN system to support recurrent
execution. As long as the network factory specifies a recurrent model for the
Q-Networks used in MADQN, the system can be made recurrent by changing the
executor and trainer functions. This code change is shown in the top part of
Block 3. Next, we add communication. To do this, we adapt the system code by
wrapping the architecture fed to the system with a communication module, in
this case broadcasted communication. This code change is shown in the bottom
part of Block 3, where we can set whether the communication channel is shared
between agents, the size of the channel and the amount of noise injected into
the channel (which can sometimes help with generalisation).

Figure 5:  Environments: From left to right, top to bottom: switch game,
speaker-listener, spread, 3v3 marine level on Starcraft and Multi-Walker.

– Switch game.  Using this new MADQN system with communication, we run
experiments on the switch riddle game introduced by Foerster et al. (2016) to
specifically test a system’s communication capabilities. In this simple game,
agent’s are sent randomly to an interrogation room one by one and are witness
to a switch either being on or off. Once in the room, each agent has the
option to guess whether it thinks all agents have already visited the
interrogation room or not. The switches are modelled as messages sent between
the agents and the actions are either the final guess that all agents have
been in the room or a no-op action. The guess is final, since once any agent
makes a first guess the game ends and all agents receive either a reward of 1,
if the guess is correct, or a reward of -1, if it is incorrect. Even though
the game is relatively simple, the size of the space of possible policies is
significant, calculated by Foerster et al. (2016) to be $4^{3N^{O(N)}}$, where
$N$ is the number of agents in the game. Our results for the switch game are
shown in the top plot in Figure 4, demonstrating the benefit of the added
communication module. Most Mava modules work by wrapping the system
architecture. For example, we could also add a replay memory stabilisation
module to our MADQN system using the fingerprint technique proposed by
Foerster et al. (2017c). This module may help deal with non-stationarity in
the replay distribution. To add the module, we can import the stabilisation
module and wrap the system architecture as follows:
stabilising.FingerPrintStabalisation(architecture).

– StarCraft Multi-Agent Challenge (SMAC).  In our next example use case, we
test on the SMAC environment (Samvelyan et al., 2019). This environment offers
a set of decentralised unit micromanagement levels where a MARL system is able
to control a number of starcraft units and is pitted against the game
heuristic AI in a battle scenario. Value decomposition networks (VDN) and
related methods such as QMIX have been demonstrated to work well on Starcraft.
These methods all center around the concept of building a $Q$-value mixing
network to learn a decomposed joint value function for all the agents in the
system. In VDN the "network" is only a sum of $Q$-values, but in more complex
cases such QMIX and derivative systems, the mixing is done using a neural
network where the weights are being determined by a type of hypernetwork. In
Mava, we can add mixing as a module, typically to a MADQN based system, by
importing the module and wrapping the architecture: e.g.
mixing.AdditiveMixing(architecture). The bottom plot in Figure 4 shows episode
returns for our implementation of a VDN system on the 3v3 marine level
compared to standard independent MADQN using feedforward networks.
Unfortunately, we have been unable to reach similar performance on SMAC using
our implementation of QMIX compared to the results presented in the original
paper (Rashid et al., 2018).

Architectures.  To demonstrate the use of different system architectures we
switch to using an actor-critic system, suitable for both discrete and
continuous control. Specifically, we use MAD4PG as an example, where the
running changes to our program will now be associated with the code shown in
Block 4. Here, we import the mad4pg system and pass the system architecture
type, in this case a decentralised (independent agents) $Q$-value actor-critic
architecture, along with its trainer function. As a test run, we show in
Figure 6 the performance of decentralised MAD4PG compared to MADDPG using
weight sharing on the simple agent navigation levels spread and speaker-
listener (Lowe et al., 2017). Here, both systems are able to obtain a mean
episode return similar to previously reported performances (Lowe et al.,
2017).

⬇

# MAD4PG system

program = mad4pg.MAD4PG(

...

DecentralisedQValueCritic

MADDPGFeedForwardExecutor,

MAD4PGDecentralisedTrainer,

num_executors=2,

...

).build()

Block 4 MAD4PG

Figure 6:  MAD4PG system code and experiments. Left: Code for MAD4PG.
Evaluation run of MAD4PG "solving" the Multi-Walker environment. Right:
Experiments on MPE and Multi-Walker environments using different architectures
and number of executors.

– Multi-Walker.  The multi-agent bipedal environment, Multi-Walker (Gupta et
al., 2017), presents a more difficult continuous control task, where agents
are required to balance a beam while learning to walk forward in unison. As
mentioned before, a popular training paradigm for actor-critic MARL systems is
to share information between agents using a centralised critic. To change the
architecture of our system we can import a different supported architecture
type and specify the corresponding trainer. For example, to convert our MAD4PG
system to use a centralised critic, we replace the architecture with a
CentralisedQValueCritic architecture and set the trainer to a
MAD4PGCentralisedTrainer. We compare centralised versus decentralised training
on Multi-Walker in Figure 6. Even though decentralised MAD4PG is able to solve
the environment, centralised training does not seem to help, however, we note
this is consistent with the findings of Gupta et al. (2017). In a similar way
to changing to a centralised architecture, we can have the system be a
networked system. Given a network specification detailing the connections
between agents, we can replace the architecture of the system with a networked
architecture using the NetworkedQValueCritic class.

Distribution.  In our our final experiment, we demonstrate the effects of
distributed training on the time to convergence. The bottom right plot in
Figure 6 shows the results of distributed training for MAD4PG on Multi-Walker.
Specifically, we compare evaluation performance against training time for
different numbers of executors. We see a marked difference in early training
as we increase num_executors beyond one, but less of a difference between
having two compared to four executors.

##  6 Conclusion

In this work, we presented Mava, a research framework for distributed multi-
agent reinforcement learning. We highlighted some of the useful abstractions
and tools in Mava for building scalable MARL systems through a set of examples
and use cases. We believe this work is a step in the right direction, but as
with any framework, Mava is an ongoing work in progress and will remain in
active development as the MARL community grows to support the myriad of its
innovations and use cases.

## References

  * Hoffman et al. (2020) M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas, “Acme: A research framework for distributed reinforcement learning,” _arXiv preprint arXiv:2006.00979_ , 2020\. [Online]. Available: https://arxiv.org/abs/2006.00979
  * Sutton and Barto (2018) R. S. Sutton and A. G. Barto, _Introduction to reinforcement learning_. Second edition. MIT press Cambridge, 2018.
  * Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * Silver et al. (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton _et al._ , “Mastering the game of go without human knowledge,” _nature_ , vol. 550, no. 7676, pp. 354–359, 2017.
  * Berner et al. (2019) C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse _et al._ , “Dota 2 with large scale deep reinforcement learning,” _arXiv preprint arXiv:1912.06680_ , 2019\.
  * Vinyals et al. (2019) O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, “Grandmaster level in StarCraft II using multi-agent reinforcement learning,” _Nature_ , vol. 575, no. 7782, pp. 350–354, 2019.
  * Mao et al. (2016) H. Mao, M. Alizadeh, I. Menache, and S. Kandula, “Resource management with deep reinforcement learning,” in _Proceedings of the 15th ACM workshop on hot topics in networks_ , 2016, pp. 50–56.
  * Gregurić et al. (2020) M. Gregurić, M. Vujić, C. Alexopoulos, and M. Miletić, “Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data,” _Applied Sciences_ , vol. 10, no. 11, p. 4011, 2020.
  * Goodfellow et al. (2016) I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1.
  * Liang et al. (2018) E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. E. Gonzalez, M. I. Jordan, and I. Stoica, “RLlib: Abstractions for distributed reinforcement learning,” in _International Conference on Machine Learning (ICML)_ , 2018.
  * Lin et al. (2018) K. Lin, R. Zhao, Z. Xu, and J. Zhou, “Efficient large-scale fleet management via multi-agent deep reinforcement learning,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 1774–1783.
  * Shoham and Leyton-Brown (2008) Y. Shoham and K. Leyton-Brown, _Multiagent systems: Algorithmic, game-theoretic, and logical foundations_. Cambridge University Press, 2008.
  * Cassirer et al. (2021) A. Cassirer, G. Barth-Maron, E. Brevdo, S. Ramos, T. Boyd, T. Sottiaux, and M. Kroiss, “Reverb: A framework for experience replay,” _arXiv preprint arXiv:2102.04736_ , 2021. [Online]. Available: https://arxiv.org/abs/2102.04736
  * Yang et al. (2021) F. Yang _et al._ , “Launchpad,” https://github.com/deepmind/launchpad, 2021.
  * Terry et al. (2020) J. K. Terry, B. Black, M. Jayakumar, A. Hari, L. Santos, C. Dieffendahl, N. L. Williams, Y. Lokesh, R. Sullivan, C. Horsch, and P. Ravi, “Pettingzoo: Gym for multi-agent reinforcement learning,” _arXiv preprint arXiv:2009.14471_ , 2020.
  * Samvelyan et al. (2019) M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson, “The StarCraft Multi-Agent Challenge,” _CoRR_ , vol. abs/1902.04043, 2019\.
  * Mohanty et al. (2020) S. Mohanty, E. Nygren, F. Laurent, M. Schneider, C. Scheller, N. Bhattacharya, J. Watson, A. Egli, C. Eichenberger, C. Baumberger, G. Vienken, I. Sturm, G. Sartoretti, and G. Spigler, “Flatland-rl : Multi-agent reinforcement learning on trains,” 2020.
  * Lanctot et al. (2019) M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zambaldi, S. Upadhyay, J. Pérolat, S. Srinivasan, F. Timbers, K. Tuyls, S. Omidshafiei, D. Hennes, D. Morrill, P. Muller, T. Ewalds, R. Faulkner, J. Kramár, B. D. Vylder, B. Saeta, J. Bradbury, D. Ding, S. Borgeaud, M. Lai, J. Schrittwieser, T. Anthony, E. Hughes, I. Danihelka, and J. Ryan-Davis, “OpenSpiel: A framework for reinforcement learning in games,” _CoRR_ , vol. abs/1908.09453, 2019. [Online]. Available: http://arxiv.org/abs/1908.09453
  * Duan et al. (2016) Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” in _International conference on machine learning_. PMLR, 2016, pp. 1329–1338.
  * Dhariwal et al. (2017) P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov, “Openai baselines,” https://github.com/openai/baselines, 2017.
  * Achiam (2018) J. Achiam, “Spinning Up in Deep Reinforcement Learning,” 2018.
  * Castro et al. (2018) P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare, “Dopamine: A Research Framework for Deep Reinforcement Learning,” 2018. [Online]. Available: http://arxiv.org/abs/1812.06110
  * Guadarrama et al. (2018) S. Guadarrama, A. Korattikara, O. Ramirez, P. Castro, E. Holly, S. Fishman, K. Wang, E. Gonina, N. Wu, E. Kokiopoulou, L. Sbaiz, J. Smith, G. Bartók, J. Berent, C. Harris, V. Vanhoucke, and E. Brevdo, “TF-Agents: A library for reinforcement learning in tensorflow,” https://github.com/tensorflow/agents, 2018, [Online; accessed 25-June-2019]. [Online]. Available: https://github.com/tensorflow/agents
  * Fan et al. (2018) L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei, “Surreal: Open-source reinforcement learning framework and robot manipulation benchmark,” in _Conference on Robot Learning_ , 2018.
  * Espeholt et al. (2019) L. Espeholt, R. Marinier, P. Stanczyk, K. Wang, and M. Michalski, “Seed rl: Scalable and efficient deep-rl with accelerated central inference,” 2019.
  * Budden et al. (2020) D. Budden, M. Hessel, J. Quan, S. Kapturowski, K. Baumli, S. Bhupatiraju, A. Guy, and M. King, “RLax: Reinforcement Learning in JAX,” 2020. [Online]. Available: http://github.com/deepmind/rlax
  * Song et al. (2020) Y. Song, A. Wojcicki, T. Lukasiewicz, J. Wang, A. Aryan, Z. Xu, M. Xu, Z. Ding, and L. Wu, “Arena: A general evaluation platform and building toolkit for multi-agent intelligence.” in _AAAI_ , 2020, pp. 7253–7260.
  * Moritz et al. (2017) P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: A distributed framework for emerging AI applications,” _CoRR_ , vol. abs/1712.05889, 2017. [Online]. Available: http://arxiv.org/abs/1712.05889
  * Lowe et al. (2017) R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in _Advances in neural information processing systems_ , 2017, pp. 6379–6390.
  * Rashid et al. (2018) T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” _arXiv preprint arXiv:1803.11485_ , 2018\.
  * Sunehag et al. (2017) P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls _et al._ , “Value-decomposition networks for cooperative multi-agent learning,” _arXiv preprint arXiv:1706.05296_ , 2017.
  * Chu et al. (2020) T. Chu, S. Chinchali, and S. Katti, “Multi-agent reinforcement learning for networked system control,” _International Conference on Learning Representations_ , 2020.
  * Foerster et al. (2016) J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” in _Advances in neural information processing systems_ , 2016, pp. 2137–2145.
  * Sukhbaatar et al. (2016) S. Sukhbaatar, R. Fergus _et al._ , “Learning multiagent communication with backpropagation,” in _Advances in neural information processing systems_ , 2016, pp. 2244–2252.
  * Andreae (1969) J. Andreae, “Learning machines: A unified view,” _Encyclopaedia of Linguistics, Information and Control, Pergamon Press_ , pp. 261–270, 01 1969.
  * Watkins (1989) C. J. C. H. Watkins, “Learning from delayed rewards,” 1989.
  * Watkins and Dayan (1992) C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * Williams (1992) R. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning.” _Machine Learning_ , pp. 229–256, 1992.
  * Auer et al. (2002) P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” _Machine learning_ , vol. 47, no. 2-3, pp. 235–256, 2002.
  * Sutton et al. (2000) R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in _Advances in neural information processing systems_ , 2000, pp. 1057–1063.
  * Mnih et al. (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * Hessel et al. (2017) M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” _arXiv preprint arXiv:1710.02298_ , 2017\.
  * Lillicrap et al. (2015) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” _arXiv preprint arXiv:1801.01290_ , 2018.
  * Arulkumaran et al. (2017) K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “A brief survey of deep reinforcement learning,” _arXiv preprint arXiv:1708.05866_ , 2017.
  * Li (2017) Y. Li, “Deep reinforcement learning: An overview,” _arXiv preprint arXiv:1701.07274_ , 2017.
  * Mnih et al. (2016) V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_ , 2016, pp. 1928–1937.
  * Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning _et al._ , “Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1407–1416.
  * Kapturowski et al. (2018) S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney, “Recurrent experience replay in distributed reinforcement learning,” in _International conference on learning representations_ , 2018.
  * Barth-Maron et al. (2018) G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” _arXiv preprint arXiv:1804.08617_ , 2018\.
  * Littman (1994) M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in _Machine learning proceedings 1994_. Elsevier, 1994, pp. 157–163.
  * Tan (1993) M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents,” in _Proceedings of the tenth international conference on machine learning_ , 1993, pp. 330–337.
  * Tampuu et al. (2017) A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” _PloS one_ , vol. 12, no. 4, p. e0172395, 2017.
  * Claus and Boutilier (1998) C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems,” _AAAI/IAAI_ , vol. 1998, no. 746-752, p. 2, 1998.
  * Foerster et al. (2017a) J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” _arXiv preprint arXiv:1705.08926_ , 2017.
  * Foerster et al. (2017b) J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. S. Torr, P. Kohli, and S. Whiteson, “Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning,” _ICML_ , 2017.
  * Das et al. (2019) A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “TarMAC: Targeted multi-agent communication,” _36th International Conference on Machine Learning, ICML 2019_ , vol. 2019-June, pp. 2776–2784, 2019\.
  * de Witt et al. (2020) C. S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, “Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control,” _arXiv_ , p. 14, 2020. [Online]. Available: http://arxiv.org/abs/2003.06709
  * Wang et al. (2020a) Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang, “Off-policy multi-agent decomposed policy gradients,” _arXiv preprint arXiv:2007.12322_ , 2020.
  * Powers et al. (2003) R. Powers, Y. Shoham, and T. Grenager, “Multi-Agent Reinforcement Learning : a critical survey,” pp. 1–13, 2003.
  * Busoniu et al. (2008) L. Busoniu, R. Babuska, and B. D. Schutter, “A comprehensive survey of multi-agent reinforcement learning,” _Tech Report_ , p. 18, 2008.
  * Hernandez-Leal et al. (2019a) P. Hernandez-Leal, B. Kartal, and M. E. Taylor, “A survey and critique of multiagent deep reinforcement learning,” _Autonomous Agents and Multi-Agent Systems_ , vol. 33, no. 6, pp. 750–797, 2019.
  * Hernandez-Leal et al. (2019b) P. Hernandez-Leal, M. Kaisers, T. Baarslag, and E. M. de Cote, “A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity,” _arXiv_ , pp. 1–64, 2019. [Online]. Available: http://arxiv.org/abs/1707.09183
  * OroojlooyJadid and Hajinezhad (2019) A. OroojlooyJadid and D. Hajinezhad, “A review of cooperative multi-agent deep reinforcement learning,” _arXiv preprint arXiv:1908.03963_ , 2019.
  * Nguyen et al. (2019) T. T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications,” _arXiv_ , pp. 1–27, 2019. [Online]. Available: http://arxiv.org/abs/1812.11794
  * Zhang et al. (2019) K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _arXiv preprint arXiv:1911.10635_ , 2019.
  * Yang and Wang (2020) Y. Yang and J. Wang, “An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective,” pp. 1–122, 2020. [Online]. Available: http://arxiv.org/abs/2011.00583
  * Muldal et al. (2019) A. Muldal, Y. Doron, J. Aslanides, T. Harley, T. Ward, and S. Liu, “dm_env: A python interface for reinforcement learning environments,” 2019. [Online]. Available: http://github.com/deepmind/dm_env
  * Brockman et al. (2016) G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * Foerster et al. (2017c) J. Foerster, N. Nardelli, G. Farquhar, T. Afouras, P. H. Torr, P. Kohli, and S. Whiteson, “Stabilising experience replay for deep multi-agent reinforcement learning,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1146–1155.
  * Mahajan et al. (2019) A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson, “Maven: Multi-agent variational exploration,” _arXiv preprint arXiv:1910.07483_ , 2019.
  * Rashid et al. (2020) T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” _Journal of Machine Learning Research_ , vol. 21, no. 178, pp. 1–51, 2020.
  * Son et al. (2019) K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, “Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning,” _arXiv preprint arXiv:1905.05408_ , 2019.
  * Son et al. (2020) K. Son, S. Ahn, R. Delos, S. Jinwoo, and Y. Yung, “Qtran++: Improved value transformation for cooperative multi-agent reinforcement learning,” _arXiv preprint arXiv:2006.12010_ , 2020.
  * Wang et al. (2020b) J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “Qplex: Duplex dueling multi-agent q-learning,” _arXiv preprint arXiv:2008.01062_ , 2020.
  * Iqbal et al. (2020) S. Iqbal, C. A. S. de Witt, B. Peng, W. Böhmer, S. Whiteson, and F. Sha, “Ai-qmix: Attention and imagination for dynamic multi-agent reinforcement learning,” _arXiv preprint arXiv:2006.04222_ , 2020.
  * Gupta et al. (2020) T. Gupta, A. Mahajan, B. Peng, W. Böhmer, and S. Whiteson, “Uneven: Universal value exploration for multi-agent reinforcement learning,” _arXiv preprint arXiv:2010.02974_ , 2020.
  * Silver et al. (2014) D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in _International conference on machine learning_. PMLR, 2014, pp. 387–395.
  * Bellemare et al. (2017) M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 449–458.
  * Gupta et al. (2017) J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” in _International Conference on Autonomous Agents and Multiagent Systems_. Springer, 2017, pp. 66–83.


* Bartolo et al. [2004] N. Bartolo, E. Komatsu, Sabino Matarrese, and A. Riotto.  Non-Gaussianity from inflation: Theory and observations.  _Phys. Rept._ , 402:103–266, 2004.  doi: 10.1016/j.physrep.2004.08.022.  arXiv: astro-ph/0406398.
  * Akrami et al. [2020] Y. Akrami et al.  Planck 2018 results. IX. Constraints on primordial non-Gaussianity.  _Astron. Astrophys._ , 641:A9, 2020.  doi: 10.1051/0004-6361/201935891.  arXiv: 1905.05697.
  * Raccanelli et al. [2017] Alvise Raccanelli, Maresuke Shiraishi, Nicola Bartolo, Daniele Bertacca, Michele Liguori, Sabino Matarrese, Ray P. Norris, and David Parkinson.  Future Constraints on Angle-Dependent Non-Gaussianity from Large Radio Surveys.  _Phys. Dark Univ._ , 15:35–46, 2017.  doi: 10.1016/j.dark.2016.10.006.  arXiv: 1507.05903.
  * Komatsu and Spergel [2001] Eiichiro Komatsu and David N. Spergel.  Acoustic signatures in the primary microwave background bispectrum.  _Phys. Rev. D_ , 63:063002, 2001.  doi: 10.1103/PhysRevD.63.063002.  arXiv: astro-ph/0005036.
  * Dalal et al. [2008] Neal Dalal, Olivier Dore, Dragan Huterer, and Alexander Shirokov.  The imprints of primordial non-gaussianities on large-scale structure: scale dependent bias and abundance of virialized objects.  _Phys. Rev. D_ , 77:123514, 2008.  doi: 10.1103/PhysRevD.77.123514.  arXiv: 0710.4560.
  * Matarrese and Verde [2008] Sabino Matarrese and Licia Verde.  The effect of primordial non-Gaussianity on halo bias.  _Astrophys. J. Lett._ , 677:L77–L80, 2008.  doi: 10.1086/587840.  arXiv: 0801.4826.
  * Desjacques and Seljak [2010] V. Desjacques and U. Seljak.  Primordial non-Gaussianity from the large scale structure.  _Class. Quant. Grav._ , 27:124011, 2010.  doi: 10.1088/0264-9381/27/12/124011.  arXiv: 1003.5020.
  * Bruni et al. [2012] Marco Bruni, Robert Crittenden, Kazuya Koyama, Roy Maartens, Cyril Pitrou, and David Wands.  Disentangling non-Gaussianity, bias and GR effects in the galaxy distribution.  _Phys. Rev. D_ , 85:041301, 2012.  doi: 10.1103/PhysRevD.85.041301.  arXiv: 1106.3999.
  * Raccanelli et al. [2013] Alvise Raccanelli, Daniele Bertacca, Davide Pietrobon, Fabian Schmidt, Lado Samushia, Nicola Bartolo, Olivier Dore, Sabino Matarrese, and Will J. Percival.  Testing Gravity Using Large-Scale Redshift-Space Distortions.  _Mon. Not. Roy. Astron. Soc._ , 436:89–100, 2013.  doi: 10.1093/mnras/stt1517.  arXiv: 1207.0500.
  * Riquelme et al. [2023] Walter Riquelme et al.  Primordial non-Gaussianity with angular correlation function: integral constraint and validation for DES.  _Mon. Not. Roy. Astron. Soc._ , 523(1):603–619, 2023.  doi: 10.1093/mnras/stad1429.  arXiv: 2209.07187.
  * Sheth and Tormen [1999] Ravi K. Sheth and Giuseppe Tormen.  Large scale bias and the peak background split.  _Mon. Not. Roy. Astron. Soc._ , 308:119, 1999.  doi: 10.1046/j.1365-8711.1999.02692.x.  arXiv: astro-ph/9901122.
  * Sheth et al. [2001] Ravi K. Sheth, H. J. Mo, and Giuseppe Tormen.  Ellipsoidal collapse and an improved model for the number and spatial distribution of dark matter haloes.  _Mon. Not. Roy. Astron. Soc._ , 323:1, 2001.  doi: 10.1046/j.1365-8711.2001.04006.x.  arXiv: astro-ph/9907024.
  * Barreira [2020] Alexandre Barreira.  On the impact of galaxy bias uncertainties on primordial non-Gaussianity constraints.  _JCAP_ , 12:031, 2020.  doi: 10.1088/1475-7516/2020/12/031.  arXiv: 2009.06622.
  * Fondi et al. [2024] Emanuele Fondi, Licia Verde, Francisco Villaescusa-Navarro, Marco Baldi, William R. Coulton, Gabriel Jung, Dionysios Karagiannis, Michele Liguori, Andrea Ravenni, and Benjamin D. Wandelt.  Taming assembly bias for primordial non-Gaussianity.  _JCAP_ , 02:048, 2024.  doi: 10.1088/1475-7516/2024/02/048.  arXiv: 2311.10088.
  * Bahr-Kalus et al. [2021] Benedict Bahr-Kalus, Daniele Bertacca, Licia Verde, and Alan Heavens.  The Kaiser-Rocket effect: three decades and counting.  _JCAP_ , 11:027, 2021.  doi: 10.1088/1475-7516/2021/11/027.  arXiv: 2107.00351.
  * Limber [1954] D. Nelson Limber.  The Analysis of Counts of the Extragalactic Nebulae in Terms of a Fluctuating Density Field. II.  _Astrophys. J._ , 119:655, 1954.  doi: 10.1086/145870.
  * Kaiser [1992] Nick Kaiser.  Weak gravitational lensing of distant galaxies.  _Astrophys. J._ , 388:272, 1992.  doi: 10.1086/171151.
  * LoVerde and Afshordi [2008] Marilena LoVerde and Niayesh Afshordi.  Extended Limber Approximation.  _Phys. Rev. D_ , 78:123506, 2008.  doi: 10.1103/PhysRevD.78.123506.  arXiv: 0809.5112.
  * Grasshorn Gebhardt and Jeong [2018] Henry S. Grasshorn Gebhardt and Donghui Jeong.  Fast and accurate computation of projected two-point functions.  _Phys. Rev. D_ , 97(2):023504, 2018.  doi: 10.1103/PhysRevD.97.023504.  arXiv: 1709.02401.
  * Elkhashab et al. [2021] Mohamed Yousry Elkhashab, Cristiano Porciani, and Daniele Bertacca.  The large-scale monopole of the power spectrum in a Euclid-like survey: wide-angle effects, lensing, and the ‘finger of the observer’.  _Mon. Not. Roy. Astron. Soc._ , 509(2):1626–1645, 2021.  doi: 10.1093/mnras/stab3010.  arXiv: 2108.13424.
  * Hui et al. [2007] Lam Hui, Enrique Gaztanaga, and Marilena LoVerde.  Anisotropic Magnification Distortion of the 3D Galaxy Correlation. 1. Real Space.  _Phys. Rev. D_ , 76:103502, 2007.  doi: 10.1103/PhysRevD.76.103502.  arXiv: 0706.1071.
  * Montanari and Durrer [2015] Francesco Montanari and Ruth Durrer.  Measuring the lensing potential with tomographic galaxy number counts.  _JCAP_ , 10:070, 2015.  doi: 10.1088/1475-7516/2015/10/070.  arXiv: 1506.01369.
  * Bertacca [2015] Daniele Bertacca.  Observed galaxy number counts on the light cone up to second order: III. Magnification bias.  _Class. Quant. Grav._ , 32(19):195011, 2015.  doi: 10.1088/0264-9381/32/19/195011.  arXiv: 1409.2024.


# Comment on ”Tetraquarks as diquark-antidiquark bound systems”

M. R. Hadizadeh  Institute of Nuclear and Particle Physics and Department of
Physics and Astronomy, Ohio University, Athens, OH 45701, USA

###### Abstract

The author argues that the calculated masses of heavy tetraquarks obtained by
solution of the spin-independent homogeneous Lippmann-Schwinger integral
equation in a diquark-antidiquark picture reported by M. Monemzadeh et al.,
Phys. Lett. B 741, 124 (2015), are incorrect. We have reexamined all of the
published results and we believe that not only the reported tetraquark masses
for states with zero total angular momentum are incorrect, the reported masses
for states with non-zero total angular momentum are quite misleading, because
these states cannot be predicted by a spin-independent formalism.

In a recent letter by Monemzadeh et al. Monemzadeh-PLB , the tetraquark bound
state is studied as a two-body problem in a diquark-antidiquark picture. The
nonrelativistic $s-$wave bound state of a diquark-antidiquark system with
reduced mass $\mu_{D\bar{D}}$ and the binding energy $E_{T}$, interacting with
pair force $V(r)$ can be described in configuration space by homogeneous
Lippmann-Schwinger (LS) integral equation

$\psi(r)=-\frac{2\,\mu_{D\bar{D}}}{4\pi}\int\,d^{3}{\bf
r}^{\prime}\,\frac{e^{-\kappa|{\bf r}-{\bf r}^{\prime}|}}{|{\bf r}-{\bf
r}^{\prime}|}\,V(r^{\prime})\,\psi(r^{\prime}),$ (1)

where $\kappa=\sqrt{2\,\mu_{D\bar{D}}|E_{T}|}$. The corresponding equation in
published letter Monemzadeh-PLB  (i.e. Eq. 2 and consequently Eqs. 4, 9 and
11) is missing a factor of $(2\pi)^{-1.5}$ which should be a typo, because our
numerical analysis demonstrates that missing this factor leads to completely
unreasonable tetraquark masses111To verify derivation of LS integral equation
(1) and test our numerical algorithm and code, we have solved it for spin-
independent Malfliet-Tjon Yukawa potential Malfliet-Tjon  and we have
reproduced the same deuteron binding energy obtained from solution of LS
equation in momentum space..

In order to study tetraquark bound states, the spin-independent part of the
diquark-antidiquark potential given in Ref. Ebert-PLB634  is used, with the
strong coupling constant $\alpha_{s}$ of Ref. ebert2011masses

$\alpha_{s}(\mu)=\frac{4\pi}{\beta_{0}}\frac{1}{\ln\left({\frac{4\mu^{2}_{D\bar{D}}+M_{0}^{2}}{\Lambda^{2}}}\right)},$
(2)

with $\beta_{0}=11-\frac{2}{3}n_{f}$ ($n_{f}$ is the number of flavor quarks),
$M_{0}=2.24\sqrt{A}=0.95\,\text{GeV}$ and $\Lambda=0.413\,\text{GeV}$.

In Table 1 we have listed our numerical results for $1s$ heavy charm
tetraquarks for states with total angular momentum equal to zero. As it is
shown our calculated masses are all larger than the predicted results of Ref.
Ebert-PLB634  which are obtained from solution of relativistic and spin-
dependent LS integral equation in momentum space. Our results seem to be quite
reasonable, because our numerical analysis indicates that the relativistic
effects lead to a reduction in the calculated masses Hadizadeh-in-preparation
. Clearly the reported results in Ref. Monemzadeh-PLB  can not be trusted,
because as it is shown in Table 1, not only $J=0$ states are not calculated
correctly, they have reported the masses for non zero $J$ states, like
$\frac{S\bar{A}\pm A\bar{S}}{\sqrt{2}}$ and $A\bar{A}$, which cannot be
obtained in this formalism. It is a serious challenge and the authors should
clarify how this spin-independent formalism can distinguish different spin
states and consequently the solution of spin-independent LS integral equation
can predict the masses of tetraquarks with non zero total angular momentum.

Table 1: Masses of charm diquark-antidiquark for $1s$ state with total angular momentum $J=0$ calculated from non-relativistic Lippmann-Schwinger equation. $S$ and $A$ denote the scalar and axial vector diquarks. Diquark content |  | Mass [GeV] |
---|---|---|---
|  | $cq\bar{c}\bar{q}$ |
| present | Ref. Monemzadeh-PLB  | Ref. Ebert-PLB634
$S\bar{S}$ | 3.885 | 3.70314 | 3.812
$A\bar{A}$ | 4.013 | 3.83908 | 3.852
|  | $cs\bar{c}\bar{s}$ |
| present | Ref. Monemzadeh-PLB  | Ref. Ebert-PLB634
$S\bar{S}$ | 4.117 | 4.05390 | 4.051
$A\bar{A}$ | 4.250 | 4.09962 | 4.110

Moreover, from numerical point of view the following issue should be
considered. Since a regularized form of the diquark-antidiquark potential is
used to overcome the singularity of the confining potential at large
distances, the calculated masses should be independent of the regularization
cutoff $r_{c}$. In Ref. Monemzadeh-PLB  the authors have chosen the non-zero
root of the potential as regularization cutoff and consequently the potential
is fixed equal to zero for $r\geq r_{c}$. As we have shown in Table 2 clearly
this cutoff is not high enough and in order to achieve the cutoff-independent
results, converged with four significant digits, one needs to choose a cutoff
at least equal to $5$ GeV-1. Choosing a regularization cutoff equal to non
zero root of the diquark-antidiquark potential leads to smaller tetraquark
masses which are shown with bold numbers in Table 2, even with the same
regularization cutoff the reported masses in Ref. Monemzadeh-PLB  are
different from our results.

We should mention that in our calculations we have used $100$ mesh points of
Gauss-Legendre quadrature for integration over angle between ${\bf r}$ and
${\bf r}^{\prime}$ with a linear mapping, whereas the integration over
$r^{\prime}$ is done by a hyperbolic-linear mapping with the sub-intervals
$[0,\frac{r_{c}}{2}]\,+[\frac{r_{c}}{2},r_{c}]+[r_{c},20]$ GeV-1 using 150
mesh points to achieve the converged masses with four significant digits .

Table 2: Masses of charm diquark-antidiquark for $1s$ state with total angular momentum $J=0$ as a function of regularization cutoff $r_{c}$. The bold numbers are calculated for values of $r_{c}$ which are non zero root of the diquark-antidiquark potential and numbers in parentheses are corresponding masses from Ref. Monemzadeh-PLB . $r_{c}$ [GeV-1] |  | Mass [GeV] |
---|---|---|---
|  | $cq\bar{c}\bar{q}$ |
| $S\bar{S}$ |  | $A\bar{A}$
2.831 | 3.858 |  |  3.987 (3.83908)
2.844 |  3.859 (3.70314) |  | 3.987
2.85 | 3.859 |  | 3.988
2.9 | 3.862 |  | 3.991
3.0 | 3.867 |  | 3.995
3.5 | 3.879 |  | 4.007
4.0 | 3.883 |  | 4.011
5.0 | 3.885 |  | 4.013
6.0 | 3.885 |  | 4.013
|  | $cs\bar{c}\bar{s}$ |
| $S\bar{S}$ |  | $A\bar{A}$
2.808 | 4.092 |  |  4.226 (4.09962)
2.821 |  4.093 (4.05390) |  | 4.227
2.85 | 4.094 |  | 4.228
2.9 | 4.097 |  | 4.231
3.0 | 4.101 |  | 4.235
3.5 | 4.112 |  | 4.245
4.0 | 4.116 |  | 4.248
5.0 | 4.117 |  | 4.250
6.0 | 4.117 |  | 4.250

Although the conclusions based on the incorrect results are misleading but we
don’t understand this part of their conclusion: ”our results are in good
agreement with the results derived from complicated relativistic methods and
can be a good replacement for them”. How a nonrelativistic spin-independent
method can be an alternative to a relativistic spin-dependent method, even
when the results of both methods be in good agreement?

In conclusion, our calculations demonstrate that all of the results published
in recent letter by M. Monemzadeh et al. Monemzadeh-PLB , for masses of
tetraquarks with total angular momentum equal to zero, are incorrect and the
letter has serious numerical problems. The correct results should be as
results given in Table 1. Moreover, beside these incorrect results, we have
difficulty to understand how the authors have obtained the masses of
tetraquarks for non zero total angular momentum in a spin-independent
formalism!

## Acknowledgments

The author would like to thank V.O. Galkin for providing the values of strong
coupling constant for different tetraquark states. This work is supported by
National Science Foundation under contract NSF-PHY-1005587 with Ohio
University. Partial support was also provided by the Institute of Nuclear and
Particle Physics at Ohio University.

## References

  * (1) M. Monemzadeh, N. Tazimi, P. Sadeghi, Phys. Lett. B 741, 124 (2015).
  * (2) R. A. Malfliet and J. A. Tjon, Nucl. Phys. A 127, 161 (1969).
  * (3) D. Ebert, R. N. Faustov, V. O. Galkin, Phys. Lett. B 634, 214 (2006).
  * (4) D. Ebert, R. Faustov, V. Galkin, Phys. Lett. B 696, 241 (2011).
  * (5) M. R. Hadizadeh, A. Khaledi-Nasab, in preparation.