# ELIGN: Expectation Alignment
as a Multi-Agent Intrinsic Reward

Zixian Ma1, Rose Wang1, Li Fei-Fei1, Michael Bernstein1, Ranjay Krishna1,2
Stanford University1, University of Washington2
<EMAIL_ADDRESS>

###### Abstract

Modern multi-agent reinforcement learning frameworks rely on centralized
training and reward shaping to perform well. However, centralized training and
dense rewards are not readily available in the real world. Current multi-agent
algorithms struggle to learn in the alternative setup of decentralized
training or sparse rewards. To address these issues, we propose a self-
supervised intrinsic reward ELIGN - expectation alignment -  inspired by the
self-organization principle in Zoology. Similar to how animals collaborate in
a decentralized manner with those in their vicinity, agents trained with
expectation alignment learn behaviors that match their neighbors’
expectations. This allows the agents to learn collaborative behaviors without
any external reward or centralized training. We demonstrate the efficacy of
our approach across 6 tasks in the multi-agent particle and the complex Google
Research football environments, comparing ELIGN to sparse and curiosity-based
intrinsic rewards. When the number of agents increases, ELIGN scales well in
all multi-agent tasks except for one where agents have different capabilities.
We show that agent coordination improves through expectation alignment because
agents learn to divide tasks amongst themselves, break coordination
symmetries, and confuse adversaries. These results identify tasks where
expectation alignment is a more useful strategy than curiosity-driven
exploration for multi-agent coordination, enabling agents to do zero-shot
coordination.

##  1 Introduction

Many real world AI applications can be formulated as multi-agent systems,
including autonomous vehicles (Cao et al., 2012), resource management (Ying &
Dayong, 2005), traffic control (Sunehag et al., 2017), robot swarms (Swamy et
al., 2020), and multi-player video games (Berner et al., 2019). Agents must
adapt their behaviors to each other in order to coordinate successfully in
these systems. However, adaptive coordination algorithms are challenging to
develop because each agent is not privy to other agents’ intentions and their
future behaviors (Foerster et al., 2017).

These challenges are more acute in decentralized training under partial
observability than centralized training or full observability. In the real
world, agents act under partial observability and learn in a decentralized
manner: they do not learn collaborative behaviors with a single centralized
algorithm with a complete knowledge of the environment (Iqbal & Sha, 2019; Liu
et al., 2020). Unfortunately, the most successful multi-agent algorithms train
agents with a centralized critic, assuming access to all agents’ observations
and actions (Foerster et al., 2018; Rashid et al., 2018; Sunehag et al., 2017;
Lowe et al., 2017). The most successful multi-agent algorithms for
decentralized training and partial observability assume task-specific reward
shaping (Jain et al., 2020; Iqbal & Sha, 2019), which is expensive to
generate. These algorithms struggle to learn with sparse reward structure.

Consider a cooperative navigation task, where $N$ agents aim to simultaneously
occupy $N$ goal locations. A centralized algorithm with full observability is
capable of optimally assigning the nearest goal location to each respective
agent. However, with partial observability, agents can see only a handful of
goal locations and other agents. They are unaware of others’ observations,
actions, and intentions with decentralized training. We observe that agents
simultaneously occupy the same goal; they fail to collaborate because they do
not predict which goal each agent is expected to occupy. To overcome instances
of miscoordination, decentralized algorithms have adapted single-agent
curiosity-based intrinsic rewards (Pathak et al., 2017; Stadie et al., 2015).
Multi-agent curiosity-based rewards incentivize agents to explore novel states
(Iqbal & Sha, 2020). Although curiosity helps agents discover new goal
locations, it doesn’t solve the challenge of coordination, such as assigning
goals to each agent. Only a few attempts explore other forms of multi-agent
intrinsic rewards (Iqbal & Sha, 2020; Böhmer et al., 2019; Schafer, 2019).

Figure 1: We introduce ELIGN, i.e, expectation alignment, a task-agnostic
intrinsic reward to improve multi-agent systems. Intuitively, ELIGN encourages
agents to become more predictable to their neighbors. An agent (e.g., agent
$i$ here) learns to behave in ways that match its neighbors’ (e.g., agent
$j$’s) predictions of its next observation. Here, agent $j$ expects agent $i$
to move up instead of down, moving closer to a point of interest above it.
Agent $i$ attains (a) a higher reward when its action (e.g., upward) aligns
with this expectation or (b) a lower reward when its action (e.g., downward)
is misaligned.

In this work, we propose ELIGN as a novel multi-agent self-supervised
intrinsic reward, enabling decentralized training under partial observability.
Intuitively, expectation alignment encourages agents to elicit behaviors that
decrease future uncertainty for their team: it encourages each agent to choose
actions that match their teammates’ expectations. Going back to the
cooperative navigation task, expectation alignment encourages each agent to
move to goals others expect it to occupy, like goals that are either closest
to the agent or goals that other agents aren’t moving towards (Figure 1). We
take inspiration from the self-organization principle in Zoology (Couzin,
2007). This principle hypothesizes that collective animal intelligence emerges
because groups synchronize their behaviors using only their local environment;
they do not rely on complete information about other agents and can coordinate
successfully by predicting the dynamics of agents within their field-of-view
(Collett et al., 1998; Theraulaz & Bonabeau, 1995; Ben-Jacob et al., 1994;
Buhl et al., 2006). Similarly, expectation alignment as an intrinsic reward is
calculated based on the agent’s local observations and its approximation of
neighboring agents’ expectations. It does not require a centralized controller
nor full observability. ELIGN is task-agnostic and we apply it to both
collaborative and competitive multi-agent tasks.

We demonstrate the efficacy of our approach in the multi-agent particle and
Google Research football environments, two popular benchmarks for multi-agent
reinforcement learning (Lowe et al., 2017; Kurach et al., 2019). We evaluate
ELIGN under partial and full observability, with decentralized and centralized
training, and in terms of scalability. We observe that expectation alignment
outperforms sparse and curiosity-based intrinsic rewards (Ndousse et al.,
2021; Stadie et al., 2015; Iqbal & Sha, 2020), especially under partial
observability with decentralized training. We additionally test expectation
alignment as a way to perform zero-shot coordination with new agent partners,
and investigate why ELIGN improves coordination. We show that agent
coordination improves through expectation alignment because agents learn to
divide tasks amongst themselves and break coordination symmetries (Hu et al.,
2020).

##  2 Related Work

Our formulation of expectation alignment, a task-agnostic intrinsic reward for
multi-agent training, draws inspiration from the self-organization principle
in Zoology, which posits that synchronized group behavior is mediated by local
behavioral rules (Couzin, 2007) and not by a centralized controller (Camazine
et al., 2020). Group cohesion emerges by predicting and adjusting one’s
behavior to that of near neighbors (Buhl et al., 2006). This principle
underlies the coordination found in multi-cellular organisms (Camazine et al.,
2020), the migration of wingless locusts (Collett et al., 1998), the
collective swarms of bacteria (Ben-Jacob et al., 1994), the construction of
bridge structures by ants (Theraulaz & Bonabeau, 1995), and some human
navigation behaviors (Couzin, 2007).

Intrinsic motivation for single agents. Although we draw inspiration from
Zoology for formalizing expectation alignment as an intrinsic reward, there is
a rich body of work on intrinsic rewards within the single-agent reinforcement
learning community. To incentivize exploration, even when non-optimal
successful trajectories are uncovered first, scholars have argued for the use
of intrinsic motivation (Schmidhuber, 1991). Single-agent intrinsic motivation
has focused on exploring previously unencountered states (Pathak et al., 2017;
Burda et al., 2018a), which works particularly well in discrete domains. In
continuous domains, identifying unseen states requires keeping track of an
intractable number of visited states; instead, literature has recommended
learning a forward dynamics model to predict future states and identify novel
states using the uncertainty of this model (Achiam & Sastry, 2017). Other
formulations encourage re-visiting states where the dynamics model’s
prediction of future states errs (Stadie et al., 2015; Pathak et al., 2017).
Follow up papers have improved how uncertainty (Kim et al., 2020) and model
errors (Burda et al., 2018b; Sekar et al., 2020) are calculated.

Intrinsic motivation for multiple agents. Most multi-agent intrinsic rewards
have been adapted from single-agent curiosity-based incentives
(iqbal2019coordinated; Böhmer et al., 2019; Schafer, 2019) and have primarily
focused on cooperative tasks. They propose intrinsic rewards to improve either
coordination, collaboration, or deception: These rewards either maximize
information conveyed by an agent’s actions (iqbal2019coordinated; Chitnis et
al., 2020; Wang et al., 2019), shape the influence of an agent (Jaques et al.,
2019; Foerster et al., 2017), incentivize agents to hide intentions (Strouse
et al., 2018), build accurate models of other agents’ policies (Hernandez-Leal
et al., 2019; Jaques et al., 2019), or break extrinsic rewards for better
credit assignment (Du et al., 2019).

Several multi-agent intrinsic rewards (Hernandez-Leal et al., 2019; Jaques et
al., 2019), including ours, rely on the ability to model others’ dynamics in a
shared environment. This ability is a key component to coordination, closely
related to Theory of Mind (Tomasello et al., 2005). Our work can be
interpreted as using a Theory of Mind model of others’ behaviors to calculate
an intrinsic motivation loss. Unlike existing Theory of Mind methods that
learn a model per collaborator (Roy et al., 2020), we learn a single dynamics
model, allowing our method to scale as the number of agents increase. Our
proposal is related to model-based reinforcement learning (Jaderberg et al.,
2016; Wang et al., 2020a); however, instead of learning a dynamics model for
control, we learn a dynamics model as a source of reward. Our work is closely
related to a recently proposed auxiliary loss on predicting an agent’s own
future states (Ndousse et al., 2021). However, there are three key
differences. First, their work predicts ego-agent observations, whereas our
work additionally predicts future observations from the other agents’ point of
view. Second, their loss optimizes state embeddings while ours optimizes
agents’ policies. Third, their work focuses on cooperative tasks whereas ours
applies to both cooperative and competitive domains.

Multi-agent reinforcement learning algorithms. Today, the predominant deep
multi-agent framework uses actor-critic methods with a centralized critic and
decentralized execution (Lowe et al., 2017; Foerster et al., 2018; Iqbal &
Sha, 2019; Liu et al., 2020; Rashid et al., 2018). This framework allows a
critic to access the observations and actions of all agents to ease training.
However, there are several situations where centralized training may not be
desirable or possible. Examples include low bandwidth communication
restrictions or human-robot tasks where observations cannot be easily shared
between agents (Ying & Dayong, 2005; Cao et al., 2012; Huang et al., 2015).
Decentralized training is therefore the most practical training paradigm but
it suffers from unstable training: the environment is nonstationary from a
single-agent’s perspective (Lowe et al., 2017). Our work uses a decentralized
training framework and tackles the nonstationarity challenge with an intrinsic
reward designed to improve an agent’s ability to model others. We also apply
expectation alignment to centralized training and observe that it still aids
cooperative and some competitive tasks.

##  3 Background

We formulate our setting as a partially observable Markov game
$(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},r_{\textrm{ex}},N)$
(Littman, 1994). A Markov game for $N$ agents is defined by a state space
$\mathcal{S}$ describing the possible configurations of the environment. The
observation space for agents is
$\mathcal{O}=(\mathcal{O}_{1},\ldots,\mathcal{O}_{N})$ and the action space is
$\mathcal{A}=(\mathcal{A}_{1},\ldots,\mathcal{A}_{N})$. Each agent $i$
observes $\mathbf{o}_{i}\in\mathcal{O}_{i}$, a private partial view of the
state, and performs actions $a_{i}\in\mathcal{A}_{i}$. Using the observation,
each agent uses a stochastic policy
$\pi_{\theta_{i}}:\mathcal{O}_{i}\times\mathcal{A}_{i}\rightarrow[0,1]$, where
$\theta_{i}$ parameterizes the policy. The environment changes according to
the state transition function which transitions to the next state using the
current state and each agent’s actions,
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$. The team of
agents obtains a shared extrinsic reward as a function of the environment
state, $r_{\textrm{ex}}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$.
The team’s goal is to maximize the total expected return:
$R=\sum_{t=0}^{T}\gamma^{t}r_{\textrm{ex}}^{t}$ where $0\leq\gamma\leq 1$ is
the discount factor, $t$ is the time step, and $T$ is the time horizon. The
environment may also contain adversarial agents who have their own reward
structure.

##  4 Expectation Alignment

To understand expectation alignment intuitively, let’s revisit the cooperative
navigation task, where $N$ agents are rewarded for simultaneously occupying as
many goal locations as possible. In Figure 1, agent $i$ has a dynamics model
trained on its past experiences. It predicts how future states will evolve
from the point of view of agent $j$, who is within $i$’s view. In this
example, $j$ will expect $i$ to move towards the goal since $i$ is closer to
it. ELIGN encourages $i$ to pursue the action that $j$ expects (Figure 1(a)).
In turn, $j$ can now assume that the observed goal location will eventually be
occupied by $i$ and should therefore explore to find another goal. By aligning
shared expectations, agent behaviors become more predictable. Conversely, when
neighbors behave opposite to an agent’s predictions, the agent can infer about
the environment outside of its own receptive field (Krause et al., 2002). For
example, in Figure 1 (b), if agent $j$ observes $i$ running away from a goal,
this surprising behavior might indicate the existence of an adversary outside
$j$’s receptive field.

Our training algorithm consists of three interwoven phases of learning a
dynamics model, calculating the ELIGN reward, and optimizing the agent’s
policy (Algorithm 1).

1: Initialize replay buffer $D$ and $D^{\prime}$

2: Initialize $N$ agents with random $\theta_{i}$: $i\in[1,N]$

3: while not converged do

4: for $b=1\ldots B$ do

5: Populate buffer $D$ with episode using policies
$(\pi_{\theta_{1}},\ldots,\pi_{\theta_{N}})$

6: end for

7: // Train dynamics model

8: for agent $i=1\ldots N$ do

9: Sample transitions: $\\{(o_{i},a_{i},r_{\textrm{ex}},o^{\prime}_{i})\\}\sim
D_{i}$

10: Predict $\hat{o}_{i}^{\prime}=f_{\theta_{i}}(o_{i},a_{i})$

11: Update dynamics $\theta_{i}$ using $o^{\prime}_{i}$.

12: end for

13: // Calculate ELIGN reward

14: for agent $i=1\ldots N$ do

15: Sample $B$ transitions:
$\\{(o_{i},a_{i},r_{\textrm{ex}},o^{\prime}_{i})\\}\sim D_{i}$

16: Compute intrinsic rewards $r_{\textrm{in}}(o_{i},a_{i})$

17: Add $\\{(o_{i},a_{i},r_{\textrm{ex}}+\beta
r_{\textrm{in}},o^{\prime}_{i})\\}$ to $D^{\prime}_{i}$

18: end for

19: // Policy learning

20: Update all $\theta_{i}$s using transitions from $D^{\prime}$

21: end while

Algorithm 1 ELIGN: Expectation Alignment

###  4.1 Training the dynamics model

Similar to prior work (Wang et al., 2018; Kidambi et al., 2020), each agent
$i$ learns a dynamics model $f_{\theta_{i}}$ to predict the next observation
$\hat{o}^{\prime}_{i}$ given its current observation and action $o_{i},a_{i}$,
i.e,

$\hat{o}^{\prime}_{i}=f_{\theta_{i}}(o_{i},a_{i}).$

We use a three-layer Multi-Layer Perceptron with ReLU non-linearities as the
dynamics model. We minimize the mean squared error between its prediction and
ground truth next observation $o^{\prime}_{i}$.

###  4.2 Calculating intrinsic reward

The intrinsic reward captures how well agent $i$ aligns to its neighbors’
(e.g., agent $j$’s) expectations on its next state. Calculating this reward
requires $j$ to accurately predict $i$’s behavior, simulating a Theory of Mind
(Tomasello et al., 2005). As suggested by the self-organization principle, $i$
must learn to align to $j$’s predictions. Ideally, the ELIGN intrinsic reward
is calculated as:

$r_{\textrm{in}}(o_{i},a_{i})=-\frac{1}{|\mathcal{N}(i)|}\sum_{j\in\mathcal{N}(i)}{\|o^{\prime}_{i}-f_{\theta_{j}}(o_{i},a_{i})\|}$

where $\mathcal{N}(i)$ is the set of neighbors within $i$’s receptive field,
including $i$ itself. The ELIGN reward is high when the average $L_{2}$ loss
is small, i.e, when $i$’s actual next observation is close to agent $j$’s
predicted observation of $i$ for all $j$ in its neighbors. In that case, $i$
has chosen an action that aligns with $j$’s expectations of how $i$ should
act.

In a decentralized training setup, however, $i$ doesn’t have access to $j$’s
dynamics model $f_{\theta_{j}}$, so $i$ approximates $j$’s dynamic model with
a proxy: its own dynamics model $f_{\theta_{i}}$ and the knowledge of agent
$j$’s observation radius. Such an approximation is ecologically valid since we
often approximate others’ behaviors using a second-order cognitive Theory of
Mind (Morin, 2006). Additionally, $i$ doesn’t have access to $j$’s entire
observation; so, we restrict the future prediction from $j$’s point of view by
using the portion of $j$’s observation $i$ can see: $o_{i\cap j}=o_{i}\odot
o_{j}$. Agent $i$’s decentralized intrinsic reward then becomes:

$r_{\textrm{in}}(o_{i},a_{i})=-\frac{1}{|\mathcal{N}(i)|}\sum_{j\in\mathcal{N}(i)}{\|o^{\prime}_{i\cap
j}-f_{\theta_{i}}(o_{i\cap j},a_{i})\|}$

We found that the approximation of $f_{\theta_{j}}$ using $f_{\theta_{i}}$
works well empirically. Dynamics model losses for all agents quickly decrease
within 5-10 training epochs. we validate its applicability in small-scale
heterogeneous multi-agent tasks where agents have variable capabilities,
although we find the methods perform similarly when more heterogeneous agents
are added.

###  4.3 Policy learning

Once the ELIGN rewards are calculated, the total rewards at each step for each
agent $i$ is: $r_{i}=r_{\textrm{ex}}+\beta r_{\textrm{in}}(o_{i},a_{i})$ where
$r_{\textrm{ex}}$ is the extrinsic reward provided by the environment and
$\beta$ is a hyperparameter for weighing the intrinsic reward in the agent’s
overall reward calculation. In practice, we set $\beta$ to be
$\frac{1}{|\mathcal{O}_{i}|}$ where $|\mathcal{O}_{i}|$ is the observation
dimension; we find this scale generalizes well across tasks. Since our
contribution is agnostic to any particular multi-agent training algorithm, the
team of agents can now be trained using any multi-agent training algorithm to
maximize returns $R=\sum_{t=0}^{T}\gamma^{t}r$.

Both centralized and decentralized training algorithms can make use of these
rewards. We primarily use the multi-agent decentralized variant of the soft-
actor critic algorithm in our experiments (Haarnoja et al., 2018; Iqbal & Sha,
2019). Compared to centralized joint-action training, whose action space grows
exponentially in $N$ agents, our decentralized method has linear space
complexity. Further, decentralized training can parallelize training time to
be less than linear with respect to $N$. Although we present results with one
centralized training framework, studying the impact of expectation alignment
with all the centralized critic frameworks is out of scope for this paper.

###  4.4 Extending expectation alignment to competitive tasks

We extend the ELIGN formulation to competitive tasks where a team of agents
compete against adversaries. In this case, agents are encouraged to misalign
with their adversaries’ expectations, i.e, agents are incentivized to be
unpredictable to their adversaries within its receptive field
($\mathcal{N}_{\text{adv}}(i)$):

$\displaystyle r_{\text{in}}$
$\displaystyle=\frac{1}{|\mathcal{N}_{\text{adv}}(i)|}\sum_{k\in\mathcal{N}_{\text{adv}}(i)}{\|o^{\prime}_{i\cap
k}-f_{\theta_{i}}(o_{i\cap k},a_{i}))\|}$

##  5 Experiments

Our experiments explore the utility of using expectation alignment as an
intrinsic reward compared to sparse and curiosity-based intrinsic rewards. We
primarily focus on decentralized training under partial observability.
However, we also demonstrate that ELIGN can easily augment centralized methods
and assist in fully observable tasks. We vary the number of agents in the
multi-agent particle tasks to test scalability. We end by investigating how
and why ELIGN improves coordination by designing three evaluation conditions.
First, does expectation alignment improve coordination by breaking symmetries
(Hu et al., 2020; Wang et al., 2020b)? Second, does ELIGN enable zero-shot
generalization to new partners? Lastly, is the dynamics model critical in
aligning agent behaviors?

###  5.1 Environments

We evaluate ELIGN across both cooperative and competitive tasks in the multi-
agent particle environment (Mordatch & Abbeel, 2017; Lowe et al., 2017) and
the Google Research football environment (Kurach et al., 2019).

State and action space The multi-agent particle environment is a two-
dimensional world. The Google Research football environment is a three-
dimensional world. Both environments have continuous state spaces and discrete
action spaces. Particle agents observe all agents’ positions and velocities.
They can “stay” or change their velocity in one of the four cardinal
directions. Each football agent controls one player. Players observe the ball,
other players’ positions and directions. They can apply one of ten actions
from “top_left”, “top”, “top_right”, “right”, “bottom_right”, “bottom”,
“bottom_left”, “sprint”, and “dribble”.

Observability The original environments assume full observability, where each
agent can observe the position $p=(x,y)$ and velocity $v=(\Delta x,\Delta y)$
of all agents; each agent’s observation vector is thus
$\mathbf{o}_{i,\text{full}}=[p_{1},\ldots,p_{N},v_{1},\ldots,v_{N}]$. We
extend these environments to be partially observable, where agent $i$ observes
only the portion within its receptive field; like prior work with partial
observability (Corder et al., 2019), we hide the position and velocity
information of any agent $j$ outside of agent $i$’s receptive field; i.e, if
the Euclidean distance between agent $i$ and $j$ surpasses a vicinity
threshold $\tau$, then $p_{j}$ and $v_{j}$ are $0$ in
$\mathbf{o}_{i,\text{partial}}$. We set $\tau=0.5$ for partially observable
and $\infty$ in the original fully observable case, where the world’s width
and height are $2.0$ in the multi-agent particle environment and $0.84:2.00$
in the Google Research football environment. Both environments also contain
objects such as obstacles, goals and a ball; they are similarly hidden if out
of sight. Partial observability is a more ecologically valid training
condition since most agents in real-world tasks can only observe a small
portion of their environment at a given time.

Table 1: We report the mean test episode extrinsic rewards and standard errors
of decentralized training under partial observability in multi-agent particle
and Google Research football environments.
$\textrm{{elign}}_{\textrm{self/team}}$ outperform sparse and both curiosity-
based intrinsic rewards. $\textrm{{elign}}_{\textrm{adv}}$ achieves the best
performance among all competitive tasks except for Physical deception, where
$\textrm{{elign}}_{\textrm{team}}$ is the best. These results demonstrate the
benefit of using alignment as intrinsic reward to train better decentralized
policies under partial observability.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt# v Adv#) | Coop nav. (3v0) |  | Hetero nav. (4v0) |  | Phy decep. (2v1) |  | Pred-prey (2v2) |  | Keep-away (2v2) |  | 3v1 w/ keeper (3v2)
$\textsc{sparse}^{1}$ | $139.07\pm 13.63$ |  | $284.42\pm 12.83$ |  | $93.60\pm 8.61$ |  | $-4.72\pm 2.4$ |  | $4.58\pm 3.27$ |  | $0.020\pm 0.001$
$\textrm{{curio}}_{\textrm{self}}^{2}$ | $133.93\pm 7.66$ |  | $286.22\pm 9.97$ |  | $68.80\pm 7.93$ |  | $-6.50\pm 2.18$ |  | $11.88\pm 2.88$ |  | $0.024\pm 0.004$
$\textrm{{curio}}_{\textrm{team}}^{3}$ | $125.42\pm 11.95$ |  | $262.28\pm 22.59$ |  | $85.31\pm 11.93$ |  | $-3.57\pm 1.75$ |  | $9.54\pm 5.04$ |  | $0.021\pm 0.002$
$\textrm{{elign}}_{\textrm{self}}$ | $\mathbf{155.88}\pm\mathbf{5.11}$ |  | $292.34\pm 9.24$ |  | $69.91\pm 4.51$ |  | $-7.58\pm 2.55$ |  | $12.84\pm 4.29$ |  | $0.003\pm 0.018$
$\textrm{{elign}}_{\textrm{team}}$ | $141.04\pm 8.04$ |  | $\mathbf{311.67}\pm\mathbf{10.88}$ |  | $\mathbf{101.72}\pm\mathbf{6.31}$ |  | $-7.69\pm 2.69$ |  | $2.96\pm 4.03$ |  | $0.022\pm 0.001$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $92.20\pm 4.23$ |  | $\mathbf{-2.51}\pm\mathbf{1.70}$ |  | $\mathbf{19.46}\pm\mathbf{5.05}$ |  | $\mathbf{0.025}\pm\mathbf{0.001}$
$\textsc{Hand-crafted}^{1}$ | $75.56\pm 18.90$ |  | $228.48\pm 18.88$ |  | $94.25\pm 14.75$ |  | $\mathbf{-0.77}\pm\mathbf{0.17}$ |  | $\mathbf{52.14}\pm\mathbf{3.11}$ |  | $-$
1 Lowe et al. (2017); Kurach et al. (2019),2 Stadie et al. (2015),3 Iqbal & Sha (2020) |  |

Figure 2: We plot the average test occupancy/collision count per step of
decentralized algorithms under partially observable multi-agent particle
tasks. On these metrics, $\textrm{{elign}}_{\textrm{self}}$ and
$\textrm{{elign}}_{\textrm{adv}}$ perform the best on cooperative and
competitive tasks respectively.

###  5.2 Tasks

Multi-agent particle environment We use the following tasks from the multi-
agent particle environment (Lowe et al., 2017; Liu et al., 2020). We choose
$N$ based on prior work.

Cooperative navigation: $N$ agents must cooperate to reach a set of $N$ goal
locations. Agents are collectively rewarded based on the occupancy of any
agent on any goal location.

Heterogeneous navigation: $N$ agents must reach $N$ goals but they differ in
speeds and sizes. $\frac{N}{2}$ agents are slow and big, and the other
$\frac{N}{2}$ agents are fast and small.

Physical deception: $N$ agents cooperate to reach a single goal location and
are rewarded if any one occupies the goal. However, they are penalized when
any of $\frac{N}{2}$ adversaries occupies the goal and gets rewarded. The
adversaries do not know which landmark is the goal and must infer it from the
agents’ behavior. The agents should learn to deceive the adversaries by
covering all the landmarks.

Keep-away: There are $N$ landmarks, one of which is the goal and known to $N$
agents. Agents are rewarded for occupying it and preventing $M$ adversaries
from reaching it. Adversaries are rewarded for pushing the agents away from
the goal, but they can only infer which landmark is the goal.

Predator-prey: $N$ slow adversaries chase and capture $N$ fast cooperating
agents around a randomly generated obstacle-filled environment. Each time an
adversary catches an agent, the agent is penalized and the adversary is
rewarded.

Google Research football We use the Academy 3vs1 with Keeper competitive task
from the Google Research football environment (Kurach et al., 2019). In this
task, three agents try to score from the edge of the penalty box, one on each
side, and the other at the center. This task is initialized with the centered
agent having the ball and facing the defender. There is an adversary who plays
the keeper.

###  5.3 Training and evaluation

We train all algorithms with $5$ random seeds. All the hyperparameters used in
the training can be found in the Appendix. For the Multi-agent particle
environment, each experiment uses one Tesla K40 GPU to train until
convergence, i.e. the best evaluation episode reward hasn’t changed for $100$
epochs. Each epoch equates to $800K$ episodes of $25$ timesteps. We evaluate
the algorithms by running $1K$ test episodes of $25$ timesteps and mainly
report the mean average test episode reward and standard error across the
random seeds. We also evaluate on task-specific metrics, including agent-goal
occupancy/agent-adversary collision count, and agent-goal/agent-adversary
distance. For Google Research football, each experiment trains for $5M$
timesteps. We evaluate on and report the mean average episode rewards and the
standard errors across the seeds.

###  5.4 Baselines

Table 2: We report the mean test episode extrinsic rewards and standard errors
as the number of agents is increased and trained using of decentralized
algorithms. When the number of agents increases, one of elign still performs
the best in all tasks except for Heterogenous navigation.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (5v0) |  | Hetero nav. (6v0) |  | Phy decep. (4v2) |  | Pred-prey (4v4) |  | Keep-away (4v4)
$\textsc{sparse}^{1}$ | $459.92\pm 22.44$ |  | $616.62\pm 25.30$ |  | $166.89\pm 27.72$ |  | $-28.75\pm 7.3$ |  | $0.75\pm 1.82$
$\textrm{{curio}}_{\textrm{self}}^{2}$ | $458.45\pm 19.79$ |  | $\mathbf{702.73}\pm\mathbf{18.57}$ |  | $146.55\pm 29.05$ |  | $-25.35\pm 6.16$ |  | $10.52\pm 5.48$
$\textrm{{curio}}_{\textrm{team}}^{3}$ | $497.15\pm 11.47$ |  | $695.38\pm 12.22$ |  | $84.66\pm 16.94$ |  | $-17.21\pm 8.23$ |  | $1.40\pm 2.06$
$\textrm{{elign}}_{\textrm{self}}$ | $\mathbf{498.24}\pm\mathbf{9.77}$ |  | $646.70\pm 23.25$ |  | $137.38\pm 30.00$ |  | $\mathbf{-9.14}\pm\mathbf{5.57}$ |  | $9.83\pm 11.22$
$\textrm{{elign}}_{\textrm{team}}$ | $488.83\pm 20.82$ |  | $638.74\pm 28.93$ |  | $\mathbf{186.83}\pm\mathbf{21.92}$ |  | $-20.4\pm 5.93$ |  | $2.07\pm 4.55$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $182.61\pm 17.63$ |  | $-21.37\pm 7.02$ |  | $\mathbf{11.29}\pm\mathbf{9.02}$
1 Lowe et al. (2017),2 Stadie et al. (2015),3 Iqbal & Sha (2020)

All algorithms are trained using the same agent architecture and optimization
algorithm. They vary in task-specific extrinsic rewards and intrinsic rewards.
We use two versions of the soft actor-critic algorithm Haarnoja et al. (2018):
a decentralized one that trains each agent individually without access to
other agents’ observations and actions (ie. the original soft-actor critic
algorithm) and a centralized one with a critic that has access to other
agents’ observations and actions (Iqbal & Sha, 2019). Note, our intrinsic
reward can also be added to non-actor-critic methods, such as COMA (Foerster
et al., 2018) and VDN (Sunehag et al., 2017). We leave this to future work to
avoid conflating the effects of expectation alignment with COMA’s
counterfactual reasoning and VDN’s value decomposition.

We use sparse (Lowe et al., 2017; Kurach et al., 2019),
$\textrm{{curio}}_{\textrm{self}}$ (Stadie et al., 2015),
$\textrm{{curio}}_{\textrm{team}}$ (Iqbal & Sha, 2020), and variations of our
elign rewards. Sparse rewards agents only when they reach a goal state.
$\textrm{{curio}}_{\textrm{team}}$ is a curiosity-based multi-agent intrinsic
reward which maximizes the average $L_{2}$ loss (instead of minimizing it in
elign). It rewards agents for exploring novel states (Iqbal & Sha, 2020).
$\textrm{{curio}}_{\textrm{self}}$ also maximizes the $L_{2}$ loss but only
using agent $i$’s own observation (Stadie et al., 2015). We experiment with
three variants of ELIGN: $\textrm{{elign}}_{\textrm{self}}$, incentivizing
alignment to one’s own expectation $\textrm{{elign}}_{\textrm{team}}$
incentivizing agents to align to their team, and
$\textrm{{elign}}_{\textrm{adv}}$ incentivizing misalignment to adversaries’
expectations. Note that $\textrm{{elign}}_{\textrm{self}}$ is similar to the
auxiliary loss in Ndousse et al. (2021) but we use it for policy optimization,
rather than for training a state encoder. We also add hand-crafted dense
rewards to provide oracle performance for all tasks.

###  5.5 Results in partially observable environments with decentralized
training

ELIGN outperforms baselines across cooperative and competitive tasks in the
multi-agent particle environment. Table 1 demonstrates that both
$\textrm{{elign}}_{\textrm{self}}$ and $\textrm{{elign}}_{\textrm{team}}$
outperform all sparse and $\textrm{{curio}}_{\textrm{self/team}}$ baseline
rewards in cooperative tasks. While not all elign variants surpass the
baselines in competitive tasks, we find that
$\textrm{{elign}}_{\textrm{team}}$ achieves the highest reward in Physical
deception, and $\textrm{{elign}}_{\textrm{adv}}$ performs the best in
Predator-prey and Keep-away. Similarly, Figure 2 shows that
$\textrm{{elign}}_{\textrm{self}}$ achieves the highest per-step occupancy
count in both cooperative tasks, and $\textrm{{elign}}_{\textrm{adv}}$ does
the best in all competitive tasks.

ELIGN outperforms baselines in the complex Google Research football
environment. As shown in Table 1, $\textrm{{elign}}_{\textrm{adv}}$ achieves
the best mean average episode reward in the competitive Academy 3vs1 with
keeper task. Collectively, these results provide empirical evidence that the
self-organizing principle improves coordination under partial information, a
setting that is most realistic to real world multi-agent systems.

In competitive tasks, agents benefit more from being misaligned to adversaries
than being aligned to their team members. Among the four competitive tasks in
the multi-agent particle and football environments, we find that
$\textrm{{elign}}_{\textrm{adv}}$ outperforms all sparse and
$\textrm{{curio}}_{\textrm{self/team}}$ baselines and other variants of elign
in Predator-prey, Keep-away and Academy 3vs1 with keeper. This suggests that
being misaligned to adversaries, ie taking surprising actions that conflict
with the adversary’s expectations, might be a more useful strategy in multi-
agent competitive tasks.

Figure 3: $\textrm{{elign}}_{\textrm{team}}$ achieves consistent gains
compared against sparse when the number of agents increases in the Cooperative
navigation task.

When the number of agents increases, ELIGN scales well in all multi-agent
particle tasks except for Heterogenous navigation. Table 2 shows that our
elign intrinsic reward still largely achieves the best performance when more
agents are added to cooperative and competitive tasks. The only exception is
the Heterogenous navigation task, where both
$\textrm{{elign}}_{\textrm{self}}$ and $\textrm{{elign}}_{\textrm{team}}$
outperform sparse but not $\textrm{{curio}}_{\textrm{self/team}}$. We
hypothesize that it is more difficult for agents to predict their neighbors’
future states accurately when there are more agents with different sizes and
speeds, and errors in dynamics prediction could lead to misleading alignment
signals. Further, we see a consistent increase in
$\textrm{{elign}}_{\textrm{team}}$’s performance compared against sparse even
when the number of agents scales to ten in Cooperative navigation (Figure 3).

###  5.6 Results with full observability and centralized ELIGN

We further test the utility of decentralized ELIGN in fully observable
environments and centralized ELIGN under partial observability. We find that
decentralized expectation alignment helps in fully observable Cooperative
navigation, Heterogenous navigation, and Predator-prey, tasks where
expectation alignment has been observed in nature. Similarly, centralized
ELIGN also improves agents’ performance compared against sparse and
$\textrm{{curio}}_{\textrm{self/team}}$ rewards in the same tasks with partial
observability. These results can be found in Tables 3 and 4 in the Appendix.
As full observability and centralized training are our main focus, we leave it
to future work to investigate why expectation alignment benefits these tasks
but not others.

Figure 4: We visualize the symmetry-breaking setups in three example tasks.
More details can be found in the Appendix.

###  5.7 Investigating how the ELIGN reward helps

We further investigate how expectation alignment improves coordination through
three evaluation setups.

ELIGN helps agents divide sub-tasks. A core challenge in multi-agent
collaboration is efficient task division (Wang et al., 2020b). Here, we test
whether expectation alignment improves sub-task allocation. We initialize
agents in states without an optimal sub-task allocation, necessitating
symmetry-breaking (Hu et al., 2020). Figure 4 illustrates the symmetry-
breaking setups: in cooperative navigation, when agents are initialized
equidistant to all the goal locations, there isn’t an optimal allocation of
agents to goals. We find that $\textrm{{elign}}_{\textrm{self}}$ achieves the
best performance in both cooperative tasks, while
$\textrm{{elign}}_{\textrm{team}}$ and $\textrm{{elign}}_{\textrm{adv}}$ are
the best strategies in Physical deception and Predator-prey respectively
(Figure 5). Upon a qualitative evaluation of Cooperative navigation, we
observe that agents with expectation alignment are able to predict which goals
will be covered by their collaborators and move towards their allocated one.
Without expectation alignment, agents move towards the same goal.

Figure 5: We plot the test occupancy/collision count per step of decentralized
algorithms in symmetry-breaking settings under partial observability.
$\textrm{{elign}}_{\textrm{self}}$ performs the best in both cooperative
tasks. $\textrm{{elign}}_{\textrm{team}}$ and
$\textrm{{elign}}_{\textrm{adv}}$ are the best strategies in Physical
deception and Predator-prey.

ELIGN helps agents generalize to new partners. Another core challenge in
multi-agent collaboration is zero-shot coordination, where agents are tested
to collaborate with new partners they haven’t been trained with. We study
whether expectation alignment enables better zero-shot coordination. New
partners are sampled from other training runs with different seeds and the
team is evaluated using the same metrics as before. We observe that elign
strategies enable better performance than sparse on average, and one of
$\textrm{{elign}}_{\textrm{self,team,adv}}$ performs the best in Heterogenous
navigation, Physical deception and Keep away. (Figure 7).  These results
suggest that ELIGN results in better zero-shot coordination with new partners
sampled from separate training runs.

Accuracy of the dynamics model affects ELIGN.

Figure 6: Test performance decreases with dynamics model loss ($R^{2}=0.242$),
implying that ELIGN requires an accurate dynamics model.

We investigate the accuracy of the dynamics model in calculating useful
intrinsic rewards. Since ELIGN uses a dynamics model to calculate rewards, we
test whether an inaccurate model misleads agents towards unaligned behaviors.
We train agents on noisy dynamics models by adding Gaussian noise
$\epsilon\sim\mathcal{N}(0,\sigma)$ to the predictions made by the dynamics
model. We run experiments with multiple $\sigma$ values to study how
performance changes as the dynamics model becomes more noisy:
$\sigma\in[0.5,1.0,2.0]$. Our experiments cover one cooperative and one
competitive task. Figure 6 plots the final dynamics loss against the reward
change from the noiseless run. As the dynamics model degrades, we observe that
the task performance also drops. This study identifies the importance of an
accurate dynamics model, suggesting that expectation alignment should used in
environments where an accurate dynamics model can be learned.

Figure 7:  We plot the test occupancy/collision count per step of
decentralized algorithms in zero-shot evaluation with new partners. We find
that one of $\textrm{{elign}}_{\textrm{self,team,adv}}$ achieves the best
performance in Heterogenous navigation, Physical deception and Keep away.

##  6 Discussion

Limitations and future work. While curiosity has proven useful for exploration
in single-agent tasks, we find that expectation alignment—which mathematically
encourages agents to be more predictable instead of finding
novelty—outperforms curiosity in numerous multi-agent tasks. We hypothesize
that our results arise because today’s multi-agent task state space requires
significantly less exploration than those used for single-agent (e.g. Atari
games). Our findings are limited to the multi-agent particle and Google
Research football environments, which have a smaller action space than most
ecologically valid scenarios.

Language, motion, and human gesture are all combinatorially vast; in such
action spaces, expectation alignment might develop social dynamics that hinder
non-optimal multi-agent behaviors. Similarly, photorealistic environments have
a larger state space, where teams perform common household activities (e.g.,
cooking, cleaning, etc.) or drive together in crowded cities (Srivastava et
al., 2021). Future work should develop new multi-agent environments that
demand exploration complexity and where both curiosity and expectation
alignment would be necessary for collaboration. For example, in a search and
rescue task where a single agent is unable to carry the injured, curiosity
would encourage “search” while ELIGN would speed up “rescue”. In the end, we
envision that both these forms of rewards would be necessary for successful
collaboration. However, choosing when to encourage curiosity versus
expectation alignment is an open research problem.

Additionally, enabling stable multi-agent training without centralized
training could open up future opportunities for legible (Dragan et al., 2013)
agents in human environments. Agents with interpretable actions can induce
more faithful human mental models, improving human-AI interaction; however,
predictability does not imply legibility. Future work could explore the role
of legibility in designing intrinsic rewards.

Future work should also explore the use of expectation alignment in massive
collaboration settings with hundreds of agents. Drawing on Zoology research
Couzin (2007) expectation alignment should scale in such settings if agents
align their behaviors only to their nearest neighbors and not to the entire
cohort.

Conclusion. Inspired by the self-organizing principle in Zoology, we introduce
ELIGN, i.e, expectation alignment, a simple, task-agnostic, and self-
supervised intrinsic reward for multi-agent systems. ELIGN rewards agents when
they act predictably to their teammates and unpredictably to their
adversaries. ELIGN improves multi-agent performance across six cooperative and
competitive tasks in the multi-agent particle and Google Research football
environments, especially for decentralized training under partial
observability. It also scales well, helps agents break symmetries, and
generalize to new partners.

## References

  * Achiam & Sastry (2017) Achiam, J. and Sastry, S.  Surprise-based intrinsic motivation for deep reinforcement learning.  _arXiv preprint arXiv:1703.01732_ , 2017.
  * Ben-Jacob et al. (1994) Ben-Jacob, E., Schochet, O., Tenenbaum, A., Cohen, I., Czirok, A., and Vicsek, T.  Generic modelling of cooperative growth patterns in bacterial colonies.  _Nature_ , 368(6466):46–49, 1994.
  * Berner et al. (2019) Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Böhmer et al. (2019) Böhmer, W., Rashid, T., and Whiteson, S.  Exploration with unreliable intrinsic reward in multi-agent reinforcement learning.  _arXiv preprint arXiv:1906.02138_ , 2019.
  * Buhl et al. (2006) Buhl, J., Sumpter, D. J., Couzin, I. D., Hale, J. J., Despland, E., Miller, E. R., and Simpson, S. J.  From disorder to order in marching locusts.  _Science_ , 312(5778):1402–1406, 2006.
  * Burda et al. (2018a) Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A.  Large-scale study of curiosity-driven learning.  _arXiv preprint arXiv:1808.04355_ , 2018a.
  * Burda et al. (2018b) Burda, Y., Edwards, H., Storkey, A., and Klimov, O.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018b.
  * Camazine et al. (2020) Camazine, S., Deneubourg, J.-L., Franks, N. R., Sneyd, J., Theraula, G., and Bonabeau, E.  _Self-organization in biological systems_.  Princeton university press, 2020.
  * Cao et al. (2012) Cao, Y., Yu, W., Ren, W., and Chen, G.  An overview of recent progress in the study of distributed multi-agent coordination.  _IEEE Transactions on Industrial informatics_ , 9(1):427–438, 2012.
  * Chitnis et al. (2020) Chitnis, R., Tulsiani, S., Gupta, S., and Gupta, A.  Intrinsic motivation for encouraging synergistic behavior.  _arXiv preprint arXiv:2002.05189_ , 2020.
  * Collett et al. (1998) Collett, M., Despland, E., Simpson, S. J., and Krakauer, D. C.  Spatial scales of desert locust gregarization.  _Proceedings of the National Academy of Sciences_ , 95(22):13052–13055, 1998.
  * Corder et al. (2019) Corder, K., Vindiola, M. M., and Decker, K.  Decentralized multi-agent actor-critic with generative inference.  _arXiv preprint arXiv:1910.03058_ , 2019.
  * Couzin (2007) Couzin, I.  Collective minds.  _Nature_ , 445(7129):715–715, 2007.
  * Dragan et al. (2013) Dragan, A. D., Lee, K. C., and Srinivasa, S. S.  Legibility and predictability of robot motion.  In _2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pp. 301–308. IEEE, 2013.
  * Du et al. (2019) Du, Y., Han, L., Fang, M., Dai, T., Liu, J., and Tao, D.  Liir: learning individual intrinsic reward in multi-agent reinforcement learning.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pp. 4403–4414, 2019.
  * Foerster et al. (2018) Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S.  Counterfactual multi-agent policy gradients.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Foerster et al. (2017) Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I.  Learning with opponent-learning awareness.  _arXiv preprint arXiv:1709.04326_ , 2017.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _ICML_ , 2018.
  * Hernandez-Leal et al. (2019) Hernandez-Leal, P., Kartal, B., and Taylor, M. E.  Agent modeling as auxiliary task for deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , volume 15, pp. 31–37, 2019\.
  * Hu et al. (2020) Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J.  “other-play” for zero-shot coordination.  In _International Conference on Machine Learning_ , pp. 4399–4410. PMLR, 2020.
  * Huang et al. (2015) Huang, C.-M., Cakmak, M., and Mutlu, B.  Adaptive coordination strategies for human-robot handovers.  In _Robotics: science and systems_ , volume 11. Rome, Italy, 2015\.
  * Iqbal & Sha (2019) Iqbal, S. and Sha, F.  Actor-attention-critic for multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pp. 2961–2970. PMLR, 2019.
  * Iqbal & Sha (2020) Iqbal, S. and Sha, F.  Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning, 2020.  URL https://openreview.net/forum?id=rkltE0VKwH.
  * Jaderberg et al. (2016) Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K.  Reinforcement learning with unsupervised auxiliary tasks.  _arXiv preprint arXiv:1611.05397_ , 2016.
  * Jain et al. (2020) Jain, U., Weihs, L., Kolve, E., Farhadi, A., Lazebnik, S., Kembhavi, A., and Schwing, A.  A cordial sync: Going beyond marginal policies for multi-agent embodied tasks.  In _European Conference on Computer Vision_ , pp. 471–490. Springer, 2020.
  * Jaques et al. (2019) Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and De Freitas, N.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 3040–3049. PMLR, 2019.
  * Kidambi et al. (2020) Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T.  Morel: Model-based offline reinforcement learning.  _arXiv preprint arXiv:2005.05951_ , 2020.
  * Kim et al. (2020) Kim, K., Sano, M., De Freitas, J., Haber, N., and Yamins, D.  Active world model learning with progress curiosity.  In _International conference on machine learning_ , pp. 5306–5315. PMLR, 2020.
  * Krause et al. (2002) Krause, J., Ruxton, G. D., Ruxton, G., Ruxton, I. G., et al.  _Living in groups_.  Oxford University Press, 2002.
  * Kurach et al. (2019) Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., and Gelly, S.  Google research football: A novel reinforcement learning environment.  _CoRR_ , abs/1907.11180, 2019.  URL http://arxiv.org/abs/1907.11180.
  * (31) Liang, E., Liaw, R., Moritz, P., Nishihara, R., Fox, R., Goldberg, K., Gonzalez, J. E., Jordan, M. I., and Stoica, I.  URL https://arxiv.org/abs/1712.09381.
  * Littman (1994) Littman, M. L.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_ , pp. 157–163. Elsevier, 1994.
  * Liu et al. (2020) Liu, I.-J., Yeh, R. A., and Schwing, A. G.  Pic: permutation invariant critic for multi-agent deep reinforcement learning.  In _Conference on Robot Learning_ , pp. 590–602. PMLR, 2020.
  * Lowe et al. (2017) Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Advances in Neural Information Processing Systems_ , 2017.
  * Mordatch & Abbeel (2017) Mordatch, I. and Abbeel, P.  Emergence of grounded compositional language in multi-agent populations.  _arXiv preprint arXiv:1703.04908_ , 2017.
  * Morin (2006) Morin, A.  Levels of consciousness and self-awareness: A comparison and integration of various neurocognitive views.  _Consciousness and cognition_ , 15(2):358–371, 2006.
  * Ndousse et al. (2021) Ndousse, K., Eck, D., Levine, S., and Jaques, N.  Emergent social learning via multi-agent reinforcement learning, 2021\.
  * Pathak et al. (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pp. 2778–2787. PMLR, 2017.
  * Rashid et al. (2018) Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning, 2018.
  * Roy et al. (2020) Roy, J., Barde, P., Harvey, F., Nowrouzezahrai, D., and Pal, C.  Promoting coordination through policy regularization in multi-agent deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:15774–15785, 2020.
  * Schafer (2019) Schafer, L.  _Curiosity in multi-agent reinforcement learning_.  PhD thesis, Master’s thesis, The University of Edinburgh, 2019.
  * Schmidhuber (1991) Schmidhuber, J.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pp. 222–227, 1991.
  * Sekar et al. (2020) Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D.  Planning to explore via self-supervised world models.  In _International Conference on Machine Learning_ , pp. 8583–8592. PMLR, 2020.
  * Srivastava et al. (2021) Srivastava, S., Li, C., Lingelbach, M., Martín-Martín, R., Xia, F., Vainio, K., Lian, Z., Gokmen, C., Buch, S., Liu, C. K., et al.  Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments.  _arXiv preprint arXiv:2108.03332_ , 2021.
  * Stadie et al. (2015) Stadie, B. C., Levine, S., and Abbeel, P.  Incentivizing exploration in reinforcement learning with deep predictive models.  _arXiv preprint arXiv:1507.00814_ , 2015.
  * Strouse et al. (2018) Strouse, D., Kleiman-Weiner, M., Tenenbaum, J., Botvinick, M., and Schwab, D. J.  Learning to share and hide intentions using information regularization.  _Advances in Neural Information Processing Systems_ , 31:10249–10259, 2018.
  * Sunehag et al. (2017) Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al.  Value-decomposition networks for cooperative multi-agent learning.  _arXiv preprint arXiv:1706.05296_ , 2017.
  * Swamy et al. (2020) Swamy, G., Reddy, S., Levine, S., and Dragan, A. D.  Scaled autonomy: Enabling human operators to control robot fleets.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 5942–5948. IEEE, 2020.
  * Theraulaz & Bonabeau (1995) Theraulaz, G. and Bonabeau, E.  Coordination in distributed building.  _Science_ , 269(5224):686–688, 1995.
  * Tomasello et al. (2005) Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H.  Understanding and sharing intentions: The origins of cultural cognition.  _Behavioral and brain sciences_ , 28(5):675–691, 2005.
  * Wang et al. (2020a) Wang, R. E., Kew, J. C., Lee, D., Lee, T.-W. E., Zhang, T., Ichter, B., Tan, J., and Faust, A.  Model-based reinforcement learning for decentralized multiagent rendezvous, 2020a.
  * Wang et al. (2020b) Wang, R. E., Wu, S. A., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman-Weiner, M.  Too many cooks: Bayesian inference for coordinating multi-agent collaboration, 2020b.
  * Wang et al. (2019) Wang, T., Wang, J., Wu, Y., and Zhang, C.  Influence-based multi-agent exploration.  _arXiv preprint arXiv:1910.05512_ , 2019.
  * Wang et al. (2018) Wang, X., Xiong, W., Wang, H., and Wang, W. Y.  Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 37–53, 2018.
  * Weng et al. (2021) Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, H., and Zhu, J.  Tianshou: A highly modularized deep reinforcement learning library.  _arXiv preprint arXiv:2107.14171_ , 2021.
  * Ying & Dayong (2005) Ying, W. and Dayong, S.  Multi-agent framework for third party logistics in e-commerce.  _Expert Systems with Applications_ , 29(2):431–436, 2005.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] See Section 5 for the results
mentioned in the abstract.

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 6 for a
discussion on the limitations.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See the Appendix

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes] We have read the ethics review guidelines.

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See the Appendix

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See the Experiments 5 and Appendix sections.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] See the Experiments 5 section.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the
Experiments 5 and Appendix sections.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] See the
Alignment 4 and Experiments 5 sections.

    2. (b)

Did you mention the license of the assets? [Yes] See the Appendix

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] See the Appendix

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A] We did not collect any data.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A] We did not collect any
data.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A] We did not crowdsource any data.

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A] There were no human
subjects experiments in our work.

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A] No human subjects were hired.

##  Appendix A Appendix

###  A.1 Code

We upload our code for training and evaluating agents with and without
expectation alignment in both the multi-agent particle and Google Research
football environments here: https://github.com/StanfordVL/alignment.

###  A.2 Symmetry-breaking initializations

We create a symmetry-breaking version of each task for evaluation by
initializing the environment in the following ways:

Cooperative Navigation and Heterogenous Navigation: All agents are initialized
at the origin (i.e. center of the world), and target landmarks are placed
randomly on a circle perimeter with the maximum radius (i.e. world radius -
the greatest landmark size) so that each agent is equidistant from each target
landmark.

Physical Deception: Both agents and adversaries start at the origin. All the
landmarks, including the goal, are randomly initialized on a circle perimeter.

Predator-prey: The collaborative agents are initialized at the center while
the adversaries are placed randomly on a circle perimeter. All the landmarks
are randomly initialized in the world.

Keep-away: All the cooperative agents are placed at the origin. Adversaries
and landmarks, including the goal, are randomly initialized on a circle
perimeter. In this task setup, we do not initialize the adversaries at the
center because they are awarded for colliding with the cooperative agents.

###  A.3 Learning curves

Figure 8:  Learning curves of test occupancy/collision in all five tasks in
the multi-agent particle environment. On average, it takes the best elign
variant 65 epochs to reach the maximum score of the best curio method at 100
epochs, which means elign requires on average 35% fewer samples than curio.
Figure 9:  Learning curves of test episode rewards in all five tasks in the
multi-agent particle environment.

Compared to the best baseline’s highest performance at the 100th epoch, we
find that it only takes the best ELIGN variant 31 (Coop nav), 74 (Hetero nav),
30 (Physical dec), 97 (Predator-prey) and 92 (Keep-away) epochs respectively
in the five multi-agent particle tasks (8). This means that on average the
best ELIGN variant requires $35\pm 32$ fewer training steps to reach the same
performance as curiosity or spare methods.

###  A.4 Assets and licenses

We use four assets in total, two of which are existing multi-agent
reinforcement learning environments, and the other two are libraries for
training reinforcement learning algorithms.

We conduct our evaluation on the multi-agent particle (Lowe et al., 2017) and
Google Research football (Kurach et al., 2019) environments, which are under
the MIT license and Apache-2.0 license respectively.

We adapt the tianshou (Weng et al., 2021) and rllib (part of the ray package)
(Liang et al., ) libraries to our experiments, and they are under the MIT
license and Apache-2.0 license respectively.

###  A.5 Societal impacts

While developing new intrinsic rewards to improve decentralized multi-agent
training can help develop and deploy agents in a variety of applications, we
foresee no immediate societal consequences of this work. However, our
experiments thus far have not studied the possible degradation of behaviors
when agents align to malicious teammates. We have also not tested how emergent
properties promote better or worse human collaborators.

###  A.6 Additional tables

We include 20 tables of additional results that quantify the agents’
performance under fully observable environments, with centralized training,
and beyond extrinsic reward.

Table 3 reports the test episode rewards of decentralized methods under the
fully observable multi-agent particle environment.

Table 4 reports the test episode rewards of centralized methods under partial
observability.

Table 5 and 6 report two sets of metrics of decentralized methods trained with
different intrinsic rewards in both partially and fully observable settings.
Table 5 reports the average number of agent-target occupancies per step (or,
we can understand it as: on average, the total number of goals occupied by the
agents at any given timestep throughout an episode) and agent-adversary
collisions in Predator-prey. Higher scores are better for the occupancy
metric, and lower scores are better for collision. Table 6 reports the average
minimum agent-to-target distance and agent-to-adversary distance. Agent-to-
target distances measure the closest distance an agent achieves to the target
location; lower scores are better on this metric. Agent-to-adversary distances
measure the closest distance an adversary gets to a good agent; higher scores
are better on this metric. Note that these distance-based metrics are not
included in the reward functions, and should mainly be used to make
comparisons in the case where primary metrics (i.e, reward and
occupancy/collision count) have the same values.

Table 7 and 8 report the same metrics as 5 and 6 respectively, but in scaled
environments with more agents.

Tables 9, 10 and 11 report the test episode reward and additional metrics of
decentralized algorithms in the symmetry-breaking experiments conducted under
“Investigating how alignment reward helps”. Table 12, 13 and 14 report the
same set of metrics but from experiments conducted in scaled and symmetry-
breaking environments.

Table 15 reports the test mean episode rewards of centralized algorithms with
different intrinsic rewards under full observability. Table 16 and 17 show the
other two sets of metrics (i.e, occupancy/collision count and agent-
target/agent-adversary distance) of centralized algorithms. Table 18, 19, and
20 contain the same metrics as 15, 16 and 17 respectively, but in scaled
environments.

Finally, Tables 21 and 22 report the test episode reward values and secondary
distance-based metrics for the zero-shot generalization experiments conducted
under “Investigating how ELIGN reward helps”. These experiments measure how
well agents trained on different seeds generalized to new partners trained on
other seeds.

Table 3: We report the mean test episode extrinsic rewards and standard errors
of decentralized methods with different intrinsic rewards in fully observable
environments.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (3v0) |  | Hetero nav. (4v0) |  | Phy decep. (2v1) |  | Pred-prey (2v2) |  | Keep-away (2v2)
Full observability | $\textsc{sparse}^{1}$ | $154.00\pm 10.51$ |  | $274.75\pm 19.74$ |  | $82.97\pm 12.23$ |  | $-10.48\pm 4.20$ |  | $\mathbf{4.95}\pm\mathbf{2.96}$
$\textrm{{curio}}_{\textrm{self}}$ | $154.71\pm 8.00$ |  | $268.85\pm 15.61$ |  | $\mathbf{100.66}\pm\mathbf{15.14}$ |  | $-8.74\pm 4.62$ |  | $-2.00\pm 1.24$
$\textrm{{elign}}_{\textrm{self}}$ | $\mathbf{161.70}\pm\mathbf{4.52}$ |  | $\mathbf{280.16}\pm\mathbf{17.12}$ |  | $87.50\pm 15.40$ |  | $\mathbf{-5.60}\pm\mathbf{2.60}$ |  | $0.40\pm 1.92$
1 Lowe et al. (2017)

Table 4: We report the mean test episode extrinsic rewards and standard errors
of centralized methods with different intrinsic rewards under partial and full
observability.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (3v0) |  | Hetero nav. (4v0) |  | Phy decep. (2v1) |  | Pred-prey (2v2) |  | Keep-away (2v2)
Partial observability | sparse1 | $113.25\pm 8.10$ |  | $178.62\pm 9.62$ |  | $\mathbf{117.45}\pm\mathbf{10.63}$ |  | $-1.96\pm 1.45$ |  | $\mathbf{35.79}\pm\mathbf{14.93}$
$\textrm{{curio}}_{\textrm{self}}^{2}$ | $128.77\pm 7.70$ |  | $190.30\pm 7.73$ |  | $111.08\pm 10.09$ |  | $-1.63\pm 1.27$ |  | $13.94\pm 12.56$
$\textrm{{curio}}_{\textrm{team}}^{3}$ | $114.13\pm 11.84$ |  | $189.80\pm 11.81$ |  | $114.32\pm 5.46$ |  | $-3.04\pm 1.09$ |  | $6.01\pm 3.36$
$\textrm{{elign}}_{\textrm{self}}$ | $\mathbf{137.14}\pm\mathbf{3.63}$ |  | $169.58\pm 14.99$ |  | $93.27\pm 3.70$ |  | $-0.41\pm 0.28$ |  | $22.77\pm 9.91$
$\textrm{{elign}}_{\textrm{team}}$ | $119.10\pm 10.89$ |  | $\mathbf{210.81}\pm\mathbf{9.70}$ |  | $96.49\pm 6.46$ |  | $-0.92\pm 0.72$ |  | $24.94\pm 12.58$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $102.37\pm 6.98$ |  | $\mathbf{-0.13}\pm\mathbf{0.03}$ |  | $8.70\pm 4.44$
1 Lowe et al. (2017),2 Stadie et al. (2015),3 Iqbal & Sha (2020)

Table 5:  The average test occupancy/collision count per step and standard
errors of decentralized methods with different intrinsic rewards under partial
and full observability. Higher scores are better for the occupancy metric
($\uparrow$), and lower scores are better for the collision metric
($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\uparrow$ |  |  Hetero nav. (4v0) $\uparrow$ |  |  Phy decep. (2v1) $\uparrow$ |  |  Pred-prey (2v2) $\downarrow$ |  |  Keep-away (2v2) $\uparrow$
Partial observability | sparse | $0.46\pm 0.05$ |  | $0.57\pm 0.01$ |  | $0.98\pm 0.07$ |  | $0.02\pm 0.01$ |  | $0.07\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.43\pm 0.03$ |  | $0.60\pm 0.01$ |  | $0.99\pm 0.03$ |  | $0.02\pm 0.01$ |  | $0.14\pm 0.02$
$\textrm{{curio}}_{\textrm{team}}$ | $0.42\pm 0.05$ |  | $0.59\pm 0.01$ |  | $0.95\pm 0.01$ |  | $0.02\pm 0.01$ |  | $0.10\pm 0.03$
$\textrm{{elign}}_{\textrm{self}}$ | $0.52\pm 0.03$ |  | $0.61\pm 0.01$ |  | $0.95\pm 0.02$ |  | $0.03\pm 0.01$ |  | $0.10\pm 0.02$
$\textrm{{elign}}_{\textrm{team}}$ | $0.44\pm 0.04$ |  | $0.58\pm 0.02$ |  | $0.99\pm 0.07$ |  | $0.03\pm 0.01$ |  | $0.07\pm 0.02$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $1.00\pm 0.06$ |  | $0.01\pm 0.01$ |  | $0.15\pm 0.03$
Full observability | sparse | $0.46\pm 0.11$ |  | $0.57\pm 0.01$ |  | $0.88\pm 0.09$ |  | $0.03\pm 0.01$ |  | $0.06\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.50\pm 0.07$ |  | $0.59\pm 0.02$ |  | $1.09\pm 0.13$ |  | $0.03\pm 0.01$ |  | $0.02\pm 0.00$
$\textrm{{elign}}_{\textrm{self}}$ | $0.48\pm 0.11$ |  | $0.58\pm 0.02$ |  | $0.83\pm 0.10$ |  | $0.02\pm 0.01$ |  | $0.04\pm 0.01$

Table 6:  The average test agent-to-target (agt-target) and agent-to-adversary
(agt-adv) distances and standard errors of decentralized methods with
different intrinsic rewards under partial and full observability. Lower scores
are better for agt-target ($\downarrow$), and higher scores are better for
agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\downarrow$ |  |  Hetero nav. (4v0) $\downarrow$ |  |  Phy decep. (2v1) $\downarrow$ |  |  Pred-prey (2v2) $\uparrow$ |  |  Keep-away (2v2) $\downarrow$
Partial observability | sparse | $0.30\pm 0.02$ |  | $0.23\pm 0.00$ |  | $0.26\pm 0.01$ |  | $1.45\pm 0.11$ |  | $1.41\pm 0.07$
$\textrm{{curio}}_{\textrm{self}}$ | $0.32\pm 0.02$ |  | $0.25\pm 0.01$ |  | $0.25\pm 0.00$ |  | $1.36\pm 0.06$ |  | $1.14\pm 0.09$
$\textrm{{curio}}_{\textrm{team}}$ | $0.31\pm 0.01$ |  | $0.25\pm 0.01$ |  | $0.26\pm 0.00$ |  | $1.48\pm 0.13$ |  | $1.31\pm 0.10$
$\textrm{{elign}}_{\textrm{self}}$ | $0.33\pm 0.03$ |  | $0.25\pm 0.01$ |  | $0.26\pm 0.00$ |  | $1.39\pm 0.12$ |  | $1.26\pm 0.09$
$\textrm{{elign}}_{\textrm{team}}$ | $0.33\pm 0.02$ |  | $0.23\pm 0.01$ |  | $0.25\pm 0.01$ |  | $1.38\pm 0.13$ |  | $1.38\pm 0.09$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.25\pm 0.01$ |  | $1.54\pm 0.08$ |  | $1.14\pm 0.09$
Full observability | sparse | $0.32\pm 0.09$ |  | $0.23\pm 0.00$ |  | $0.26\pm 0.01$ |  | $1.23\pm 0.12$ |  | $1.27\pm 0.09$
$\textrm{{curio}}_{\textrm{self}}$ | $0.28\pm 0.04$ |  | $0.22\pm 0.01$ |  | $0.23\pm 0.01$ |  | $1.37\pm 0.15$ |  | $1.53\pm 0.03$
$\textrm{{elign}}_{\textrm{self}}$ | $0.30\pm 0.07$ |  | $0.23\pm 0.01$ |  | $0.27\pm 0.01$ |  | $1.40\pm 0.13$ |  | $1.41\pm 0.10$

Table 7:  The average test occupancy/collision count per step and standard
errors of decentralized methods with different intrinsic rewards in scaled
environments under partial and full observability. Higher scores are better
for the occupancy metric ($\uparrow$), and lower scores are better for the
collision metric ($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\uparrow$ |  |  Hetero nav. (6v0) $\uparrow$ |  |  Phy decep. (4v2) $\uparrow$ |  |  Pred-prey (4v4) $\downarrow$ |  |  Keep-away (4v4) $\uparrow$
Partial observability | sparse | $0.50\pm 0.04$ |  | $0.46\pm 0.08$ |  | $1.20\pm 0.10$ |  | $0.11\pm 0.02$ |  | $0.08\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.48\pm 0.03$ |  | $0.63\pm 0.01$ |  | $1.20\pm 0.08$ |  | $0.07\pm 0.02$ |  | $0.15\pm 0.06$
$\textrm{{curio}}_{\textrm{team}}$ | $0.53\pm 0.03$ |  | $0.60\pm 0.02$ |  | $1.20\pm 0.09$ |  | $0.05\pm 0.02$ |  | $0.06\pm 0.01$
$\textrm{{elign}}_{\textrm{self}}$ | $0.49\pm 0.03$ |  | $0.67\pm 0.00$ |  | $1.30\pm 0.23$ |  | $0.04\pm 0.02$ |  | $0.14\pm 0.08$
| $\textrm{{elign}}_{\textrm{team}}$ | $0.56\pm 0.04$ |  | $0.56\pm 0.00$ |  | $1.21\pm 0.09$ |  | $0.08\pm 0.02$ |  | $0.10\pm 0.02$
| $\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $1.23\pm 0.10$ |  | $0.08\pm 0.02$ |  | $0.16\pm 0.07$
Full observability | sparse | $0.52\pm 0.11$ |  | $0.46\pm 0.08$ |  | $0.99\pm 0.09$ |  | $0.21\pm 0.01$ |  | $0.03\pm 0.00$
$\textrm{{curio}}_{\textrm{self}}$ | $0.39\pm 0.13$ |  | $0.56\pm 0.01$ |  | $0.86\pm 0.04$ |  | $0.16\pm 0.03$ |  | $0.04\pm 0.00$
| $\textrm{{elign}}_{\textrm{self}}$ | $0.55\pm 0.11$ |  | $0.56\pm 0.00$ |  | $1.04\pm 0.07$ |  | $0.15\pm 0.03$ |  | $0.06\pm 0.02$

Table 8:  The average test agent-to-target (agt-target) and agent-to-adversary
(agt-adv) distances and standard errors of decentralized methods with
different intrinsic rewards in scaled environments under partial and full
observability. Lower scores are better for agt-target ($\downarrow$), and
higher scores are better for agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\downarrow$ |  |  Hetero nav. (6v0) $\downarrow$ |  |  Phy decep. (4v2) $\downarrow$ |  |  Pred-prey (4v4) $\uparrow$ |  |  Keep-away (4v4) $\downarrow$
Partial observability | sparse | $0.22\pm 0.01$ |  | $0.27\pm 0.05$ |  | $0.23\pm 0.02$ |  | $2.03\pm 0.15$ |  | $2.97\pm 0.17$
$\textrm{{curio}}_{\textrm{self}}$ | $0.30\pm 0.02$ |  | $0.21\pm 0.01$ |  | $0.24\pm 0.01$ |  | $2.18\pm 0.13$ |  | $2.70\pm 0.25$
$\textrm{{curio}}_{\textrm{team}}$ | $0.23\pm 0.02$ |  | $0.22\pm 0.01$ |  | $0.23\pm 0.02$ |  | $2.29\pm 0.12$ |  | $3.14\pm 0.08$
$\textrm{{elign}}_{\textrm{self}}$ | $0.29\pm 0.03$ |  | $0.19\pm 0.00$ |  | $0.24\pm 0.02$ |  | $2.39\pm 0.11$ |  | $2.97\pm 0.30$
| $\textrm{{elign}}_{\textrm{team}}$ | $0.23\pm 0.04$ |  | $0.21\pm 0.00$ |  | $0.23\pm 0.01$ |  | $2.16\pm 0.12$ |  | $2.88\pm 0.19$
| $\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.22\pm 0.01$ |  | $2.12\pm 0.16$ |  | $2.66\pm 0.23$
Full observability | sparse | $0.23\pm 0.06$ |  | $0.27\pm 0.05$ |  | $0.21\pm 0.02$ |  | $1.64\pm 0.02$ |  | $3.28\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.33\pm 0.09$ |  | $0.21\pm 0.01$ |  | $0.22\pm 0.01$ |  | $1.81\pm 0.12$ |  | $3.24\pm 0.08$
| $\textrm{{elign}}_{\textrm{self}}$ | $0.20\pm 0.04$ |  | $0.21\pm 0.00$ |  | $0.21\pm 0.01$ |  | $1.82\pm 0.10$ |  | $2.97\pm 0.17$

Table 9: We report the mean test episode extrinsic rewards and standard errors
of decentralized methods with different intrinsic rewards in symmetry-breaking
settings under partial and full observability.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (3v0) |  | Hetero nav. (4v0) |  | Phy decep. (2v1) |  | Pred-prey (2v2) |  | Keep-away (2v2)
Partial observability | sparse | $97.45\pm 10.49$ |  | $184.18\pm 7.63$ |  | $59.39\pm 21.10$ |  | $-1.89\pm 1.69$ |  | $3.85\pm 4.25$
$\textrm{{curio}}_{\textrm{self}}$ | $85.23\pm 10.88$ |  | $184.07\pm 9.99$ |  | $54.17\pm 27.40$ |  | $-2.86\pm 1.19$ |  | $19.57\pm 4.92$
$\textrm{{curio}}_{\textrm{team}}$ | $81.50\pm 15.78$ |  | $141.78\pm 20.04$ |  | $41.12\pm 13.37$ |  | $-2.80\pm 1.91$ |  | $10.21\pm 6.34$
$\textrm{{elign}}_{\textrm{self}}$ | $110.29\pm 9.67$ |  | $176.98\pm 6.38$ |  | $98.90\pm 17.71$ |  | $-4.00\pm 2.14$ |  | $9.47\pm 3.99$
$\textrm{{elign}}_{\textrm{team}}$ | $92.41\pm 10.70$ |  | $187.42\pm 11.29$ |  | $74.06\pm 21.58$ |  | $-2.00\pm 1.39$ |  | $3.32\pm 3.04$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $87.55\pm 15.35$ |  | $-1.40\pm 1.25$ |  | $13.77\pm 3.58$
Full observability | sparse | $150.42\pm 15.18$ |  | $250.41\pm 14.23$ |  | $69.06\pm 14.06$ |  | $-7.62\pm 3.50$ |  | $3.50\pm 4.00$
$\textrm{{curio}}_{\textrm{self}}$ | $149.48\pm 9.42$ |  | $241.69\pm 19.58$ |  | $52.69\pm 17.97$ |  | $-10.40\pm 6.33$ |  | $-1.10\pm 0.59$
$\textrm{{elign}}_{\textrm{self}}$ | $152.08\pm 6.68$ |  | $275.69\pm 7.49$ |  | $75.79\pm 24.54$ |  | $-4.44\pm 2.05$ |  | $0.96\pm 3.14$

Table 10:  The average test occupancy/collision count per step and standard
errors of decentralized methods with different intrinsic rewards in symmetry-
breaking settings under partial and full observability. Higher scores are
better for the occupancy metric ($\uparrow$), and lower scores are better for
the collision metric ($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\uparrow$ |  |  Hetero nav. (4v0) $\uparrow$ |  |  Phy decep. (2v1) $\uparrow$ |  |  Pred-prey (2v2) $\downarrow$ |  |  Keep-away (2v2) $\uparrow$
Partial observability | sparse | $0.26\pm 0.04$ |  | $0.27\pm 0.02$ |  | $0.67\pm 0.09$ |  | $0.02\pm 0.01$ |  | $0.04\pm 0.03$
$\textrm{{curio}}_{\textrm{self}}$ | $0.22\pm 0.01$ |  | $0.28\pm 0.04$ |  | $0.61\pm 0.06$ |  | $0.02\pm 0.01$ |  | $0.15\pm 0.04$
$\textrm{{curio}}_{\textrm{team}}$ | $0.26\pm 0.06$ |  | $0.29\pm 0.02$ |  | $0.65\pm 0.05$ |  | $0.01\pm 0.01$ |  | $0.08\pm 0.04$
$\textrm{{elign}}_{\textrm{self}}$ | $0.29\pm 0.05$ |  | $0.32\pm 0.03$ |  | $0.62\pm 0.02$ |  | $0.02\pm 0.01$ |  | $0.08\pm 0.03$
$\textrm{{elign}}_{\textrm{team}}$ | $0.27\pm 0.04$ |  | $0.27\pm 0.02$ |  | $0.72\pm 0.10$ |  | $0.02\pm 0.01$ |  | $0.05\pm 0.04$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.68\pm 0.07$ |  | $0.00\pm 0.00$ |  | $0.14\pm 0.04$
Full observability | sparse | $0.45\pm 0.12$ |  | $0.54\pm 0.01$ |  | $0.89\pm 0.11$ |  | $0.03\pm 0.01$ |  | $0.05\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.48\pm 0.08$ |  | $0.54\pm 0.01$ |  | $1.13\pm 0.14$ |  | $0.04\pm 0.02$ |  | $0.00\pm 0.00$
$\textrm{{elign}}_{\textrm{self}}$ | $0.46\pm 0.11$ |  | $0.54\pm 0.01$ |  | $0.86\pm 0.12$ |  | $0.02\pm 0.01$ |  | $0.02\pm 0.02$

Table 11:  The average test agent-to-target (agt-target) and agent-to-
adversary (agt-adv) distances and standard errors of decentralized methods
with different intrinsic rewards in symmetry-breaking settings under partial
and full observability. Lower scores are better for agt-target ($\downarrow$),
and higher scores are better for agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\downarrow$ |  |  Hetero nav. (4v0) $\downarrow$ |  |  Phy decep. (2v1) $\downarrow$ |  |  Pred-prey (2v2) $\uparrow$ |  |  Keep-away (2v2) $\downarrow$
Partial observability | sparse | $0.53\pm 0.02$ |  | $0.57\pm 0.02$ |  | $0.35\pm 0.03$ |  | $1.49\pm 0.14$ |  | $1.57\pm 0.15$
$\textrm{{curio}}_{\textrm{self}}$ | $0.57\pm 0.04$ |  | $0.55\pm 0.04$ |  | $0.37\pm 0.02$ |  | $1.29\pm 0.06$ |  | $1.07\pm 0.17$
$\textrm{{curio}}_{\textrm{team}}$ | $0.53\pm 0.03$ |  | $0.55\pm 0.03$ |  | $0.35\pm 0.02$ |  | $1.49\pm 0.13$ |  | $1.37\pm 0.18$
$\textrm{{elign}}_{\textrm{self}}$ | $0.68\pm 0.05$ |  | $0.52\pm 0.03$ |  | $0.34\pm 0.01$ |  | $1.39\pm 0.13$ |  | $1.26\pm 0.18$
$\textrm{{elign}}_{\textrm{team}}$ | $0.55\pm 0.05$ |  | $0.56\pm 0.02$ |  | $0.31\pm 0.02$ |  | $1.41\pm 0.10$ |  | $1.53\pm 0.17$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.33\pm 0.04$ |  | $1.61\pm 0.08$ |  | $1.08\pm 0.15$
Full observability | sparse | $0.45\pm 0.12$ |  | $0.29\pm 0.00$ |  | $0.25\pm 0.01$ |  | $1.28\pm 0.15$ |  | $1.30\pm 0.15$
$\textrm{{curio}}_{\textrm{self}}$ | $0.33\pm 0.06$ |  | $0.30\pm 0.01$ |  | $0.22\pm 0.01$ |  | $1.47\pm 0.17$ |  | $1.71\pm 0.05$
$\textrm{{elign}}_{\textrm{self}}$ | $0.46\pm 0.11$ |  | $0.30\pm 0.00$ |  | $0.25\pm 0.02$ |  | $1.49\pm 0.16$ |  | $1.53\pm 0.16$

Table 12: We report the mean test episode extrinsic rewards and standard
errors of decentralized methods with different intrinsic rewards in scaled and
symmetry-breaking settings.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (5v0) |  | Hetero nav. (6v0) |  | Phy decep. (4v2) |  | Pred-prey (4v4) |  | Keep-away (4v4)
Partial observability | sparse | $328.24\pm 24.17$ |  | $405.08\pm 21.53$ |  | $172.87\pm 32.43$ |  | $-35.40\pm 8.63$ |  | $1.37\pm 3.48$
$\textrm{{curio}}_{\textrm{self}}$ | $295.48\pm 20.54$ |  | $436.17\pm 26.30$ |  | $202.39\pm 26.06$ |  | $-11.19\pm 3.65$ |  | $9.24\pm 8.49$
$\textrm{{curio}}_{\textrm{team}}$ | $316.33\pm 14.44$ |  | $422.71\pm 13.24$ |  | $229.50\pm 28.29$ |  | $-11.56\pm 6.37$ |  | $-1.29\pm 1.58$
$\textrm{{elign}}_{\textrm{self}}$ | $357.40\pm 19.52$ |  | $412.39\pm 12.63$ |  | $129.07\pm 51.08$ |  | $-7.34\pm 5.12$ |  | $11.97\pm 13.30$
| $\textrm{{elign}}_{\textrm{team}}$ | $354.14\pm 19.53$ |  | $417.94\pm 22.29$ |  | $184.21\pm 23.16$ |  | $-19.37\pm 6.44$ |  | $4.05\pm 5.78$
| $\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $148.69\pm 31.79$ |  | $-23.42\pm 8.32$ |  | $18.71\pm 14.78$
Full observability | sparse | $466.17\pm 28.16$ |  | $471.19\pm 16.23$ |  | $233.61\pm 25.44$ |  | $-39.24\pm 6.63$ |  | $-5.10\pm 0.26$
$\textrm{{curio}}_{\textrm{self}}$ | $509.91\pm 14.10$ |  | $606.07\pm 7.55$ |  | $256.13\pm 41.13$ |  | $-38.66\pm 13.38$ |  | $-6.58\pm 1.29$
| $\textrm{{elign}}_{\textrm{self}}$ | $520.25\pm 9.68$ |  | $510.18\pm 25.71$ |  | $222.31\pm 15.39$ |  | $-30.56\pm 9.87$ |  | $-4.27\pm 2.53$

Table 13:  The average test occupancy/collision count per step and standard
errors of decentralized methods with different intrinsic rewards in scaled and
symmetry-breaking settings. under partial and full observability. Higher
scores are better for the occupancy metric ($\uparrow$), and lower scores are
better for the collision metric ($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\uparrow$ |  |  Hetero nav. (6v0) $\uparrow$ |  |  Phy decep. (4v2) $\uparrow$ |  |  Pred-prey (4v4) $\downarrow$ |  |  Keep-away (4v4) $\uparrow$
Partial observability | sparse | $0.37\pm 0.05$ |  | $0.38\pm 0.03$ |  | $0.75\pm 0.04$ |  | $0.13\pm 0.03$ |  | $0.04\pm 0.02$
$\textrm{{curio}}_{\textrm{self}}$ | $0.29\pm 0.03$ |  | $0.43\pm 0.02$ |  | $0.66\pm 0.06$ |  | $0.05\pm 0.02$ |  | $0.12\pm 0.08$
$\textrm{{curio}}_{\textrm{team}}$ | $0.32\pm 0.02$ |  | $0.34\pm 0.01$ |  | $0.78\pm 0.07$ |  | $0.05\pm 0.02$ |  | $0.02\pm 0.01$
$\textrm{{elign}}_{\textrm{self}}$ | $0.29\pm 0.03$ |  | $0.39\pm 0.02$ |  | $0.96\pm 0.20$ |  | $0.04\pm 0.03$ |  | $0.11\pm 0.10$
| $\textrm{{elign}}_{\textrm{team}}$ | $0.37\pm 0.03$ |  | $0.40\pm 0.03$ |  | $0.72\pm 0.03$ |  | $0.07\pm 0.03$ |  | $0.06\pm 0.03$
| $\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.63\pm 0.09$ |  | $0.07\pm 0.03$ |  | $0.15\pm 0.10$
Full observability | sparse | $0.52\pm 0.11$ |  | $0.43\pm 0.09$ |  | $0.86\pm 0.07$ |  | $0.19\pm 0.02$ |  | $0.00\pm 0.00$
$\textrm{{curio}}_{\textrm{self}}$ | $0.39\pm 0.14$ |  | $0.54\pm 0.00$ |  | $0.81\pm 0.06$ |  | $0.14\pm 0.04$ |  | $0.00\pm 0.00$
| $\textrm{{elign}}_{\textrm{self}}$ | $0.55\pm 0.11$ |  | $0.55\pm 0.00$ |  | $0.94\pm 0.08$ |  | $0.12\pm 0.03$ |  | $0.02\pm 0.01$

Table 14:  The average test agent-to-target (agt-target) and agent-to-
adversary (agt-adv) distances and standard errors of decentralized methods
with different intrinsic rewards in scaled and symmetry-breaking settings
under partial and full observability. Lower scores are better for agt-target
($\downarrow$), and higher scores are better for agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\downarrow$ |  |  Hetero nav. (6v0) $\downarrow$ |  |  Phy decep. (4v2) $\downarrow$ |  |  Pred-prey (4v4) $\uparrow$ |  |  Keep-away (4v4) $\downarrow$
Partial observability | sparse | $0.36\pm 0.01$ |  | $0.42\pm 0.02$ |  | $0.35\pm 0.01$ |  | $2.04\pm 0.18$ |  | $3.10\pm 0.29$
$\textrm{{curio}}_{\textrm{self}}$ | $0.50\pm 0.04$ |  | $0.37\pm 0.02$ |  | $0.38\pm 0.03$ |  | $2.35\pm 0.12$ |  | $2.70\pm 0.35$
$\textrm{{curio}}_{\textrm{team}}$ | $0.43\pm 0.03$ |  | $0.45\pm 0.01$ |  | $0.35\pm 0.02$ |  | $2.39\pm 0.15$ |  | $3.37\pm 0.19$
$\textrm{{elign}}_{\textrm{self}}$ | $0.52\pm 0.05$ |  | $0.41\pm 0.02$ |  | $0.37\pm 0.04$ |  | $2.52\pm 0.15$ |  | $3.22\pm 0.42$
| $\textrm{{elign}}_{\textrm{team}}$ | $0.42\pm 0.03$ |  | $0.41\pm 0.02$ |  | $0.37\pm 0.01$ |  | $2.25\pm 0.15$ |  | $3.00\pm 0.33$
| $\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.41\pm 0.05$ |  | $2.27\pm 0.16$ |  | $2.62\pm 0.30$
Full observability | sparse | $0.29\pm 0.07$ |  | $0.37\pm 0.06$ |  | $0.26\pm 0.01$ |  | $1.81\pm 0.04$ |  | $3.69\pm 0.05$
$\textrm{{curio}}_{\textrm{self}}$ | $0.42\pm 0.11$ |  | $0.29\pm 0.00$ |  | $0.26\pm 0.02$ |  | $2.02\pm 0.16$ |  | $3.63\pm 0.12$
| $\textrm{{elign}}_{\textrm{self}}$ | $0.26\pm 0.05$ |  | $0.29\pm 0.00$ |  | $0.27\pm 0.01$ |  | $2.10\pm 0.12$ |  | $3.24\pm 0.26$

Table 15: We report the mean test episode extrinsic rewards and standard
errors of centralized methods with different intrinsic rewards under full
observability.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (3v0) |  | Hetero nav. (4v0) |  | Phy decep. (2v1) |  | Pred-prey (2v2) |  | Keep-away (2v2)
Full observability | sparse | $106.02\pm 20.95$ |  | $123.17\pm 18.77$ |  | $130.90\pm 6.59$ |  | $-1.90\pm 1.61$ |  | $12.49\pm 9.83$
$\textrm{{curio}}_{\textrm{self}}$ | $86.52\pm 16.02$ |  | $108.84\pm 6.89$ |  | $107.84\pm 13.67$ |  | $-1.69\pm 0.60$ |  | $23.70\pm 12.95$
$\textrm{{elign}}_{\textrm{self}}$ | $120.47\pm 12.26$ |  | $134.30\pm 5.84$ |  | $105.74\pm 9.72$ |  | $-2.37\pm 1.39$ |  | $22.92\pm 7.00$

Table 16:  The average test occupancy/collision count per step and standard
errors of centralized methods with different intrinsic rewards under partial
and full observability. Higher scores are better for the occupancy metric
($\uparrow$), and lower scores are better for the collision metric
($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\uparrow$ |  |  Hetero nav. (4v0) $\uparrow$ |  |  Phy decep. (2v1) $\uparrow$ |  |  Pred-prey (2v2) $\downarrow$ |  |  Keep-away (2v2) $\uparrow$
Partial observability | sparse | $0.29\pm 0.10$ |  | $0.50\pm 0.03$ |  | $0.94\pm 0.06$ |  | $0.00\pm 0.00$ |  | $0.36\pm 0.12$
$\textrm{{curio}}_{\textrm{self}}$ | $0.28\pm 0.09$ |  | $0.47\pm 0.03$ |  | $0.94\pm 0.03$ |  | $0.01\pm 0.00$ |  | $0.17\pm 0.10$
$\textrm{{curio}}_{\textrm{team}}$ | $0.33\pm 0.10$ |  | $0.47\pm 0.04$ |  | $0.92\pm 0.01$ |  | $0.01\pm 0.00$ |  | $0.08\pm 0.03$
$\textrm{{elign}}_{\textrm{self}}$ | $0.21\pm 0.10$ |  | $0.50\pm 0.01$ |  | $0.92\pm 0.02$ |  | $0.00\pm 0.00$ |  | $0.25\pm 0.08$
$\textrm{{elign}}_{\textrm{team}}$ | $0.23\pm 0.09$ |  | $0.55\pm 0.02$ |  | $0.90\pm 0.07$ |  | $0.01\pm 0.00$ |  | $0.24\pm 0.11$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.94\pm 0.04$ |  | $0.00\pm 0.00$ |  | $0.10\pm 0.05$
Full observability | sparse | $0.34\pm 0.10$ |  | $0.33\pm 0.07$ |  | $0.88\pm 0.04$ |  | $0.01\pm 0.00$ |  | $0.26\pm 0.11$
$\textrm{{curio}}_{\textrm{self}}$ | $0.30\pm 0.07$ |  | $0.32\pm 0.05$ |  | $0.82\pm 0.02$ |  | $0.01\pm 0.01$ |  | $0.33\pm 0.16$
$\textrm{{elign}}_{\textrm{self}}$ | $0.30\pm 0.11$ |  | $0.40\pm 0.04$ |  | $0.88\pm 0.05$ |  | $0.01\pm 0.01$ |  | $0.30\pm 0.07$

Table 17:  The average test agent-to-target (agt-target) and agent-to-
adversary (agt-adv) distances and standard errors of centralized methods with
different intrinsic rewards under partial and full observability. Lower scores
are better for agt-target ($\downarrow$), and higher scores are better for
agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (3v0) $\downarrow$ |  |  Hetero nav. (4v0) $\downarrow$ |  |  Phy decep. (2v1) $\downarrow$ |  |  Pred-prey (2v2) $\uparrow$ |  |  Keep-away (2v2) $\downarrow$
Partial observability | sparse | $0.42\pm 0.05$ |  | $0.29\pm 0.02$ |  | $0.27\pm 0.01$ |  | $1.54\pm 0.02$ |  | $1.38\pm 0.13$
$\textrm{{curio}}_{\textrm{self}}$ | $0.42\pm 0.05$ |  | $0.29\pm 0.01$ |  | $0.27\pm 0.01$ |  | $1.46\pm 0.05$ |  | $1.40\pm 0.13$
$\textrm{{curio}}_{\textrm{team}}$ | $0.41\pm 0.06$ |  | $0.29\pm 0.02$ |  | $0.28\pm 0.01$ |  | $1.49\pm 0.04$ |  | $1.43\pm 0.14$
$\textrm{{elign}}_{\textrm{self}}$ | $0.50\pm 0.07$ |  | $0.29\pm 0.01$ |  | $0.27\pm 0.01$ |  | $1.60\pm 0.04$ |  | $1.26\pm 0.12$
$\textrm{{elign}}_{\textrm{team}}$ | $0.45\pm 0.05$ |  | $0.27\pm 0.01$ |  | $0.28\pm 0.01$ |  | $1.52\pm 0.04$ |  | $1.35\pm 0.14$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.28\pm 0.01$ |  | $1.55\pm 0.03$ |  | $1.45\pm 0.10$
Full observability | sparse | $0.38\pm 0.07$ |  | $0.34\pm 0.04$ |  | $0.25\pm 0.00$ |  | $1.59\pm 0.06$ |  | $1.43\pm 0.09$
$\textrm{{curio}}_{\textrm{self}}$ | $0.36\pm 0.05$ |  | $0.32\pm 0.02$ |  | $0.25\pm 0.01$ |  | $1.53\pm 0.09$ |  | $1.08\pm 0.15$
$\textrm{{elign}}_{\textrm{self}}$ | $0.43\pm 0.08$ |  | $0.30\pm 0.02$ |  | $0.25\pm 0.00$ |  | $1.51\pm 0.08$ |  | $1.18\pm 0.15$

Table 18: We report the mean test episode extrinsic rewards and standard
errors of centralized methods with different intrinsic rewards in scaled
environments.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (5v0) |  | Hetero nav. (6v0) |  | Phy decep. (4v2) |  | Pred-prey (4v4) |  | Keep-away (4v4)
Partial observability | sparse | $100.63\pm 19.36$ |  | $346.16\pm 18.95$ |  | $-38.99\pm 16.18$ |  | $-17.33\pm 4.29$ |  | $-2.50\pm 2.64$
$\textrm{{elign}}_{\textrm{self}}$ | $112.15\pm 19.69$ |  | $375.21\pm 26.10$ |  | $13.71\pm 29.53$ |  | $-20.12\pm 1.42$ |  | $-4.68\pm 1.21$
$\textrm{{elign}}_{\textrm{team}}$ | $97.93\pm 25.23$ |  | $372.41\pm 44.28$ |  | $60.07\pm 13.26$ |  | $-27.87\pm 0.99$ |  | $1.72\pm 3.79$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $21.67\pm 48.17$ |  | $-17.68\pm 5.59$ |  | $-4.92\pm 1.81$
Full observability | sparse | $50.60\pm 13.10$ |  | $153.76\pm 19.81$ |  | $97.32\pm 17.95$ |  | $-38.25\pm 5.06$ |  | $-3.39\pm 2.77$
$\textrm{{elign}}_{\textrm{self}}$ | $186.55\pm 53.15$ |  | $127.97\pm 13.02$ |  | $103.46\pm 28.91$ |  | $-23.29\pm 5.00$ |  | $-4.90\pm 0.67$

Table 19:  The average test occupancy/collision count per step and standard
errors of centralized methods with different intrinsic rewards in scaled
environments under partial and full observability. Higher scores are better
for the occupancy metric ($\uparrow$), and lower scores are better for the
collision metric ($\downarrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\uparrow$ |  |  Hetero nav. (6v0) $\uparrow$ |  |  Phy decep. (4v2) $\uparrow$ |  |  Pred-prey (4v4) $\downarrow$ |  |  Keep-away (4v4) $\uparrow$
Partial observability | sparse | $0.11\pm 0.02$ |  | $0.29\pm 0.06$ |  | $0.56\pm 0.05$ |  | $0.06\pm 0.02$ |  | $0.07\pm 0.02$
$\textrm{{elign}}_{\textrm{self}}$ | $0.23\pm 0.09$ |  | $0.33\pm 0.04$ |  | $0.56\pm 0.06$ |  | $0.08\pm 0.00$ |  | $0.05\pm 0.00$
$\textrm{{elign}}_{\textrm{team}}$ | $0.27\pm 0.10$ |  | $0.33\pm 0.05$ |  | $0.50\pm 0.08$ |  | $0.09\pm 0.00$ |  | $0.09\pm 0.03$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.60\pm 0.09$ |  | $0.05\pm 0.02$ |  | $0.05\pm 0.01$
Full observability | sparse | $0.10\pm 0.04$ |  | $0.16\pm 0.04$ |  | $0.50\pm 0.03$ |  | $0.12\pm 0.01$ |  | $0.06\pm 0.01$
$\textrm{{elign}}_{\textrm{self}}$ | $0.16\pm 0.09$ |  | $0.11\pm 0.00$ |  | $0.55\pm 0.02$ |  | $0.10\pm 0.02$ |  | $0.04\pm 0.01$

Table 20:  The average test agent-to-target (agt-target) and agent-to-
adversary (agt-adv) distances and standard errors of centralized methods with
different intrinsic rewards in scaled environments under partial and full
observability. Lower scores are better for agt-target ($\downarrow$), and
higher scores are better for agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\downarrow$ |  |  Hetero nav. (6v0) $\downarrow$ |  |  Phy decep. (4v2) $\downarrow$ |  |  Pred-prey (4v4) $\uparrow$ |  |  Keep-away (4v4) $\downarrow$
Partial observability | sparse | $0.33\pm 0.01$ |  | $0.29\pm 0.02$ |  | $0.34\pm 0.02$ |  | $2.27\pm 0.08$ |  | $3.12\pm 0.18$
$\textrm{{elign}}_{\textrm{self}}$ | $0.32\pm 0.04$ |  | $0.28\pm 0.01$ |  | $0.36\pm 0.02$ |  | $2.32\pm 0.09$ |  | $3.25\pm 0.05$
$\textrm{{elign}}_{\textrm{team}}$ | $0.30\pm 0.04$ |  | $0.28\pm 0.01$ |  | $0.37\pm 0.03$ |  | $2.29\pm 0.08$ |  | $3.01\pm 0.20$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.33\pm 0.04$ |  | $2.44\pm 0.08$ |  | $3.23\pm 0.13$
Full observability | sparse | $0.37\pm 0.03$ |  | $0.37\pm 0.02$ |  | $0.29\pm 0.02$ |  | $1.98\pm 0.07$ |  | $3.13\pm 0.17$
$\textrm{{elign}}_{\textrm{self}}$ | $0.36\pm 0.04$ |  | $0.39\pm 0.00$ |  | $0.27\pm 0.01$ |  | $1.98\pm 0.08$ |  | $3.25\pm 0.12$

Table 21:  We sample agents from different decentralized training runs and
evaluate their zero-shot performance in scaled environments under partial
observability. We report the mean test episode extrinsic rewards and standard
errors of decentralized methods with different intrinsic rewards.

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) | Coop nav. (5v0) |  | Hetero nav. (6v0) |  | Phy decep. (4v2) |  | Pred-prey (4v4) |  | Keep-away (4v4)
Partial observability | sparse | $434.68\pm 6.42$ |  | $561.16\pm 31.63$ |  | $128.64\pm 17.31$ |  | $-32.12\pm 3.63$ |  | $-2.80\pm 2.91$
$\textrm{{elign}}_{\textrm{self}}$ | $471.07\pm 5.00$ |  | $676.01\pm 16.53$ |  | $248.16\pm 6.62$ |  | $-16.77\pm 2.25$ |  | $-5.03\pm 1.06$
$\textrm{{elign}}_{\textrm{team}}$ | $511.97\pm 6.95$ |  | $699.56\pm 11.64$ |  | $190.06\pm 29.10$ |  | $-19.40\pm 2.86$ |  | $-3.10\pm 3.09$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $228.53\pm 25.03$ |  | $-31.03\pm 3.13$ |  | $27.24\pm 4.48$

Table 22:  We sample agents from different decentralized training runs and
evaluate their zero-shot performance in scaled environments under partial
observability. We report the average test agent-to-target (agt-target) and
agent-to-adversary (agt-adv) distances and standard errors of decentralized
methods with different intrinsic rewards. Lower scores are better for agt-
target ($\downarrow$), and higher scores are better for agt-adv ($\uparrow$).

| Cooperative |  | Competitive
---|---|---|---
Task (Agt # vs. Adv #) |  Coop nav. (5v0) $\downarrow$ |  |  Hetero nav. (6v0) $\downarrow$ |  |  Phy decep. (4v2) $\downarrow$ |  |  Pred-prey (4v4) $\uparrow$ |  |  Keep-away (4v4) $\downarrow$
Partial observability | sparse | $0.22\pm 0.00$ |  | $0.23\pm 0.00$ |  | $0.23\pm 0.00$ |  | $1.93\pm 0.00$ |  | $3.16\pm 0.01$
$\textrm{{elign}}_{\textrm{self}}$ | $0.19\pm 0.00$ |  | $0.20\pm 0.00$ |  | $0.17\pm 0.00$ |  | $2.33\pm 0.00$ |  | $3.31\pm 0.01$
$\textrm{{elign}}_{\textrm{team}}$ | $0.43\pm 0.01$ |  | $0.19\pm 0.00$ |  | $0.24\pm 0.00$ |  | $2.04\pm 0.01$ |  | $3.15\pm 0.01$
$\textrm{{elign}}_{\textrm{adv}}$ | — |  | — |  | $0.21\pm 0.00$ |  | $2.11\pm 0.01$ |  | $2.31\pm 0.01$

###  A.7 Model architecture and hyperparameters

Table 23:  Model and training hyperparameters Parameter | Multi-agent particle | Google Research football
---|---|---
SAC actor model architecture | FC layers [128,128] | FC layers [256,256]
SAC critic model architecture | FC layers [128,128] | FC layers [256,256]
World model architecture | FC layers [128,128] | FC layers [128,128]
Replay buffer size | 1,000,000 | 1,000,000
Batch size | 1,024 | 256
Actor learning rate | 0.001 | 0.0003
Critic learning rate | 0.001 | 0.0003
Discount factor gamma | 0.95 | 0.99
SAC soft update coefficient | 0.01 | 0.005
SAC policy entropy regularization coefficient | 0.1 | 1.0 (initial)

Table 23 presents the model architecture and hyperparameters used to train the
algorithms in the multi-agent particle and Google Research football
environments.