# Efficient Communication via Self-supervised Information Aggregation for
Online and Offline Multi-agent Reinforcement Learning

Cong Guan  National Key Laboratory for Novel Software Technology, Nanjing
University
Email: {guanc, chenf<EMAIL_ADDRESS>{zzzhang<EMAIL_ADDRESS>Feng
Chen  National Key Laboratory for Novel Software Technology, Nanjing
University
Email: {guanc, chenf<EMAIL_ADDRESS>{zzzhang<EMAIL_ADDRESS>Lei
Yuan  National Key Laboratory for Novel Software Technology, Nanjing
University
Email: {guanc, chenf<EMAIL_ADDRESS>{zzzhang<EMAIL_ADDRESS>Zongzhang Zhang  National Key Laboratory for Novel Software Technology,
Nanjing University
Email: {guanc, chenf<EMAIL_ADDRESS>{zzzhang<EMAIL_ADDRESS>Yang
Yu Corresponding author National Key Laboratory for Novel Software Technology,
Nanjing University
Email: {guanc, chenf<EMAIL_ADDRESS>{zzzhang<EMAIL_ADDRESS>
###### Abstract

Utilizing messages from teammates can improve coordination in cooperative
Multi-agent Reinforcement Learning (MARL). Previous works typically combine
raw messages of teammates with local information as inputs for policy.
However, neglecting message aggregation poses significant inefficiency for
policy learning. Motivated by recent advances in representation learning, we
argue that efficient message aggregation is essential for good coordination in
cooperative MARL. In this paper, we propose Multi-Agent communication via
Self-supervised Information Aggregation (MASIA), where agents can aggregate
the received messages into compact representations with high relevance to
augment the local policy. Specifically, we design a permutation invariant
message encoder to generate common information-aggregated representation from
messages and optimize it via reconstructing and shooting future information in
a self-supervised manner. Hence, each agent would utilize the most relevant
parts of the aggregated representation for decision-making by a novel message
extraction mechanism. Furthermore, considering the potential of offline
learning for real-world applications, we build offline benchmarks for multi-
agent communication, which is the first as we know. Empirical results
demonstrate the superiority of our method in both online and offline settings.
We also release the built offline benchmarks in this paper as a testbed for
communication ability validation to facilitate further future research.

##  1 Introduction

Multi-Agent Reinforcement Learning (MARL) [24] has attracted widespread
attention [24, 10] recently, achieving remarkable success in many complex
domains [2], such as traffic signal control [11], droplet control [36], active
voltage control [65], and dynamic algorithm configuration [72]. For better
coordination on further applications, some issues like non-stationarity [47],
scalability [4] remain to be solved. To solve the non-stationarity caused by
the concurrent learning of multiple policies and scalability as the agent
number increases, most recent works on MARL adopt the Centralized Training and
Decentralized Execution (CTDE) [28, 39] paradigm, which includes both value-
based methods [57, 52, 64, 76] and policy gradient methods [16, 38, 68, 75],
or other techniques like transformer [69]. Under the CTDE paradigm, however,
the coordination ability of the learned policies can be fragile due to the
partial observability in the multi-agent environment, which is a common
challenge in many multi-agent tasks [41]. While recurrent neural networks
could in principle relieve this issue by conditioning the policy on action-
observation history [23], the uncertainty of other agents (e.g., states and
actions) at execution time can result in catastrophic miscoordination and even
sub-optimality [67, 8].

Communication shows great potential in solving these problems [78, 84], with
which agents can share information such as observations, intentions, or
experiences to stabilize the learning process, leading to a better
understanding of the environment (or the other agents) and better coordination
as a result. Previous communication methods either focus on generating
meaningful information [67, 27, 82] for the message senders, or design
techniques such as attention mechanism [6, 44], message gate [40, 8] to filter
the most relevant information on raw received messages. These approaches treat
the received information as a black box and tacitly assume that policy
networks can automatically extract the most critical information from multiple
raw messages during policy learning. On this occasion, with the only signal
given by reinforcement learning, the extraction process may be reasonably
inefficient, especially in complex scenarios.

Motivated by recent advances in state representation learning [54, 33], which
reveals that auxiliary representation objectives could facilitate policy
learning [14], we aim at efficiently aggregating information as compact
representations for policy by designing a novel communication framework Multi-
Agent communication via Self-supervised Information Aggregation (MASIA).
Specifically, representations are optimized through self-supervised
objectives, which encourages the representations to be both abstract of the
true states and predictive of the future information. Since agents are guided
towards higher cumulative rewards during policy learning, correlating
representations with true states and future information could intensify the
learning signals in policy learning. In this way, the efficiency of policy
learning could be encouraged. Also, considering that permutation invariance of
representations can also promote efficiency, we design a self-attention
mechanism to maintain the invariance of obtained representations. We also
design a network that weighs the aggregated representation for individual
agents to derive unique and highly relevant representation to augment local
policies for efficient coordination.

On the other hand, the application of Reinforcement Learning (RL) in real-
world scenarios faces significant challenges as the interaction with
environments is typically costly or even impossible [35]. Offline RL is
recently proposed to help solve this concern, which only learns the agent
policy from a fixed offline dataset without interaction with the environment.
Despite the significance and popularity of this topic, current works primarily
focus on single agent setting [35, 49, 18], or multi-agent coordination
without communication [74, 46, 58, 5, 79]. Even communication plays a crucial
role in multi-agent coordination, especially in partially observable scenarios
[84], there are neither any efficient approaches for this issue nor any
testbeds to validate this setting. Based on this situation, we construct an
offline testbed based on some popular scenarios where communication is indeed
necessary to test the communication ability for different communication
approaches. We hope these benchmarks can be utilized to benchmark the
performance of various multi-agent communication algorithms and trigger more
research about offline reinforcement learning concerning multi-agent
communication.

To evaluate our method, we conduct extensive experiments on various
cooperative multi-agent benchmarks, including Hallway [67], Level-Based
Foraging [48], Traffic Junction [6], and two maps from StarCraft Multi-Agent
Challenge (SMAC) [67] to validate the communication effectiveness in online
and offline settings. The online experimental results show that MASIA
outperforms previous approaches, strong baselines, and ablations of our
method, demonstrating the effectiveness of MASIA for online learning. We also
build the offline dataset based on the four environments above. Further
experiments concerning offline learning show that MASIA also performs well in
the offline setting.

Our main contributions are:

  * •

We propose a novel framework that uses a message aggregation network to
extract from multiple messages generated by various teammates, with which we
acquire a permutation invariant information aggregation representation. Agents
can then use a novel focusing network to extract the most relevant information
for decision-making.

  * •

Two representation objectives are introduced to make the information
representation compact and sufficient, including the state reconstruction and
multi-step future states prediction.

  * •

We construct an offline dataset for multi-agent communication, which considers
multiple environments and various dataset settings. This dataset is set up to
benchmark different communication algorithms under offline learning and
encourage more research on offline multi-agent communication learning.

  * •

Sufficient online results on various benchmarks and communication conditions
demonstrate that our proposed approach significantly improves the
communication performance, and visualization results further reveal why it
works. Additional experimental results on the offline dataset justify the
effectiveness of our approach in the offline setting, inspiring further
research in this field.

##  2 Problem Formulation

This paper considers a fully cooperative MARL communication problem, which can
be modeled as Decentralised Partially Observable Markov Decision Process under
Communication (Dec-POMDP-Com) [45] and formulated as a tuple
$\langle\mathcal{N},\mathcal{S},\mathcal{A},P,\Omega,O,R,\gamma,\mathcal{M}\rangle$,
where $\mathcal{N}=\\{1,\dots,n\\}$ is the set of agents, $\mathcal{S}$ is the
set of global states, $\mathcal{A}$ is the set of actions, $\Omega$ is the set
of observations, $O$ is the observation function, $R$ represents the reward
function, $\gamma\in[0,1)$ stands for the discounted factor, and $\mathcal{M}$
indicates the set of messages. At each time step, due to partial
observability, each agent $i\in\mathcal{N}$ can only acquire the observation
$o_{i}\in\Omega$ drawn from the observation function $O(s,i)$ with
$s\in\mathcal{S}$, each agent holds an individual policy
$\pi(a_{i}\mid\tau_{i},m_{i})$, where $\tau_{i}$ represents the history
$(o_{i}^{1},a_{i}^{1},\dots,o_{i}^{t-1},a_{i}^{t-1},o_{i}^{t})$ of agent $i$
at current timestep $t$, and $m_{i}\in\mathcal{M}$ is the message received by
the agent $i$. The joint action $\bm{a}=\langle a_{1},\dots,a_{n}\rangle$
leads to next state $s^{\prime}\sim P(s^{\prime}\mid s,\bm{a})$ and the global
reward $R(s,\bm{a})$. The formal objective is to find a joint policy
$\bm{\pi}(\bm{\tau},\bm{a})$ to maximize the global value function $Q_{\rm
tot}^{\bm{\pi}}(\bm{\tau},\bm{a})=\mathbb{E}_{s,\bm{a}}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s,\bm{a})\mid
s_{0}=s,\bm{a_{0}}=\bm{a},\bm{\pi}\right]$, with
$\bm{\tau}=\langle\tau_{1},\dots,\tau_{n}\rangle$. As each agent can behave as
a message sender as well as a message receiver, this paper considers learning
useful message representation in the received end, and agents only use local
information $o_{i}$ as message to share within the team.

We optimize the policy by value-based MARL, where deep Q-learning [43]
implements the action-value function $Q(s,\bm{a})$ with a deep neural network
$Q(\bm{\tau},\bm{a};\bm{\theta})$ parameterized by $\bm{\theta}$. This paper
follows the CTDE paradigm. In the centralized training phase, deep Q-learning
uses a replay memory $\mathcal{D}$ to store the transition tuple
$\langle\bm{\tau},\bm{a},r,\bm{\tau}^{\prime}\rangle$. We use
$Q(\bm{\tau},\bm{a};\bm{\theta})$ to approximate $Q(s,\bm{a};\bm{\theta})$ to
relieve the partial observability. Thus, the parameters $\bm{\theta}$ are
learnt by minimizing the expected Temporal Difference (TD) error:

$\mathcal{L}(\bm{\theta})=\mathbb{E}_{\left(\bm{\tau},\bm{a},r,\bm{\tau}^{\prime}\right)\in\mathcal{D}}\left[\left(r+\gamma
V\left(\bm{\tau}^{\prime};\bm{\theta}^{-}\right)-Q(\bm{\tau},\bm{a};\bm{\theta})\right)^{2}\right],$

where
$V\left(\tau^{\prime};\bm{\theta}^{-}\right)=\max_{\bm{a}^{\prime}}Q\left(\bm{\tau}^{\prime},\bm{a}^{\prime};\bm{\theta}^{-}\right)$
is the expected future return of the TD target and $\bm{\theta}^{-}$ are
parameters of the target network periodically updated with $\bm{\theta}$.

When considering the offline setting, we have an offline dataset which is
denoted as $\mathcal{B}$. The dataset is collected by specific behavior
policy, and it keeps fixed during the whole training process. For Dec-POMDP
problems, the offline dataset $\mathcal{B}$ is typically decomposed of a
number of trajectories, that is to say,
$\mathcal{B}:=\left\\{(s^{t},\\{o_{i}^{t}\\}_{\mathcal{N}},\\{a_{i}^{t}\\}_{\mathcal{N}},r^{t})_{t=1}^{T}\right\\}$,
where $T$ denotes the length of the trajectory, $r^{t}$ means the reward
obtained at the $t$-th timestep. When further considering multi-agent
communication, we additionally record the receivers’ id of the messages sent
by each agent. This information portrays the communication channels allowed by
the current problem.

##  3 Method

Figure 1: Structure of MASIA. (a) The overall architecture. (b) Information
aggregation and extraction. (c) Information aggregation optimization. (d)
Transition model learning.

In this paper, we propose efficient Multi-Agent communication via Self-
supervised Information Aggregation (MASIA), a novel multi-agent communication
mechanism for promoting cooperation performance. Redundant communications
could increase the burden of information processing for each agent to make
decisions and pose new challenges for information extraction since plenty of
irrelevant information is contained in raw messages. To design an efficient
communication mechanism, we believe two properties are of vital importance -
sufficiency and compactness, where sufficiency means a rich amount of
information, and compactness calls for higher information density.

To meet the standard of sufficiency, a global encoder, which we call
Information Aggregation Encoder (IAE), is shared among agents to aggregate the
information broadcasted by agents into a common representation. With proper
training, this representation could reflect the global observation so that
each agent could obtain sufficient information from it to make decisions. As
for compactness, we first design an auxiliary loss on the global
representation to correlate it with the policy learning process, and make each
agent only focus on the part of the representation related to its performance
and coordination by the designed focusing network through excluding the
unrelated parts. The entire framework of our method is shown in Figure 1.
Furthermore, a description of the training and execution processes can be
found in Appendix B.2.

###  3.1 Information Aggregation and Extraction

Information Aggregation.  Believing that the true state should be reflected
from combined messages, we design the aggregation encoder to be capable of
subsuming all the messages sent from agents. Also, the communication system in
multi-agent systems is flexible and permutation invariant in nature, which
calls for a permutation invariant structure for the aggregation encoder. Based
on these beliefs, we apply a self-attention mechanism to aggregate multiple
messages from different teammates:

$\displaystyle\bm{Q},\bm{K},\bm{V}$
$\displaystyle=\mathtt{MLP}_{Q,K,V}([o^{t}_{1},\dots,o^{t}_{i},\dots,o^{t}_{n}]),$
(1) $\displaystyle\bm{H}$
$\displaystyle=\mathtt{softmax}(\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{k}}})\bm{V},$
(2)

where the learnable matrices $Q$, $K$, and $V$ transform the perception from
all agents into the corresponding query $\bm{Q}$ , key $\bm{K}$, and value
$\bm{V}$, which are the concepts defined in the attention mechanism [61].
Specifically, each row vector of $\bm{H}$ can be seen as a querying result of
one agent for all available information, and the hidden state $\bm{H}$ will be
fed into the subsequent integration network to finally obtain the output
aggregated representation $z^{t}$. A detailed discussion about the design of
the integration network can be found in Appendix B.1.

Figure 2: A toy experiment for information redundancy on the task of Traffic
Junction.

In the centralized training phase, we use the aggregated representation
$z^{t}$ as extra input in addition to the individual observation to feed the
value function. Since $z^{t}$ contains the information required to determine
the true state, taking $z^{t}$ as extra input could reduce the uncertainty
about the environment states and produce better estimations on the Q-values
for value functions under any value-based policy learning algorithm.

Information Extraction.  Similar to the decision process of human beings,
global messages are usually redundant for an individual agent to make good
coordination in communication systems. For example, on the task of Traffic
Junction [6], one natural idea is that the information of neighboring cars are
more important for agents to perceive than those distant ones, and the
unrelated information in global message may sometimes even confuse the agents
and impede the learning when the map is large. A toy experiment shown in
Figure 2 supports this idea. We apply the QMIX algorithm in Traffic Junction
tasks with different sight settings. The results show that the agents learn
better policies when in a small-sight setting (sight-1) than both in a super-
limited-sight setting (sight-0) and full-sight setting (sight-full),
motivating a demand of efficient message extraction.

To make each agent capable of deciding its own perceptive area, we employ the
focusing network to weigh the aggregated representations for each agent. The
focusing network is designed as a Multi-Layer Perception (MLP) with the
Sigmoid output activation function to ensure that each dimension of
$w_{i}^{t}$ is bounded between $0$ and $1$. By taking element-wise
multiplication with $z^{t}$, a unique representation could be distilled for
individual agents. In this way, if the focusing network produces higher
weights on specific dimensions, changes in aggregated representations on these
dimensions would be more significant and thus, the agent would be more
sensitive to aggregated representation on these parts. On the contrary, if
some near-zero weights are outputted on some dimensions, information on those
dimensions would be filtered out. In particular, although the information
extraction process can, to some extent, reflect the specificity of each agent,
we stress the local information by feeding it into the subsequent network
together with the extracted representation.

###  3.2 Information Representation Optimization

As for the learning process of aggregated representation, we consider two
typical objectives in global encoder training: reconstruction and multi-step
prediction, which constrain the representations produced by the global encoder
to be sufficient and compact, respectively. For the reconstruction objective,
we employ an additional decoder, which aims to reconstruct the global state by
the aggregated representation to allow self-supervision on the global encoder.
Specifically, the decoder is optimized together with the aggregation encoder
by reconstructing the global states $s^{t}$ from the multiple received
messages $\bm{o}^{t}$:

$\mathcal{L}_{ae}(\theta,\eta)=\mathbb{E}_{\bm{o}^{t},s^{t}}\|g_{\eta}(z^{t})-s^{t}\|^{2}_{2},\quad~{}z^{t}=f_{\theta}(\bm{o}^{t}),~{}$
(3)

where $f_{\theta},g_{\eta}$ denote the encoder network parameterized by
$\theta$ and the decoder network parameterized by $\eta$, respectively. This
loss term resembles a classical auto-encoder loss, while the decoder here is
not to reconstruct the input, but to recover the global state from
representations instead. By utilizing this loss, we guide the encoder to
extract observational features that can help infer the global state and let
$z^{t}=f_{\theta}(\bm{o}^{t})$ be a sufficient representation.

As for the multi-step prediction objective, we constrain the produced
representation to be predictive of future information. Specifically, we design
a transition model
$h_{\psi}:\mathcal{Z}\times\mathcal{A}^{n}\rightarrow\mathcal{Z}$
parameterized by $\psi$ as auxiliary model, which predicts the aggregated
representation $z^{t+1}$ on next step $t+1$ through the aggregated
representation $z^{t}$ and joint action $\bm{a}^{t}$ on current step $t$. We
regress the predicted aggregated representation after $k$-step rollout on the
actual aggregated representation of future messages $\bm{o}^{t+k}$, updating
both the aggregation encoder and the auxiliary model via the multi-step
prediction loss:

$\mathcal{L}_{m}(\theta,\psi)=\mathbb{E}_{\bm{o}^{t},s^{t},\bm{a}^{t},\dots,\bm{a}^{t+K-1},\bm{o}^{t+K},s^{t+K}}\left[\sum_{k=1}^{K}\|\hat{z}^{t+k}-\tilde{z}^{t+k}\|_{2}^{2}\right],$
(4) $\displaystyle\hat{z}^{t+1}=h_{\psi}(\tilde{z}^{t},\bm{a}^{t}),~{}$
$\displaystyle\hat{z}^{t+k}=h_{\psi}(\hat{z}^{t+k-1},~{}\bm{a}^{t+k-1}),\quad
k=2,\dots,K,$ $\displaystyle\tilde{z}^{t+k}=f_{\theta}(\bm{o}^{t+k}),\quad
k=0,\dots,K.$

To further stabilize the learning process, we apply the double network
technique, which employs two networks with the same architecture but different
update frequencies, for the aggeration encoder. The target network is updated
via Exponential Moving Average (EMA) like in SPR [54]. By forcing the
aggregated representation to be predictive of its future states, the
aggregated representation could be more correlated with the information
required for its decision-making, which meets the compactness standard.
Combining these two objectives allows the aggregation encoder to extract more
helpful information for agents to coordinate better. To improve the capability
of information extraction on individual agents, we also enhance the learning
process of these components with an RL objective. Specifically, we consider
minimizing the TD loss:

$\displaystyle\mathcal{L}_{rl}(\theta,\phi)=\mathbb{E}_{\left(\bm{\tau},\bm{a},r,\bm{\tau}^{\prime}\right)\in\mathcal{D}}\Bigg{[}\Bigg{(}r+\gamma\max_{\bm{a}^{\prime}}Q_{\rm
tot}\left(\bm{\tau}^{\prime},\bm{a}^{\prime};\theta^{-},\phi^{-}\right)$ (5)
$\displaystyle-Q_{\rm
tot}(\bm{\tau},\bm{a};\theta,\phi)\Bigg{)}^{2}\Bigg{]},{}$

where $Q_{\rm tot}$ is computed with individual $Q$-values. The computation of
Q-values is actually dependent on the specific value-based learning algorithm.
We apply it to prevalent methods, including VDN [57], QMIX [52], and QPLEX
[64]. Moreover, the updating of the focusing network is coupled to the RL
objective, making the weights produced by the focusing network could be task-
sensitive, which could also facilitate policy learning.

###  3.3 Representation Pre-training for Offline Learning

Considering the significance of offline learning for real-world applications,
we also cover the offline setting in this paper. When applying our approach to
offline communication learning, we can only access a fixed offline dataset
without interacting with the environment. Thus, the difficulty lies in how to
fully utilize the limited offline data to learn multi-agent communication
policies efficiently. On the other hand, as the aggregation encoder is updated
together with the Q-networks, the representation space $\mathcal{Z}$ it
derives is, to some extent, dynamic. Usually the encoder requires a certain
amount of data for training to obtain a relatively stable representation
space.

Figure 3: The total training workflow of MASIA for offline learning. Which
includes two training stages, pre-training and joint training, respectively.
During the stage of pre-training, we only optimize the unsupervised
representation loss with the offline data, while for joint training we
optimize the representation loss and RL loss together. Besides, we utilize
agent$(o_{1}^{t},...,o_{n}^{t})$ to indicate the network inference process of
the agent policy, which is to output action or Q-values.

Motivated by these analyses, we propose doing representation pre-training
before optimizing the agent policy with the offline dataset. The core idea is
to firstly obtain a relatively good aggregation encoder, of which the derived
representation space is to some extent stable. Then we further update the
Q-networks with the offline dataset and jointly fine-tune the aggregation
encoder. The total workflow is depicted in Figure 3. In fact, this practice
brings some advantages: (1) the representation pre-training process typically
optimizes the unsupervised learning objectives and updates only the
aggregation encoder network, which does not require considering the
Q-divergence problem; (2) pre-training the aggregation encoder network brings
a more stable aggregation representation space, which will facilitate the
following optimization of RL objective. With this practice, we aim to better
utilize the offline dataset by considering the property of our approach. In
some sense, it can also be considered a benefit of our approach’s mechanism,
and we adopt this practice in all offline experiments. Further ablation
studies in Section 4.2.2 justifies the effectiveness of this practice.

##  4 Experiment

To evaluate the effectiveness of our approach both in the online and offline
problem setting, we first conduct online communication learning on four
benchmarks to validate the effectiveness of MASIA. Further, we build an
offline dataset to support offline communication learning. Its purpose is to
check the convergence performance of various multi-agent communication
learning algorithms in the offline setting and justify whether MASIAcan still
perform well with only a fixed offline dataset.

###  4.1 Online Multi-Agent Communication Learning

We conduct experiments on various benchmarks with different communication
request levels111The codes are available at https://github.com/chenf-ai/MASIA.
Specifically, we aim to answer the following questions in this section: 1) How
does our method perform when compared with multiple baselines in various
scenarios (Section 4.1.1)? 2) What kind of knowledge has been learned by the
information aggregation encoder (Section 4.1.2)? 3) How can the information
extraction module extract the most relevant information for the individual
from the learned embedding space (Section 4.1.3)? 4) Can MASIA be applied to
different value decomposition baselines to improve their coordination ability
and robustness in various communication conditions (Section 4.1.4)?

Figure 4: Multiple benchmarks used in our experiments.

(a) Hallway: 4x6x10

(b) LBF: 11x11-6p-4f-s1

(c) TJ: medium

(d) SMAC: 1o10b_vs_1r

(e) Hallway: 3x5-4x6x10

(f) LBF: 20x20-10p-6f-s1

(g) TJ: hard

(h) SMAC: 1o2r_vs_4r

Figure 5: Performance comparison with baselines on multiple benchmarks.

We compare MASIA against a variety of baselines, including communication-free
methods and some state-of-the-art communication approaches. QMIX [52] is a
strong communication-free baseline, and we use the implementation by
PyMARL222Our experiments are all based on the PyMARL framework, which uses
SC2.4.6.2.6923. for comparison, which has shown excellent performance on
diverse multi-agent benchmarks [53]. TarMAC utilizes an attention mechanism to
select messages according to their relative importance. The implementation we
used is provided by [67], denoted as TarMAC + QMIX. NDQ [67] aims at learning
nearly decomposable Q functions via generating meaningful messages and
communication minimization. TMC [82] applies a temporal smoothing technique at
the message sender end to drastically reduce the amount of information
exchanged between agents. For the ablation study, we design a baseline only
different in the communication protocol, which adopts a full communication
paradigm, where each agent gets message from all other teammates at each
timestep, denoted as FullComm.

We evaluate our proposed method on multiple benchmarks shown in Figure 4.
Hallway [67] is a cooperative environment under partial observability, where
$m$ agents are randomly initialized at different positions and required to
arrive at the goal $g$ simultaneously. We consider two scenarios with various
agents and different groups, and different groups have to arrive at different
times. Level Based Foraging (LBF) [48] is another cooperative partially
observable grid world game, where agents coordinate to collect food
concurrently. Traffic Junction (TJ) [6] is a popular benchmark used to test
communication ability, where many cars move along two-way roads with one or
more road junctions following the predefined routes, and we test on the medium
and hard maps. Two maps named 1o2r_vs_4r and 1o10b_vs_1r from SMAC [67]
require the agents to cooperate and communicate to get the position of the
enemies. For evaluation, all results are reported with median performance with
$95\%$ confidence interval on $5$ random seeds. Details about benchmarks,
network architecture and hyper-parameter choices of our method are all
presented in Appendices A, and B.1, respectively.

####  4.1.1 Communication Performance

Figure 6: (a) Information aggregation visualization. Each plotted dot
represents an aggregated representation. The three different colors
respectively represent three different initialization situations for the enemy
entities. For example, type $0$ shows the case when the enemies initialize at
the lower right corner. To distinguish aggregated representations on different
timesteps, we mark larger timesteps by darker shades of the dots. (b)
Visualization of variations of selected agents’ focus weight $w_{i}^{t}$ in a
single episode. We use the horizontal axes for timesteps in one single episode
and vertical axes for dimensions of the aggregated representation. The weight
is reflected through luminance, and the darker the cell, the larger the
weight.

We first compare MASIA against multiple baselines to investigate the
communication efficiency on various benchmarks. As illustrated in Figure 5,
MASIA achieves the best performance with low variance on all benchmarks,
indicating MASIA’s strong applicability in scenarios with various
difficulties. In Hallway (Figure 5(a) & Figure 5(e)), where frequent
communications are required for good performance (method without communication
such as QMIX fails), other communication methods such as TarMAC, NDQ, and TMC
achieve low performance or even fail in this environment. This indicates that
inappropriate message generation or message selection would injure the
learning process. We believe the reason why FullComm succeeds is that there is
hardly any redundancy in agents’ observations in Hallway. Our MASIA also
succeeds in this environment, showing superiority over others. The dominating
performance of MASIA is even more significant in extended Hallway (Figure
5(e)), where agents are separated into different groups. In this environment,
MASIA can help agents extract information about their teammates and learn a
coordination pattern more efficiently. In LBF (Figure 5(b) & Figure 5(f)),
existing communication-based MARL methods like NDQ, Fullcomm, and TarMAC
struggle due to the sparsity of rewards, especially when the foods are more
sparsely distributed (Figure 5(f)). In contrast to the performance of QMIX in
Hallway, QMIX performs well in LBF, which is attributed to the fact that
agents can observe the grids near them, and the mixing network of QMIX can
help improve the coordination ability of the fixed group of agents in the
training phase. Our method achieves comparable performance with QMIX and
TarMAC, showing its strong coordination ability even in sparse reward
scenarios. In Traffic Junction (Figure 5(c) & Figure 5(g)), TarMAC and NDQ
have high variance due to the instability of their messages, while MASIA gains
high sample efficiency and can generate steady messages since it aims to
reconstruct the state. On the SMAC benchmarks (Figure 5(d) & Figure 5(h)), we
test on two complex scenarios requiring communication to succeed, where one
overseer is in active service to get the information of the enemies. Messages
are demanded since the agents have limited sight, so other teammates need the
overseer’s messages to identify the enemies’ positions. Our method MASIA can
maintain the high efficiency of learning and always have competitive
performance when converged, which is superior to other baselines.

####  4.1.2 Insights into Information Aggregation Encoder

To determine what kind of knowledge the encoder has learned through training,
we conduct a visualization analysis on the map 1o2r_vs_4r from SMAC to
demonstrate the information contained in the aggregation representation
$z^{t}$. We project the aggregation representation vectors into two-
dimensional plane by t-SNE [60] in Figure 6a. We take trajectories from $3$
scenarios of different types of initialization under a task where agents have
to seek the enemy at the start, discover the enemy, and finally battle with it
for better performance. It can be observed that (1) the aggregated
representations could be well distinguished by phases. Projected
representations in the seeking phase are far from those in the battle phase
and closing phase. This implies that our learned representations could well
reflect the true states. (2) the aggregated representations are first
divergent in the seeking phase when enemies have been initialized, but become
increasingly interlaced later until the closing phase, when enemies have been
wiped out after a fierce battle. Since enemies are highly related to decision
making, such a result verifies that the aggregated representations exploit
also reward information. To sum up, the visualization results show that MASIA
can extract valuable global information with these representations.

####  4.1.3 Study about Individual Information Extraction

To demonstrate the effectiveness of our information extraction module, we
analyze the weights computed by the focusing network. Specifically, we select
the TJ (medium) task for evaluation and compare the weights produced by two
different agents. In this environment, agents could dynamically enter or leave
the plane, making the agents staying in the environment flexible through time.
It can be observed that agent $3$ and agent $6$ put focus on similar areas of
the aggregated representation. Especially after timestep $15$, when agents $3$
and $6$ are in similar situations and distant from the intersection, their
focuses are nearly the same. This verifies that the global state information
has been successfully extracted to individual agents. Also, on the top of the
figure, we draw a figure to measure the relationship between the cosine
similarity of weight vectors of different agents against timesteps. It reveals
that the similarity of their focus rises after these two agents begin to
proceed in the same line (indicated by the render images posted on the lower
parts of Figure 6b). This also conforms to the intuition that similar messages
should be extracted for similar observations.

####  4.1.4 Generality of Our Method

Table 1: The format of our offline dataset for multi-agent communication.

step $0$ | $\cdots$ |  step $t$ |  step $t+1$ | $\cdots$ |  step $T$
---|---|---|---|---|---
$s^{0}$ | $\cdots$ | $s^{t}$ | $s^{t+1}$ | $\cdots$ | $s^{T}$
$r^{0}$ | $\cdots$ | $r^{t}$ | $r^{t+1}$ | $\cdots$ | $r^{T}$
agent${}_{i=1}^{0}$ | $\cdots$ | agent${}_{i=1}^{t}$ | $o_{1}^{t}$ | agent${}_{i=1}^{t+1}$ | $o_{1}^{t+1}$ | $\cdots$ | agent${}_{i=1}^{T}$
$a_{1}^{t}$ | $a_{1}^{t+1}$
$done_{1}^{t}$ | $done_{1}^{t+1}$
${receiver\ id}_{1}^{t}$ | ${receiver\ id}_{1}^{t+1}$
$\cdots$ | $\cdots$ | $\cdots$ | $\cdots$ | $\cdots$ | $\cdots$
agent${}_{i=n}^{0}$ | $\cdots$ | agent${}_{i=n}^{t}$ | $o_{n}^{t}$ | agent${}_{i=n}^{t+1}$ | $o_{n}^{t+1}$ | $\cdots$ | agent${}_{i=n}^{T}$
$a_{n}^{t}$ | $a_{n}^{t+1}$
$done_{n}^{t}$ | $done_{n}^{t+1}$
${receiver\ id}_{n}^{t}$ | ${receiver\ id}_{n}^{t+1}$

We aim to verify that the proposed approach is agnostic to various sight
ranges and applied value-based MARL methods. We first conduct experiments on
the map 1o10b_vs_1r to show that MASIA could also generalize well on agents
with limited observations. The results in Section 4.1.1 show the performance
of MASIA when the agents have a sight range of $9$. When we narrow the agents’
sight ranges, as shown in Figure 7a, by receiving and aggregating messages
from teammates, the performance of MASIA does not suffer from a significant
drop. Our information aggregation and extraction modules prevent the agent
from forfeiting knowledge about the state when the sight range is further
limited.

Figure 7: (a) Performance comparison with varying sights, where s$n$ means the
sight range is $n$. (b) The increase of winning rates brought about by MASIA
on map 1o10b_vs_1r.

To show the generality of the MASIA framework, we also carry out experiments
to integrate MASIA with current baselines, including VDN, QMIX, and QPLEX. As
illustrated in Figure 7b, when integrated with MASIA, the performance of these
baselines can be vastly improved on the map 1o10b_vs_1r from SMAC. In this
scenario, one overseer is in service to monitor the enemies. Without
communication, the other agents have to search the map for the enemies
exhaustively. While with reliable communication, they could communicate with
each other and the overseer for better coordination. The results demonstrate
that MASIA can efficiently aggregate the messages and improve the agents’
coordination ability for these value-based MARL methods.

####  4.1.5 Ablation Studies for Online Learning

In our work, we propose two representation objectives to make the aggregated
information representation compact and sufficient. To further justify the
effectiveness of these two objectives, we conduct ablation studies for online
experiments. Specifically, we design three ablations:
(1)$\lambda_{1}=1,\lambda_{2}=0$; (2)$\lambda_{1}=0,\lambda_{2}=1$;
(3)$\lambda_{1}=0,\lambda_{2}=0$, which respectively corresponds to (1) only
use encoder-decoder learning loss; (2) only use latent model learning loss;
(3) neither loss is used. While our method (MASIA) corresponds to
$\lambda_{0}=1,\lambda_{1}=1$. The experimental results for the tasks of
Hallway are illustrated in Figure 8.

Figure 8: Ablation experiments for online learning in Hallway.

From the experimental results, we can see that MASIA with two representation
objectives outperforms the ablations. The proposed two represetantion
objectives help MASIA learn to solve the task faster in both settings of
4x6x10 and 3x5-4x6x10. Especially, some random seeds of the third ablation
even fail to solve the task 3x5-4x6x10 within 2M samples, which shows the
indispensable roles of these two representation objectives. They offer good
guides and accelerate the task learning.

###  4.2 Offline Multi-Agent Communication Learning

Currently, in the field of multi-agent communicative reinforcement learning,
there is still a lack of appropriate evaluation criteria for offline learning.
Thus, to explore the possibility of learning multi-agent communication
policies with offline dataset and to test the effectiveness of various multi-
agent communication algorithms in the offline setting, we construct a set of
offline datasets and conduct experiments on them. In specific, to make our
constructed dataset as close to the real-world data as possible, we learn from
D4RL [17] which offers a benchmark for single-agent offline learning, we
collect data in a similar way to build our dataset. Different from D4RL, our
dataset considers the property of Dec-POMDP, which means that it also contains
observations for each agent. Besides, as we focus on multi-agent communication
setting, the information of message receivers at each timestep is also
included in the dataset. We provide a description of the whole dataset
structure in Table 1. From this table, we can see that each trajectory in the
dataset consists of the individual observations and individual actions for
each agent. Besides, the variable ${receiver\ id}_{i}^{t}$ indicates who can
receive the messages sent by agent $i$ at timestep $t$. In specific, the data
is typically stored as vectors. For example, ${receiver\ id}_{i}^{t}$ may be
stored as [0, 1] which indicates that agent 0 and agent 1 can receive the
messages sent by agent $i$ at timestep $t$.

Table 2: The range of evaluation return values for the three different levels
of behavior policies.

Environment Level | Good | Medium | Poor
---|---|---|---
Hallway: 4x6x10 | 0.75 - 1.0 | 0.5 - 0.75 | 0.0 - 0.5
Hallway: 3x5-4x6x10 | 1.5 - 2.0 | 1.0 - 1.5 | 0.0 - 1.0
LBF: 11x11-6p-4f-s1 | 0.75 - 1.0 | 0.5 - 0.75 | 0.0 - 0.5
LBF: 20x20-10p-6f-s1 | 0.75 - 1.0 | 0.5 - 0.75 | 0.0 - 0.5
SMAC: 1o2r_vs_4r | 15 - 20 | 10 - 15 | 0 - 10
SMAC: 1o10b_vs_1r | 15 - 20 | 10 - 15 | 0 - 10
TJ: easy | 0 - (-7) | (-7) - (-17) | (-17) - (-600)
TJ: medium | 0 - (-7) | (-7) - (-17) | (-17) - (-600)

Considering that the real-world data is typically of greatly varying quality,
we design three different dataset settings: 1) expert, 2) noisy, and 3)
replay, respectively. These data-collection schemes aim to systematically
cover different real-world settings. More details are listed as below:

  * •

Expert Dataset. We train an online policy until convergence and greedily
sample data with the final expert policy. Such practice is also adopted in Fu
et al. [17], Gulcehre et al. [22] and Kumar et al. [31].

  * •

Noisy Dataset. The noisy dataset is generated with an expert policy that
selects actions via $\varepsilon$-greedy method with $\varepsilon=0.2$.
Creating a dataset from a fixed noisy policy is similar to the dataset
collection process in Fujimoto et al. [18], Kumar et al. [31] and Gulcehre et
al. [22]. This scheme aims to simulate the real-world scenarios where experts
may occasionally make mistakes.

  * •

Replay Dataset. This dataset contains the samples that represent different
periods of the online policy learning, to which similar scheme can be found in
Agarwal et al. [1] and Fujimoto et al. [18]. To consider offline data of
different qualities, we include three forms of Replay dataset. Concretely, we
save some intermediate policies during online training, and artificially
divide these saved policies into three levels (Poor, Medium and Good) based on
the evaluation return values as shown in Table 2. Then we collect three forms
of datasets with different mixtures of behavior policies: 1) Replay (Poor)
consists entirely of data collected by Poor-Level behavior policies (Poor-
Level data); 2) Replay (Medium) consists of 80% Medium-Level data and 20%
Poor-Level data; (3) Replay (Good) consists of 60% Good-Level data, 20%
Medium-Level data and 20% Poor-Level data.

Table 3: Experimental results of different methods under different benchmarks.

Environment/Dataset Method | ICQ+MASIA | ICQ+Fullcomm | ICQ+NDQ | ICQ+TarMAC | ICQ+TMC | ICQ
---|---|---|---|---|---|---
Hallway: 4x6x10 | Expert | 100.00 $\pm$ 0.00 | 100.00 $\pm$ 0.00 |  85.10 $\pm$ 5.55 |  97.80 $\pm$ 4.28 |  0.04 $\pm$ 0.24 |  0.25 $\pm$ 0.64
Noisy | 99.90 $\pm$ 0.07 |  99.60 $\pm$ 0.95 |  99.50 $\pm$ 0.74 |  99.80 $\pm$ 0.51 |  14.10 $\pm$ 11.62 |  1.55 $\pm$ 2.82
Replay (Poor) | 94.50 $\pm$ 3.59 |  87.90 $\pm$ 8.95 |  62.20 $\pm$ 9.21 |  83.60 $\pm$ 7.92 |  0.52 $\pm$ 0.94 |  0.11 $\pm$ 0.35
Replay (Medium) | 97.40 $\pm$ 3.60 |  94.40 $\pm$ 5.07 |  57.40 $\pm$ 8.24 |  92.80 $\pm$ 5.51 |  1.39 $\pm$ 1.69 |  0.12 $\pm$ 0.40
Replay (Good) | 99.70 $\pm$ 0.72 |  91.10 $\pm$ 8.46 |  11.30 $\pm$ 13.95 |  97.70 $\pm$ 2.59 |  0.33 $\pm$ 1.01 |  0.02 $\pm$ 0.17
Hallway: 3x5-4x6x10 | Expert | 99.70 $\pm$ 0.57 |  98.90 $\pm$ 0.48 |  86.50 $\pm$ 5.17 |  98.50 $\pm$ 2.26 |  1.18 $\pm$ 1.48 |  0.36 $\pm$ 0.73
Noisy | 99.90 $\pm$ 0.10 |  99.90 $\pm$ 0.20 |  99.10 $\pm$ 1.17 |  99.80 $\pm$ 1.84 |  7.67 $\pm$ 4.43 |  5.97 $\pm$ 3.71
Replay (Poor) | 73.40 $\pm$ 10.63 |  40.80 $\pm$ 20.46 |  10.20 $\pm$ 16.61 |  53.40 $\pm$ 26.28 |  0.00 $\pm$ 0.00 |  0.00 $\pm$ 0.00
Replay (Medium) | 81.90 $\pm$ 11.54 |  38.60 $\pm$ 15.92 |  2.78 $\pm$ 4.02 |  35.60 $\pm$ 24.91 |  0.00 $\pm$ 0.00 |  0.00 $\pm$ 0.00
Replay (Good) | 95.20 $\pm$ 4.02 |  77.50 $\pm$ 13.98 |  76.50 $\pm$ 12.65 |  88.30 $\pm$ 8.35 |  0.01 $\pm$ 0.10 |  0.06 $\pm$ 0.46
LBF: 11x11-6p-4f-s1 | Expert | 88.70 $\pm$ 13.09 |  76.50 $\pm$ 15.61 |  82.60 $\pm$ 4.33 |  88.30 $\pm$ 11.37 |  80.50 $\pm$ 14.13 |  82.90 $\pm$ 12.53
Noisy | 89.60 $\pm$ 9.42 |  78.40 $\pm$ 14.59 |  80.60 $\pm$ 4.93 |  88.80 $\pm$ 10.09 |  82.10 $\pm$ 10.94 |  80.10 $\pm$ 11.59
Replay (Poor) |  77.20 $\pm$ 15.14 |  66.40 $\pm$ 15.53 |  70.10 $\pm$ 5.38 | 78.10 $\pm$ 13.81 |  70.90 $\pm$ 15.14 |  71.20 $\pm$ 12.55
Replay (Medium) | 82.70 $\pm$ 10.92 |  70.70 $\pm$ 15.93 |  74.80 $\pm$ 5.74 |  80.50 $\pm$ 11.75 |  73.60 $\pm$ 13.45 |  75.70 $\pm$ 13.13
Replay (Good) | 89.10 $\pm$ 11.41 |  76.90 $\pm$ 13.43 |  79.30 $\pm$ 4.60 |  87.40 $\pm$ 8.96 |  78.20 $\pm$ 12.09 |  78.30 $\pm$ 11.92
LBF: 20x20-10p-6f-s1 | Expert | 54.70 $\pm$ 10.39 |  42.90 $\pm$ 10.53 |  45.80 $\pm$ 4.27 |  52.50 $\pm$ 10.83 |  46.40 $\pm$ 10.26 |  44.90 $\pm$ 10.53
Noisy |  54.10 $\pm$ 10.66 |  43.30 $\pm$ 11.89 |  47.30 $\pm$ 5.30 | 54.60 $\pm$ 12.71 |  48.70 $\pm$ 11.72 |  46.60 $\pm$ 11.92
Replay (Poor) | 48.90 $\pm$ 12.75 |  36.80 $\pm$ 11.49 |  44.50 $\pm$ 4.85 |  44.78 $\pm$ 10.23 |  45.10 $\pm$ 12.29 |  45.10 $\pm$ 11.72
Replay (Medium) | 50.20 $\pm$ 10.92 |  39.90 $\pm$ 11.04 |  46.40 $\pm$ 5.13 |  48.10 $\pm$ 11.85 |  45.70 $\pm$ 10.45 |  44.70 $\pm$ 10.56
Replay (Good) | 51.50 $\pm$ 12.78 |  41.70 $\pm$ 10.38 |  45.60 $\pm$ 5.50 |  48.90 $\pm$ 11.81 |  46.30 $\pm$ 10.54 |  45.20 $\pm$ 10.19
SMAC: 1o2r_vs_4r | Expert |  83.60 $\pm$ 17.68 | 83.60 $\pm$ 15.04 |  79.30 $\pm$ 12.71 |  80.70 $\pm$ 19.69 |  45.60 $\pm$ 25.81 |  43.70 $\pm$ 25.58
Noisy | 81.10 $\pm$ 15.08 |  76.70 $\pm$ 21.21 |  77.40 $\pm$ 10.59 |  79.90 $\pm$ 18.75 |  57.90 $\pm$ 26.88 |  41.50 $\pm$ 25.26
Replay (Poor) | 85.70 $\pm$ 15.61 |  83.60 $\pm$ 19.86 |  82.10 $\pm$ 10.76 |  84.70 $\pm$ 15.56 |  40.30 $\pm$ 25.85 |  45.40 $\pm$ 25.34
Replay (Medium) |  79.60 $\pm$ 18.17 | 80.50 $\pm$ 20.29 |  77.70 $\pm$ 10.86 |  79.20 $\pm$ 18.55 |  50.20 $\pm$ 26.39 |  45.20 $\pm$ 25.71
Replay (Good) | 80.10 $\pm$ 15.26 |  75.90 $\pm$ 23.41 |  77.90 $\pm$ 10.78 |  19.60 $\pm$ 21.23 |  48.80 $\pm$ 26.75 |  43.80 $\pm$ 26.68
SMAC: 1o10b_vs_1r | Expert | 84.10 $\pm$ 21.36 |  77.30 $\pm$ 21.48 |  82.80 $\pm$ 12.41 |  74.30 $\pm$ 23.08 |  11.90 $\pm$ 16.75 |  9.34 $\pm$ 18.37
Noisy | 84.60 $\pm$ 17.77 |  82.40 $\pm$ 18.89 |  78.70 $\pm$ 10.19 |  58.20 $\pm$ 26.63 |  13.30 $\pm$ 19.76 |  15.60 $\pm$ 18.11
Replay (Poor) | 82.90 $\pm$ 17.19 |  78.70 $\pm$ 21.93 |  77.70 $\pm$ 10.05 |  80.90 $\pm$ 20.91 |  7.70 $\pm$ 11.57 |  9.79 $\pm$ 18.18
Replay (Medium) |  82.90 $\pm$ 21.26 |  82.80 $\pm$ 18.54 |  81.70 $\pm$ 8.90 | 85.40 $\pm$ 16.14 |  9.19 $\pm$ 14.65 |  7.77 $\pm$ 13.97
Replay (Good) | 85.60 $\pm$ 17.79 |  80.90 $\pm$ 20.71 |  81.10 $\pm$ 10.19 |  80.30 $\pm$ 23.97 |  10.30 $\pm$ 17.21 |  6.44 $\pm$ 12.16
TJ: easy | Expert | 96.70 $\pm$ 10.04 |  68.60 $\pm$ 29.36 |  89.70 $\pm$ 7.51 |  96.70 $\pm$ 12.93 |  92.00 $\pm$ 16.31 |  91.20 $\pm$ 18.64
Noisy | 97.90 $\pm$ 9.93 |  93.60 $\pm$ 12.65 |  96.20 $\pm$ 4.39 |  96.40 $\pm$ 11.47 |  95.80 $\pm$ 10.24 |  95.10 $\pm$ 16.29
Replay (Poor) |  91.10 $\pm$ 19.53 |  75.60 $\pm$ 27.95 |  95.30 $\pm$ 5.22 |  92.20 $\pm$ 16.53 |  90.20 $\pm$ 20.76 | 95.50 $\pm$ 13.71
Replay (Medium) |  87.30 $\pm$ 21.93 |  71.90 $\pm$ 28.52 |  93.60 $\pm$ 7.89 |  89.80 $\pm$ 18.95 | 98.10 $\pm$ 9.69 |  96.80 $\pm$ 12.41
Replay (Good) |  90.80 $\pm$ 20.92 |  75.60 $\pm$ 27.01 |  89.20 $\pm$ 12.01 | 96.70 $\pm$ 10.23 |  94.40 $\pm$ 17.71 |  96.30 $\pm$ 13.93
TJ: medium | Expert | 88.60 $\pm$ 19.98 |  15.80 $\pm$ 24.56 |  78.70 $\pm$ 14.59 |  75.50 $\pm$ 31.17 |  81.50 $\pm$ 28.94 |  87.90 $\pm$ 22.49
Noisy |  72.70 $\pm$ 29.63 |  48.20 $\pm$ 32.49 |  71.30 $\pm$ 17.58 |  56.40 $\pm$ 34.31 |  76.10 $\pm$ 28.35 | 78.10 $\pm$ 26.14
Replay (Poor) | 99.60 $\pm$ 3.04 |  69.90 $\pm$ 30.91 |  96.30 $\pm$ 5.30 |  98.10 $\pm$ 6.89 |  97.50 $\pm$ 9.80 |  98.10 $\pm$ 9.24
Replay (Medium) |  89.90 $\pm$ 19.21 |  49.20 $\pm$ 33.36 |  83.70 $\pm$ 11.69 |  87.80 $\pm$ 21.61 | 91.20 $\pm$ 17.81 |  80.10 $\pm$ 27.11
Replay (Good) |  92.60 $\pm$ 14.81 |  72.20 $\pm$ 30.12 |  90.30 $\pm$ 6.89 |  85.90 $\pm$ 23.69 | 96.10 $\pm$ 12.88 |  91.10 $\pm$ 20.63

####  4.2.1 Communication Performance

(a) Replay (Poor)

(b) Replay (Medium)

(c) Replay (Medium)

(d) Replay (Good)

Figure 9: Learning curves for offline experiments. (a) and (b) are conducted
on Hallway: 3x5-4x6x10, while (c) and (d) are conducted on SMAC: 1o2r_vs_4r.

Considering the Q-divergence problem which is well-studied [35] in offline
learning, we propose to combine these communication algorithms with an offline
MARL algorithm, ICQ [74], which adopts a conservative learning paradigm to
alleviate the Q-divergence issue. Specifically, we integrate the design of
different communication algorithms into the actor network structure, thus
letting the agents learn message-based policy. That is to say agents make
decisions based on communication messages. We also add the experimental
results of the ICQ algorithm itself for an ablation, thus to validate the
effectiveness of agent communication. The overall results are listed in Table
3. Although Off-Policy Evaluation (OPE) [62] is commonly applied in offline RL
to test the final policy performance, in order to provide more accurate
evaluation, we directly evaluate the policy in the environment and report the
evaluation results. Actually, it is exciting to find that MASIA obtains the
best communication performance on most tasks under different offline dataset
settings. For example, when learning with the Expert offline dataset, MASIA
achieves the best performance in all scenarios except for SMAC: 1o2r_vs_4r.
While on the task SMAC: 1o2r_vs_4r, MASIA achieves as good average performance
as Full-Comm with a slightly larger variance. Again taking the task of
Hallway: 3x5-4x6x10 as an example, MASIA attains the highest success rates on
all modes of offline dataset, especially with a huge performance advantage
over all other baselines on Replay dataset. The good performance of MASIA on
Replay (Poor) and Replay (Medium) datasets demonstrates the robustness of
MASIA as it can still learn good communication policies with low quality data.

To further compare the learning trends of MASIA and other baselines on
different tasks, we also selectively show the learning curves for offline
experiment on part of datasets of Hallway: 3x5-4x6x10 and SMAC: 1o2r_vs_4r in
Figure 9. As we can see from Figure 9a and 9b, MASIA exhibits better learning
speed and convergence in the learning curves of Hallway: 3x5-4x6x10. MASIA is
the only method that achieves over 60% median test success rates within 4M
learning samples on Replay (Poor) dataset. Similarly, on Replay (Medium)
dataset of Hallway: 3x5-4x6x10, only MASIA steadily converges to a success
rate of over 80%, while all other methods fail. Again on SMAC: 1o2r_vs_4r,
MASIA shows faster convergence rate and better convergence performance, on par
with Full-Comm algorithm.

####  4.2.2 Ablation Studies for Offline Learning

(a) Replay (Medium)

(b) Replay (Poor)

Figure 10: Ablation experiments for offline learning on the task of Hallway:
3x5-4x6x10.

In the previous experiments, we have proved the importance of our proposed two
unsupervised representation objectives under online setting. To study how
these two objectives work in the offline setting, we further conduct ablation
studies for the offline experiments. The ablation results on task of Hallway:
3x5-4x6x10 with dataset Replay (Medium) and Replay (Poor) are depicted in
Figure 10.

Similar to the ablation studies for online experiments, here we ablate the two
unsupervised objectives, respectively. From the results we can see that,
ablating any of these two objectives will result in a drop in the final
communication performance, indicating that the designed two unsupervised
objectives play an indispensable role in the offline setting. Actually, it can
be seem that the impact of ablating the unsupervised learning objectives in
the offline setting is relatively larger than that in the online experiments.
The main reason for this phenomenon is that it is more challenging to learn a
good communication policy in the offline setting as no exploration in the
environment is allowed. Besides, the problem of the dataset quality,
especially for dataset Replay (Poor), strengthens the problem.

Moreover, we also add the ablation experiments for the representation pre-
training practice in the offline experiments. From the results we can see that
the ablation of the representation pre-training also causes great harm to the
performance, which justifies the effectiveness of this practice. More ablation
experiments and analyses can be found in Appendix C.2.

##  5 Related Work

Multi-Agent Reinforcement Learning (MARL). MARL has made prominent progress
these years. Having emerged under the CTDE paradigm, many methods are designed
to relieve the non-stationarity issue, and have made noticeable progress these
years. Most of them can be roughly divided into policy-based and value-based
methods. Typical policy gradient-methods involve MADDPG [38], COMA [16], MAAC
[25], SQDDPG [66], FOP [83], and HAPPO [29] which explore the optimization of
multi-agent policy gradient methods, while value-based methods mainly focus on
the factorization of the global value function [63, 9]. VDN [57] applies a
simple additive factorization to decompose the joint value function into
agent-wise value functions. QMIX [52] structurally enforces the learned joint
value function to be monotonic to the agent’s utilities, which can represent a
more affluent class of value functions. QPLEX [64] further takes a duplex
dueling network architecture to factorize the joint value function, achieving
a full expressiveness power of Individual Global Maximization (IGM) [55].

Multi-agent Communication. Communication plays a promising role in multi-agent
coordination under partial observability [13, 84]. Extensive research have
been made on learning communication protocols to improve performance on
cooperative tasks [20, 15, 34, 73, 12, 37, 77, 71, 21]. Previous works can be
divided into two categories. One focuses on generating a meaningful message
for the message senders. The simplest way is to treat the raw local
observation, or the local information history as message [15, 56]. VBC [81]
and TMC [82] apply techniques, such as variance-based control and temporal
smoothing, in the sender end to make the generated messages meaningful and
valuable for policy learning. NDQ [67] generates minimized messages for
different teammates to learn nearly decomposable value functions, and optimize
the message generator based on two different information-theory-based
regularizers to achieve expressive communication. On the contrary, other works
try to learn efficiently to extract the most useful message on the receiver
end, and they design mechanisms to differentiate the importance of messages.
I2C [8] and ACML [40] employ the gate mechanism to be selective on received
messages. There are also works inspired by the broad application of the
attention mechanism [3, 7]. TarMAC [6] achieves targeted communication via a
simple signature-based soft-attention mechanism, where the sender broadcasts a
key encoding the properties of the agents, then the receiver attends to all
received messages for a weighted sum of messages for decision marking. SARNet
[51] and MAGIC [44] further remove the signature in TarMAC and leverage
attention-based networks to learn efficient and interpretable relations
between entities, decide when and with whom to communicate.

Offline MARL. Offline reinforcement learning [35] attracts tremendous
attention for its data-driven training paradigm without interactions with the
environment [50]. Previous work [19] discusses the distribution shift issue in
offline learning and considers learning behavior-constrained policies to
relieve extrapolation error from unseen data estimations [70, 30, 32]. Offline
MARL is a promising research direction [80] that trains policies from a static
dataset. Following online MARL methods that either extend policy gradient
algorithms to multi-agent cases [38, 16, 68] or adopt Q-learning paradigms
with value decomposition [57, 52, 55, 64], existing offline MARL methods try
to exploit offline data with policy constraints. ICQ [74] effectively
alleviates the extrapolation error by only trusting offline data. MABCQ [26]
introduces a fully decentralized offline MARL setting and utilizes techniques
of value deviation and transition normalization for efficient learning. OMAR
[46] combines first-order policy gradients and zeroth-order optimization
methods to avoid the uncoordinated local optima. MADT [42] leverages
transformer’s modelling ability of sequence modelling and integrates it
seamlessly with both offline and online MARL tasks. [58] investigates offline
MARL with explicit consideration on the diversity of agent-wise trajectories
and proposes a novel framework called Shared Individual Trajectories (SIT) to
address this problem. [59] proposes to first train a teacher policy who has
the privilege to access every agent’s observations, actions, and rewards.
After the teacher policy has identified and recombined the "good" behavior in
the dataset, they create separate student policies and distill not only the
teacher policy’s features but also its structural relations among different
agents’ features to student policies. ODIS [79] proposes a novel Offline MARL
algorithm to Discover coordInation Skills (ODIS) from multi-task data. [5]
recently releases a framework Off-the-Grid MARL (OG-MARL) for generating
offline MARL datasets and algorithms without communication, which releases an
initial set of datasets and baselines for cooperative offline MARL, along with
a standardised evaluation protocol.

To the best of our knowledge, none of the existing MARL communication methods
explicitly consider how the multiple received messages can be optimized for
efficient policy learning. Agents may be confused by redundant information
from teammates, and simply augmenting the local policy with the raw message
may burden the learning. Meanwhile, there is no testbed for offline multi-
agent communication setting. Our proposed method applies a message aggregation
module to learn a compact information representation and extracts the most
relevant part for decision-making in online and offline settings.

##  6 Conclusion and Future Work

In this paper, we investigate the information representation for multi-agent
communication. Previous works either focus on generating meaningful messages
or designing a mechanism to select the most relevant message in a raw way,
ignoring the aggregation of the message, resulting in low sample efficiency in
complex scenarios. Our approach improves communication efficiency by learning
a compact information representation to ground the true state and optimizing
it in a self-supervised way. Also, we apply a focusing network to extract the
most relevant part for decision-making. We conduct sufficient experiments in
various benchmarks to verify the efficiency of the proposed methods, both in
online and offline settings, and more visualization results further reveal why
our approach works. We further release the newly built offline benchmark for
multi-agent communication, hoping to facilitate the real-world application of
MARL communication. For future work, more results on image input and solving
the scalability issue when facing environments with hundreds or thousands of
agents by techniques like agent grouping would be of great interest. Also, how
to obtain a robust communication policy when suffering from a distribution
shift in online policy deployment is an urgent topic.

## Acknowledgments

This work is supported by the National Key Research and Development Program of
China (2020AAA0107200), the National Science Foundation of China (61921006,
61876119, 62276126), the Natural Science Foundation of Jiangsu (BK20221442),
and the program B for Outstanding Ph.D. candidate of Nanjing University. We
thank Lichao Zhang and Chuneng Fan for their useful support, suggestions, and
discussions.

## References

  * [1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In International Conference on Machine Learning, pages 104–114. PMLR, 2020.
  * [2] Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò.  Multi-agent reinforcement learning: A review of challenges and applications.  Applied Sciences, 11(11):4948, 2021.
  * [3] Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath.  An attentive survey of attention models.  arXiv preprint arXiv:1904.02874, 2019.
  * [4] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht.  Scaling multi-agent reinforcement learning with selective parameter sharing.  In International Conference on Machine Learning, pages 1989–1998, 2021.
  * [5] Formanek Claude, Jeewa Asad, Shock Jonathan, and Arnu Pretorius.  Off-the-grid MARL: a framework for dataset generation with baselines for cooperative offline multi-agent reinforcement learning.  arXiv preprint arXiv:2302.00521, 2023.
  * [6] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau.  TarMAC: Targeted multi-agent communication.  In International Conference on Machine Learning, pages 1538–1546, 2019.
  * [7] Alana de Santana Correia and Esther Luna Colombini.  Attention, please! A survey of neural attention models in deep learning.  arXiv preprint arXiv:2103.16775, 2021.
  * [8] Ziluo Ding, Tiejun Huang, and Zongqing Lu.  Learning individually inferred communication for multi-agent cooperation.  In Advances in Neural Information Processing Systems, pages 22069–22079, 2020.
  * [9] Zehao Dou, Jakub Grudzien Kuba, and Yaodong Yang.  Understanding value decomposition algorithms in deep cooperative multi-agent reinforcement learning.  arXiv preprint arXiv:2202.04868, 2022.
  * [10] Wei Du and Shifei Ding.  A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications.  Artificial Intelligence Review, 54(5):3215–3238, 2021.
  * [11] Xin Du, Jiahai Wang, Siyuan Chen, and Zhiyue Liu.  Multi-agent deep reinforcement learning with spatio-temporal feature fusion for traffic signal control.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 470–485, 2021.
  * [12] Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang.  Learning correlated communication topology in multi-agent reinforcement learning.  In International Conference on Autonomous Agents and MultiAgent Systems, pages 456–464, 2021.
  * [13] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel.  Biases for emergent communication in multi-agent reinforcement learning.  In Advances In Neural Information Processing Systems, pages 13111–13121, 2019.
  * [14] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales.  Self-supervised representation learning: Introduction, advances and challenges.  arXiv preprint arXiv:2110.09327, 2021.
  * [15] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson.  Learning to communicate with deep multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 2137–2145, 2016.
  * [16] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.  Counterfactual multi-agent policy gradients.  In AAAI Conference on Artificial Intelligence, pages 2974–2982, 2018.
  * [17] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  arXiv preprint arXiv:2004.07219, 2020.
  * [18] Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau.  Benchmarking batch deep reinforcement learning algorithms.  arXiv preprint arXiv:1910.01708, 2019.
  * [19] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In International Conference on Machine Learning, pages 2052–2062, 2019.
  * [20] C Lee Giles and Kam-Chuen Jim.  Learning communication for multi-agent systems.  In Workshop on Radical Agent Concepts, pages 377–390. Springer, 2002.
  * [21] Cong Guan, Feng Chen, Lei Yuan, Chenghe Wang, Hao Yin, Zongzhang Zhang, and Yang Yu.  Efficient multi-agent communication via self-supervised information aggregation.  In Advances in Neural Information Processing Systems, 2022.
  * [22] Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas.  Regularized behavior value estimation.  arXiv preprint arXiv:2103.09575, 2021.
  * [23] Matthew J. Hausknecht and Peter Stone.  Deep recurrent Q-learning for partially observable MDPs.  In AAAI Conference on Artificial Intelligence, pages 29–37, 2015\.
  * [24] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor.  A survey and critique of multiagent deep reinforcement learning.  Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019\.
  * [25] Shariq Iqbal and Fei Sha.  Actor-attention-critic for multi-agent reinforcement learning.  In International Conference on Machine Learning, pages 2961–2970, 2019.
  * [26] Jiechuan Jiang and Zongqing Lu.  Offline decentralized multi-agent reinforcement learning.  arXiv preprint arXiv:2108.01832, 2021.
  * [27] Woojun Kim, Jongeui Park, and Youngchul Sung.  Communication in multi-agent reinforcement learning: Intention sharing.  In International Conference on Learning Representations, 2020.
  * [28] Landon Kraemer and Bikramjit Banerjee.  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  Neurocomputing, 190:82–94, 2016.
  * [29] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang.  Trust region policy optimisation in multi-agent reinforcement learning.  arXiv preprint arXiv:2109.11251, 2021.
  * [30] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy Q-learning via bootstrapping error reduction.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative Q-learning for offline reinforcement learning.  Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  * [32] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative Q-learning for offline reinforcement learning.  In Advances in Neural Information Processing Systems, volume 33, pages 1179–1191, 2020.
  * [33] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  CURL: Contrastive unsupervised representations for reinforcement learning.  In International Conference on Machine Learning, pages 5639–5650, 2020.
  * [34] Angeliki Lazaridou and Marco Baroni.  Emergent multi-agent communication in the deep learning era.  arXiv preprint arXiv:2006.02419, 2020.
  * [35] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643, 2020.
  * [36] Tung-Che Liang, Jin Zhou, Yun-Sheng Chan, Tsung-Yi Ho, Krishnendu Chakrabarty, and Cy Lee.  Parallel droplet control in meda biochips using multi-agent reinforcement learning.  In International Conference on Machine Learning, pages 6588–6599, 2021.
  * [37] Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola.  Learning to ground multi-agent communication with autoencoders.  In Advances in Neural Information Processing Systems, pages 15230–15242, 2021.
  * [38] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.
  * [39] Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato.  Contrasting centralized and decentralized critics in multi-agent reinforcement learning.  arXiv preprint arXiv:2102.04402, 2021.
  * [40] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni.  Learning agent communication under limited bandwidth by message pruning.  In AAAI Conference on Artificial Intelligence, pages 5142–5149, 2020.
  * [41] Weichao Mao, Kaiqing Zhang, Erik Miehling, and Tamer Başar.  Information state embedding in partially observable cooperative multi-agent reinforcement learning.  In IEEE Conference on Decision and Control, pages 6124–6131, 2020\.
  * [42] Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu.  Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.  arXiv preprint arXiv:2112.02845, 2021.
  * [43] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [44] Yaru Niu, Rohan Paleja, and Matthew Gombolay.  Multi-agent graph-attention communication and teaming.  In International Conference on Autonomous Agents and MultiAgent Systems, pages 964–973, 2021.
  * [45] Frans A Oliehoek and Christopher Amato.  A Concise Introduction to Decentralized POMDPs.  Springer, 2016.
  * [46] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu.  Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.  In International Conference on Machine Learning, pages 17221–17237, 2022.
  * [47] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht.  Dealing with non-stationarity in multi-agent deep reinforcement learning.  arXiv preprint arXiv:1906.04737, 2019.
  * [48] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht.  Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks.  In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
  * [49] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  arXiv preprint arXiv:1910.00177, 2019.
  * [50] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini.  A survey on offline reinforcement learning: Taxonomy, review, and open problems.  arXiv preprint arXiv:2203.01387, 2022.
  * [51] Murtaza Rangwala and Ryan Williams.  Learning multi-agent communication through structured attentive reasoning.  In Advances in Neural Information Processing Systems, pages 10088–10098, 2020.
  * [52] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson.  QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning.  In International Conference on Machine Learning, pages 4295–4304, 2018.
  * [53] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob N Foerster, and Shimon Whiteson.  The starcraft multi-agent challenge.  In International Conference on Autonomous Agents and MultiAgent Systems, pages 2186–2188, 2019.
  * [54] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman.  Data-efficient reinforcement learning with self-predictive representations.  In International Conference on Learning Representations, 2020.
  * [55] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi.  QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.  In International Conference on Machine Learning, pages 5887–5896, 2019.
  * [56] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus.  Learning multiagent communication with backpropagation.  In Advances in Neural Information Processing Systems, pages 2244–2252, 2016.
  * [57] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, and Thore Graepel.  Value-decomposition networks for cooperative multi-agent learning based on team reward.  In International Conference on Autonomous Agents and MultiAgent Systems, pages 2085–2087, 2018.
  * [58] Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang.  Learning from good trajectories in offline multi-agent reinforcement learning.  arXiv preprint arXiv:2211.15612, 2022.
  * [59] Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola.  Offline multi-agent reinforcement learning with knowledge distillation.  In Advances in Neural Information Processing Systems, 2022.
  * [60] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research, 9(11):2579–2605, 2008.
  * [61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [62] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue.  Empirical study of off-policy policy evaluation for reinforcement learning.  In Advances in Neural Information Processing Systems, 2021.
  * [63] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang.  Towards understanding cooperative multi-agent Q-learning with value factorization.  In Advances in Neural Information Processing Systems, pages 29142–29155, 2021.
  * [64] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang.  QPLEX: Duplex dueling multi-agent Q-learning.  In International Conference on Learning Representations, 2021.
  * [65] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green.  Multi-agent reinforcement learning for active voltage control on power distribution networks.  In Advances in Neural Information Processing Systems, pages 3271–3284, 2021.
  * [66] Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu.  Shapley Q-value: A local reward approach to solve global reward games.  In AAAI Conference on Artificial Intelligence, pages 7285–7292, 2020.
  * [67] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang.  Learning nearly decomposable value functions via communication minimization.  In International Conference on Learning Representations, 2020.
  * [68] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang.  DOP: Off-policy multi-agent decomposed policy gradients.  In International Conference on Learning Representations, 2021.
  * [69] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang.  Multi-agent reinforcement learning is a sequence modeling problem.  In Advances in Neural Information Processing Systems, 2022.
  * [70] Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  arXiv preprint arXiv:1911.11361, 2019.
  * [71] Di Xue, Lei Yuan, Zongzhang Zhang, and Yang Yu.  Efficient multi-agent communication via shapley message value.  In International Joint Conference on Artificial Intelligence, pages 578–584, 2022.
  * [72] Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu.  Multi-agent dynamic algorithm configuration.  In Advances in Neural Information Processing Systems, 2022.
  * [73] Wanqi Xue, Wei Qiu, Bo An, Zinovi Rabinovich, Svetlana Obraztsova, and Chai Kiat Yeo.  Mis-spoke or mis-lead: Achieving robustness in multi-agent communicative reinforcement learning.  In International Conference on Autonomous Agents and MultiAgent Systems, pages 1418–1426, 2022.
  * [74] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao.  Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning.  In Advances in neural information processing systems, 2021.
  * [75] Jianing Ye, Chenghao Li, Jianhao Wang, and Chongjie Zhang.  Towards global optimality in cooperative MARL with sequential transformation.  arXiv preprint arXiv:2207.11143, 2022.
  * [76] Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu.  Multi-agent concentrative coordination with decentralized task representation.  In International Joint Conference on Artificial Intelligence, pages 599–605, 2022.
  * [77] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang.  Multi-agent incentive communication via decentralized teammate modeling.  In AAAI Conference on Artificial Intelligence, pages 9466–9474, 2022.
  * [78] Mohamed Salah Zaïem and Etienne Bennequin.  Learning to communicate in multi-agent reinforcement learning: A review.  arXiv preprint arXiv:1911.05438, 2019.
  * [79] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang.  Discovering generalizable multi-agent coordination skills from multi-task offline data.  In International Conference on Learning Representations, 2023.
  * [80] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar.  Finite-sample analysis for decentralized batch multiagent reinforcement learning with networked agents.  IEEE Transactions on Automatic Control, 66(12):5925–5940, 2021\.
  * [81] Sai Qian Zhang, Qi Zhang, and Jieyu Lin.  Efficient communication in multi-agent reinforcement learning via variance based control.  In Advances in Neural Information Processing Systems, pages 3230–3239, 2019.
  * [82] Sai Qian Zhang, Qi Zhang, and Jieyu Lin.  Succinct and robust multi-agent communication with temporal message control.  In Advances in Neural Information Processing Systems, pages 17271–17282, 2020.
  * [83] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu.  FOP: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning.  In International Conference on Machine Learning, pages 12491–12500, 2021.
  * [84] Changxi Zhu, Mehdi Dastani, and Shihan Wang.  A survey of multi-agent reinforcement learning with communication.  arXiv preprint arXiv:2203.08975, 2022.

##  Appendix A Details about Benchmarks and Algorithms Involved

We introduce four types of testing environments as shown in Figure 4 and Table
4, including Hallway [67], Level-Based Foraging (LBF) [48], Traffic Junction
(TJ) [6], and two maps named 1o2r_vs_4r and 1o10b_vs_1r requiring
communication from StarCraft Multi-Agent Challenge (SMAC) [67].

Table 4: Properties for each environment.

Environment Property | narrow | undirected | multi-task | sparse rewards | suboptimal | realistic | partial observability
---|---|---|---|---|---|---|---
Hallway: 4x6x10 | ✓ | ✓ |  | ✓ |  |  | ✓
Hallway: 3x5-4x6x10 | ✓ | ✓ |  | ✓ |  |  | ✓
LBF: 11x11-6p-4f-s1 | ✓ |  | ✓ |  | ✓ |  | ✓
LBF: 20x20-10p-6f-s1 | ✓ |  | ✓ |  | ✓ |  | ✓
SMAC: 1o2r_vs_4r | ✓ |  |  |  | ✓ |  | ✓
SMAC: 1o10b_vs_1r | ✓ |  |  |  | ✓ |  | ✓
TJ: easy | ✓ |  |  |  | ✓ |  | ✓
TJ: medium | ✓ |  |  |  | ✓ |  | ✓

Hallway: We design two instances of the Hallway environment. In the first
instance, we apply three hallways with lengths of $4,6$, and $10$,
respectively. That means we let three agents $a,b,c$ respectively initialized
randomly at states $a_{1}$ to $a_{4}$, $b_{1}$ to $b_{6}$, and $c_{1}$ to
$c_{10}$, and require them to arrive at state $g$ simultaneously. In the
second instance, we divide $5$ agents into two groups. The first group has
hallways with lengths of $3$ and $5$, and the second group has hallways with
lengths of $4,6$, and $10$. A reward of $1$ will be given if one group arrives
at the goal $g$ simultaneously. However, if both groups reach the goal
simultaneously, a penalty of $-0.5$ will be given.

Level-Based Foraging (LBF): We use a variant version of the original
environment used in [48], where we define the state to be a data structure
that can represent the true global state instead of concatenating the
observations of all agents directly. On this basis, we use two environment
instances with different configurations, of which one is an $11\times 11$ grid
world with $6$ agents, $4$ foods, and the other is a $20\times 20$ grid world
with $10$ agents, $6$ foods. In both instances, the observation of agents is a
$3\times 3$ field of view around it.

Traffic Junction (TJ): We use the medium and hard versions of the Traffic
Junction environments. The $medium$ version has an agent number limit of $10$,
and the road dimension is $14$, while the $hard$ version has an agent number
limit of $20$, and the road dimension is $18$. In both of these two instances,
the sight of the agent is limited to $0$, which means each agent can only
observe a $1\times 1$ field of view around it.

StarCraft Multi-Agent Challenge (SMAC): We use two maps named 1o2r_vs_4r and
1o10b_vs_1r in SMAC, which are introduced in NDQ [67]. In 1o2r_vs_4r, an
Overseer finds $4$ Reapers, and the ally units, $2$ Roaches, need to reach
enemies and kill them. Similarly, 1o10b_vs_1r is a map full of cliffs, where
an Overseer detects a Roach, and the randomly spawned ally units, $10$
Banelings, are required to reach and kill the enemy.

Table 5: Comparison of various algorithms used in this paper.

Name |  | Type of communication
---
| Where to process the information
---
(the Sender/Receiver)
| Matched scenarios
---
MASIA(ours) | Full | Receiver | No Restrictions
Full-Comm | Full | No | Without redundant information
NDQ | Full | Sender | Value function is nearly decomposable
TarMAC+QMIX | Full | Receiver | Message with relative importance
TMC | Time-Partial | Sender & Receiver | Message with transmission loss
QMIX | No | No | Full observation or easy coordination

Several algorithms are involved in our work, including Full-Comm, NDQ [67],
TarMAC [6], TMC [82], and QMIX [52]. All these methods follow the setting of
Dec-POMDP in our experiments, which means that each agent can only have access
to its individual partial observation at each timestep. Some algorithms among
them (Full-Comm, NDQ, TarMAC, and TMC) do communication to base each agent’s
decision-making on richer information, while QMIX has no communication and
always let agents make decisions based on their local observations (or
observation histories). In Table 5, we offer a comparison between these
baselines and our method from different dimensions.

##  Appendix B Implementation Details

###  B.1 Network Architecture and Hyper-parameters

Integration Network. The Information Aggregation Encoder (IAE) in our approach
consists of a self-attention network and an integration network. Here we
describe the details of the integration network. We introduce $3$ different
kinds of integration networks as shown in Figure 11.

Figure 11: Three versions of implementation for integration network. The
dotted lines in (c) indicate that these GRU networks share the same
parameters.

Among these three versions of integration networks, $(a)$ and $(b)$ apply mean
and sum pooling operations, respectively, on the output vectors of the self-
attention network, while in the third version $(c)$, we let the self-attention
network’s output vectors pass through a shared GRU network separately and
finally flatten them as the obtained aggregation representation. In $(a)$ and
$(b)$, no matter how we permute the agents, we will always obtain the same
aggregation representation. In $(c)$, the permutation of the agents will only
affect the order of some dimensions of the aggregation representation instead
of obtaining a totally different representation in some vanilla designs, such
as networks with Multi-Layer Perceptions. In practice, we find that design
$(c)$ achieves the best performance among these three integration networks,
and all the experiment results shown in our paper are based on this
implementation.

Hyper-parameters. Our implementation of MASIA is based on the
EPyMARL333https://github.com/uoe-agents/epymarl [53] with StarCraft
2.4.6.2.69232 and uses its default hyper-parameter settings. For example, we
apply the default $\epsilon$-greedy action selection algorithm to each method,
which means $\epsilon$ decays from $1$ to $0.05$ in $50$K timesteps. The
selection of the additional parameters introduced in our approach is listed in
Table 6. We use this set of parameters in all experiments shown in this paper
except for the ablations.

Table 6: Hyper-parameters in experiments.

name | value
---|---
hidden dimension for query and key in self-attention module | 16
output dimension of self-attention module | 32
$\lambda_{3}$ (whether to warm up the representation learning) | 1
$\lambda_{2}$ (coefficient of latent model learning loss) | 1
$\lambda_{1}$ (coefficient of encoder-decoder learning loss) | 1
action embedding dimension in latent model | 8
dimension for each agent in the aggregation representation | 8
observation embedding dimension before concatenated with $z$ | 32
whether to predict the residuals of the next state | True
prediction length K in latent model | 2
hidden dimension for hidden states in latent model | 64

Algorithm 1 Training Framework

1: Initialize replay buffer $\mathcal{D}$.

2: Initialize information aggregation encoder with random parameters $\theta$
and state prediction decoder with random parameters $\eta$.

3: Initialize $Q$ network with random parameters $\phi$ and latent model with
random parameters $\psi$.

4: Initialize parameters of target encoder $\theta^{-}=\theta$, and target $Q$
network $\phi^{-}=\phi$.

5: for $\mbox{episode}=1$ to $M$ do

6: Roll out one trajectory $\bm{\tau}$ with $\epsilon$-greedy policy in the
environment.

7: Store the trajectory $\bm{\tau}$ in $\mathcal{D}$.

8: if $|\mathcal{D}|$ is larger than batch size $m$ then

9: Sample a minibatch $\mathcal{B}$ of $m$ trajectories from $\mathcal{D}$.

10: Compute the encoder-decoder loss:
$\mathcal{L}_{ae}(\theta,\eta)=\sum_{\rm
traj\in\mathcal{B}}\sum_{t=1}^{T}\|g_{\eta}(z^{t})-s^{t}\|_{2}^{2},~{}z^{t}=f_{\theta}(\bm{o}^{t}).$

11: Compute the latent model loss:
$\displaystyle\mathcal{L}_{m}(\theta,\psi)=\sum_{\rm
traj\in\mathcal{B}}\sum_{t=1}^{T-K}\sum_{k=1}^{K}\|\hat{z}^{t+k}-z^{t+k}\|_{2}^{2},~{}$
$\displaystyle\hat{z}^{k+1}=h_{\psi}(z^{t},\bm{a}^{t}),~{}$
$\displaystyle\hat{z}^{t+k}=h_{\psi}(\hat{z}^{t+k-1},\bm{a}^{t+k-1}),k=2,\dots,K,~{}$
$\displaystyle z^{t+k}=f_{\theta^{-}}(\bm{o}^{t+k}),k=0,\dots,K,$

12: Compute the reinforcement learning loss:
$\displaystyle\mathcal{L}_{rl}(\theta,\phi)=\sum_{\rm
traj\in\mathcal{B}}\sum_{t=1}^{T-1}\Big{(}r+$
$\displaystyle\gamma\max_{\bm{a}^{\bm{\prime}}}Q_{\rm
tot}(\bm{\tau}^{t+1},\bm{a}^{\bm{\prime}};\theta^{-},\phi^{-})-Q_{\rm
tot}(\bm{\tau}^{t},\bm{a}^{t};\theta,\phi)\Big{)}^{2}.$

13: Update $\theta,\eta,\phi,\psi$ by minimizing
$\mathcal{L}_{rl}(\theta,\phi)+\lambda_{1}\mathcal{L}_{ae}(\theta,\eta)+\lambda_{2}\mathcal{L}_{m}(\theta,\psi)$.

14: end if

15: Update target network parameters $\theta^{-},\phi^{-}$ with the EMA
method.

16: end for

###  B.2 The Overall Flow for Training and Testing

Firstly, to offer a direct impression of our work, we first talk about how
agents behave differently during decentralized execution vs. centralized
training in our method (MASIA). The main differences are that we would use
state information to help compute the auto-encoder loss and estimate the
$Q_{\rm{tot}}$ (if there is a mixing network in the $Q$-network, e.g.
MASIA+QMIX) during training phase. Besides, multi-step prediction loss is also
only computed and optimized in the training phase. During decentralized
execution, the state decoder, the latent model, and the possible mixing
network are all thrown away. We remain the IAE, the focusing network, and
individual Q-networks to ensure the decentralized execution process. The
representation loss terms we designed are aimed at training the encoder
network well.

To illustrate the process of training, the overall training flow of MASIA is
shown in Algorithm 1. Lines 5$\sim$16 express the whole training process,
where we apply an off-policy learning algorithm and iteratively update the
parameters of the model. Specifically, we compute encoder-decoder loss, latent
model loss and temporal difference loss in Lines 10, 11 and 12, respectively,
and do parameter updating together in Line 13. Besides, the execution flow of
MASIA is shown in Algorithm 2. In the execution phase, the agents first
broadcast their observations to each other, and then each agent calculates its
own action $a_{i}^{t}$ by applying the focusing network and individual $Q$
network learned during training, which is described in Lines 4 and 5.

Algorithm 2 Execution Flow

0: information aggregation encoder with parameters $\theta$, individual $Q$
network with parameters $\phi_{i}$ for each $Q_{i}$, focusing networks with
parameter $\omega_{i}$ for each agent and agent number $n$.

1: for $\mbox{step}=0$ to episode_limit do

2: Each agent broadcasts its observational information $o_{i}^{t}$ at timestep
$t$, and then each agent feed collected observations
$\bm{o}^{t}=\\{o_{i}^{t}\\}_{n}$ into the information aggregation network,
obtaining $z^{t}=f_{\theta}(\bm{o}^{t})$.

3: for $i=1$ to $n$ do

4: Agent $i$ calculates $w_{i}^{t}=F_{w_{i}}(o_{i}^{t})$ by using the focusing
network, and obtains extracted information $\bar{z}_{i}^{t}=w_{i}^{t}\cdot
z^{t}$.

5: The $o_{i}^{t}$ and $\bar{z}_{i}^{t}$ are fed into the individual $Q$
network to calculate $Q_{i}(\tau_{i}^{t},\cdot)$, and agent $i$ gets action
$a_{i}^{t}=\mathop{\arg\max}_{a}Q_{i}(\tau_{i}^{t},a)$.

6: end for

7: The agent system interacts with the environment by executing actions
$\\{a_{i}^{t}\\}_{n}$.

8: end for

###  B.3 Experimental Details

Our experiments were performed on a desktop machine with 4 NVIDIA GTX 3090
GPUs. For all the performance curves in our paper, we pause training every $M$
timesteps and evaluate for $N$ episodes with decentralized greedy action
selection. The $(M,N)$ in Hallway [67], Level-Based Foraging [48], Traffic
Junction [6], and SMAC [67] are $(10\rm{K},100),(50\rm{K},100),(10\rm{K},40)$,
and $(50\rm{K},100)$, respectively. We evaluate the test win rate, the
percentage of episodes in which the agents win the game within the time limit
in $N$ testing episodes for all tasks.

##  Appendix C More Results about offline learning

###  C.1 Details about Offline Dataset

Due to the huge cost and limitations of interacting with the environment in
the real world, offline learning has been seen as a promising direction for
getting reinforcement learning off the ground. Besides, since many real-world
problems can be modeled as multi-agent systems, especially multi-agent
communication systems, advances in research concerning offline communication
learning are urgently needed. However, in the field of multi-agent
communicative reinforcement learning, there is still no good evaluation
criteria for offline learning, which largely hinders the development of this
field. Considering this situation, we construct an offline dataset, which
involves multiple multi-agent communication environments and different dataset
settings, to support the research of offline multi-agent communication
learning.

In specific, our dataset involves four environments, each including two
scenarios, for a total of eight scenarios. The included environments are
consistent with the task environments in the online experiments, which are
shown in Figure 4. As we analyzed in the experimental section (Section 4),
these environments represent communication scenarios with different
characteristics. To offer more insights of these four environments in the
offline setting, we additionally provide a description of these environments
in Table 4 with reference to several dataset properties proposed in D4RL [17].
Among these properties, all environments are narrow, not realistic and of
partial observability. The latter two are because the data are all collected
in the simulator without real human data and the agents’ sight ranges are
limited due to the communication setting. The data for Hallway environment is
undirected and of sparse rewards because the agents only obtain rewards when
they reach target together and the data can be stitched in this environment.
Besides, it is the only environment that the expert policy can generate
optimal trajectories. In addition, the data for LBF environment is multi-task
because the locations of the target fruits are randomly generated at the
beginning of each episode, thus it can be seem as including data of multiple
navigation tasks.

Table 7: Properties of different offline datasets. Environment Scenarios | Dataset | Behavior Policy Test Success% | # Samples | Return Distribution
---|---|---|---|---
Hallway: 4x6x10 | Expert |  100.00 $\pm$ 0.00 |  | 1.00
Noisy |  100.00 $\pm$ 0.00 |  | 0.40
Replay (Poor) |  32.90 $\pm$ 47.01 | 12000 | 0.33
Replay (Medium) |  62.50 $\pm$ 48.41 |  | 0.62
Replay (Good) |  87.50 $\pm$ 33.07 |  | 0.88
Hallway: 3x5-4x6x10 | Expert |  99.80 $\pm$ 0.25 |  | 2.00
Noisy |  99.80 $\pm$ 0.25 |  | 1.01
Replay (Poor) |  43.90 $\pm$ 33.31 | 12000 | 0.88
Replay (Medium) |  64.20 $\pm$ 30.41 |  | 1.28
Replay (Good) |  82.90 $\pm$ 27.74 |  | 1.66
LBF: 11x11-6p-4f-s1 | Expert |  90.20 $\pm$ 2.59 |  | 0.91
Noisy |  90.20 $\pm$ 2.59 |  | 0.83
Replay (Poor) |  46.60 $\pm$ 14.93 | 12000 | 0.48
Replay (Medium) |  67.40 $\pm$ 13.36 |  | 0.67
Replay (Good) |  83.60 $\pm$ 22.71 |  | 0.84
20x20-10p-6f-s1 | Expert |  53.80 $\pm$ 2.34 |  | 0.54
Noisy |  53.80 $\pm$ 2.34 |  | 0.45
Replay (Poor) |  38.70 $\pm$ 14.55 | 12000 | 0.41
Replay (Medium) |  45.50 $\pm$ 15.75 |  | 0.65
Replay (Good) |  49.10 $\pm$ 22.67 |  | 0.78
SMAC: 1o2r_vs_4r | Expert |  81.50 $\pm$ 5.72 |  | 18.25
Noisy |  81.50 $\pm$ 5.72 |  | 14.71
Replay (Poor) |  45.70 $\pm$ 19.25 | 12000 | 9.14
Replay (Medium) |  63.80 $\pm$ 29.03 |  | 12.78
Replay (Good) |  76.60 $\pm$ 27.69 |  | 16.53
SMAC: 1o10b_vs_1r | Expert |  82.80 $\pm$ 2.66 |  | 18.52
Noisy |  82.80 $\pm$ 2.66 |  | 12.27
Replay (Poor) |  48.20 $\pm$ 31.48 | 12000 | 9.62
Replay (Medium) |  67.30 $\pm$ 34.65 |  | 14.63
Replay (Good) |  78.70 $\pm$ 30.43 |  | 16.80
TJ: easy | Expert |  98.70 $\pm$ 1.25 |  | -3.28
Noisy |  98.70 $\pm$ 1.25 |  | -13.48
Replay (Poor) |  45.60 $\pm$ 17.28 | 10000 | -25.27
Replay (Medium) |  57.80 $\pm$ 16.57 |  | -20.61
Replay (Good) |  73.90 $\pm$ 16.98 |  | -9.58
TJ: medium | Expert |  94.30 $\pm$ 1.08 |  | -7.08
Noisy |  94.30 $\pm$ 1.08 |  | -15.53
Replay (Poor) |  42.70 $\pm$ 14.86 | 36000 | -27.77
Replay (Medium) |  52.40 $\pm$ 21.79 |  | -23.32
Replay (Good) |  67.10 $\pm$ 18.16 |  | -17.17

Moreover, considering that the real data may be of various qualities, we
include multiple dataset settings to cover different data distributions. In
Section 4.2, we introduce all there different dataset settings and how we
construct them. Actually, due to the differences in data collection
approaches, these datasets somewhat differ in the quality of data. For
example, the data in Expert Dataset are typically trajectories of high
returns. More detailed information about these datasets is listed in Table 7,
where we offer the return distributions of different datasets. Furthermore, to
offer a more intuitive presentation of the data distribution, we plot more
information in Figures 14-28. Take the task scenario of LBF: 11x11-6p-4f-s1 as
an example, the return values of the most trajectories in Expert, Noisy, and
Replay (Good) Datasets are concentrated on $1.0$ which is the optimal return
value in this task. While no trajectory in Replay (Poor) and Replay (Medium)
Datasets reaches a return value of $1.0$ and most trajectories are
concentrated at much lower return values.

###  C.2 More Results of Offline Learning Ablations

(a) Replay (Medium)

(b) Expert

Figure 12: Ablation experiments for offline learning on the task of TJ:
medium.

In Sec. 4.2.2, we conduct offline ablation studies on Replay (Medium) and
Replay (Poor) datasets of Hallway: 3x5-4x6x10 to verify the effectiveness of
two unsupervised representation losses and the practice of offline pre-
training. Here, we further provide the ablation results on the task of TJ:
medium. As we can see from Figure 12, the conclusion on TJ: medium is similar
to that we obtained in the maintext, MASIA equipped with both unsupervised
representation losses and offline pre-training achieves faster learning speed
and better convergence performance over other ablations.

Figure 14: Hallway: 4x6x10 offline dataset distribution.

Figure 16: Hallway: 3x5-4x6x10 offline dataset distribution.

Figure 18: LBF: 11x11-6p-4f-s1 offline dataset distribution.

Figure 20: LBF: 20x20-10p-6f-s1 offline dataset distribution.

Figure 22: SMAC: 1o2r_vs_4r offline dataset distribution.

Figure 24: SMAC: 1o10b_vs_1r offline dataset distribution.

Figure 26: TJ: easy offline dataset distribution.

Figure 28: TJ: medium offline dataset distribution.