# Offline Decentralized Multi-Agent
Reinforcement Learning

Jiechuan Jiang  Zongqing Lu Corresponding Author. Email:
<EMAIL_ADDRESS>Peking University

###### Abstract

In many real-world multi-agent cooperative tasks, due to high cost and risk,
agents cannot continuously interact with the environment and collect
experiences during learning, but have to learn from offline datasets. However,
the transition dynamics in the dataset of each agent can be much different
from the ones induced by the learned policies of other agents in execution,
creating large errors in value estimates. Consequently, agents learn
uncoordinated low-performing policies. In this paper, we propose a framework
for offline decentralized multi-agent reinforcement learning, which exploits
value deviation and transition normalization to deliberately modify the
transition probabilities. Value deviation optimistically increases the
transition probabilities of high-value next states, and transition
normalization normalizes the transition probabilities of next states. They
together enable agents to learn high-performing and coordinated policies.
Theoretically, we prove the convergence of Q-learning under the altered non-
stationary transition dynamics. Empirically, we show that the framework can be
easily built on many existing offline reinforcement learning algorithms and
achieve substantial improvement in a variety of multi-agent tasks.

##  1 Introduction

Reinforcement learning (RL) has shown great potential in domains, including
recommendation systems [13], games [28], and robotics [7]. When applying RL to
real-world applications, the agent needs to continuously interact with the
environment and collect the experiences for learning, which however can be
costly, risky, and time-consuming. One way to address this is offline RL,
where the agent learns the policy, without interacting with the environment,
from a fixed dataset of experiences collected by any behavior policy. The main
challenge of offline RL is the extrapolation error, an error incurred by the
mismatch between the experience distributions of the learned policy and
dataset [5]. Many offline RL methods [5, 10, 31, 11] have been proposed to
address the value overestimate caused by the mismatch between action
distributions of the learned policy and behavior policy. However, much less
attention has been paid to the mismatch between the transition dynamics in the
dataset and the real ones in execution. The main reason is that the mismatch
of transition dynamics is negligible given a large dataset, if the environment
is stationary. Nevertheless, in many real-world scenarios, there are other
learning agents in the environment, e.g., autonomous driving. That means the
policies of other agents during data collection can be significantly different
from their policies in execution, which creates the mismatch of transition
dynamics and hence undermines the learned policy.

More specifically, these scenarios can be formulated as the offline
decentralized multi-agent setting, where agents cooperate on the task but each
learns the policy from its own fixed dataset. The dataset of each agent is
independently collected by any behavior policies arbitrarily (i.e., we do not
make any assumptions on data collection) and contains its own action instead
of the joint action of all agents. This setting resembles many industrial
applications where agents belong to different companies and the actions of
other agents are not accessible, e.g., autonomous vehicles or robots.
Apparently, this setting does not fit the paradigm of offline centralized
training and decentralized execution (CTDE) in multi-agent reinforcement
learning (MARL)111Even if the datasets of all agents can be accessed in a
centralized way, a full dataset that contains the joint actions cannot be
constructed, because the datasets are fully independent without any common
information. [33], and each agent has to learn a cooperative policy in an
_offline and fully decentralized_ way. In such decentralized multi-agent
settings, from the perspective of an individual agent, other agents are a part
of the environment, thus the transition dynamics experienced by each agent
depend on the policies of other agents and change as other agents update their
policies [2]. As the learned policies of other agents will be inconsistent
with their behavior policies, the transition dynamics induced by the learned
policies of other agents in execution will be different from the transition
dynamics in the dataset, causing a large mismatch between the transition
dynamics. This mismatch can lead to a low-performing policy for each agent.
Furthermore, as agents learn in a decentralized way on the datasets with
different distributions of experiences collected by various behavior policies,
the estimated values of the same state can be much different between agents,
which causes the learned policies cannot well coordinate with each other.

In this paper, we propose a framework for offline fully decentralized multi-
agent reinforcement learning, where, to overcome the mismatch between
transition dynamics and miscoordination, value deviation and transition
normalization are introduced to deliberately modify the transition
probabilities in the dataset. During data collection, if one agent takes a
‘good’ action while other agents take ‘bad’ actions at a state, the transition
probabilities of low-value next states will be high. Thus, the Q-value of the
‘good’ action will be underestimated by the agent. As other agents are also
learning, their learned policies are expected to become better than their
behavior policies. Therefore, for each agent, the transition probabilities of
high-value next states in execution will be higher than those in the dataset.
So, we let each agent be _optimistic_ towards other agents and multiply the
transition probabilities by the deviation of the value of next state from the
expected value over all next states, to make the estimated transition
probabilities close to the transition probabilities induced by the learned
policies of other agents. To address the miscoordination caused by diverse
value estimates of agents, we normalize the transition probabilities in the
dataset to be uniform. Transition normalization helps to build the consensus
about value estimate among agents and hence promotes coordination. By
combining value deviation and transition normalization, agents are able to
learn high-performing and coordinated policies in an offline and fully
decentralized way.

Value deviation and transition normalization make the transition dynamics non-
stationary during learning. However, we mathematically prove the convergence
of Q-learning under such purposely controlled non-stationary transition
dynamics. Moreover, by derivation, we show that value deviation and transition
normalization can take effect merely as the weights of the objective function,
thus our framework can be easily built on many existing offline single-agent
RL algorithms that address the overestimation incurred by out-of-distribution
actions. Empirically, we provide an example instantiation and the thorough
analysis of our framework on BCQ [5], termed MABCQ, and also test the variants
on CQL [11] and TD3+BC [4], termed MACQL and MATD3+BC respectively.
Experimental results show that our method substantially outperforms baselines
in a variety of multi-agent tasks, including multi-agent mujoco [1], SMAC [23]
and MPE [15]. To the best of our knowledge, this is the first work for offline
and decentralized multi-agent reinforcement learning.

##  2 Preliminaries

We consider $N$ agents in multi-agent MDP [17]
$M_{\mathrm{env}}=<\mathcal{S},\mathcal{A},R,P_{\mathrm{env}},\gamma>$ with
the state space $\mathcal{S}$ and the joint action space $\mathcal{A}$. At
each timestep, each agent $i$ gets state222Global state is only for the
convenience of theoretical analysis [9] the agents could learn based on
partial observation in practice. $s$, and performs an individual action
$a_{i}$, and the environment transitions to the next state $s^{\prime}$ by
taking the joint action $\bm{a}$ with the transition probability
$P_{\mathrm{env}}\left(s^{\prime}|s,\bm{a}\right)$. All agents get a shared
reward $r=R\left(s\right)$, which is simplified to just depending on state
[24]333This simplification is only for the convenience of theoretical
analysis, but not necessary in practice.. All the agents learn to maximize the
expected return $\mathbb{E}\sum_{t=0}^{\infty}\gamma^{t}r_{t}$, where $\gamma$
is a discount factor. However, in the fully decentralized learning,
$M_{\mathrm{env}}$ is partially observable to each agent since each agent only
observes its own action $a_{i}$ instead of the joint action $\bm{a}$. During
execution, from the perspective of each agent $i$, there is an experienced MDP
$M_{\mathcal{E}_{i}}=<\mathcal{S},\mathcal{A}_{i},R,P_{\mathcal{E}_{i}},\gamma>$
with the individual action space $\mathcal{A}_{i}$ and the online transition
probability

$P_{\mathcal{E}_{i}}\left(s^{\prime}|s,a_{i}\right)={\sum}_{\bm{a}_{-i}}P_{\mathrm{env}}\left(s^{\prime}|s,a_{i},\bm{a}_{-i}\right)\bm{\pi}_{\mathcal{E}_{-i}}(\bm{a}_{-i}|s),$

where $a_{-i}$ and $\bm{\pi}_{\mathcal{E}_{-i}}$ respectively denote the joint
action and the learned joint policy of all agents except agent $i$. However,
if the agent cannot interact with other agents in the environment,
$P_{\mathcal{E}_{i}}$ is unknown.

In offline decentralized settings, each agent $i$ only has access to a fixed
offline dataset $\mathcal{B}_{i}$, which is pre-collected by behavior policies
$\bm{\pi}_{\mathcal{B}}$ and contains the tuples $(s,a_{i},r,s^{\prime})$. As
defined in BCQ [5], the visible MDP
$M_{\mathcal{B}_{i}}=<\mathcal{S},\mathcal{A}_{i},R,P_{\mathcal{B}_{i}},\gamma>$
is constructed on $\mathcal{B}_{i}$, which has the offline transition
probability

$P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)={\sum}_{\bm{a}_{-i}}P_{\mathrm{env}}\left(s^{\prime}|s,a_{i},\bm{a}_{-i}\right)\bm{\pi}_{\mathcal{B}_{-i}}(\bm{a}_{-i}|s),$

As the learned policies of other agents $\bm{\pi}_{\mathcal{E}_{-i}}$ may
greatly deviate from their behavior policies $\bm{\pi}_{\mathcal{B}_{-i}}$,
$P_{\mathcal{B}_{i}}$ can be largely biased from $P_{\mathcal{E}_{i}}$. We
define the discrepancy between $P_{\mathcal{B}_{i}}$ and $P_{\mathcal{E}_{i}}$
as transition bias. The large extrapolation errors caused by transition bias,
and the differences in value estimates between agents lead to uncoordinated
low-performing policies.

Table 1: The matrix game. |  | Agent $2$
---|---|---
|  | $a_{1}$ ($0.4$) | $a_{2}$ ($0.6$)
| $a_{1}$ ($0.8$) | 1 | 5
Agent $1$ | $a_{2}$ ($0.2$) | 6 | 1
Table 2: Transition probabilities and expected returns calculated in the
dataset.

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0.60546875,0,0}a_{1}})=0.4$ |
| $a_{1}$ | $p(5|{\color[rgb]{0.60546875,0,0}a_{1}})=0.6$ | 3.4
|  | $p(6|{\color[rgb]{0.60546875,0,0}a_{2}})=0.4$ |
Agent $1$ | $a_{2}$ | $p(1|{\color[rgb]{0.60546875,0,0}a_{2}})=0.6$ | 3.0

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0,0,0.609375}a_{1}})=0.8$ |
| $a_{1}$ | $p(6|{\color[rgb]{0,0,0.609375}a_{1}})=0.2$ | 2.0
|  | $p(5|{\color[rgb]{0,0,0.609375}a_{2}})=0.8$ |
Agent $2$ | $a_{2}$ | $p(1|{\color[rgb]{0,0,0.609375}a_{2}})=0.2$ | 4.2

To intuitively illustrate the issue, we introduce a two-player matrix game as
depicted in Table 1, where the action distributions of the behavior policies
of the two agents are $[0.8,0.2]$ and $[0.4,0.6]$, respectively. Table 2 shows
the transition probabilities and expected returns calculated independently by
the agents from the datasets collected by the behavior policies. However, as
the behavior policies are poor, when agent 1 chooses the optimal action
$a_{2}$, agent 2 has a higher probability to select the suboptimal action
$a_{2}$, which leads to low transition probabilities of true high-value next
states ($p(6|{\color[rgb]{0.60546875,0,0}a_{2}})=0.4$ vs.
$p(1|{\color[rgb]{0.60546875,0,0}a_{2}})=0.6$). So the agents underestimate
the optimal actions and converge to the suboptimal policies
$({\color[rgb]{0.60546875,0,0}a_{1}},{\color[rgb]{0,0,0.609375}a_{2}})$,
rather than the optimal policy
$({\color[rgb]{0.60546875,0,0}a_{2}},{\color[rgb]{0,0,0.609375}a_{1}})$.

##  3 Offline Decentralized MARL Framework

In fully decentralized learning, as the policies of other agents are not
accessible, it is hard for an agent to learn a policy that can coordinate well
with other agents, merely from the dataset that only contains its own action.
To tackle this challenging problem, we propose a framework, which newly
introduces value deviation and transition normalization to address transition
bias and miscoordination, and leverages the offline single-agent RL algorithm
to avoid out-of-distribution actions. The convergence under the non-stationary
transition dynamics is theoretically guaranteed, and an example instantiation
of the framework is provided.

###  3.1 Value Deviation

If the behavior policies of some agents are low-performing during data
collection, they usually take ‘bad’ actions to cooperate with the ‘good’
actions of other agents, which leads to high transition probabilities of low-
value next states. When agent $i$ performs Q-learning with the dataset
$\mathcal{B}_{i}$, the Bellman operator $\mathcal{T}$ is approximated by the
transition probability $P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)$ to
estimate the expectation over $s^{\prime}$:

$\mathcal{T}Q_{i}(s,a_{i})=\mathbb{E}_{s^{\prime}\sim
P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)}\left[r+\gamma\max_{\hat{a}_{i}}Q_{i}\left(s^{\prime},\hat{a}_{i}\right)\right].$

If $P_{\mathcal{B}_{i}}$ of a high-value $s^{\prime}$ is lower than
$P_{\mathcal{E}_{i}}$, the Q-value of the state-action pair $(s,a_{i})$ is
underestimated, which will cause large extrapolation error.

However, as the policies of other agents are also updating towards maximizing
the Q-values, $P_{\mathcal{E}_{i}}$ of high-value next states will grow higher
than $P_{\mathcal{B}_{i}}$. Thus, we let each agent be optimistic towards
other agents and modify $P_{\mathcal{B}_{i}}$ as

$P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)\cdot(\underset{\text{value
deviation}}{\underbrace{1+\frac{V_{i}^{*}(s^{\prime})-\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})}{|\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})|}}})\cdot\frac{1}{z_{i}^{vd}},$

where the state value $V_{i}^{*}(s)=\max_{a_{i}}Q_{i}(s,a_{i})$,
$1+\nicefrac{{V_{i}^{*}(s^{\prime})-\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})}}{{|\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})|}}$
is the deviation of the value of next state from the expected value over all
next states, which increases the transition probabilities of high-value next
states and decreases those of low-value next states, and
$z_{i}^{vd}=\sum_{s^{\prime}}P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)*(1+\nicefrac{{V_{i}^{*}(s^{\prime})-\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})}}{{|\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})|}})$
is a normalization term to make sure the sum of the transition probabilities
is one. Value deviation modifies the transition probabilities to be close to
$P_{\mathcal{E}_{i}}$ and hence reduces the transition bias. The optimism
towards other agents helps the agent discover potential good actions which are
hidden by the poor behavior policies of other agents.

###  3.2 Transition Normalization

As $\mathcal{B}_{i}$ of each agent is individually collected by different
behavior policies, the diverse combinations of behavior policies of all agents
lead to the value of the same state $s$ being overestimated by some agents,
while being underestimated by others. Since the agents are trained to reach
high-value states, the large disagreement on state values will cause
miscoordination of the learned policies. To overcome the problem, we normalize
$P_{\mathcal{B}_{i}}$ to be uniform over next states,

$P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)\cdot\underset{\text{transition
normalization}}{\underbrace{\frac{1}{P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)}}}\cdot\frac{1}{z_{i}^{tn}}$

where $z_{i}^{tn}$ is a normalization term that is the number of different
$s^{\prime}$ given $(s,a_{i})$ in $\mathcal{B}_{i}$. Transition normalization
enforces that each agent has the same $P_{\mathcal{B}_{i}}$ when it acts the
learned action $a_{i}^{*}$ on the same state $s$, and we have the following
proposition.

######  Proposition 1.

In episodic environments, if each agent $i$ performs Q-learning on
$\mathcal{B}_{i}$, all agents will converge to the same $V^{*}$ if they have
the same transition probability on any state where each agent $i$ acts the
learned action $a_{i}^{*}$.

The proof is provided in Appendix555Appendix is available at
https://arxiv.org/abs/2108.01832.. This proposition implies that transition
normalization can enable agents to have the same state value estimate.
However, to satisfy
$P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)=P_{\mathcal{B}_{2}}\left(s^{\prime}|s,a_{2}^{*}\right)=\ldots=P_{\mathcal{B}_{N}}\left(s^{\prime}|s,a_{N}^{*}\right)$
for all $s^{\prime}\in\mathcal{S}$ in the datasets, the agents should have the
same set of $s^{\prime}$ at $(s,a^{*})$, which is a strong assumption. In
practice, although the assumption is not strictly satisfied, transition
normalization can still normalize the transition probabilities, encouraging
the estimated state value $V^{*}$ to be close to each other.

###  3.3 Optimization Objective

Combining value deviation
$1+\nicefrac{{(V_{i}^{*}(s^{\prime})-\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime}))}}{{|\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})|}}$,
denoted as $\lambda_{{vd}_{i}}$, and transition normalization
$\nicefrac{{1}}{{P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)}}$,
denoted as $\lambda_{{tn}_{i}}$, we modify $P_{\mathcal{B}_{i}}$ as

$\hat{P}_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)=P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)\cdot\frac{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}}{z_{i}},$

where
$z_{i}=\sum_{s^{\prime}}(1+\nicefrac{{(V_{i}^{*}(s^{\prime})-\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime}))}}{{|\mathbb{E}_{\hat{s}^{\prime}}V_{i}^{*}(\hat{s}^{\prime})|}})$
is the normalization term. Indeed, $\hat{P}_{\mathcal{B}_{i}}$ makes offline
learning on $\mathcal{B}_{i}$ similar to online decentralized MARL. In the
initial stage, $\lambda_{{vd}_{i}}$ is close to $1$ since $Q_{i}(s,a_{i})$ is
not learned yet, and the transition probabilities are uniform, resembling the
start stage in online learning where all agents are acting randomly. During
training, the transition probabilities of high-value states gradually grow by
value deviation, which is an analogy of other agents improving their policies
in online learning. Therefore, $\hat{P}_{\mathcal{B}_{i}}$ encourages the
agents to learn high-performing policies and improves coordination.

Although $\hat{P}_{\mathcal{B}_{i}}$ is non-stationary (i.e.,
$\lambda_{{vd}_{i}}$ changes along with the updates of Q-value), we have the
following theorem that guarantees the convergence of Bellman operator
$\mathcal{T}$ under $\hat{P}_{\mathcal{B}_{i}}$,

$\mathcal{T}Q_{i}(s,a_{i})=\mathbb{E}_{s^{\prime}\sim\hat{P}_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)}\left[r+\gamma\max_{\hat{a}_{i}}Q_{i}\left(s^{\prime},\hat{a}_{i}\right)\right].$

######  Theorem 1.

Under the non-stationary transition probability $\hat{P}_{\mathcal{B}_{i}}$,
the Bellman operator $\mathcal{T}$ is a contraction and converges to a unique
fixed point when $\gamma<\nicefrac{{r_{\min}}}{{2r_{\max}-r_{\min}}}$, if the
reward is bounded by the positive region $[r_{\min},r_{\max}]$.

The proof is provided in Appendix. As any positive affine transformation of
the reward function does not change the optimal policy in the fixed-horizon
environments [36], Theorem 1 holds in general, and we can rescale the reward
to make $r_{\min}$ arbitrarily close to $r_{\max}$ so as to make the upper
bound of $\gamma$ close to $1$.

In deep reinforcement learning, directly modifying the transition probability
is infeasible. However, we can instead modify the sampling probability to
achieve the same effect. The optimization objective of decentralized deep
Q-learning
$\mathbb{E}_{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}|Q_{i}(s,a_{i})-y_{i}|^{2}$
is calculated by sampling the batch from $\mathcal{B}_{i}$ according to the
sampling probability $p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})$. By factorizing
$p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})$, we have

$\underset{{\color[rgb]{0.60546875,0,0}\text{sampling
probability}}}{\underbrace{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}}=p_{\mathcal{B}_{i}}(s,a_{i})\cdot\underset{{\color[rgb]{0,0,0.609375}\text{transition
probability}}}{\underbrace{P_{\mathcal{B}_{i}}(s^{\prime}|s,a_{i})}}.$

Therefore, we can modify the transition probability as
$\frac{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}}{z_{i}}P_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)$
and scale $p_{\mathcal{B}_{i}}(s,a_{i})$ with $z_{i}$. Then, the sampling
probability can be re-written as

$\underset{{\color[rgb]{0.60546875,0,0}\text{modified sampling
probability}}}{\underbrace{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}}=z_{i}p_{\mathcal{B}_{i}}(s,a_{i})\cdot\underset{{\color[rgb]{0,0,0.609375}\text{modified
transition
probability}}}{\underbrace{\frac{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}}{z_{i}}P_{\mathcal{B}_{i}}(s^{\prime}|s,a_{i})}}.$

Since $z_{i}$ is independent of $s^{\prime}$, it can be regarded as a scale
factor on $p_{\mathcal{B}_{i}}(s,a_{i})$, which will not change the expected
target value $y_{i}$. Thus, sampling batches according to the modified
sampling probability can achieve the same effect as modifying the transition
probability. Using importance sampling, the modified optimization objective is

$\begin{split}&\mathbb{E}_{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}|Q_{i}(s,a_{i})-y_{i}|^{2}\\\
&=\mathbb{E}_{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}\frac{\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}|Q_{i}(s,a_{i})-y_{i}|^{2}\\\
&=\mathbb{E}_{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}|Q_{i}(s,a_{i})-y_{i}|^{2}.\end{split}$

We can see that $\lambda_{{tn}_{i}}$ and $\lambda_{{vd}_{i}}$ simply take
effect as the weights of the objective function, which makes them easily
integrated with existing offline RL methods.

Algorithm 1 MABCQ

1: for $i\in N$ do

2: Initialize the conditional VAEs:
$G^{1}_{i}=\\{E^{1}_{i}\left(\mu^{1},\sigma^{1}|s,a\right),D^{1}_{i}\left(a|s,z^{1}\right)\\}$,
$G^{2}_{i}=\\{E^{2}_{i}\left(\mu^{2},\sigma^{2}|s,a,s^{\prime}\right),D^{2}_{i}\left(a|s,s^{\prime},z^{2}\right)\\}$.

3: Initialize Q-network $Q_{i}$, perturbation network $\xi_{i}$, and their
target networks $\hat{Q}_{i}$ and $\hat{\xi}_{i}$.

4: Fit the VAEs $G^{1}_{i}$ and $G^{2}_{i}$ using $\mathcal{B}_{i}$.

5: for $t=1,\ldots,max\\_update$ do

6: Sample a mini-batch from $\mathcal{B}_{i}$.

7: Update $Q_{i}$ by minimizing (1).

8: Update $\xi_{i}$ by maximizing (2).

9: Update the target networks $\hat{Q}_{i}$ and $\hat{\xi}_{i}$.

10: end for

11: end for

###  3.4 An Example Instantiation

Our framework can be practically instantiated on many offline single-agent RL
algorithms that address the overestimation incurred by out-of-distribution
actions. Here, we give the instantiation of the framework on BCQ [5], termed
MABCQ. To make MABCQ adapt to high-dimensional continuous spaces, for each
agent $i$, we train a Q-network $Q_{i}$, a perturbation network $\xi_{i}$, and
a conditional VAE
$G^{1}_{i}=\\{E^{1}_{i}\left(\mu^{1},\sigma^{1}|s,a\right),D^{1}_{i}\left(a|s,z^{1}\sim\left(\mu^{1},\sigma^{1}\right)\right)\\}$.
In execution, each agent $i$ generates $n$ actions by $G_{i}^{1}$, adds small
perturbations $\in[-\Phi,\Phi]$ on the actions using $\xi_{i}$, and then
selects the action with the highest value in $Q_{i}$. The policy can be
written as

$\begin{split}\pi_{i}(s)=\underset{a_{i}^{j}+\xi_{i}\left(s,a_{i}^{j}\right)}{\operatorname{argmax}}Q_{i}\left(s,a_{i}^{j}+\xi\left(s,a_{i}^{j}\right)\right),\\\
\text{where }\left\\{a_{i}^{j}\sim
G_{i}^{1}(s)\right\\}_{j=1}^{n}.\end{split}$

$Q_{i}$ is updated by minimizing

$\begin{split}\mathbb{E}_{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}|Q_{i}(s,a_{i})-y_{i}|^{2},\\\
\text{where
}y_{i}=r+\gamma\hat{Q}_{i}(s^{\prime},\hat{\pi}_{i}(s^{\prime})).\end{split}$
(1)

$y_{i}$ is calculated by the target networks $\hat{Q}_{i}$ and
$\hat{\xi}_{i}$, where $\hat{\pi}_{i}$ is correspondingly the policy induced
by $\hat{Q}_{i}$ and $\hat{\xi}_{i}$. $\xi_{i}$ is updated by maximizing

$\mathbb{E}_{p_{\mathcal{B}_{i}}(s,a_{i},s^{\prime})}\lambda_{{tn}_{i}}\lambda_{{vd}_{i}}Q_{i}\left(s,a_{i}+\xi_{i}\left(s,a_{i}\right)\right).$
(2)

To estimate $\lambda_{{vd}_{i}}$, we need
$V_{i}^{*}(s^{\prime})=\hat{Q}_{i}(s^{\prime},\hat{\pi}_{i}(s^{\prime}))$ and
$\mathbb{E}_{s^{\prime}}[V_{i}^{*}\left(s^{\prime}\right)]=\frac{1}{\gamma}(\hat{Q}_{i}(s,a_{i})-r)$,
which can be estimated from the sample without actually going through all
$s^{\prime}$. We estimate $\lambda_{{vd}_{i}}$ using the target networks to
stabilize $\lambda_{{vd}_{i}}$ along with the updates of $Q_{i}$ and
$\xi_{i}$. To avoid extreme values, we clip $\lambda_{{vd}_{i}}$ to the region
$[1-\epsilon,1+\epsilon]$, where $\epsilon$ is the optimism level.

To estimate $\lambda_{{tn}_{i}}$, we train a VAE
$G^{2}_{i}=\\{E^{2}_{i}\left(\mu^{2},\sigma^{2}|s,a,s^{\prime}\right),D^{2}_{i}\left(a|s,s^{\prime},z^{2}\sim\left(\mu^{2},\sigma^{2}\right)\right)\\}$.
Since the latent variable of VAE follows the Gaussian distribution, we use the
mean as the encoding of the input and estimate the probability density
functions: $\rho_{i}(s,a)\approx\rho_{\mathcal{N}(0,1)}(\mu^{1}_{i})$ and
$\rho_{i}(s,a,s^{\prime})\approx\rho_{\mathcal{N}(0,1)}(\mu^{2}_{i})$, where
$\rho_{\mathcal{N}(0,1)}$ is the density of unit Gaussian distribution. The
conditional density is
$\rho_{i}(s^{\prime}|a,s)\approx\nicefrac{{\rho_{\mathcal{N}(0,1)}(\mu^{2}_{i})}}{{\rho_{\mathcal{N}(0,1)}(\mu^{1}_{i})}}$
and the transition probability is
$P_{\mathcal{B}_{i}}(s^{\prime}|s,a_{i})\approx\int_{s^{\prime}-\frac{1}{2}\delta_{\mathcal{S}}}^{s^{\prime}+\frac{1}{2}\delta_{\mathcal{S}}}\rho_{i}(s^{\prime}|s,a)\mathrm{d}s^{\prime}\approx\rho_{i}(s^{\prime}|s,a)\left\|\delta_{\mathcal{S}}\right\|$
when $\left\|\delta_{\mathcal{S}}\right\|$ is a small constant. Approximately,
we have

$\lambda_{{tn}_{i}}=\frac{\rho_{\mathcal{N}(0,1)}(\mu^{1}_{i})}{\rho_{\mathcal{N}(0,1)}(\mu^{2}_{i})},$

and the constant $\left\|\delta_{\mathcal{S}}\right\|$ is considered in
$z_{i}$. In practice, we find that $\lambda_{{tn}_{i}}$ falls into the region
$[0.2,1.4]$ for almost all samples. For completeness, we summarize the
training of MABCQ in Algorithm 1.

Table 3: Transition probabilities and expected returns calculated in the
dataset using only $\lambda_{{vd}}$.

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0.60546875,0,0}a_{1}})=0.12$ |
| $a_{1}$ | $p(5|{\color[rgb]{0.60546875,0,0}a_{1}})=0.88$ | 4.52
|  | $p(6|{\color[rgb]{0.60546875,0,0}a_{2}})=0.8$ |
Agent $1$ | $a_{2}$ | $p(1|{\color[rgb]{0.60546875,0,0}a_{2}})=0.2$ | 5

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0,0,0.609375}a_{1}})=0.4$ |
| $a_{1}$ | $p(6|{\color[rgb]{0,0,0.609375}a_{1}})=0.6$ | 4
|  | $p(5|{\color[rgb]{0,0,0.609375}a_{2}})=0.95$ |
Agent $2$ | $a_{2}$ | $p(1|{\color[rgb]{0,0,0.609375}a_{2}})=0.05$ | 4.8

Table 4: Transition probabilities and expected returns calculated in the
dataset using $\lambda_{{tn}}$ and $\lambda_{{vd}}$.

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0.60546875,0,0}a_{1}})=0.17$ |
| $a_{1}$ | $p(5|{\color[rgb]{0.60546875,0,0}a_{1}})=0.83$ | 4.33
|  | $p(6|{\color[rgb]{0.60546875,0,0}a_{2}})=0.86$ |
Agent $1$ | $a_{2}$ | $p(1|{\color[rgb]{0.60546875,0,0}a_{2}})=0.14$ | 5.29

| action | transition | expected return
---|---|---|---
|  | $p(1|{\color[rgb]{0,0,0.609375}a_{1}})=0.14$ |
| $a_{1}$ | $p(6|{\color[rgb]{0,0,0.609375}a_{1}})=0.86$ | 5.29
|  | $p(5|{\color[rgb]{0,0,0.609375}a_{2}})=0.83$ |
Agent $2$ | $a_{2}$ | $p(1|{\color[rgb]{0,0,0.609375}a_{2}})=0.17$ | 4.33

Table 5: Normalized scores of MABCQ and the baselines. | MABCQ |  BCQ w/ $\lambda_{vd}$ |  BCQ w/ $\lambda_{tn}$ | BCQ | DDPG | Behavior
---|---|---|---|---|---|---
HalfCheetah | $\bm{17.6}\pm 3.3$ | $13.3\pm 4.8$ | $13.4\pm 6.5$ | $13.4\pm 5.2$ | $-3.1\pm 3.6$ | $11.3\pm 2.8$
Walker | $\bm{54.4}\pm 5.6$ | $50.1\pm 11.0$ | $41.2\pm 17.8$ | $28.8\pm 14.4$ | $1.7\pm 0.9$ | $10.0\pm 0.8$
Hopper | $\bm{43.1}\pm 14.2$ | $34.1\pm 8.2$ | $19.8\pm 8.7$ | $18.0\pm 3.4$ | $9.0\pm 15.3$ | $10.7\pm 2.3$
Ant | $\bm{60.5}\pm 3.6$ | $\bm{59.7}\pm 4.9$ | $\bm{62.9}\pm 2.1$ | $51.5\pm 12.7$ | $-48.4\pm 1.3$ | $19.3\pm 4.5$

##  4 Related Work

###  4.1 Off-policy MARL

Many off-policy MARL methods have been proposed for learning to solve
cooperative tasks in an online manner. Policy-based methods [15, 8, 37, 26,
29] extend actor-critic into multi-agent cases. Value factorization methods
[27, 22, 25, 30] decompose the joint value function into individual value
functions. All these methods follow CTDE, where the information of all agents
can be accessed in a centralized way during training. Unlike these studies, we
consider decentralized settings where global information is not available. For
decentralized learning, the key challenge is the obsolete experiences in the
replay buffer, which is considered in Fingerprints [2], Lenient-DQN [19], and
concurrent experience replay [18]. However, these methods require additional
information, e.g., training iteration number and exploration rate, which are
often not provided by the offline dataset.

###  4.2 Offline RL

Offline RL requires the agent to learn from a fixed batch of data consisting
of single-step transitions, without exploration. Most offline RL methods
consider the out-of-distribution action [12] as the fundamental challenge,
which is the main cause of the extrapolation error [5] in value estimate in
the single-agent environment. To minimize the extrapolation error, some recent
methods introduce constraints to enforce the learned policy to be close to the
behavior policy, which can be direct action constraint [5], kernel MMD [10],
Wasserstein distance [31], KL divergence [21], or $l2$ distance [4, 20]. Some
methods train a Q-function pessimistic to out-of-distribution actions to avoid
overestimation by adding a reward penalty quantified by the learned
environment model [35], by minimizing the Q-values of out-of-distribution
actions [11, 34], by weighting the update of Q-function via Monte Carlo
dropout [32], or by explicitly assigning and training pseudo Q-values for out-
of-distribution actions [16]. Our framework can be built on these methods.

MAICQ [33] studies offline MARL in the CTDE setting, which requires the joint
actions of all agents in the dataset and cannot be applied to decentralized
settings where datasets contain only individual actions. All the methods
aforementioned do not consider the extrapolation error introduced by the
transition bias, which is a fatal problem in offline decentralized MARL.

##  5 Experiments

We evaluate our framework in both fully and partially observable tasks. In
each task, we build offline dataset $\mathcal{B}_{i}$ for each agent $i$,
which does not contain actions of other agents. We will give the details about
the collection of each offline dataset. Our method and baselines have the same
neural network architectures and hyperparameters, which are available in
Appendix. All the models are trained for five runs with different random
seeds, and the results are presented in terms of mean and std.

###  5.1 Matrix Game

We perform MABCQ on the matrix game in Table 1. As shown in Table 3, if we
only use $\lambda_{vd}$ without considering transition normalization, since
the transition probabilities of high-value next states have been increased,
for agent $1$ the value of ${\color[rgb]{0.60546875,0,0}a_{2}}$ becomes higher
than that of ${\color[rgb]{0.60546875,0,0}a_{1}}$. However, due to the
unbalanced action distribution of agent $1$, the initial transition
probabilities of agent $2$ are extremely unbalanced. With $\lambda_{vd}$,
agent $2$ still underestimates the value of ${\color[rgb]{0,0,0.609375}a_{1}}$
and learns the action ${\color[rgb]{0,0,0.609375}a_{2}}$. The agents arrive at
the joint action
$({\color[rgb]{0.60546875,0,0}a_{2}},{\color[rgb]{0,0,0.609375}a_{2}})$, which
is a worse solution than the initial one (Table 2). Further, by normalizing
the transition probabilities by $\lambda_{tn}$, the agents can learn the
optimal solution
$({\color[rgb]{0.60546875,0,0}a_{2}},{\color[rgb]{0,0,0.609375}a_{1}})$ and
build the consensus about the values of learned actions, as shown in Table 4.

Table 6: Mean difference in value estimates among agents during training. It is shown that transition normalization indeed reduces the difference in value estimates.  value difference | MABCQ |  BCQ w/ $\lambda_{vd}$
---|---|---
HalfCheetah | $\bm{44.4}\pm 3.4$ | $411.7\pm 72.4$
Walker | $\bm{28.2}\pm 2.8$ | $38.7\pm 6.9$
Hopper | $\bm{24.2}\pm 0.8$ | $25.4\pm 1.3$
Ant | $\bm{60.8}\pm 2.9$ | $67.0\pm 3.1$
Table 7: Extrapolation errors. It is shown that our framework can decrease the extrapolation error. extrapolation error | MABCQ | BCQ
---|---|---
HalfCheetah | $98.4\pm 31.3$ | $97.2\pm 29.1$
Walker | $\bm{55.0}\pm 9.6$ | $91.5\pm 35.4$
Hopper | $\bm{28.1}\pm 3.4$ | $65.8\pm 6.4$
Ant | $\bm{180.2}\pm 22.2$ | $231.3\pm 47$
Table 8: Rewards on SMAC datasets. |  | MABCQ | BCQ | MACQL | CQL
---|---|---|---|---|---
random | 3m | $\bm{4.0}\pm 1.1$ | $0\pm 0$ | $\bm{13.5}\pm 1.5$ | $9.3\pm 3.0$
8m | $\bm{4.5}\pm 0.9$ | $0.8\pm 0.2$ | $\bm{8.2}\pm 1.1$ | $6.6\pm 1.2$
3s_vs_3z | $\bm{8.1}\pm 0.7$ | $0\pm 0$ | $10.2\pm 0.8$ | $10.1\pm 0.4$
3s_vs_4z | $\bm{4.1}\pm 1.5$ | $0\pm 0$ | $5.7\pm 0.6$ | $6.7\pm 0.2$
medium | 3m | $8.9\pm 1.3$ | $7.8\pm 0.4$ | $\bm{15.1}\pm 1.8$ | $13.8\pm 1.4$
8m | $\bm{7.6}\pm 0.8$ | $4.5\pm 1.2$ | $\bm{14.5}\pm 1.5$ | $12.4\pm 0.6$
3s_vs_3z | $\bm{8.7}\pm 1.1$ | $3.9\pm 0.6$ | $9.3\pm 0.9$ | $8.9\pm 0.6$
3s_vs_4z | $\bm{4.3}\pm 0.5$ | $0\pm 0$ | $6.1\pm 0.7$ | $6.8\pm 1.8$
replay | 3m | $13.2\pm 0.2$ | $12.7\pm 0.7$ | $13.8\pm 0.4$ | $13.5\pm 0.6$
8m | $\bm{15.2}\pm 1.0$ | $14.3\pm 0.9$ | $\bm{17.9}\pm 0.4$ | $16.3\pm 0.4$
3s_vs_3z | $19.4\pm 0.4$ | $19.8\pm 0.3$ | $20.0\pm 0.0$ | $20.0\pm 0.0$
3s_vs_4z | $5.3\pm 0.6$ | $5.3\pm 0.9$ | $\bm{5.9}\pm 0.3$ | $5.2\pm 0.7$
expert | 3m | $18.8\pm 0.7$ | $18.3\pm 1.1$ | $18.9\pm 0.5$ | $19.1\pm 0.5$
8m | $17.0\pm 0.8$ | $17.5\pm 1.1$ | $18.5\pm 1.2$ | $18.3\pm 1.1$
3s_vs_3z | $19.1\pm 0.5$ | $19.0\pm 0.6$ | $19.2\pm 0.6$ | $19.1\pm 0.6$
3s_vs_4z | $5.6\pm 0.9$ | $5.4\pm 1.1$ | $6.8\pm 0.7$ | $6.5\pm 0.9$

###  5.2 Multi-Agent Mujoco

To evaluate MABCQ in high-dimensional complex environments, we adopt multi-
agent mujoco [1], where each agent independently controls one or some joints
of the robot and can get the state [9] and reward of the robot. The task
illustration and the collection of offline datasets are given in Appendix.

Baselines. We compare MABCQ against

  * •

BCQ w/ ${\lambda_{vd}}$. Using $\lambda_{vd}$ alone on BCQ.

  * •

BCQ w/ ${\lambda_{tn}}$. Using $\lambda_{tn}$ alone on BCQ.

  * •

BCQ. Removing both $\lambda_{tn}$ and $\lambda_{vd}$ from MABCQ.

  * •

DDPG [14]. Each agent $i$ is trained using independent DDPG on the offline
$\mathcal{B}_{i}$ without action constraint and transition probability
modification.

  * •

Behavior. Each agent $i$ takes the action generated from the VAE $G_{i}^{1}$.

(a) 3s_vs_3z

(b) 2s3z

(c) DG

(d) CN

Figure 1: (a) and (b): Comparison with MAICQ on two SMAC datasets [33]. (c):
Performance of MATD3+BC with different $\epsilon$ on DG datasets. (d):
Learning curves on CN datasets.

Ablation. Table 5 shows the normalized scores [3] of all the methods in the
four tasks. Without action constraint, DDPG severely suffers from the large
extrapolation error and hardly learns. BCQ outperforms the behavior policies
but only arrives at mediocre performance. Using $\lambda_{tn}$ alone does not
always improve the performance, e.g., in HalfCheetah and Hopper. This is
because $\lambda_{tn}$ makes transition probabilities be uniform, which can be
far from the ones in execution, leading to large extrapolation errors. In Ant,
BCQ w/ $\lambda_{tn}$ outperforms BCQ, which is attributed to the value
consensus built by the normalized transition probabilities. By optimistically
increasing the transition probabilities of high-value next states,
$\lambda_{vd}$ mitigates the underestimation of potential good actions and
thus boosts the performance. MABCQ combines the advantages of value deviation
and transition normalization and outperforms other baselines.

Consensus on value estimates. To verify that transition normalization can
decrease the difference in value estimates among agents, we uniformly sample a
subset from the union of all agents’ states and calculate the difference in
value estimates, $\max_{i}V^{*}_{i}-\min_{i}V^{*}_{i}$, on this subset. The
mean differences during training are illustrated in Table 6. The
$\max_{i}V^{*}_{i}-\min_{i}V^{*}_{i}$ of MABCQ is indeed lower than that of
BCQ w/ $\lambda_{vd}$. If there is a consensus among agents about which states
are high-value, the agents will select the actions that most likely lead to
the common high-value states. This promotes the coordination of policies and
helps MABCQ outperform BCQ w/ $\lambda_{vd}$.

Extrapolation error. In Table 7, we present the extrapolation errors of MABCQ
and BCQ, measured by $|\frac{1}{N}\sum_{i}Q_{i}(s,a_{i})-G|$, where $G$ is the
true return evaluated by Monte Carlo. Although MABCQ greatly outperforms BCQ
(i.e., much higher return), it still achieves much smaller extrapolation
errors than BCQ in Walker, Hopper, and Ant. This empirically verifies the
claim that our method can decrease the extrapolation error.

###  5.3 SMAC

We also investigate the proposed framework on SMAC [23] tasks, including 3m,
8m, 3s_vs_3z, and 3s_vs_4z. We build random datasets that are generated by
uniform policies, medium datasets that are generated by mixed medium and
uniform policies, replay datasets that are collected in the training process
of QMIX [22], and expert datasets that are generated by expert policies
trained by QMIX. Each dataset contains $1\times 10^{4}$ episodes. We also
build our framework on CQL [11], as MACQL. As shown in Table 8. MABCQ and
MACQL achieve great performance improvement compared with the baselines,
especially in random and medium datasets, where the transition dynamics in
execution are much different from the ones in offline datasets. In expert
datasets, since the behavior policies are much deterministic, offline RL
methods avoid selecting out-of-distribution actions and thus degenerate to
behavior cloning. Therefore, all the methods perform similarly.

Offline CTDE settings. Although offline CTDE method [33] does not fit our
offline decentralized setting, our framework can work in offline CTDE
datasets. To verify, we select two replay datasets (jointly collected) in
MAICQ [33], split them into individual datasets, and test MACQL on them. As
shown in Figures 1(a) and 1(b), our decentralized method can obtain
competitive performance compared with the centralized method, MAICQ.

Table 9: Rewards on DG datasets. | MATD3+BC | TD3+BC
---|---|---
full observation | $\bm{46.5}\pm 0.9$ | $38.9\pm 0.8$
partial observation | $\bm{31.4}\pm 0.6$ | $20.7\pm 0.6$
Table 10: Normalized scores on D4RL MuJoCo datasets. |  | MATD3+BC | TD3+BC | TD3+BC (single)
---|---|---|---|---
random | halfcheetah | $\bm{14.3}\pm 2.9$ | $12.9\pm 2.6$ | $10.2\pm 1.3$
hopper | $10.3\pm 0.6$ | $10.4\pm 0.8$ | $11.0\pm 0.1$
walker2d | $4.5\pm 3.4$ | $3.7\pm 3.0$ | $1.4\pm 1.6$
medium | halfcheetah | $\bm{41.5}\pm 2.4$ | $40.4\pm 2.4$ | $42.8\pm 0.3$
hopper | $97.1\pm 1.7$ | $98.7\pm 1.2$ | $99.5\pm 1.0$
walker2d | $\bm{82.0}\pm 5.1$ | $72.3\pm 4.1$ | $79.7\pm 1.8$
replay | halfcheetah | $40.0\pm 3.2$ | $39.3\pm 2.8$ | $43.3\pm 0.5$
hopper | $\bm{28.4}\pm 4.1$ | $25.5\pm 3.3$ | $31.4\pm 3.0$
walker2d | $21.0\pm 2.9$ | $21.3\pm 1.5$ | $25.2\pm 5.1$
medium-expert | halfcheetah | $96.2\pm 5.5$ | $95.3\pm 6.5$ | $97.9\pm 4.4$
hopper | $112.0\pm 0.8$ | $112.3\pm 0.9$ | $112.2\pm 0.2$
walker2d | $88.9\pm 17.4$ | $82.0\pm 10.8$ | $105.7\pm 2.7$

###  5.4 MPE

We additionally evaluate our framework in an MPE-based [15] Differential Game
(DG), where the transition bias greatly affects the performance. Two agents
can move in the range $[-1,1]$. The action is the speed, which is in the range
$[-0.1,0.1]$. Define $l=\sqrt{x_{1}^{2}+x_{2}^{2}}$, where $x_{1}$ and $x_{2}$
are the positions of the two agents, respectively. The shared reward is set as

$r=\begin{cases}0.5\times(\cos(15\times l)+1)&\text{ if }l<0.2\\\ 0&\text{ if
}0.2\leq l\leq 0.6\\\ 0.5\times(l-0.6)^{2}&\text{ if }l>0.6.\end{cases}$

The visualization of reward function is shown in Figure 2.

Figure 2: Visualization of reward function in Differential Game.

Partial observation vs full observation. The offline datasets are collected by
uniform random policies, containing $1\times 10^{6}$ transitions. In the full
observation setting, the dataset of each agent contains the positions of both
agents. In the partial observation setting, the dataset of each agent only
contains its own position. In both settings, the datasets do not contain the
actions of the other agent. We add $\lambda_{vd}$ and $\lambda_{tn}$ to TD3+BC
[4], as MATD3+BC. As shown in Table 9, MATD3+BC obtains the substantial
improvement in both full and partial observation settings.

Hyperparameter $\bm{\epsilon}$. The optimism level $\epsilon$ controls the
strength of value deviation. If $\epsilon$ is too small, value deviation has
weak effects on the objective function. On the other hand, if $\epsilon$ is
too large, the agent will be overoptimistic about other agents’ learned
policies. Figure 1(c) shows that the performance of MATD3+BC with different
$\epsilon$, which verifies our framework is robust to $\epsilon$.

We test MATD3+BC on Cooperative Navigation in MPE, where $4$ agents learn to
cover $4$ landmarks. The reward is $-\text{sum}(\mathrm{distance}_{j})$, where
$\mathrm{distance}_{j}$ is the distance from landmark $j$ to the closest
agent. The offline datasets are collected by uniform random policies,
containing $1\times 10^{6}$ transitions. Figure 1(d) shows our framework
significantly outperforms the baseline.

###  5.5 Additional Results

We also split the D4RL [3] Mujoco datasets into decentralized multi-agent
datasets, and test MATD3+BC on them. The results are summarized in Table 10.
We find the results of decentralized methods, where the joints are controlled
by different agents, are very close to the results of single-agent method,
TD3+BC (single), where a single agent controls all joints, which could be seen
as the “upper-bound” of the decentralized methods. That is the reason that our
method does not bring significant improvement in these tasks. However,
MATD3+BC still outperforms TD3+BC on several tasks, e.g., halfcheetah-random
and walker2d-medium.

Table 11: Average time taken by one update. MABCQ | BCQ | MATD3+BC | TD3+BC
---|---|---|---
$18$ ms |  $10$ ms |  $4$ ms |  $3$ ms

To demonstrate the computation efficiency of our method, in Table 11, we
record the average time taken by one update in Halfcheetah. The experiments
are carried out on Intel i7-8700 CPU and NVIDIA GTX 1080Ti GPU. Since
$\lambda_{vd}$ and $\lambda_{tn}$ could be calculated from the sampled
experience without actually going through all next states, our framework
additionally needs only two forward passes for computing $\lambda_{vd}$ and
$\lambda_{tn}$ in the update. Since the value computation is very efficient in
TD3+BC, our framework is also efficient on it.

##  6 Conclusion

We propose a framework for offline decentralized multi-agent reinforcement
learning, to overcome the mismatch between transition dynamics. The framework
can be instantiated on many offline RL methods. Theoretically, we show that
under the purposely controlled non-stationary transition dynamics, offline
decentralized Q-learning converges to a unique fixed point. Empirically, the
framework outperforms the baselines in a variety of multi-agent offline
datasets.

This work was supported by NSF China under grants 62250068 and 61872009. The
authors would like to thank the anonymous reviewers for their valuable
comments.

## References

  * [1] Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson, ‘Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control’, arXiv preprint arXiv:2003.06709, (2020).
  * [2] Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson, ‘Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning’, in International Conference on Machine Learning (ICML), (2017).
  * [3] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine, ‘D4rl: Datasets for Deep Data-Driven Reinforcement Learning’, arXiv preprint arXiv:2004.07219, (2020).
  * [4] Scott Fujimoto and Shixiang Shane Gu, ‘A Minimalist Approach To Offline Reinforcement Learning’, in Thirty-Fifth Conference on Neural Information Processing Systems (NeurIPS), (2021).
  * [5] Scott Fujimoto, David Meger, and Doina Precup, ‘Off-Policy Deep Reinforcement Learning Without Exploration’, in International Conference on Machine Learning (ICML), (2019).
  * [6] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, ‘Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with A Stochastic Actor’, in International Conference on Machine Learning (ICML), (2018).
  * [7] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine, ‘How To Train Your Robot with Deep Reinforcement Learning: Lessons We Have Learned’, The International Journal of Robotics Research, 40(4-5), 698–721, (2021).
  * [8] Shariq Iqbal and Fei Sha, ‘Actor-Attention-Critic for Multi-Agent Reinforcement Learning’, in International Conference on Machine Learning (ICML), (2019).
  * [9] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang, ‘Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning’, International Conference on Learning Representations (ICLR), (2022).
  * [10] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine, ‘Stabilizing Off-Policy Q-Learning Via Bootstrapping Error Reduction’, in Advances in Neural Information Processing Systems (NeurIPS), (2019).
  * [11] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine, ‘Conservative Q-Learning for Offline Reinforcement Learning’, Neural Information Processing Systems (NeurIPS), (2020).
  * [12] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, ‘Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems’, arXiv preprint arXiv:2005.01643, (2020).
  * [13] Lihong Li, Wei Chu, John Langford, and Robert E Schapire, ‘A Contextual-Bandit Approach To Personalized News Article Recommendation’, in International Conference on World Wide Web (WWW), pp. 661–670, (2010).
  * [14] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, ‘Continuous Control with Deep Reinforcement Learning.’, in International Conference on Learning Representations (ICLR), (2016).
  * [15] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch, ‘Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments’, in Advances in Neural Information Processing Systems (NeurIPS), (2017).
  * [16] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu, ‘Mildly Conservative Q-learning for Offline Reinforcement Learning’, Neural Information Processing Systems (NeurIPS), (2022).
  * [17] Frans A Oliehoek and Christopher Amato, A Concise Introduction To Decentralized POMDPs, Springer, 2016.
  * [18] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian, ‘Deep Decentralized Multi-Task Multi-Agent Reinforcement Learning Under Partial Observability’, in International Conference on Machine Learning (ICML), (2017).
  * [19] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani, ‘Lenient Multi-Agent Deep Reinforcement Learning’, in International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), (2018).
  * [20] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu, ‘Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification’, arXiv preprint arXiv:2111.11188, (2021).
  * [21] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine, ‘Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning’, arXiv preprint arXiv:1910.00177, (2019).
  * [22] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson, ‘QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning’, in International Conference on Machine Learning (ICML), (2018).
  * [23] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson, ‘The StarCraft Multi-Agent Challenge’, CoRR, abs/1902.04043, (2019).
  * [24] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz, ‘Trust Region Policy Optimization’, in International Conference on Machine Learning (ICML), (2015).
  * [25] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi, ‘QTRAN: Learning To Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning’, in International Conference on Machine Learning (ICML), (2019).
  * [26] Kefan Su and Zongqing Lu, ‘Divergence-Regularized Multi-Agent Actor-Critic’, in International Conference on Machine Learning (ICML), (2022).
  * [27] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al., ‘Value-Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward’, in International Conference on Autonomous Agents and Multiagent Systems (AAMAS), (2018).
  * [28] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al., ‘Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning’, Nature, 575(7782), 350–354, (2019).
  * [29] Jiangxing Wang, Deheng Ye, and Zongqing Lu, ‘More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization’, in International Conference on Learning Representations (ICLR), (2023).
  * [30] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang, ‘Qplex: Duplex Dueling Multi-Agent Q-Learning’, in International Conference on Learning Representations (ICLR), (2021).
  * [31] Yifan Wu, George Tucker, and Ofir Nachum, ‘Behavior Regularized Offline Reinforcement Learning’, arXiv preprint arXiv:1911.11361, (2019).
  * [32] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh, ‘Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning’, arXiv preprint arXiv:2105.08140, (2021).
  * [33] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao, ‘Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning’, in Neural Information Processing Systems (NeurIPS), (2021).
  * [34] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn, ‘Combo: Conservative Offline Model-Based Policy Optimization’, arXiv preprint arXiv:2102.08363, (2021).
  * [35] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma, ‘MOPO: Model-Based Offline Policy Optimization’, Advances in Neural Information Processing Systems (NeurIPS), (2020).
  * [36] Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna.  BRAC+: Going Deeper with Behavior Regularized Offline Reinforcement Learning, 2021.
  * [37] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu, ‘FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning’, in International Conference on Machine Learning (ICML), (2021).

##  Appendix A Proofs

######  Proposition 1.

In episodic environments, if each agent $i$ performs Q-learning on
$\mathcal{B}_{i}$, all agents will converge to the same $V^{*}$ if they have
the same transition probability on any state where each agent $i$ acts the
learned action $a_{i}^{*}$.

###### Proof.

Considering the two-agent case, we define $\delta(s)$ as the difference in the
$V^{*}$.

$\displaystyle\delta(s)$ $\displaystyle=V_{1}^{*}(s)-V_{2}^{*}(s)$
$\displaystyle=\sum_{s^{\prime}}P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)\left(r+\gamma
V_{1}^{*}\left(s^{\prime}\right)\right)$
$\displaystyle\quad-\sum_{s^{\prime}}P_{\mathcal{B}_{2}}\left(s^{\prime}|s,a_{2}^{*}\right)\left(r+\gamma
V_{2}^{*}\left(s^{\prime}\right)\right)$
$\displaystyle=\sum_{s^{\prime}}P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)\left(r+\gamma
V_{2}^{*}\left(s^{\prime}\right)+\gamma
V_{1}^{*}\left(s^{\prime}\right)-\gamma
V_{2}^{*}\left(s^{\prime}\right)\right)$
$\displaystyle\quad-\sum_{s^{\prime}}P_{\mathcal{B}_{2}}\left(s^{\prime}|s,a_{2}^{*}\right)\left(r+\gamma
V_{2}^{*}\left(s^{\prime}\right)\right)$
$\displaystyle=\sum_{s^{\prime}}\left(P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)-P_{\mathcal{B}_{2}}\left(s^{\prime}|s,a_{2}^{*}\right)\right)\left(r+\gamma
V_{2}^{*}\left(s^{\prime}\right)\right)$ $\displaystyle\quad+\gamma
P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)\delta\left(s^{\prime}\right)$

For the terminal state $s_{end}$, we have $\delta(s_{end})=0$. If
$P_{\mathcal{B}_{1}}\left(s^{\prime}|s,a_{1}^{*}\right)=P_{\mathcal{B}_{2}}\left(s^{\prime}|s,a_{2}^{*}\right),\,\forall
s^{\prime}\in S$, recursively expanding the $\delta$ term, we arrive at
$\delta(s)=0+\gamma 0+\gamma^{2}0+...+0=0$. We can easily show that it also
holds in the $N$-agent case. ∎

######  Theorem 1.

Under the non-stationary transition probability $\hat{P}_{\mathcal{B}_{i}}$,
the Bellman operator $\mathcal{T}$ is a contraction and converges to a unique
fixed point when $\gamma<\frac{r_{\min}}{2r_{\max}-r_{\min}}$, if the reward
is bounded by the positive region $[r_{\min},r_{\max}]$.

###### Proof.

We initialize the Q-value to be $\eta r_{\min}$, where $\eta$ denotes
$\frac{1-\gamma^{T+1}}{1-\gamma}$. Since the reward is bounded by the positive
region $[r_{\min},r_{\max}]$, the Q-value under the operator $\mathcal{T}$ is
bounded to $[\eta r_{\min},\eta r_{\max}]$. Based on the definition of
$\hat{P}_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)$, it can be written
as
$\frac{V_{i}^{*}(s^{\prime})}{\sum_{\hat{s}^{\prime}}V_{i}^{*}({s}^{\prime})}$,
where
$V_{i}^{*}({s}^{\prime})=\max_{\hat{a}_{i}}Q_{i}\left(s^{\prime},\hat{a}_{i}\right)$.
Then, we have the following,

$\displaystyle\left\|\mathcal{T}Q_{i}^{1}-\mathcal{T}Q_{i}^{2}\right\|_{\infty}$
$\displaystyle=\max_{s,a_{i}}\left|\sum_{s^{\prime}\in\mathcal{S}}\hat{P}^{1}_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)\left[r+\gamma\max_{\hat{a}_{i}}Q_{i}^{1}\left(s^{\prime},\hat{a}_{i}\right)\right]\right.$
$\displaystyle\left.\quad-\sum_{s^{\prime}\in\mathcal{S}}\hat{P}^{2}_{\mathcal{B}_{i}}\left(s^{\prime}|s,a_{i}\right)\left[r+\gamma\max_{\hat{a}_{i}}Q^{2}_{i}\left(s^{\prime},\hat{a}_{i}\right)\right]\right|$
$\displaystyle=\max_{s,a_{i}}\gamma\left|\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{1}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}({s}^{\prime})}-\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}\right|$
$\displaystyle=\max_{s,a_{i}}\gamma\left|\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{1}}(s^{\prime}))^{2}-(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}({s}^{\prime})}\right.$
$\displaystyle\left.\quad-\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}\left(\frac{1}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}-\frac{1}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}({s}^{\prime})}\right)\right|$
$\displaystyle=\max_{s,a_{i}}\gamma\left|\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{1}}(s^{\prime})-V_{i}^{*^{2}}(s^{\prime}))(V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime}))}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}({s}^{\prime})}\right.$
$\displaystyle\left.\quad-\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}\frac{\sum_{s^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}(s^{\prime})-V_{i}^{*^{2}}(s^{\prime})}{\sum_{{s}^{\prime}}V_{i}^{*^{1}}({s}^{\prime})\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}\right|$
$\displaystyle\leq\max_{s,a_{i}}\gamma\left|\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{1}}(s^{\prime})-V_{i}^{*^{2}}(s^{\prime}))\right|\cdot\frac{1}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{1}}({s}^{\prime})}$
$\displaystyle\quad\cdot\max\left|(V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime}))-\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}\right|$
$\displaystyle\leq\gamma|\mathcal{S}|\left\|Q_{i}^{1}-Q_{i}^{2}\right\|_{\infty}\cdot\frac{1}{|\mathcal{S}|\eta
r_{\min}}\cdot\eta(2r_{\max}-r_{\min})$
$\displaystyle=\gamma(\frac{2r_{\max}}{r_{\min}}-1)\left\|Q_{i}^{1}-Q_{i}^{2}\right\|_{\infty}.$

The third term of the penultimate line is because: if
$V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime})>\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}$,

$\displaystyle
V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime})-\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}$
$\displaystyle\leq
V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime})-\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))*\eta
r_{\min}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}\leq
2\eta r_{\max}-\eta r_{\min},$

else,

$\displaystyle\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}-(V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime}))$
$\displaystyle\quad\leq\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))*\eta
r_{\max}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}\leq\eta
r_{\max}.$

Since $2\eta r_{\max}-\eta r_{\min}\geq\eta r_{\max}$, we have

$|(V_{i}^{*^{1}}(s^{\prime})+V_{i}^{*^{2}}(s^{\prime}))-\frac{\sum_{s^{\prime}\in\mathcal{S}}(V_{i}^{*^{2}}(s^{\prime}))^{2}}{\sum_{{s}^{\prime}\in\mathcal{S}}V_{i}^{*^{2}}({s}^{\prime})}|\leq
2\eta r_{\max}-\eta r_{\min}.$

Therefore, if $\gamma<\frac{r_{\min}}{2r_{\max}-r_{\min}}$, the operator
$\mathcal{T}$ is a contraction. By contraction mapping theorem, $\mathcal{T}$
converges to a unique fixed point. ∎

##  Appendix B Settings and Hyperparameters

Figure 3: Illustrations of multi-agent Mujoco: HalfCheetah, Walker, Hooper, and Ant. Table 12: Hyperparameters Hyperparameter | continues BCQ | discrete BCQ | CQL | TD3+BC
---|---|---|---|---
learning rate of $Q$ | $10^{-3}$ | $10^{-4}$ | $10^{-4}$ | $3\times 10^{-4}$
learning rate of $\xi$ | $10^{-4}$ |  |  |
learning rate of $G$ | $10^{-4}$ |  |  |
learning rate of $\pi$ |  |  | $10^{-4}$ | $3\times 10^{-4}$
$\Phi$ | $0.05$ |  |  |
$n$ | $10$ |  |  |
VAE hidden space | $10$ |  |  |
$\alpha$ |  |  | $0.2$ |
$\lambda$ |  |  |  | $2.5$
threshold |  | $\frac{0.6}{\text{action space}}$ |  |

The illustrations of multi-agent mujoco are shown in Figure 3, different
colors indicate different agents. Each agent independently controls one or
some joints of the robot and can get the state and reward of the robot, which
are defined in the original tasks. For each environment, we collect $N$
datasets for the $N$ agents. Each dataset contains $1\times 10^{6}$
transitions $(s,a_{i},r,s^{\prime},done)$. For data collection, we train an
intermediate policy and an expert policy for each agent using SAC [6]. The
offline dataset $\mathcal{B}_{i}$ is a mixture of four parts: $20\%$
transitions are split from the experiences generated by the SAC agent at the
early training, $35\%$ transitions are generated from that the agent $i$ acts
the intermediate policy while other agents act the expert policies, $35\%$
transitions are generated from that agent $i$ performs the expert policy while
other agents act the intermediate policies, $10\%$ transitions are generated
from that all agents perform the expert policies. For the last three parts, we
add a small noise to the policies to increase the diversity of the dataset.

In all tasks, we set the discount factor $\gamma=0.99$ and use ReLU
activation. In Mujoco tasks, the MLP units are $(64,64)$, and the batch size
is $1024$. In SMAC and DG, the MLP units are $(256,256)$, and the batch size
is $100$. The hyperparameter of our framework is the optimism level
$\epsilon$. We respectively set $\epsilon=0.80,0.48,0.80,0.64$ in HalfCheetah,
Walker, Hopper, and Ant, set $\epsilon=0.99$ in SMAC, and set $\epsilon=0.9$
in MPE. The hyperparameters of baselines are summarized in Table 12.

In this paper, we use SMAC (MIT license), MPE (MIT license), and Gym (MIT
license), D4RL (Apache-2.0 license). Many thanks for their contributions.