# Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function
Factorization

Wentse Chen
Carnegie Mellon University
<EMAIL_ADDRESS>
&Shiyu Huang
Zhipu AI
<EMAIL_ADDRESS>&Jeff Schneider
Carnegie Mellon University
<EMAIL_ADDRESS>

###### Abstract

Multi-agent reinforcement learning (MARL) tasks often utilize a centralized
training with decentralized execution (CTDE) framework. QMIX is a successful
CTDE method that learns a credit assignment function to derive local value
functions from a global value function, defining a deterministic local policy.
However, QMIX is hindered by its poor exploration strategy. While maximum
entropy reinforcement learning (RL) promotes better exploration through
stochastic policies, QMIX’s process of credit assignment conflicts with the
maximum entropy objective and the decentralized execution requirement, making
it unsuitable for maximum entropy RL. In this paper, we propose an enhancement
to QMIX by incorporating an additional local Q-value learning method within
the maximum entropy RL framework. Our approach constrains the local Q-value
estimates to maintain the correct ordering of all actions. Due to the
monotonicity of the QMIX value function, these updates ensure that locally
optimal actions align with globally optimal actions. We theoretically prove
the monotonic improvement and convergence of our method to an optimal
solution. Experimentally, we validate our algorithm in matrix games, Multi-
Agent Particle Environment and demonstrate state-of-the-art performance in
SMAC-v2. 111Our code: https://github.com/WentseChen/Soft-QMIX

Figure 1: Soft-QMIX combines the QMIX value decomposition mechanism with
maximum entropy RL under the CTDE framework. The figure explains the algorithm
design rationale. The left figure illustrates a naive approach to applying
maximum entropy RL in the CTDE context, where blue text represents the desired
objectives, and black text indicates corresponding constraints, It reveals
that incorporating maximum entropy RL under the CTDE framework implicitly
constrains the global value to be the sum of local values, significantly
limiting the expressiveness of the critic network. The right figure depicts
the Soft-QMIX improvement, where Soft-QMIX first employs QMIX’s value
decomposition mechanism to obtain $Q_{i}$. Given the meaningful order of
QMIX’s output $Q_{i}$, such as possessing properties like IGM, an order-
preserving transformation converts $Q_{i}$ to $f_{i}(Q_{i})$. This
transformation is trained by the loss function that minimize the difference
between $f_{i}(Q_{i})$ and $Q_{tot}.$

##  1 Introduction

In the context of solving cooperative tasks using deep multi-agent
reinforcement learning (MARL), a standard approach is centralized training
with decentralized execution (CTDE) [5, 23, 15]. Decentralized execution
implies that during the execution phase, the decision-making of each agent is
independent and limited to conditioning on local observations. Centralized
training, on the other hand, allows the algorithm to access joint actions and
global states while training the value function. Compared to decentralized
training, which treats other agents as part of the environment, centralized
training helps alleviate the instability often encountered in MARL training
[30].

In most cooperative tasks, the multi-agent system receives a single joint
reward, and MARL algorithms learn the corresponding joint value function based
on this reward. Consequently, there is a need to design a credit assignment
mechanism [2] that decomposes the joint value function into several local
value functions, guiding the updates of the actor networks through these local
value functions. The challenge in credit assignment lies in reconstructing the
joint distribution from the combination of multiple marginal distributions, a
process that typically involves approximation errors [32]. Therefore, past
research has primarily focused on exploring the impact of mechanism design on
the model’s capacity [27, 31].

Beyond model capabilities, exploration ability is also a critical factor
affecting the performance of MARL algorithms. Maximum entropy RL [14], known
for encouraging exploration [7], smoothing the landscape of the objective [1],
and enhancing robustness [4], has been widely adopted in single-agent RL.
However, integrating MARL with maximum entropy RL is not straightforward. As
illustrated in Figure 1, using maximum entropy RL within the framework of CTDE
will significantly limit the model’s abilities in a manner akin to VDN [28],
which enforces that the sum of local q-values must equal the global q-value.

In this paper, we propose a value-based maximum entropy MARL method compatible
with the CTDE framework, named Soft-QMIX. We adopt the QMIX [23] mechanism to
train a centralized critic network and utilize its credit assignment mechanism
to obtain local Q-functions. We also incorporate maximum entropy RL, utilizing
stochastic policies and entropy regularization to improve exploration and
robustness. To guarantee monotonically increasing expected returns during
training and convergence to the optimal policy, an order-preserving
transformation on the original local Q-functions is used for the final policy.
Through this design, Soft-QMIX can benefit from both maximum entropy RL and
QMIX’s credit assignment mechanism, while still ensuring convergence to the
optimal strategy.

Our main contributions can be summarized as follows:

  * •

We introduce Soft-QMIX, a value-based maximum entropy MARL algorithm that
aligns with the CTDE framework. It employs an order-preserving transformation
on local q-values, enabling it to derive meaningful order from credit
assignment and leverage the exploratory benefits provided by maximum entropy
RL.

  * •

Theoretically, we demonstrate that our approach can monotonically improve the
expected q-value and guarantee convergence to the optimal policy.

  * •

Empirically, we confirm the effectiveness of our algorithm through tests on
matrix games, MPE [18] and SMAC-v2 [3], where our method attains state-of-the-
art results.

##  2 Related Works

Multi-Agent RL. In the CTDE framework, MARL primarily involves two common
algorithmic approaches: policy gradient methods [17, 34, 15, 5, 13, 37] and
value function decomposition methods [28, 23, 31]. Policy gradient methods
first learn a centralized critic network, and then distill local policies
using losses like the KL divergence. On the other hand, Value function
decomposition addresses the credit assignment problem by decomposing the
global value function into multiple local value functions. VDN [28] assumes
that the global Q-function is the sum of local Q-functions. QMIX [23, 20]
permits the mixer function to be any function with non-negative weights. QTRAN
[27] optimizes an additional pair of inequality constraints to construct a
loss function. QPLEX [31] use the dueling architecture to decompose the
Q-function, achieving the same expressive power as QTRAN while being easier to
optimize. The expressive power of the critic networks in these algorithms
increases sequentially, but experimental results show that QMIX has better
performance [9]. Therefore, in this work, QMIX is used as the mechanism for
credit assignment.

Maximum Entropy RL. Maximum entropy RL [14] is demonstrated to have advantages
such as encouraging exploration [7] and increasing robustness [4] in single-
agent RL scenarios. In maximum entropy MARL, most works utilize an actor-
critic architecture and typically aim to minimize the KL divergence between
the local actor and the centralized critic [8, 6, 22, 36]. However, although
these methods employ QMIX to train the critic, they are only capable of
minimizing the discrepancy in expectation, thus cannot guarantee the IGM
condition, which is central to the QMIX algorithm. Our approach is value-based
and ensures that the IGM condition is met while employing maximum entropy
MARL. Moreover, our method introduces global information into the local policy
during training through a hyper-network, which most previous methods either
ignored [35] or only applied in centralized training and centralized execution
scenarios [16]. In appendix H, we provide a detailed comparison of our work
with related works.

##  3 Preliminary

###  3.1 Multi-Agent Reinforcement Learning

Cooperative MARL can be formalized as a Decentralized Partially Observable
Markov Decision Process (Dec-POMDP) [19]. Formally, a Dec-POMDP is represented
as a tuple $(\mathcal{A},S,U,T,r,O,G,\gamma)$, where
$\mathcal{A}\equiv\\{1,...,n\\}$ is the set of $n$ agents. $S$, $U$, $O$, and
$\gamma$ are state space, action space, observation space, and discount
factor, respectively.

During each discrete time step $t$, each agent $i\in\mathcal{A}$ chooses an
action $u^{i}\in U$, resulting in a collective joint action
$\mathbf{u}\in\mathbf{U}\equiv U^{n}$. The function $r(s,\mathbf{u})$ defines
the immediate reward for all agents when the collective action $\mathbf{u}$ is
taken in the state $s$.
$\mathcal{P}(s,\mathbf{u},s^{\prime}):S\times\mathbf{U}\times
S\rightarrow[0,\infty)$ is the state-transition function, which defines the
probability density of the succeeding state $s^{\prime}$ after taking action
$\mathbf{u}$ in state $s$. In a Dec-POMDP, each agent receives only partial
observations $o\in O$ according to the observation function
$G(s,i):S\times\mathcal{A}\rightarrow O$. For simplicity, if a function
depends on both $o^{i}$ and $s$, we will disregard $o^{i}$. Each agent uses a
policy $\pi_{i}(u^{i}|o^{i})$ to produce its action $u^{i}$ from its local
observation $o^{i}$. Note that the policy should be conditioned on the history
$o^{i}_{1:t}$. For simplicity, we refer to $o^{i}_{1:t}$ as $o^{i}_{t}$ or
$o^{i}$ in the whole paper. The joint policy is denoted as $\pi_{jt}$.
$\rho_{\pi}(s_{t},\mathbf{u_{t}})$ is used to represent the state-action
marginals of the trajectory distribution induced by a policy $\pi$.

###  3.2 Individual-Global Condition

In this section, we define two key concepts in credit assignment.

######  Definition 3.1.

Individual-Global-Max (IGM) For joint q-function $Q_{tot}$, if there exist
individual q-functions $\left[Q_{i}\right]_{i=1}^{n}$ such that the following
holds:

$\underset{\mathbf{u_{t}}}{\mathrm{argmax}}\
Q_{tot}(\mathbf{u_{t}},s_{t})=\begin{pmatrix}\underset{u_{t}^{1}}{\mathrm{argmax}}\
Q_{1}(u^{1}_{t}|o^{1}_{t})\\\ \vdots\\\ \underset{u_{t}^{n}}{\mathrm{argmax}}\
Q_{n}(u^{n}_{t}|o^{n}_{t})\end{pmatrix}.$ (1)

Then, we say that $[Q_{i}]_{i=1}^{n}$ satisfy IGM for $Q_{tot}$ under $s_{t}$.

In QMIX, to ensure the validity of IGM, the algorithm constrains the
expressive capability of $Q_{tot}$ to the following form: $\frac{\partial
Q_{tot}}{\partial Q_{i}}\geq 0,\forall i\in\mathcal{A}.$ Restricting the
capacity of $Q_{tot}$ using the above inequality results in $Q_{tot}$ being a
monotonic function. The monotonicity constraint serves as a sufficient, but
not necessary, condition for IGM. Consequently, the monotonicity constraint is
more stringent than those required by IGM.

######  Definition 3.2.

Individual-Global-Independence (IGI) For joint policy $\pi_{jt}$, if there
exist individual policies $\left[\pi_{i}\right]_{i=1}^{n}$ such that the
following holds:

$\pi_{jt}(\mathbf{u_{t}}|s_{t})=\prod_{i=1}^{n}\pi_{i}(u^{i}_{t}|o^{i}_{t}).$
(2)

Then, we say that $[\pi_{i}]_{i=1}^{n}$ satisfies IGI for $\pi_{jt}$ under
$s_{t}$.

Definition 3.1 characterizes the properties we expect a deterministic policy
to possess. Definition 3.2, on the other hand, describes the independence of
decision-making among agents during decentralized execution. IGI and the IGO
[36] condition are very similar; we show in Appendix B that IGI is a
sufficient condition for IGO.

###  3.3 Naive Maximum Entropy MARL Algorithm

In this section, we informally introduce a naive MARL algorithm. We use it to
illustrate the issue that naively applying credit assignment to the maximum
entropy RL algorithm will lead to limited credit assignment capability.

Assume a value-based framework, where $Q_{i}$ denotes the local Q-function and
the function for credit assignment is denoted as $f^{c}$. Hence, the global
Q-function, $Q_{tot}$, can be expressed as
$Q_{tot}(\mathbf{u},s)=f^{c}\big{(}Q_{1}(u^{i},o^{i}),...,Q_{n}(u^{n},o^{n}),s\big{)}$.
Based on maximum entropy RL, the global policy is defined as
$\pi_{jt}(\cdot|s)\propto\text{softmax}\big{(}Q_{tot}(\cdot,s)\big{)}$ and the
local policy as
$\pi_{i}(\cdot|o^{i})\propto\text{softmax}\big{(}f^{t}_{i}(Q_{i}(\cdot,o^{i}),s)\big{)}$,
where $f^{t}_{i}$ is an arbitrary function. Due to the IGI condition, we have
$f^{c}(Q_{1},...,Q_{n},s)=\sum_{i=1}^{n}f^{t}_{i}(Q_{i},s)$. When $f^{t}_{i}$
is an identity mapping, the aforementioned algorithm is equivalent to VDN
[28]. When $f^{t}_{i}$ represents a linear transformation, the algorithm
becomes analogous to the DOP, an actor-critic method [33]. In fact,
$f^{t}_{i}$ can be any function of $u^{i}$ and $o^{i}$, but cannot be a
function of $u^{j}$ or $o^{j}$, where $j\neq i$. This constraint limits the
expressiveness of the critic network.

##  4 Soft-QMIX

In this paragraph, we first briefly introduce the motivation behind the Soft-
QMIX algorithm. Then, we detail the process of the Soft-QMIX algorithm and
show that the expected returns of the centralized training are monotonically
increasing and converge to the optimum. Finally, we propose a practical
algorithm.

###  4.1 Motivation

As shown in Figure 2, we divide the decision-making process into two stages
(green and blue blocks). In the first stage, agents rank the Q-values for all
actions. We employ the value decomposition mechanism from QMIX to obtain local
Q-values but only utilize the order of their outputs, not the specific
Q-function values. In the second stage, without changing the order, we assign
specific Q-values corresponding to each action. When selecting actions in
decentralized execution, only the maximum value is required, necessitating
just the first stage for execution. The second stage serves primarily to
obtain an accurate estimated q-value. During this phase, we can utilize global
information, as it is only used during training.

###  4.2 Soft Policy Iteration

The objective function of Maximum Entropy RL can be represented in the
following form:

$\displaystyle J(\pi_{jt})$
$\displaystyle=\mathbb{E}_{(s_{t},\mathbf{u}_{t})\sim\rho_{\pi_{jt}}}\big{[}r(s_{0},\mathbf{u}_{0})+\sum_{t=1}^{\infty}\gamma^{t}\big{(}r(s_{t},\mathbf{u}_{t})+\alpha
H\left(\pi_{jt}(\cdot|s_{t})\big{)}\right)\big{]},$ (3)

where $H(\pi_{jt})$ denotes the entropy of the joint policy $\pi_{jt}$, and
the temperature parameter $\alpha$ is used to control the degree of randomness
in the policy.

In the policy evaluation step, we aim to approximate the objective function
described in Equation 3 using
$Q_{tot}:S\times\mathbf{U}\rightarrow\mathbb{R}$. The soft state value
function $V_{tot}:S\rightarrow\mathbb{R}$ is defined as follows:

$V_{tot}(s)=\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[Q_{tot}(s,\mathbf{u})-\alpha\log\pi_{jt}(\mathbf{u}|s)].$
(4)

Repeatedly applying the Bellman operation to update $Q_{tot}$ results in the
soft $Q_{tot}$ function for policy $\pi_{jt}$. The Bellman operation is
defined as follows:

$\mathcal{T}^{\pi_{jt}}Q_{tot}(s,\mathbf{u})=r(s,\mathbf{u})+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}}[V_{tot}(s^{\prime})].$
(5)

In reference to the QMIX, a monotonicity constraint is imposed on the capacity
of the critic network. Specifically, $Q_{\text{tot}}$ can be formulated as
follows:

$Q_{tot}(s,\mathbf{u})=\text{Mixer}\left(Q_{1}(o^{1},u^{1}),...,Q_{n}(o^{n},u^{n}),s\right).$
(6)

The network structure of the Mixer function is the same as QMIX’s, and is
elaborated in Appendix G.

Due to the monotonicity constraint, there is moderate error in estimating of
$Q_{tot}$. To enhance the precision of value estimation, we introduce the
TD($\lambda$) [24] method. The estimation error between the maximum entropy
objective function and the Bellman equation objective can be expressed as
follows:

$\displaystyle\delta_{t}^{V}$
$\displaystyle=-Q_{tot}(s_{t},\mathbf{u}_{t})+r_{t}+\gamma V_{tot}(s_{t+1}),$
(7) $\displaystyle\epsilon_{t}^{td(\lambda)}$
$\displaystyle=\mathcal{T}^{\pi_{jt}}_{\lambda}Q_{tot}(s_{t},\mathbf{u_{t}})-Q_{tot}(s_{t},\mathbf{u_{t}})=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{V}_{t+l}.$

Figure 2: The overall network architecture. Functions $g_{i}$ and $f_{i}$ are
order-preserving transformations, ensuring the input and output dimensions are
identical and have the same argmax values. The output of the $f_{i}$ function,
after undergoing a softmax operation, becomes the policy, from which action
$u_{t}^{i}$ is sampled. For decentralized execution, only the green network is
needed, and the maximum values of their output are selected.

The derivation and implementation details of Equation 7 are provided in
Appendix D. When combining off-policy methods with TD($\lambda$), adopting
$\lambda>0$ requires importance sampling corrections [10]. Following previous
work [9], we opt for a small $\lambda$ and assume that
$\mathcal{T}^{\pi_{jt}}_{\lambda}$ is equivalent to $\mathcal{T}^{\pi_{jt}}$
in expectation. For simpilicity, we will use $\mathcal{T}^{\pi_{jt}}$ instead
of $\mathcal{T}^{\pi_{jt}}_{\lambda}$ in theoretical discussions.

The following lemma demonstrates that by iteratively applying the Bellman
equation, convergence can be achieved towards the maximum entropy objective
$Q_{\text{tot}}$ for $\pi_{jt}$.

######  Lemma 4.1 (Joint Soft Policy Evaluation).

Let $Q^{0}_{jt}:S\times\mathcal{U}\to\mathbb{R}$ be a bounded mapping where
$|\mathcal{U}|<\infty$. Define the iterative process
$Q^{k+1}_{tot}=\mathcal{T}^{\pi_{jt}}(Q^{k}_{tot})$. It follows that the
sequence $\\{Q^{k}_{tot}\\}_{k=0}^{\infty}$ converges to the joint soft-Q-
function for policy $\pi_{jt}$ as $k\rightarrow\infty$.

Proof. See Appendix C. ∎

In Section 4.1, it was mentioned that decision-making is divided into two
stages. During training, the policy is defined as
$\pi^{train}_{i}(\cdot|o^{i},s)=\text{softmax}(f^{\text{stage2}}_{i}(f^{\text{stage1}}_{i}(\cdot|o^{i}),s)/\alpha)$,
where $f^{\text{stage1}}_{i}$ is a function dependent on local observations,
responsible for the first stage of decision-making. $f^{\text{stage2}}_{i}$,
on the other hand, is a non-linear order-preserving transformation,
responsible for the second stage of decision-making. Although
$\pi^{train}_{i}$ depends on the global state, which conflicts with the
definition in Dec-POMDP, it remains a valid option because it is utilized
solely during training. In the subsequent paragraph, $\pi$ will denote
$\pi^{train}_{i}$. $f^{\text{stage2}}_{i}$ can be further decomposed into two
order-preserving transformations, $g_{i}$ and $f_{i}$. The output of $g_{i}$
is $Q_{i}$, and $f_{i}$ further transforms $Q_{i}$ into the final policy. The
form of order-preserving transformation (OPT) is as follows:

$\displaystyle OPT_{i}(x,s)=W_{{}_{(1\times
d_{1})}}^{2}\sigma(W^{1}_{{}_{(d_{1}\times 1)}}x_{{}_{(1\times
d_{2})}}+b^{1})+b^{2},$ (8)

where $d_{1}$ is the hidden dimension, $d_{2}$ is the dimension of actions,
$b^{1}$ and $b^{2}$ are the bias, $\sigma$ is an activation function, and
$W^{1}$ and $W^{2}$ are non-negative weights. The implementation of the order-
preserving transformation is similar to QMIX’s and is elaborated in Appendix
G, where its weights and biases are provided by a hypernetwork. This
hypernetwork takes the global state $s$ as input, making $g_{i}$ and $f_{i}$
functions dependent on the global state. Figure 2 presents the overall
framework, where Agents $1$ to Agents $n$ correspond to
$f_{i}^{\text{stage1}}$, and $g_{i}$ and $f_{i}$ correspond to
$f_{i}^{\text{stage2}}$.

Since $g_{i}$ is an order-preserving transformation, and $[Q_{i}]_{i=1}^{n}$
satisfy the IGM condition for $Q_{tot}$, it follows that
$\underset{\mathbf{u}}{\text{argmax}}\,Q_{tot}(\mathbf{u},s)=\left(\underset{u^{1}}{\text{argmax}}\,f_{1}^{\text{stage1}}(u^{1}|o^{1}),\ldots,\underset{u^{n}}{\text{argmax}}\,f_{n}^{\text{stage1}}(u^{n}|o^{n})\right)$.
Therefore, during execution, only $f_{i}^{\text{stage1}}$ is needed.

During policy improvement, we update the joint policy towards the exponential
of the joint Q-function. We constrain $\pi\in\Pi$, where $\Pi$ is the family
of functions that satisfy the IGI condition. In the updating process, the
updated policy is projected onto $\Pi$. Specifically, the update formula is as
follows:

$\pi_{jt}^{\text{new}}=\underset{\pi^{\prime}_{jt}\in\Pi}{\mathrm{arg\,min}}\
D_{KL}\left(\pi^{\prime}_{jt}(\cdot|s_{t})\middle\|\frac{\exp(Q_{tot}^{\pi_{\text{old}}}(s_{t},\cdot)/\alpha)}{Z^{\pi_{\text{old}}}(s_{t})}\right),$
(9)

where
$Z^{\pi_{\text{old}}}(s_{t})=\int_{\mathbf{u}}\exp(Q_{tot}^{\pi_{\text{old}}}(s_{t},\mathbf{u})/\alpha)d\mathbf{u}$.

In value-based methods, Equation 9 is intractable. Therefore, in the policy
improvement step, we update the policy in this way:

$f_{i}^{\text{new}}=\underset{f^{\prime}_{i}}{\mathrm{argmin}}\frac{1}{2}\left(\sum_{i}f^{\prime}_{i}(Q_{i}(o^{i}_{t},u^{i}_{t},s_{t}))-y_{t}\right)^{2},$
(10)

where $y_{t}$ can be any unbiased estimate of
$\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}Q_{tot}(s_{t},\mathbf{u}_{t})\right]$.
Here, we assume222This does not necessarily hold in practical implementations,
but the difference between these is a term in the loss function of our
algorithm. For further discussion, please see the Appendix F. that
$\mathcal{T}^{\pi_{jt}}Q_{tot}(s_{t},\mathbf{u_{t}})$ is an unbiased estimate
of
$\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}\sum_{i}f_{i}(Q_{i}(s_{t},\mathbf{u_{t}}))\right]$.
Under this assumption, we can replace $Q_{\theta}$ and $y_{t}$ in Theorem 4.2
with $\sum_{i}f_{i}(Q_{i})$ and $Q_{tot}$ respectively, and get that
minimizing the L2 norm described in Equation 10 is equivalent to minimizing
the KL divergence described in Equation 9 and minimizing the TD loss of value
function at the same time. Therefore, in value-based methods, Equation 10 can
be used in place of Equation 9.

######  Theorem 4.2.

If $Q$-function is parameterized as $Q_{\theta}$, and
$V_{\theta},\pi_{\theta}$ are defined as below:

$\displaystyle V_{\theta}(s)$
$\displaystyle\coloneq\alpha\log\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\exp\left(Q_{\theta}(s,\mathbf{u})/\alpha\right)\right],$
(11) $\displaystyle\pi_{jt_{\theta}}(\mathbf{u}|s)$
$\displaystyle\coloneq\exp\left(Q_{\theta}(s,\mathbf{u})-V_{\theta}(s)\right)/\alpha.$

and $y_{t}$ is an unbiased estimator of
$\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}Q_{\theta}(s_{t},\mathbf{u_{t}})\right].$
Then, we have

$\displaystyle\nabla_{\theta}\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}$
$\displaystyle\left[\|Q_{\theta}(s,\mathbf{u})-y\|^{2}\right]$ (12)
$\displaystyle=\alpha\nabla_{\theta}D_{KL}\big{(}\pi_{jt}(\cdot|s)\|\frac{exp(y(\cdot,s)/\alpha)}{Z(s)}\big{)}+\nabla_{\theta}\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\|V_{\theta}(s)-y\|^{2}\right]$

Proof. See Appendix E.

The subsequent theorem ensures that updating the policy using Equation 9
results in a monotonic increase of the Q-function under the IGI condition.

######  Lemma 4.3 (Joint Soft Policy Improvement).

Suppose $\pi_{jt}^{\text{old}}\in\Pi$ and consider $\pi_{jt}^{\text{new}}$ to
be the solution to Equation 9. Then
$Q^{\pi_{jt}^{\text{new}}}(s_{t},\mathbf{u}_{t})\geq
Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})$ for all
$(s_{t},\mathbf{u}_{t})\in S\times\mathbf{U}$ with $|\mathbf{U}|<\infty$.

Proof. See Appendix C.

Theorem 4.4 demonstrates that iteratively applying policy evaluation and
policy improvement leads to a globally optimal policy under the IGI condition.

######  Theorem 4.4 (Joint Soft Policy Iteration).

By continually applying soft policy evaluation and soft policy improvement to
any $\pi_{jt}\in\Pi$ converges to a policy $\pi^{*}$ such that
$Q^{\pi^{*}}(s_{t},\mathbf{u}_{t})\geq Q^{\pi}(s_{t},\mathbf{u}_{t})$ for all
$\pi_{jt}\in\Pi$ and $(s_{t},\mathbf{u}_{t})\in S\times\mathbf{U}$, assuming
$|\mathbf{U}|<\infty$.

Proof. See Appendix C.

###  4.3 The Soft-QMIX Algorithm

Figure 2 illustrates the network architecture of the Soft-QMIX algorithm,
which comprises four types of networks. The Mixer network is responsible for
handling credit assignment and is parameterized along with its corresponding
hypernetwork as $\text{Mixer}_{\theta}$. The $[g_{i}]_{i=1}^{n}$ and
$[f_{i}]_{i=1}^{n}$ are $2n$ order-preserving transformations, with each
$g_{i},f_{i}$ and their corresponding hypernetwork parameterized as
$g_{i,\theta}$ and $f_{i,\theta}$. Agents 1 to $n$ share parameters, also
referred to as $f_{i}^{\text{stage1}}$ in the previous section, and are
parameterized as $f^{\text{stage1}}_{\theta}$. The updated rules for these
parameters are described as follows.

Soft-QMIX aims to minimize two loss functions. $\text{Mixer}_{\theta}$,
$g_{i,\theta}$ and $f^{\text{stage1}}_{\theta}$ are updated using the
following loss function:

$L_{Q}(\theta)=\frac{1}{2}\left(Q_{tot_{\theta}}(s_{t},\mathbf{u}_{t})-\mathcal{T}^{\pi_{jt}}_{\lambda}Q_{tot}(s_{t},\mathbf{u}_{t})\right)^{2},$
(13)

where
$Q_{tot_{\theta}}(\mathbf{u},s)=\text{Mixer}_{\theta}\big{(}g_{1,\theta}(f^{\text{stage1}}_{\theta}(o^{1},u^{1}),s),\ldots,$
$g_{n_{\theta}}(f^{\text{stage1}}_{\theta}(o^{n},u^{n}),s),s\big{)}$, and
$\mathcal{T}^{\pi_{jt}}_{\lambda}Q_{tot}(s_{t},\mathbf{u}_{t})$ is defined in
Equation 7. We utilize the technique of target Q networks to stabilize
training, where the target Q network is parameterized as $\bar{\theta}$, with
$\bar{\theta}$ being the exponentially weighted moving average of $\theta$.
Subsequently, the parameter $\phi$ is updated using the following formula:

$L_{\pi}(\phi)=\frac{1}{2}\left(\sum_{i=1}^{n}f_{i,\phi}(Q_{i}(o^{i}_{t},u^{i}_{t}),s_{t})-Q_{tot}(s_{t},\mathbf{u}_{t})\right)^{2}.$
(14)

In this process, the function $f_{i,\phi}$ takes $Q_{i}(o^{i}_{t},u^{i}_{t})$
as input, but during backpropagation, gradients are truncated at $Q_{i}$. This
truncation ensures that this loss function updates only $f_{i,\phi}$. After
the update, the policy can be expressed in the following form:

$\pi_{i}(u^{i}|o^{i},s)=\text{softmax}(\frac{f_{i}\left(g_{i}\big{(}f^{stage1}(o^{i}),s\big{)},s\right)}{\alpha})$
(15)

It’s important to note that during execution, we only need to call
$f^{stage1}$ and apply the argmax operation to it. In this paper, $\alpha$ is
fixed during training. Finally, the algorithmic procedure of Soft-QMIX is
shown in appendix A.

##  5 Experiments

In this section, we evaluate our algorithm using matrix games from prior works
[27], demonstrating its ability to consistently reach the global optimum.
Then, we compare Soft-QMIX with QMIX on MPE [18] and empirically demonstrate
Soft-QMIX algorithm achieves state-of-the-art performance on the SMAC-v2 [3]
benchmark. We compare our algorithm with four baseline algorithms. MAPPO [34]
adapts the on-policy, single-agent PPO [26] algorithm for multi-agent use,
utilizing a centralized value function. QMIX [23] is a value-based method
employing a monotonic value decomposition to create multiple local
deterministic policies from a shared value function. It uses an
$\epsilon$-greedy strategy for exploration. FOP [36] implements a maximum
entropy policy gradient approach with value factorization similar to QPLEX.
HASAC [16] offers an alternative multi-agent soft actor-critic framework,
facilitating sequential decision-making by agents in a centralized training
with centralized execution context. We elaborate implementation details and
hyperparameters in Appendix J.

###  5.1 Matrix Games

In this section, we demonstrate the effectiveness of our algorithm in the
classic one-step matrix games. Table 1 presents a classic non-monotonic payoff
matrix, often used in past papers [27, 31] to discuss the representational
capability of critic networks. QMIX, due to its monotonicity constraint, often
incurs moderate errors in fitting such payoff matrices. However, in practice,
during the training process, we are primarily concerned with the value
estimates of those joint actions that are frequently visited. For suboptimal
joint actions with lower Q values, their errors do not significantly impact
the estimated expected returns. As we use TD-$\lambda$ to estimate returns in
our experiments, such errors can be further reduced. Conversely, we are more
concerned with whether the joint actions with the highest probability of being
selected have the globally maximal value function, as we aim for the highest
returns when executing with a deterministic policy. For suboptimal joint
actions, we also wish to visit them with a certain probability to ensure
adequate exploration.

Table 1: Payoff matrix of the one-step matrix game. | A | B | C
---|---|---|---
A | 8 | -12 | -12
B | -12 | 0 | 0
C | -12 | 0 | 0

In the matrix game shown in Table 2, QMIX exhibits high estimation error due
to its monotonicity constraints, and its deterministic policy lacks effective
exploration, which leads it to learn only suboptimal joint actions with a
reward of $0$. FOP, with its network structure similar to QPLEX, can
accurately estimate value functions, yet the corresponding policy learned
tends to select suboptimal joint actions more frequently. Soft-QMIX has
smaller estimation errors for optimal joint actions and larger errors for
suboptimal ones. However, the joint action with the maximum Q value in Soft-
QMIX matches the true optimal joint action, and the local policy has the
highest probability of selecting the optimal joint action. Soft-QMIX possesses
this capability because its decision-making is divided into two steps, and in
the first step, it ensures the order of probabilities of each action being
selected in the policy.

Table 2:  (a)-(c) respectively demonstrate the results of Soft-QMIX, FOP, and
QMIX in the one-step matrix game. The first row and column display the
policies adopted by the row and column players after $10k$ steps of training,
respectively. The 3$\times$3 matrix in the bottom right corner represents the
$Q_{\text{tot}}$ ultimately learned by the algorithm. The results indicate
that Soft-QMIX is the only method among all that consistently chooses the
optimal action with the highest probability.

| 1. | 0. | 0.
---|---|---|---
1. | 8 | -12 | -14
0. | -8 | -28 | -30
0. | -10 | -30 | -32
(a) result of Soft-QMIX

| .21 | .39 | .40
---|---|---|---
.21 | 8 | -12 | -12
.40 | -12 | 0 | 0
.39 | -12 | 0 | 0
(b) result of FOP

| $\epsilon$ | 1-$2\epsilon$ | $\epsilon$
---|---|---|---
$\epsilon$ | -8.5 | -8.5 | -8.5
1-$2\epsilon$ | -8.5 | 0.2 | 0.2
$\epsilon$ | -8.5 | 0.2 | 0.2
(c) result of QMIX

###  5.2 Multi-Agent Particle Environment

In this section, we test our algorithm on three custumed Multi-Agent Particle
Environment(MPE) [18] scenarios and show that Soft-QMIX do better exploration
compare to QMIX and ablate through different choice of $\alpha$. Specifically,
we use three different Spread map, where in each map, agents are tasked to
navigate to all the landmarks without collide with each other. Harder map
means longer initial distance between agents and landmarks. Details are
available in Appendix I. Table 3 shows that Soft-QMIX significantly
outperforms QMIX on the Hard and Medium map which require more exploration,
but shows only a slight advantage on the Simple map. Additionally, we find
that higher $\alpha$ values enhance exploration and performance. These
findings support our methodological claim that integrating maximum entropy RL
with MARL algorithms boosts exploration capabilities. For each scenario, we
conduct four experiments across five seeds, reporting the average evaluation
episode rewards after 10M environment steps and their standard deviation in
our results table.

Table 3: Experiment result in three MPE scenarios, where Soft-QMIX has higher performance gain from QMIX on Hard and Medium map, which need more exploration. Higher $\alpha$ also lead to better exploration and higher performance gain. eval episode | QMIX | Soft-QMIX(ours)
---|---|---
rewards |  | $\alpha=0.1$ | $\alpha=0.3$ | $\alpha=1.0$
Spread(Hard) | -171.7 $\pm$ 3.8 | -173.7 $\pm$ 3.0 | -170.8 $\pm$ 3.1 | -168.7 $\pm$ 4.6 (+1.8%)
Spread(Medium) | -138.4 $\pm$ 2.9 | -139.0 $\pm$ 2.0 | -137.4 $\pm$ 3.3 | -134.9 $\pm$ 1.8 (+2.5%)
Spread(Simple) | -111.9 $\pm$ 1.6 | -116.2 $\pm$ 2.7 | -114.4 $\pm$ 2.0 | -111.4 $\pm$ 0.9 (+0.4%)

###  5.3 SMAC-v2

We evaluated Soft-QMIX on the SMACv2 [3] benchmark, including six scenarios:
protoss_5_vs_5, terran_5_vs_5, zerg_5_vs_5, protoss_10_vs_10, terran_10_vs_10,
and zerg_10_vs_10. The results, as shown in Figure 3, were obtained by
training each scenario for $10$ million environmental steps, with each curve
representing the average of 5 seeds, and HASAC with an average of 3 seeds.
Soft-QMIX outperforms the baseline algorithms in all scenarios. MAPPO, being
an on-policy algorithm, has lower sampling efficiency, resulting in lower win
rates for models trained under 10M environmental steps. Training with MAPPO
for 60M environmental steps can achieve win rates comparable to 10M QMIX.
QMIX, lacking exploration, tends to converge to local optima, especially in
the two Zerg scenarios. HASAC and FOP perform poorly in all the scenarios.
Ultimately, Soft-QMIX improves win rates by $5\%$ to $24\%$ in all scenarios
compared to baseline algorithms.

Figure 3: Experiment result in six SMAC-v2 scenarios, where Soft-QMIX achieved
faster convergence speed and higher performance compared to baseline
algorithms.

###  5.4 Ablation Study

We evaluate the effectiveness of different components in our algorithm through
an ablation study. We begin with QMIX, which utilizes epsilon-greedy
exploration. We then introduce +entropy, which incorporates softmax sampling
(employing $\text{softmax}(Q_{i})$ as the policy) along with a maximum entropy
objective function to the QMIX framework. Note that this version of the
algorithm does not ensure convergence, leading to a performance decline in two
out of three scenarios. The subsequent modification, +function f, integrates
the function $f_{i}$ (using $\text{softmax}(f_{i}(Q_{i})$) as the policy)
building upon the +entropy setup. Empirical results show that function f
yields the most substantial performance improvement across all components. Our
study culminates with Soft-QMIX, our finalized algorithm. For each
modification, we conduct four experiments across three seeds, reporting the
average winning rate and its standard deviation in our results table.

Table 4: The ablation studies reveal that each component enhances performance, with function $f$ contributing most significantly to the performance improvement. eval win rate | QMIX | +entropy | +function f | Soft-QMIX(ours)
---|---|---|---|---
protoss 5vs5 | 0.68 $\pm$ 0.04 | 0.73 $\pm$ 0.05 | 0.74 $\pm$ 0.04 | 0.76 $\pm$ 0.04
terran 5vs5 | 0.68 $\pm$ 0.05 | 0.67 $\pm$ 0.04 | 0.70 $\pm$ 0.05 | 0.72 $\pm$ 0.04
zerg 5vs5 | 0.41 $\pm$ 0.05 | 0.40 $\pm$ 0.04 | 0.48 $\pm$ 0.05 | 0.51 $\pm$ 0.04

##  6 Conclusion

In this paper, we introduced Soft-QMIX, an innovative approach within the CTDE
framework, integrating the maximum entropy RL framework with the QMIX-based
credit assignment mechanism. Theoretically, we established the monotonic
improvement and convergence of Soft-QMIX to a globally optimal solution.
Experimentally, our algorithm can solve matrix games and demonstrate state-of-
the-art performance on the SMAC-v2 benchmark. Currently, Soft-QMIX is a value-
based method, thus one of its limitations is that it can currently only be
deployed in tasks with discrete action spaces. In the future, we will try to
extend it to tasks with continuous action spaces.

## References

  * Ahmed et al. [2019] Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans.  Understanding the impact of entropy on policy optimization.  In _International conference on machine learning_ , pages 151–160. PMLR, 2019.
  * Chang et al. [2003] Y.-H. Chang, T. Ho, and L. Kaelbling.  All learning is local: Multi-agent learning in global reward games.  _Advances in neural information processing systems_ , 16, 2003.
  * Ellis et al. [2022] B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson.  Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.  _arXiv preprint arXiv:2212.07489_ , 2022.
  * Eysenbach and Levine [2021] B. Eysenbach and S. Levine.  Maximum entropy rl (provably) solves some robust rl problems.  _arXiv preprint arXiv:2103.06257_ , 2021.
  * Foerster et al. [2018] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson.  Counterfactual multi-agent policy gradients.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Guo and Wu [2022] F. Guo and Z. Wu.  Learning maximum entropy policies with qmix in cooperative marl.  In _2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI)_ , pages 357–361. IEEE, 2022.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * He et al. [2022] Z. He, L. Dong, C. Song, and C. Sun.  Multiagent soft actor-critic based hybrid motion planner for mobile robots.  _IEEE transactions on neural networks and learning systems_ , 2022.
  * Hu et al. [2021] J. Hu, S. Jiang, S. A. Harding, H. Wu, and S.-w. Liao.  Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning.  _arXiv preprint arXiv:2102.03479_ , 2021.
  * Huang et al. [2021] S. Huang, B. Wang, H. Su, D. Li, J. Hao, J. Zhu, and T. Chen.  Off-policy training for truncated td ($\lambda$) boosted soft actor-critic.  In _Pacific Rim International Conference on Artificial Intelligence_ , pages 46–59. Springer, 2021.
  * Huang et al. [2023] S. Huang, W. Chen, Y. Sun, F. Bie, and W.-W. Tu.  Openrl: A unified reinforcement learning framework.  _arXiv preprint arXiv:2312.16189_ , 2023.
  * Kim et al. [2021] T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun.  Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation.  _arXiv preprint arXiv:2105.08919_ , 2021.
  * Kuba et al. [2021] J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang.  Trust region policy optimisation in multi-agent reinforcement learning.  _arXiv preprint arXiv:2109.11251_ , 2021.
  * Levine [2018] S. Levine.  Reinforcement learning and control as probabilistic inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Lin et al. [2023] F. Lin, S. Huang, T. Pearce, W. Chen, and W.-W. Tu.  Tizero: Mastering multi-agent football with curriculum learning and self-play.  _arXiv preprint arXiv:2302.07515_ , 2023.
  * Liu et al. [2023] J. Liu, Y. Zhong, S. Hu, H. Fu, Q. Fu, X. Chang, and Y. Yang.  Maximum entropy heterogeneous-agent mirror learning.  _arXiv preprint arXiv:2306.10715_ , 2023.
  * Lowe et al. [2017] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Advances in neural information processing systems_ , 30, 2017.
  * Mordatch and Abbeel [2018] I. Mordatch and P. Abbeel.  Emergence of grounded compositional language in multi-agent populations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Oliehoek and Amato [2016] F. A. Oliehoek and C. Amato.  _A concise introduction to decentralized POMDPs_.  Springer, 2016.
  * Peng et al. [2021] B. Peng, T. Rashid, C. Schroeder de Witt, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson.  Facmac: Factored multi-agent centralised policy gradients.  _Advances in Neural Information Processing Systems_ , 34:12208–12221, 2021.
  * Pinsker [1964] M. S. Pinsker.  Information and information stability of random variables and processes.  _Holden-Day_ , 1964.
  * Pu et al. [2021] Y. Pu, S. Wang, R. Yang, X. Yao, and B. Li.  Decomposed soft actor-critic method for cooperative multi-agent reinforcement learning.  _arXiv preprint arXiv:2104.06655_ , 2021.
  * Rashid et al. [2020] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson.  Monotonic value function factorisation for deep multi-agent reinforcement learning.  _The Journal of Machine Learning Research_ , 21(1):7234–7284, 2020.
  * Schulman et al. [2015] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015.
  * Schulman et al. [2017a] J. Schulman, X. Chen, and P. Abbeel.  Equivalence between policy gradients and soft q-learning.  _arXiv preprint arXiv:1704.06440_ , 2017a.
  * Schulman et al. [2017b] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017b.
  * Son et al. [2019] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi.  Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.  In _International conference on machine learning_ , pages 5887–5896. PMLR, 2019.
  * Sunehag et al. [2017] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al.  Value-decomposition networks for cooperative multi-agent learning.  _arXiv preprint arXiv:1706.05296_ , 2017.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tan [1993] M. Tan.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  In _Proceedings of the tenth international conference on machine learning_ , pages 330–337, 1993.
  * Wang et al. [2020a] J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang.  Qplex: Duplex dueling multi-agent q-learning.  _arXiv preprint arXiv:2008.01062_ , 2020a.
  * Wang et al. [2021] J. Wang, Z. Ren, B. Han, J. Ye, and C. Zhang.  Towards understanding cooperative multi-agent q-learning with value factorization.  _Advances in Neural Information Processing Systems_ , 34:29142–29155, 2021.
  * Wang et al. [2020b] Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang.  Dop: Off-policy multi-agent decomposed policy gradients.  In _International conference on learning representations_ , 2020b.
  * Yu et al. [2022] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu.  The surprising effectiveness of ppo in cooperative multi-agent games.  _Advances in Neural Information Processing Systems_ , 35:24611–24624, 2022.
  * Zhang et al. [2024] M. Zhang, W. Tong, G. Zhu, X. Xu, and E. Q. Wu.  Sqix: Qmix algorithm activated by general softmax operator for cooperative multiagent reinforcement learning.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2024.
  * Zhang et al. [2021] T. Zhang, Y. Li, C. Wang, G. Xie, and Z. Lu.  Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pages 12491–12500. PMLR, 2021.
  * Zhong et al. [2023] Y. Zhong, J. G. Kuba, X. Feng, S. Hu, J. Ji, and Y. Yang.  Heterogeneous-agent reinforcement learning, 2023.
  * Zhou et al. [2022] H. Zhou, T. Lan, and V. Aggarwal.  Pac: Assisted value factorization with counterfactual predictions in multi-agent reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35:15757–15769, 2022.

##  Appendix A Soft-QMIX

Algorithm 1 Soft-QMIX

1: Input: $\theta,\phi$

2: Initiate: $\bar{\theta}\leftarrow\theta,D\leftarrow\emptyset$

3: for each iteration do

4: for each environment step do

5: get $\pi_{i}(u^{i}_{t}|o^{i}_{t},s_{t})$ via Eq. 15

6: $u^{i}_{t}\sim\pi_{i}(u^{i}_{t}|o^{i}_{t},s_{t}),\forall i\in\\{1,..,n\\}$

7: $s_{t+1}\sim p(s_{t+1}|s_{t},\mathbf{u}_{t})$

8: $D\leftarrow
D\cup\\{(s_{t},\mathbf{u}_{t},r(s_{t},\mathbf{u}_{t}),s_{t+1})\\}$

9: end for

10: for each gradient step do

11: calculate $\mathcal{T}^{\pi_{jt}}_{\lambda}Q_{tot}$ via Eq. 24

12: $\theta\leftarrow\theta-\lambda_{Q}\nabla_{\theta}L_{Q}(\theta)$

13: $\phi\leftarrow\phi-\lambda_{\pi}\nabla_{\phi}L_{\pi}(\phi)$

14: $\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}$

15: end for

16: end for

##  Appendix B Individual-Global-Independence (IGI)

In this section, we first introduce two concepts, IGI and IGO, where IGI is
proposed in our work, and IGO was proposed by A. We then explain that IGI is a
sufficient condition for IGO.

######  Definition B.1.

Individual-Global-Independence (IGI) For joint policy $\pi_{jt}$, if their
exist individual policies $\left[\pi_{i}\right]_{i=1}^{n}$ such that the
following holds:

$\pi_{jt}(\mathbf{u}|s)=\prod_{i=1}^{n}\pi_{i}(u^{i}|o^{i}).$ (16)

Then, we say that $[\pi_{i}]_{i=1}^{n}$ satisfies IGI for $\pi_{jt}$ under
$s_{t}$.

######  Definition B.2.

Individual-Global-Optimal (IGO) For optimal joint policy $\pi^{*}_{jt}$, if
their exist individual optimal policies $\left[\pi^{*}_{i}\right]_{i=1}^{n}$
such that the following holds:

$\pi^{*}_{jt}(\mathbf{u}|s)=\prod_{i=1}^{n}\pi^{*}_{i}(u^{i}|o^{i}).$ (17)

Then, we say that $[\pi_{i}]_{i=1}^{n}$ satisfies IGO for $\pi_{jt}$ under
$s_{t}$.

IGO defines that the optimal joint policy equals the product of optimal local
policies. In IGI, it is defined that any joint policy should equal the product
of local policies. If finding the optimal policy is viewed as an optimization
problem, IGI defines the feasible set of this problem. Hence, any policy not
equaling the product of local policies is not a feasible solution and thus
cannot be optimal. Therefore, all optimal policies will satisfy the joint
policy equals the product of local optimal policies, thereby meeting the IGO
condition.

Note that under the assumption of a decentralized execution setting, IGI and
IGO are equivalent.

##  Appendix C Joint Policy Iteration

######  Lemma C.1 (Joint Soft Policy Evaluation).

Let $Q^{0}_{tot}:S\times\mathbf{U}\to\mathbb{R}$ be a bounded mapping where
$|\mathbf{U}|<\infty$. Define the iterative process
$Q^{k+1}_{tot}=\mathcal{T}^{\pi_{jt}}_{\lambda}(Q^{k}_{tot})$. It follows that
the sequence $\\{Q^{k}_{tot}\\}_{k=0}^{\infty}$ converges to the joint soft-Q-
function for policy $\pi_{jt}$ as $k\rightarrow\infty$.

###### Proof.

Initiate by defining the entropy-augmented reward function:

$r_{\pi_{jt}}(s_{t},\mathbf{u}_{t})\coloneqq
r(s_{t},\mathbf{u}_{t})+\mathbb{E}_{s_{t+1}}\left[H[\pi_{jt}(\cdot|s_{t+1})]\right].$
(18)

Follow previous work [9], we choose a lower $\lambda$ and can assume that
$\mathcal{T}^{\pi_{jt}}_{\lambda}$ is an unbiased estimator of
$\mathcal{T}^{\pi_{jt}}$. Subsequently, the update mechanism is articulated
as:

$\displaystyle Q_{tot}(s_{t},\mathbf{u}_{t})$ $\displaystyle\leftarrow
r_{\pi_{jt}}(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1},\mathbf{u}_{t+1}\sim\pi_{jt}}[Q_{tot}(s_{t+1},\mathbf{u}_{t+1})].$
(19)

Conclude by invoking the established convergence theorems pertinent to policy
evaluation as explicated by Sutton and Barto. ∎

######  Lemma C.2 (Joint Soft Policy Improvement).

Suppose $\pi_{jt}^{\text{old}}\in\Pi$ and consider $\pi_{jt}^{\text{new}}$ to
be the solution to Equation 9. Then
$Q^{\pi_{jt}^{\text{new}}}(s_{t},\mathbf{u}_{t})\geq
Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})$ for all
$(s_{t},\mathbf{u}_{t})\in S\times\mathbf{U}$ with $|\mathbf{U}|<\infty$.

###### Proof.

Let $\pi_{jt}^{\text{old}}\in\Pi$ and let $Q^{\pi_{jt}^{\text{old}}}$ and
$V^{\pi_{jt}^{\text{old}}}$ be the corresponding soft state-action value and
soft state value. The update rule of $\pi_{\text{new}}$ can be defined as

$\displaystyle\pi_{\text{new}}(\cdot\mid s_{t})$
$\displaystyle=\arg\min_{\pi^{\prime}\in\Pi}\text{D}_{KL}\left(\pi^{\prime}(\cdot)\parallel\exp(Q^{\pi_{jt}^{\text{old}}}(s_{t},\cdot))-\log
Z^{\pi_{jt}^{\text{old}}}(s_{t})\right)$ (20)
$\displaystyle=\arg\min_{\pi^{\prime}\in\Pi}J_{\pi_{jt}^{\text{old}}}\left(\pi^{\prime}(\cdot\mid
s_{t})\right).$

Since we can always choose $\pi_{\text{new}}=\pi_{\text{old}}\in\Pi$, the
following inequality always hold true:
$J_{\pi_{jt}^{\text{old}}}(\pi_{\text{new}}(\cdot\mid s_{t}))\leq
J_{\pi_{jt}^{\text{old}}}(\pi_{jt}^{\text{old}}(\cdot\mid s_{t}))$. Hence

$\displaystyle\mathbb{E}_{\mathbf{u}_{t}\sim\pi_{jt}^{\text{new}}}\left[\log(\pi_{jt}^{\text{new}}(\mathbf{u}_{t}\mid
s_{t}))-Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})+\log
Z^{\pi_{jt}^{\text{old}}}(s_{t})\right]$ (21)
$\displaystyle\leq\mathbb{E}_{\pi_{jt}^{\text{old}}}\left[\log(\pi_{jt}^{\text{old}}(\mathbf{u}_{t}\mid
s_{t}))-Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})+\log
Z^{\pi_{jt}^{\text{old}}}(s_{t})\right],$

and the inequality reduces to the following form since partition function
$Z^{\pi_{jt}^{\text{old}}}$ depends only on the state,

$\mathbb{E}_{\mathbf{u}_{t}\sim\pi_{jt}^{\text{new}}}\left[Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})-\log\pi_{jt}^{\text{new}}(\mathbf{u}_{t}\mid
s_{t})\right]\geq V^{\pi_{jt}^{\text{old}}}(s_{t}).$ (22)

Next, consider the soft Bellman equation:

$\displaystyle Q^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})$
$\displaystyle=r(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p}\left[V^{\pi_{jt}^{\text{old}}}(s_{t+1})\right]$ (23) $\displaystyle\leq
r(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}Q^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})-\log\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}\mid
s_{t+1})\big{]}\big{]}$ $\displaystyle\vdots$ $\displaystyle\leq
Q^{\pi_{jt}^{\text{new}}}(s_{t},\mathbf{u}_{t}),$

where we can repeatedly expand $Q^{\pi_{jt}^{\text{old}}}$ on the RHS by
applying the soft Bellman equation and the bound in Equation 23. Convergence
to $Q^{\pi_{jt}^{\text{new}}}$ follows from Lemma B.1.

∎

######  Theorem C.3 (Joint Soft Policy Iteration).

By continually applying joint soft policy evaluation and joint soft policy
improvement to any $\pi_{jt}\in\Pi$ converges to a policy $\pi^{*}_{jt}$ such
that $Q^{\pi^{*}_{jt}}(s_{t},\mathbf{u}_{t})\geq
Q^{\pi_{jt}}(s_{t},\mathbf{u}_{t})$ for all $\pi_{jt}\in\Pi$ and
$(s_{t},\mathbf{u}_{t})\in S\times\mathbf{U}$, assuming $|\mathbf{U}|<\infty$.

###### Proof.

Let $\pi_{jt}^{i}$ represent the policy after the $i$-th update. By Lemma B.2,
the sequence $Q^{\pi_{jt}^{i}}$ is monotonically increasing. Given that
$Q^{\pi}$ is bounded above for $\pi_{jt}\in\Pi$ due to bounded rewards and
entropy terms, the sequence converges to some $\pi^{*}_{jt}$. It remains to
demonstrate that the policy $\pi^{*}_{jt}$ is indeed the optimal policy. At
convergence, we infer that $J_{\pi^{*}_{jt}}(s)\geq J_{\pi_{jt}}(s)$ for all
$\pi_{jt}\in\Pi$, if $\pi_{jt}\neq\pi^{*}_{jt}$. Using the same iterative
argument as in the proof of Lemma B.2, we get
$Q^{\pi^{*}_{jt}}(s,\mathbf{u})\geq Q^{\pi}(s,a)$ for all $(s,\mathbf{u})\in
S\times\mathbf{U}$, that is, the soft value of any other policy in $\Pi$ is
lower than that of the converged policy. Hence $\pi^{*}_{jt}$ is optimal in
$\Pi$. ∎

##  Appendix D TD($\lambda$)

In this paragraph, we will derive the expression of TD($\lambda$) in the
context of Maximum Entropy Reinforcement Learning. Specifically, we aim to
demonstrate the derivation of the following equation:

$\displaystyle\mathcal{T}^{\pi_{jt}}_{\lambda}Q_{tot}(s_{t},\mathbf{u_{t}})-Q_{tot}(s_{t},\mathbf{u_{t}})=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta^{V}_{t+l}$
(24)

where $\delta_{t}^{V}=-Q_{tot}(s_{t},\mathbf{u}_{t})+r_{t}+\gamma
V_{tot}(s_{t+1})$.

###### Proof.

The one-step TD error can be expressed in the following form:

$\displaystyle\delta_{t}^{V}=-Q_{tot}(s_{t},\mathbf{u}_{t})+r_{t}+\gamma
V_{tot}(s_{t+1})$ (25)
$\displaystyle\delta_{t+1}^{V}=-Q_{tot}(s_{t+1},\mathbf{u}_{t+1})+r_{t+1}+\gamma
V_{tot}(s_{t+2})$

Next, the k-step TD error can be represented in the following form:

$\displaystyle\epsilon_{t}^{(1)}:=-Q_{tot}(s_{t},\mathbf{u}_{t})+r_{t}+\gamma
V_{tot}(s_{t+1})=\delta^{V}_{t}$ (26)
$\displaystyle\epsilon_{t}^{(2)}:=-Q_{tot}(s_{t},\mathbf{u}_{t})+r_{t}+\gamma
r_{t+1}{\color[rgb]{1,0,0}-\log\pi_{jt}(\mathbf{u_{t+1}}|s_{t+1})}+\gamma^{2}V_{tot}(s_{t+1})=\delta^{V}_{t}+\gamma\delta^{V}_{t+1}$
$\displaystyle\epsilon_{t}^{(k)}=\sum_{l=0}^{k-1}\gamma^{l}\delta^{V}_{t+l}$

$\displaystyle\epsilon_{t}^{td(\lambda)}:=(1-\lambda)\left(\epsilon_{t}^{(1)}+\lambda\epsilon_{t}^{(2)}+\lambda^{2}\epsilon_{t}^{(3)}+\cdots\right)$
(27)
$\displaystyle=(1-\lambda)\big{(}\delta_{t}^{V}+\lambda(\delta_{t}^{V}+\gamma\delta_{t+1}^{V})+\lambda^{2}(\delta_{t}^{V}+\gamma\delta_{t+1}^{V}+\gamma^{2}\delta_{t+2}^{V})+\cdots\big{)}$
$\displaystyle=(1-\lambda)\left(\delta_{t}^{V}(1+\lambda+\lambda^{2}+\cdots)+\gamma\delta_{t+1}^{V}(\lambda+\lambda^{2}+\cdots)\right.\left.+\gamma^{2}\delta_{t+2}^{V}(\lambda^{2}+\lambda^{3}+\lambda^{4}+\cdots)+\cdots\right)$
$\displaystyle=(1-\lambda)\left(\delta_{t}^{V}\left(\frac{1}{1-\lambda}\right)+\gamma\delta_{t+1}^{V}\left(\frac{\lambda}{1-\lambda}\right)+\cdots\right)$
$\displaystyle=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+l}^{V}$

∎

It is important to note that when computing the TD lambda error in the context
of a reverse view and maximum entropy reinforcement learning, it is necessary
to subtract the $\log(\pi)$ term from the $t+1$ step’s return as well as
subtract $\log(\pi)$ from $Q_{t+1}$ in order to obtain results consistent with
Equation 27.

##  Appendix E Soft-Q Learning and Policy Gradient

######  Theorem E.1.

If $Q$-function is parameterized as $Q_{\theta}$, and
$V_{\theta},\pi_{\theta}$ are defined as below:

$\displaystyle V_{\theta}(s)$
$\displaystyle\coloneq\alpha\log\mathbb{E}_{\mathbf{u}\sim\pi_{jt}}\left[\exp\left(Q_{\theta}(s,\mathbf{u})/\alpha\right)\right],$
(28) $\displaystyle\pi_{jt_{\theta}}(\mathbf{u}|s)$
$\displaystyle\coloneq\exp\left(Q_{\theta}(s,\mathbf{u})-V_{\theta}(s)\right)/\alpha.$

and $y_{t}$ is an unbiased estimator of
$\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}Q_{\theta}(s_{t},\mathbf{u_{t}})\right].$
Then, we have

$\displaystyle\nabla_{\theta}\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}$
$\displaystyle\left[\|Q_{\theta}(s,\mathbf{u})-y\|^{2}\right]$ (29)
$\displaystyle=\alpha\nabla_{\theta}D_{KL}\big{(}\pi_{jt}(\cdot|s)\|\frac{exp(y(\cdot,s)/\alpha)}{Z(s)}\big{)}$
$\displaystyle+\nabla_{\theta}\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\|V_{\theta}(s)-y\|^{2}\right]$

###### Proof.

The proof is heavily based on [25]. From assumptions described in Eq. 11 we
have:

$\displaystyle
Q_{\theta}(s,\mathbf{u})=V_{\theta}(s)+\alpha\log\pi_{jt_{\theta}}(\mathbf{u}|s)$
(30)

For simplicity, we further define notation $\Delta_{t}$ as below.

$\displaystyle\Delta_{t}$
$\displaystyle=\mathcal{T}^{\pi_{jt}}Q_{\theta}(s_{t},\mathbf{u_{t}})-Q_{\theta}(s_{t},\mathbf{u_{t}}).$

From assumption, we have $y_{t}$ is an unbiased estimator of
$\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}Q_{\theta}(s_{t},\mathbf{u_{t}})\right]$.
Thus, the relationship between $\Delta_{t}$ and $y_{t}$ can be writen as:

$\displaystyle\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}[y_{t}]$
$\displaystyle=\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}Q_{\theta}(s_{t},\mathbf{u_{t}})\right]$
(31)
$\displaystyle=\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}[Q_{\theta}(s_{t},\mathbf{u_{t}})+\Delta_{t}]$
$\displaystyle=\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt}}[V_{\theta}(s_{t})+\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]+\Delta_{t}].$

Here are two equations that will be used in the following proof:

$\displaystyle\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt_{\theta}}}\big{[}\big{(}\nabla_{\theta}V_{\theta}(s)\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u}|s)\big{)}\big{]}$
(32) $\displaystyle=$
$\displaystyle\mathbb{E}_{s,\mathbf{u}\sim\pi_{jt_{\theta}}}\big{[}\big{(}\nabla_{\theta}V_{\theta}(s)\big{)}\big{(}\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u}|s)]\big{)}\big{]}.$

$\displaystyle\mathbb{E}_{a\sim\pi}\big{[}\nabla\log\pi(a)f(s)\big{]}=f(s)\sum_{a}\pi(a)\frac{\nabla\pi(a)}{\pi(a)}$
(33) $\displaystyle=f(s)\sum_{a}\nabla\pi(a)=f(s)\nabla\sum_{a}\pi(a)=0,$

where $f$ can be any function that does not conditioned on $a$, and
$\sum_{a}\pi(a)=1$. Let’s consider the gradient of the $Q$-function’s MSE
loss:

$\displaystyle\nabla_{\theta}\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\left[\frac{1}{2}\left(Q_{\theta}(s_{t},\mathbf{u_{t}})-y_{t}\right)^{2}\right]\bigg{|}_{\pi=\pi_{\theta}}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\left[\nabla_{\theta}Q_{\theta}(s_{t},\mathbf{u_{t}})(Q_{\theta}(s_{t},\mathbf{u_{t}})-y_{t})\right]\bigg{|}_{\pi=\pi_{\theta}}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\left[\nabla_{\theta}Q_{\theta}(s_{t},\mathbf{u_{t}})\left(V_{\theta}(s_{t})+\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})-y_{t}\right)\right]$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}\nabla_{\theta}Q_{\theta}(s_{t},\mathbf{u_{t}})\big{(}V_{\theta}(s_{t})+\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})-V_{\theta}(s_{t})-\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]-\Delta_{t}\big{)}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}\nabla_{\theta}Q_{\theta}(s_{t},\mathbf{u_{t}})\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})-\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]-\Delta_{t}\big{)}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}\big{(}\nabla_{\theta}V_{\theta}(s_{t})+\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u}|s)\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})-\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]-\Delta_{t}\big{)}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}\big{(}\nabla_{\theta}V_{\theta}(s_{t})\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}-\big{(}\nabla_{\theta}V_{\theta}(s_{t})\big{)}\big{(}\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]\big{)}$
$\displaystyle\qquad-\big{(}\nabla_{\theta}V_{\theta}(s_{t})\big{)}\Delta_{t}+\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}$
$\displaystyle\qquad-\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\big{(}\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]\big{)}-\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\Delta_{t}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}-\big{(}\nabla_{\theta}V_{\theta}(s_{t})\big{)}\Delta_{t}+\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}-$
$\displaystyle\qquad\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\big{(}\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})]\big{)}-\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\Delta_{t}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}-\big{(}\nabla_{\theta}V_{\theta}(s_{t})\big{)}\Delta_{t}+\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\big{(}\alpha\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}-\big{(}\alpha\nabla_{\theta}\log\pi_{jt_{\theta}}(\mathbf{u_{t}}|s_{t})\big{)}\Delta_{t}\big{]}.$

for simplicity, we ignore the $|_{\pi=\pi_{\theta}}$ after the first two lines
of equations. The second equation use Eq. 30. The third equation use Eq. 31.
The fourth equation cancled out $V_{\theta}(s_{t})$. The fifth equation use
Eq. 30 again. The sixth equation expand out terms. The seventh equation use
Eq. 32. The last equation use Eq. 33.

Let’s consider the gradient of the KL divergence.

$\displaystyle\nabla_{\theta}\alpha
D_{KL}\big{(}\pi_{jt}(\cdot|s)\|\frac{exp(y(\cdot,s)/\alpha)}{Z(s)}\big{)}$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{\pi_{jt}}\big{[}\alpha\log\pi_{jt}(\mathbf{u}|s)-y(\mathbf{u},s)+\alpha\log
Z(s)\big{]}$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{\pi_{jt}}\big{[}\alpha\log\pi_{jt}(\mathbf{u}|s)-y(\mathbf{u},s)\big{]}$
$\displaystyle=\nabla_{\theta}[\alpha\sum_{\mathbf{u}}\pi_{jt}(\mathbf{u}|s)\log\pi_{jt}(\mathbf{u}|s)]-\nabla_{\theta}[\sum_{\mathbf{u}}\pi_{jt}(\mathbf{u}|s)y(\mathbf{u},s)]$
$\displaystyle=\alpha\sum_{\mathbf{u}}\nabla_{\theta}\pi_{jt}(\mathbf{u}|s)\log\pi_{jt}(\mathbf{u}|s)-\sum_{\mathbf{u}}\nabla_{\theta}\pi_{jt}(\mathbf{u}|s)y(\mathbf{u},s)$
$\displaystyle=\mathbb{E}_{\pi_{jt}}\big{[}\alpha\big{(}\nabla_{\theta}\log\pi_{jt}(\mathbf{u}|s)\big{)}\log\pi_{jt}(\mathbf{u}|s)-\big{(}\nabla_{\theta}\log\pi_{jt}(\mathbf{u}|s)\big{)}y(\mathbf{u},s)\big{]}.$

Note that
$V_{\theta}(s)+\alpha\mathbb{E}_{\mathbf{u}\sim\pi_{jt}(\cdot|s)}[\log\pi_{jt_{\theta}}(\mathbf{u}|s)]$
is not a function of $\mathbf{u}$. With Eq. 33, we have:

$\displaystyle\nabla_{\theta}\alpha
D_{KL}\big{(}\pi_{jt}(\cdot|s)\|\frac{exp(y(\cdot,s)/\alpha)}{Z(s)}\big{)}$
(34)
$\displaystyle=\mathbb{E}_{\pi_{jt}}\big{[}\alpha\big{(}\nabla_{\theta}\log\pi_{jt}(\mathbf{u}|s)\big{)}\log\pi_{jt}(\mathbf{u}|s)-\big{(}\nabla_{\theta}\log\pi_{jt}(\mathbf{u}|s)\big{)}\Delta_{t}\big{]}$

Finally, we got:

$\displaystyle\nabla_{\theta}\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\left[\frac{1}{2}\left(Q_{\theta}(s_{t},\mathbf{u_{t}})-y_{t}\right)^{2}\right]\bigg{|}_{\pi=\pi_{\theta}}$
$\displaystyle=\nabla_{\theta}\alpha^{2}D_{KL}\big{(}\pi_{jt}(\cdot|s_{t})\|\frac{exp(y(s)/\alpha)}{Z(s)}\big{)}+\frac{1}{2}\nabla_{\theta}\mathbb{E}_{s_{t},\mathbf{u_{t}}\sim\pi_{jt_{\theta}}}\big{[}(V_{\theta}(s_{t})-y_{t})^{2}\big{]},$

where $y(s)$ is the distribution of $y_{t}$ at state $s$.

∎

##  Appendix F Unbiased Assumption

We assume that $\mathcal{T}^{\pi_{jt}}Q_{tot}(s_{t},\mathbf{u_{t}})$ is an
unbiased estimate of
$E_{s,\mathbf{u}\sim\pi_{jt}}\left[\mathcal{T}^{\pi_{jt}}\sum_{i}f_{i}(Q_{i}(s_{t},\mathbf{u_{t}}))\right]$
in the main paper. Practically, this assumption might fail, leading to non-
monotonic increases in the policy improvement step. Here, we initially explain
the impact of this error and derive an upper bound for it.

Conclusion: the error’s upper bound is proportional to
$|Q_{tot}-\sum_{i}f_{i}(Q_{i})|$ and
$\sqrt{D_{KL}\big{(}\pi_{jt}^{\text{old}}|\pi_{jt}^{\text{new}}\big{)}}.$ We
use Equation 14 as the loss function, ensuring that the first term decreases
throughout training. Additionally, by selecting a sufficiently small learning
rate, the second term will also remain minimal.

Derivative: We can replace both $Q_{\theta}$ and $y_{t}$ in Theorem 4.2 with
$\sum_{i}f_{i}(Q_{i})$, and get that minimizing the L2 norm described in
Equation 10 is equivalent to minimizing the td loss of value function and
minimizing the KL divergence described as follow,

$\pi_{jt}^{\text{new}}=\underset{\pi^{\prime}_{jt}\in\Pi}{\mathrm{arg\,min}}\
D_{KL}\left(\pi^{\prime}_{jt}(\cdot|s_{t})\middle\|\frac{\exp(\sum_{i}f_{i}(Q_{i}^{\pi_{\text{old}}}(o^{i}_{t},\cdot)/\alpha)}{Z^{\pi_{\text{old}}}(s_{t})}\right),$
(35)

where
$Z^{\pi_{\text{old}}}(s_{t})=\int_{\mathbf{u}}\exp(\sum_{i}f_{i}(Q_{i}^{\pi_{\text{old}}}(o^{i}_{t},u^{i}))/\alpha)d\mathbf{u}$
and the value function is define as
$V^{\prime}(s)=\sum_{i}f_{i}(Q_{i}(o^{i},u^{i}))-\alpha\log\pi_{jt}(\mathbf{u}|s).$

Similar to Equation 22, during policy improvement step, we have:

$\mathbb{E}_{\mathbf{u}_{t}\sim\pi_{jt}^{\text{new}}}\left[\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t},u^{i}_{t}))-\log\pi_{jt}^{\text{new}}(\mathbf{u}_{t}\mid
s_{t})\right]\leq V^{\prime\pi_{jt}^{\text{old}}}(s_{t}).$ (36)

During policy evaluation step, we have:

$\displaystyle Q_{tot}^{\pi_{jt}^{\text{old}}}(s_{t},\mathbf{u}_{t})$
$\displaystyle=r(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1}\sim
p}\left[V_{tot}^{\pi_{jt}^{\text{old}}}(s_{t+1})\right]$ (37)
$\displaystyle=r(s_{t},\mathbf{u}_{t})+\gamma\big{(}\mathbb{E}_{s_{t+1}\sim
p}\left[V^{\prime\pi_{jt}^{\text{old}}}(s_{t+1})\right]+\epsilon_{1}\big{)}$
$\displaystyle\leq
r(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1}}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))-\log\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}\mid
s_{t+1})\big{]}\big{]}+\gamma\epsilon_{1}$
$\displaystyle=r(s_{t},\mathbf{u}_{t})+\gamma\mathbb{E}_{s_{t+1}}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})-\log\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}\mid
s_{t+1})\big{]}\big{]}+\gamma(\epsilon_{1}+\epsilon_{2})$
$\displaystyle\vdots$ $\displaystyle\leq
Q_{tot}^{\pi_{jt}^{\text{new}}}(s_{t},\mathbf{u}_{t})+\epsilon,$

where $\epsilon$ is s the cumulative sum of $\epsilon_{1}$ and $\epsilon_{2}$
over time.$\epsilon_{1}$ and $\epsilon_{2}$ are defined as follows:

$\displaystyle\epsilon_{1}$ $\displaystyle=\mathbb{E}_{s_{t+1}\sim
p}\left[V_{tot}^{\pi_{jt}^{\text{old}}}(s_{t+1})-V^{\prime\pi_{jt}^{\text{old}}}(s_{t+1})\right]$
(38) $\displaystyle\epsilon_{2}$ $\displaystyle=\mathbb{E}_{s_{t+1}\sim
p}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))-Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})\big{]}\big{]}$

Then, we have:

$\displaystyle\epsilon_{1}+\epsilon_{2}$
$\displaystyle=\mathbb{E}_{s_{t+1}\sim
p}\left[V_{tot}^{\pi_{jt}^{\text{old}}}(s_{t+1})-V^{\prime\pi_{jt}^{\text{old}}}(s_{t+1})\right]$
(39) $\displaystyle\quad+\mathbb{E}_{s_{t+1}\sim
p}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))-Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})\big{]}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t+1}\sim
p}\left[\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{old}}}\big{[}Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})-\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))\big{]}\right]$
$\displaystyle\quad+\mathbb{E}_{s_{t+1}\sim
p}\big{[}\mathbb{E}_{\mathbf{u}_{t+1}\sim\pi_{jt}^{\text{new}}}\big{[}\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))-Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})\big{]}\big{]}$
$\displaystyle=\mathbb{E}_{s_{t+1}\sim
p}\big{[}\sum_{\mathbf{u}_{t+1}}\big{(}\pi_{jt}^{\text{old}}(\mathbf{u}_{t+1}|s_{t+1})-\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}|s_{t+1})\big{)}\big{(}Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})-\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))\big{)}\big{]}$
$\displaystyle\leq\mathbb{E}_{s_{t+1}\sim
p}\big{[}C\cdot\sum_{\mathbf{u}_{t+1}}\big{(}\pi_{jt}^{\text{old}}(\mathbf{u}_{t+1}|s_{t+1})-\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}|s_{t+1})\big{)}\big{]},$

where
$C=\underset{\mathbf{u}_{t+1}}{\mathrm{arg\,max}}|Q_{tot}^{\pi_{\text{old}}}(s_{t+1},\mathbf{u}_{t+1})-\sum_{i}f_{i}(Q_{i}^{\pi_{jt}^{\text{old}}}(o^{i}_{t+1},u^{i}_{t+1}))|.$
Due to the Pinsker’s inequality [21], we have:

$\displaystyle|\epsilon_{1}+\epsilon_{2}|$
$\displaystyle\leq\mathbb{E}_{s_{t+1}\sim
p}\big{[}C\cdot\sum_{\mathbf{u}_{t+1}}|\pi_{jt}^{\text{old}}(\mathbf{u}_{t+1}|s_{t+1})-\pi_{jt}^{\text{new}}(\mathbf{u}_{t+1}|s_{t+1})|\big{]}$
(40) $\displaystyle\leq\mathbb{E}_{s_{t+1}\sim
p}\big{[}C\cdot\sqrt{2D_{KL}\big{(}\pi_{jt}^{\text{old}}(\cdot|s_{t+1})|\pi_{jt}^{\text{new}}(\cdot|s_{t+1})\big{)}}\big{]}$

##  Appendix G Hyper-network Structure

(a)

(b)

Figure 4: (a) The overall network architecture. Agent 1 to Agent n takes local
observations as input and outputs a distribution over local actions $u^{i}$.
Functions $g_{i}$ and $f_{i}$ are order-preserving transformations, ensuring
the input and output dimensions are identical and have the same argmax values.
Both the input and output of the $g_{i}$ and $f_{i}$ networks are
k-dimensional, where $k$ is the action dimension. The output of the $f_{i}$
function, after undergoing a softmax operation, becomes the policy, from which
action $u_{t}^{i}$ is sampled. The Mixer network takes local $Q_{i}$ as input
and outputs the global $Q_{tot}$. Green blocks utilize a recurrent neural
network as the backbone, with all agents sharing parameters. For decentralized
execution, only the green network is needed, and the maximum values of their
output are selected. (b) The blue blocks’ network structure. The red blocks
represent the hyper-network, which takes the global state $s_{t}$ as input and
outputs the network’s weights and biases. The weights are constrained to be
non-negative. The activation function is denoted by $\sigma(\cdot)$. The mixer
network takes an n-dimensional Q function as input and outputs a one-
dimensional $Q_{tot}$.

Figure 4 shows the overall network structure. Specifically, function $f_{i}$
and function $g_{i}$ share the same network structure as the mixer network,
but while the mixer network’s input dimension is (batch_size, agent_num) and
its output dimension is (batch_size, 1), both the input and output dimensions
of function $f_{i}$ and function $g_{i}$ are (batch_size, 1). Additionally,
function $g_{i}$ introduces an additional residual from input to output. The
network hyperparameters are as shown in the table below.

layer name | hypernet_embed | embed_dim | num_layer
---|---|---|---
mixer | 64 | 32 | 2
function $g_{i}$ | 64 | 32 | 2
function $f_{i}$ | 64 | - | 1
Table 5: Hyper-parameters used for hyper-networks.

##  Appendix H Comparison with Baselines

The following paragraphs present the differences between our method and those
based on the AC:

1\. Using existing maximum entropy MARL algorithms may result in the loss of
action order learned through credit assignment. Achieving the IGM condition,
which outlines the order of local value functions, is crucial in credit
assignment. Our work presents a value-based algorithm that introduces order-
preserving transformations, ensuring that the order of the local actor matches
the order of the local value functions. However, the commonly used maximum
entropy MARL methods, employing the actor-critic framework, may not guarantee
alignment between the actor’s action order and that of the local Q-function
due to approximation errors.

2\. Actor-critic methods utilize the KL divergence as the loss function, while
our approach uses the MSE loss. In the CTDE framework, all methods maintain a
more accurate centralized Q-function alongside multiple imprecise local
policies. The core idea is to distill knowledge from the centralized
Q-function to the local policies. Actor-critic methods typically minimize the
KL divergence between the actor and the softmax critic, whereas our method
minimizes the MSE loss between their logits. [12] pointed out that, in
distillation contexts, MSE is a superior loss function compared to KL
divergence.

3\. AC methods can use global information when training the critic, resulting
in higher model capacity. Our method introduces global information through
hyper-networks and the mixer network, which results in lower model capacity.

4\. In fact, it’s challenging to theoretically prove that value-based methods
are inherently superior to AC methods. However, through experiments, we have
demonstrated that our method surpasses both FOP [36] and HASAC [16].

Below are comparisons with individual papers.

mSAC [22] introduces a method similar to MASAC. It employs the AC
architecture, thus encountering issues 1 and 2 mentioned above, but does not
fully utilize global information to train the critic, hence lacking the
advantage noted in point 3.

FOP [36] shares a similar issue with [22], in that the AC framework it
utilizes can actually train the critic with global state information.
Furthermore, FOP employs a credit assignment mechanism similar to that of
QPLEX [31]. [9] experimentally shown that QMIX outperforms QPLEX.
Additionally, FOP’s proof concept is that under the IGO conditions, if we
locally move local policies towards local optimal policies, the distance
between the joint policy and the optimal joint policy also decreases. In
contrast, our proof approach first establishes that our method is equivalent
to the single-agent SAC algorithm and then completes the proof using existing
conclusions.

HASAC [16] is a CTCE version of MASAC, with its core contribution being the
introduction of sequential decision-making. Therefore, we do not compare it
here.

PAC [38] utilizes global information during centralized training, possessing
the advantages noted in point 3. However, it still encounter issues 1 and 2
mentioned above.

##  Appendix I MPE setup

In the MPE [18] Spread scenario, three agents and three landmarks are present.
Agents work together to cover all landmarks while avoiding collisions. We
created three different maps: Spread(Simple), which matches the original
benchmark’s simple_spread; Spread(Medium) and Spread(Hard), which increase the
initial distances between agents and landmarks by 1.5X and 2X, respectively.

##  Appendix J Hyper-parameters

For all baseline algorithms, we implemented them using the corresponding open-
source frameworks and chose the hyperparameters provided by these frameworks
for replication. We used MAPPO implemented by Huang et al., QMIX by Hu et al.,
FOP by Zhang et al., and HARL by Zhong et al.. Our method, Soft-QMIX, shares
the same code base as QMIX. The experiments were conducted on a computer
equipped with 92 GB of RAM, a 40-core CPU, and a GeForce RTX 2080 Ti GPU. The
following are the hyperparameters used in the experiments.

Hyperparameter | Value
---|---
Learning Rate Actor | 5e-4
Learning Rate Critic | 1e-3
Data Chunk Length | 8
Env Num | 8
Episode Length | 400
PPO Epoch | 5
Actor Train Interval Step | 1
Use Recurrent Policy | True
Use Adv Normalize | True
Use Value Active Masks | False
Use Linear LR Decay | True
Table 6: MAPPO hyper-parameters used for SMACv2. We utilize the hyperparameters used in SMACv2 [3]. Parameter Name | Value
---|---
n_rollout_threads | 8
num_env_steps | 10000000
warmup_steps | 10000
train_interval | 50
update_per_train | 1
use_valuenorm | False
use_linear_lr_decay | False
use_proper_time_limits | True
hidden_sizes | [256, 256]
activation_func | relu
use_feature_normalization | True
final_activation_func | tanh
initialization_method | orthogonal
gain | 0.01
lr | 0.0003
critic_lr | 0.0005
$\text{auto\\_}\alpha$ | True
$\alpha\text{\\_lr}$ | 0.0003
$\gamma$ | 0.99
buffer_size | 1000000
batch_size | 1000
polyak | 0.005
n_step | 20
use_huber_loss | False
use_policy_active_masks | True
share_param | False
fixed_order | False
Table 7: HASAC hyper-parameters used for SMACv2. We use the hyperparameters for SMAC as specified in the original paper [16]. Parameter Name | Value
---|---
Action Selector | epsilon greedy
$\epsilon$-Start | 1.0
$\epsilon$-Finish | 0.05
$\epsilon$-Anneal Time | 1e5 (5e5 for MPE)
Runner | parallel
Batch Size Run | 4
Buffer Size | 5000
Batch Size | 128
Optimizer | Adam
$t_{\max}$ | 1005000
Target Update Interval | 200
Mac | n_mac
Agent | n_rnn
Agent Output Type | q
Learner | nq_learner
Mixer | qmix
Mixing Embed Dimension | 32
Hypernet Embed Dimension | 64
Learning Rate | 0.001
$\lambda$ | 0.4
Table 8: QMIX hyper-parameters used for SMACv2 and MPE. We utilize the hyperparameters used in SMACv2 [3]. Parameter Name | Value
---|---
$\alpha$($5\\_\text{vs}\\_5$) | 0.03
$\alpha$($10\\_\text{vs}\\_10$) | 0.005
Table 9: Soft-QMIX hyper-parameters used for SMACv2. Note that besides the hyperparameters mentioned in the table, all other values for Soft-QMIX are the same as those for QMIX. Parameter Name | Value
---|---
Action Selector | Multinomial
$\epsilon$-Start | 1.0
$\epsilon$-Finish | 0.05
$\epsilon$-Anneal Time | 50000
Buffer Size | 5000
$t_{\max}$ | 1005000
Target Update Interval | 200
Agent Output Type | pi_logits
Learner | fop_learner
Head Number | 4
Mixing Embed Dimension | 32
Learning Rate | 0.0005
Burn In Period | 100
$\lambda$ | 0.4
Table 10: FOP hyper-parameters used for SMACv2. We use the hyperparameters for
SMAC as specified in the original paper [36].

##  Appendix K Impact Statements

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here.