# S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent
Reinforcement Learning?

Shuang Luo<EMAIL_ADDRESS>Zhejiang UniversityHangzhouChina ,  Yinchuan
Li<EMAIL_ADDRESS>Huawei Noah’s Ark LabBeijingChina ,  Jiahui Li
<EMAIL_ADDRESS>Zhejiang UniversityHangzhouChina ,  Kun Kuang
<EMAIL_ADDRESS>Zhejiang UniversityHangzhouChina ,  Furui Liu
<EMAIL_ADDRESS>,  Yunfeng Shao<EMAIL_ADDRESS>Huawei Noah’s Ark
LabBeijingChina and  Chao Wu<EMAIL_ADDRESS>Zhejiang
UniversityHangzhouChina

(2022)

###### Abstract.

Collaborative multi-agent reinforcement learning (MARL) has been widely used
in many practical applications, where each agent makes a decision based on its
own observation. Most mainstream methods treat each local observation as an
entirety when modeling the decentralized local utility functions. However,
they ignore the fact that local observation information can be further divided
into several entities, and only part of the entities is helpful to model
inference. Moreover, the importance of different entities may change over
time. To improve the performance of decentralized policies, the attention
mechanism is used to capture features of local information. Nevertheless,
existing attention models rely on dense fully connected graphs and cannot
better perceive important states. To this end, we propose a _sparse state
based MARL_ (S2RL) framework, which utilizes a sparse attention mechanism to
discard irrelevant information in local observations. The local utility
functions are estimated through the self-attention and sparse attention
mechanisms separately, then are combined into a standard joint value function
and auxiliary joint value function in the central critic. We design the S2RL
framework as a plug-and-play module, making it general enough to be applied to
various methods. Extensive experiments on StarCraft II show that S2RL can
significantly improve the performance of many state-of-the-art methods.

deep learning; multi-agent reinforcement learning; sparse attention

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††price: 15.00††doi:
10.1145/3534678.3539481††isbn: 978-1-4503-9385-0/22/08††ccs: Computing
methodologies Multi-agent reinforcement learning

##  1\. Introduction

Multi-agent reinforcement learning (MARL) provides a framework for multiple
agents to solve complex sequential decision-making problems, with broad
applications including robotics control (Li et al., 2021; Lillicrap et al.,
2016), video gaming (Vinyals et al., 2019; Liu et al., 2019), traffic light
control (Wu et al., 2017) and autonomous driving (Kiran et al., 2022; Cao et
al., 2013). In the paradigm of centralized training with decentralized
execution (CTDE) (Lowe et al., 2017; Rashid et al., 2018), each local agent
models a policy that treats the local observation as input. However, the role
of entities is underestimated by most mainstream methods. Entities are defined
as fine-grained tokens of observations, e.g., _obstacles, landmarks, enemies_
, which determine the inference process of the model. Specifically, they treat
all entities observed as a whole and contribute indiscriminately to the
estimation of the value function. But in some cases, the importance of each
entity changes dynamically over time steps.

(a) Six friendly Hydralisks face 8 enemy Zealots.

(b) Dense attention distribution

(c) Sparse attention distribution

Figure 1. A visualization example of agent performance on the StarCraft II
super-hard scenario $6h\\_vs\\_8z$. As shown in (a), the closest to the green
agent H3 are the red enemies Z0 and Z5. Thus the corresponding policy is that
H3 only needs to focus on Z0 and Z5, which are more likely to be annihilated.
(b) shows the softmax attention distribution of the H3 observations, finding
that some weights are still assigned to irrelevant entities. In contrast, the
sparse attention in (c) only focuses on Z0 and Z5.

To better leverage the observation information, the attention mechanism has
been adopted (Vaswani et al., 2017) for its ability to learn the interaction
relationship among entities and dynamically focus on the crucial parts. Most
existing attention mechanisms compute importance weights based on dense fully-
connected graphs, where all participants are assigned scores according to
their contribution to model decisions. In practice, however, not all entities
are helpful for model inference, and discarding redundant entities can
sometimes improve overall performance. Therefore, it is crucial for agents to
learn to select valuable observations and exclude others. To better illustrate
this phenomenon, a visualization of the StarCraft II scene and the
corresponding attention distribution is shown in Figure 1. The green agent H3
is very close to the red enemies Z0 and Z5. Hence agent H3 focusing only on
enemies Z0 and Z5 is more effective. However, from the traditional dense
attention distribution of H3, we can see that H3 assigns much attention to
irrelevant entities. Note that the large state space brings great difficulties
to policy learning for a more complex MARL environment.

An ideal way to solve this issue is to replace the traditional attention
mechanism with sparse attention. From Figure 1(c), we can see that adopting
sparse attention can well guide H3 only to perceive Z0 and Z5, reducing the
observation space that the agent needs to perceive. However, simply applying
sparse attention to local agents will corrupt the training. The main reason is
that the network cannot distinguish which entity is more important at the
beginning of training. If the agents only focus on critical entities
initially, it may lead to an inadequate exploration of the environment and
thus converge to a suboptimal policy. More specifically, temporarily
discarding some entities can be seen as a policy exploration behavior.
Meanwhile, local policies need to execute their exploration strategies. When
these two strategies are executed simultaneously, it is difficult for the
model to converge.

In this paper, we propose a _Sparse State based MARL_ (S2RL) framework, where
the sparse attention mechanism is utilized as the auxiliary for guiding the
local agent training. In particular, we model the local utility function using
a traditional self-attention mechanism. Then, we construct a corresponding
auxiliary utility function for each agent, which is implemented by a sparse
attention mechanism. The local utility and auxiliary utility functions
respectively form the joint value and auxiliary value functions, which are
further used to train the entire network. Since the sparse attention mechanism
is considered auxiliary and thus does not corrupt the training process, the
auxiliary value function is also used to update the entire framework. To this
end, local agents can learn patterns to focus on essential entities while
ignoring redundancy.

Our main contributions are summarized as follows:

  * •

To the best of our knowledge, this paper is the first attempt that uses
enhanced awareness of crucial states as the auxiliary in MARL to improve
convergence rate and performance.

  * •

We propose the S2RL framework for local agents to perceive crucial states
while preserving all states. The proposed framework thus addresses the
inability to converge using only a small number of partial observations.

  * •

We design the S2RL framework as a plug-and-play module, making it general
enough to be applied to various methods in the CTDE paradigm.

  * •

The extensive experiments on StarCraft II show that S2RL brings remarkable
improvements to existing methods, especially in complicated scenarios.

The remainder of the paper is organized as follows. In Section 2, we introduce
the background of MARL and the CTDE framework. In Section 3, we propose our
S2RL framework. Experimental results are presented in Section 4. Related works
are presented in Section 5. Section 6 concludes the paper.

Figure 2. Illustration of the proposed S2RL method. We use the value-based
MARL framework under the CTDE paradigm and apply the S2RL method to an agent
utility network. The core of S2RL is composed of the dense attention module
and sparse attention module, where sparse attention serves as an auxiliary for
guiding the dense attention training.

##  2\. PRELIMINARIES

###  2.1. Dec-POMDP

A fully cooperative multi-agent sequential task can be described as a
decentralized partially observable Markov decision process (Dec-POMDP)
(Oliehoek and Amato, 2016), which is canonically formulated by the tuple:

(1) $\displaystyle
M=<\mathcal{I},\mathcal{S},\mathcal{U},P,R,\Omega,G,\gamma>.$

In the process, $\mathcal{I}\equiv\\{1,2,\dots,N\\}$ is the finite set of
agents and $s\in\mathcal{S}$ represents the global state of the environment.
At each time step, each agent $i\in\mathcal{I}$ receives an individual partial
observation $o^{i}\in{\Omega}$ according to the observation function $G(s,i)$
and selects an action $u^{i}\in\mathcal{U}$, forming a joint action
$\boldsymbol{u}$. This results in a transition to the next state $s^{\prime}$
according to the state transition function
$P(s^{\prime}|s,\boldsymbol{u}):\mathcal{S}\times\mathcal{U}\times\mathcal{S}\rightarrow\left[0,1\right]$.
All agents share the same global reward $r$ based on the reward function
$R(s,\boldsymbol{u}):\mathcal{S}\times\mathcal{U}\rightarrow\mathbb{R}$, and
$\gamma\in\left[0,1\right)$ is the discount factor. Due to partially
observable setting, each agent has an action-observation history
$\tau^{i}\in\mathcal{T}\equiv(\Omega\times\mathcal{U})^{*}$ and learns its
individual policy $\pi^{i}\left(u^{i}|\tau^{i}\right)$ to jointly maximize the
discounted return. The joint policy $\boldsymbol{\pi}$ induces a joint action-
value function:
$Q^{tot}_{\boldsymbol{\pi}}(s,\boldsymbol{u})=\mathbb{E}_{s_{0:\infty},\boldsymbol{u}_{0:\infty}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\mid
s_{0}=s,\boldsymbol{u}_{0}=\boldsymbol{u},\boldsymbol{\pi}\right]$.

###  2.2. CTDE Framework

The centralized training and decentralized execution (CTDE) is a popular
paradigm used in deep multi-agent reinforcement learning (Foerster et al.,
2016; Rashid et al., 2018; Mao et al., 2020; Yang et al., 2020a; Foerster et
al., 2018), which enables agents to learn their individual policies in a
centralized way. During the centralized training process, the learning model
can access the state and provide global information to assist the agents in
exploring and training. However, each agent only makes decisions based on its
local action-observation history during decentralized execution. The
Individual-Global-Max principle (Son et al., 2019) guarantees the consistency
between joint and local greedy action selections. Agents can obtain the
optimal global reward by maximizing the individual utility function of each
agent. Thus a more robust individual value function can benefit the whole team
in cooperative multi-agent tasks.

The global Q-function $Q^{tot}_{\boldsymbol{\pi}}$ is calculated by all
individual value functions:
$Q^{tot}_{\boldsymbol{\pi}}(\boldsymbol{\tau},\mathbf{u})=F([Q_{\pi}^{i}(\tau^{i},u^{i})]_{i=1}^{N},s;\theta)$
, where $\boldsymbol{\tau}\equiv\mathcal{T}^{n}$ is a joint action-observation
history and $\boldsymbol{u}$ is a joint action, $F$ is the credit assignment
function parameterized by $\theta$ to learn value function factorization. Each
agent learns its own utility function by maximizing the global value function
$Q^{tot}$, which is trained end-to-end to minimise the following TD loss:

(2)
$\displaystyle\mathcal{L}(\theta)=\mathbb{E}_{\mathcal{D}}\left[\left(y^{tot}-Q^{tot}(\boldsymbol{\tau},\boldsymbol{u},s;\theta)\right)^{2}\right],$

where $\mathcal{D}$ is the replay buffer,
$y^{tot}=r+\gamma\max_{\boldsymbol{u}^{\prime}}Q^{tot}(\boldsymbol{\tau}^{\prime},\boldsymbol{u}^{\prime},s^{\prime};\theta^{-})$
and $\theta^{-}$ is the parameter of the target network (Mnih et al., 2015).

##  3\. Sparse state based MARL

In this section, we propose a novel sparse state based MARL framework that is
general enough to be plugged into any existing value-based multi-agent
algorithm. As shown in Figure 2, our framework adopts the CTDE paradigm, where
each agent learns its individual utility network by optimizing the TD loss of
the mixing network. During the execution, the mixing network is removed, and
each agent acts according to its local policy derived from its value function.
Distinguish from other value-based methods, our agents’ value functions or
policies carry out the process of selection and discrimination according to
the importance of different entities of state. To enable efficient and
effective learning among agents between different entities of state, our
method be described by three steps: 1) selection; 2) discrimination; 3)
learning.

###  3.1. Selection and Discrimination

It is a dynamic process to assign attentions based on the contribution of the
observed entities to the value estimation. In our framework, we adopt the
self-attention module (Vaswani et al., 2017) to capture the relational weights
between the observed entities of the agents. In particular, an agent $i$
observes $M$ other entities at time step $t$, then the corresponding input of
utility network $O_{t}^{i}$ is defined as
$O_{t}^{i}=[{\boldsymbol{o}^{i,1}_{t}},\ldots,{\boldsymbol{o}^{i,M}_{t}}]^{T}\in\mathbb{R}^{{M}\times
d_{E}}$ with $d_{E}$ being the entity dimension and
$o^{i,m}_{t}\in\mathbb{R}^{d_{E}}$ being the state information of the $m$-th
($m\in\\{1,...,M\\}$) entity. All observed entities are embedded to $d_{X}$
dimension via an embedding function
$f(\cdot):\mathbb{R}^{d_{E}}\rightarrow\mathbb{R}^{d_{X}}$ as follows:

(3) $\displaystyle
X_{t}^{i}=\left\\{f(\boldsymbol{o}_{t}^{i,1}),\ldots,f(\boldsymbol{o}_{t}^{i,M})\right\\},~{}i\in\mathcal{I}.$

Then, the embedding feature of each agent ${X}\in\mathbb{R}^{{M}\times d_{X}}$
is projected to query $Q=X{W}_{Q}$, key $K=X{W}_{K}$ and value $V=X{W}_{V}$
representation, where $\\{{W}_{Q},{W}_{K},{W}_{V}\\}\in\mathbb{R}^{d_{X}\times
d_{X}}$ are trainable weight matrices. Then, $Q$, $K$, $V$ are input into the
self-attention layer to calculate the entities importance for the model
decision, which is given by

(4)
$\displaystyle\operatorname{Attn}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{X}}}\right)V.$

One of the limitations of the softmax activation function is that the
resulting probability weights for any element never appear to be zero, which
further leads to dense output probabilities. Nevertheless, for the sake of
simplifying the exploration space and selecting valuable observations, it is
crucial for agents to reduce the number of entities to focus on. Hence, a
sparse probability distribution is desired to distinguish between critical and
irrelevant entities, which can accelerate convergence and improve performance.
To start with, inspired by sparsemax (Martins and Astudillo, 2016; Peters et
al., 2019), we consider introducing sparse states to enhance the perception of
valuable entities of agent observation and neglect the others.

We denote the products of the query with all keys ${QK^{T}}$ as
$Z\in\mathbb{R}^{M\times M}$, which consists of $M$ rows
$\\{{\boldsymbol{z}_{1}},\ldots,{\boldsymbol{z}_{M}}\\}$ with
${\boldsymbol{z}_{m}}\in\mathbb{R}^{M}$ being the logits of the $m$-th row.
Afterwards, we define a matrix sorting operator
$\operatorname{SortMat}(\cdot)$ as follows:

(5)
$\displaystyle\widetilde{Z}=\operatorname{SortMat}(Z)=[\operatorname{SortVec}(\boldsymbol{z}_{1})^{T},\dots,\operatorname{SortVec}(\boldsymbol{z}_{M})^{T}]^{T},$

where $\operatorname{SortVec}(\cdot)$ sorts the elements of vector in
descending order. Then we calculate

(6) $\displaystyle\boldsymbol{n}(\widetilde{Z})=[n_{1},\dots,n_{M}]^{T},$

where $n_{m}:=\max{\\{n\in[M]\mid 1+n\widetilde{Z}_{m,n}>\sum_{k\leq
n}\widetilde{Z}_{m,k}\\}}$ is the maximal number of crucial elements in
$\boldsymbol{z}_{m}$ that we intend to preserve, while other elements is set
to zero in the subsequent operations. We define

(7) $\displaystyle\boldsymbol{c}=[c_{1},\cdots,c_{M}]^{T}$

with $c_{m}=\sum_{k\leq n_{m}}Z_{m,k}-1$ and the scaling vector as

(8)
$\displaystyle\boldsymbol{\mu}=[\frac{1}{n_{1}},\cdots,\frac{1}{n_{M}}]^{T}.$

Then, the threshold matrix is calculated as

(9) $\displaystyle\Delta=\boldsymbol{1}_{M\times
1}(\boldsymbol{c}\odot\boldsymbol{\mu})^{T},$

where $\boldsymbol{1}_{M\times 1}\in\mathbb{R}^{M}$ is an all-one vector and
$\odot$ is the pointwise product. The sparse attention weights matrix $P$ is
obtained by

(10) $\displaystyle P=[Z-\Delta]_{+},$

where $[\cdot]_{+}:=\max\\{0,\cdot\\}$. Thus, the sparse attention is given by

(11) $\displaystyle\operatorname{sAttn}(Q,K,V)=PV,$

which can retain most of the essential properties of softmax while assigning
zero probability to low-scoring choices. Therefore, the model will pay more
attention to critical entities when making decisions, reducing the attention
to other redundant entities.

Algorithm 1 Sparse State based MARL Algorithm

Initialize: Critic network $\theta_{\rho}$, target critic
$\hat{\theta}_{\rho}$ = $\theta_{\rho}$, agents’ Q-value networks
$\theta_{\pi}=(\theta_{1},\dots,\theta_{N})$ and Replay buffer $\mathcal{D}$

1:for each training episode $e$ do

2: $t=0$, $s_{0}$ = initial state, ${o}_{0}^{i}=G({s}_{0},i)$ and
$h^{i}_{0}=0$ for $i\in\mathcal{N}$

3: while $s\neq terminal$ and $t<T$ do

4: $t=t+1$

5: for each agent $i$ do

6: Calculate dense attention $Y_{dense}$ by (12)

7: Calculate sparse attention $Y_{sparse}$ by (13)

8: Calculate trajectory encode $h_{t}^{i}$ by (14)

9: Obtain $Q_{dense}^{i}(\tau^{i},\cdot)$ by (15)

10: Obtain $Q_{sparse}^{i}(\tau^{i},\cdot)$ by (16)

11: Sample $u^{i}$ from $\pi^{i}(Q_{dense}^{i},\epsilon)$

12: end for

13: Execute actions $\boldsymbol{u}_{t}=(u^{1},\dots,u^{N})$

14: Receive reward $r_{t+1}$ and next state ${s}_{t+1}$

15: end while

16: Store episodes in replay buffer $\mathcal{D}$

17: Sample a random minibatch of episodes from $\mathcal{D}$

18: Dense Attention Loss:

19: Compute $\mathcal{L}_{td}(\theta_{\pi},\theta_{\rho})$ by (17)

20: Auxiliary Sparse Attention Loss:

21: Compute $\mathcal{L}_{aux}(\theta_{\pi},\theta_{\rho})$ by (18)

22: Update $\theta_{\pi}$ and $\theta_{\rho}$ by (19)

23: Every $C$ episodes reset $\hat{\theta}_{\rho}$ = $\theta_{\rho}$

24:end for

###  3.2. Learning with Sparse Loss

Obviously, the sparse attention mechanism can be realized by directly
replacing the traditional self-attention activation function with a sparse
distribution function. However, the model cannot distinguish which entity is
vital from the beginning. Thus directly adopting the sparse attention
mechanism will have performance regression. To address this issue, we design
the structure shown on the right side of Figure 2 to guide the training of
local agents, where we utilize two routes to exploit dense and sparse
attention, respectively. Dense attention guarantees that the algorithm can
converge, while sparse attention is a powerful auxiliary to enhance the agent
perception of critical entities, thereby improving performance.

To do this, the sparse attention module and dense attention module share the
weight matrices $\\{{W}_{Q},{W}_{K},{W}_{V}\\}$ and the GRU module. Denote the
parameters of these two networks by $\theta_{\pi}$. The projected matrix $Q$
and $K$ are fed into both dense and sparse attention. Then, we calculate the
weighted sum of $V$ to obtain the output

(12) $Y_{dense}=\operatorname{Attn}(Q,K,V)\in\mathbb{R}^{M\times d_{X}},$ (13)
$Y_{sparse}=\operatorname{sAttn}(Q,K,V)\in\mathbb{R}^{M\times d_{X}}.$

In our implementation, the GRU (Cho et al., 2014) module is utilized to encode
an agent’s history of observations and actions via

(14) $h_{t}^{i}=GRU(X_{t}^{i},h_{t-1}^{i}).$

Then, $Y_{dense}$ and $Y_{sparse}$ are concatenated with the output of GRU
separately to estimate the individual value function as follows:

(15)
$Q_{dense}^{i}(\tau^{i},\cdot)=\operatorname{Agent}^{i}(Y_{dense},h_{t}^{i}),$
(16)
$Q_{sparse}^{i}(\tau^{i},\cdot)=\operatorname{Agent}^{i}(Y_{sparse},h_{t}^{i}).$

Each agent selects the action that maximizes $Q_{dense}^{i}$ and
$Q_{sparse}^{i}$ for subsequent computations in centralized training. In
addition, the action selected by $Q_{dense}^{i}$ is executed in the
environment. For the exploration strategy, $\epsilon$-greedy is adopted, and
the exploration rate of $\epsilon$ decreases over time.

To better learn the role of entities in credit assignment, we use a mixing
network to estimate the global Q-values $Q_{dense}^{tot}$ and
$Q_{sparse}^{tot}$, using per-agent utility $Q_{dense}^{i}$ and
$Q_{sparse}^{i}$. Since the auxiliary estimation is calculated in the
individual utility function, our proposed S2RL is seamlessly integrated with
various valued-based algorithms. For example, we can use the mixing network, a
feed-forward neural network introduced by QMIX (Rashid et al., 2018). The
mixing network mixes the agent network outputs monotonically. The parameters
of the mixing network parameterized by $\theta_{\rho}$ are conditioned on the
global states and are generated by a hyper-network. Then, we minimize the
following TD loss to update the dense attention module:

(17)
$\mathcal{L}_{td}\left(\theta_{\pi},\theta_{\rho}\right)=\mathbb{E}_{\mathcal{D}}\left[\left(r+\gamma\max_{\boldsymbol{u}^{\prime}}\bar{Q}_{dense}^{tot}\left(s^{\prime},\boldsymbol{u}^{\prime}\right)-Q_{dense}^{tot}(s,\boldsymbol{u})\right)^{2}\right],$

where $\bar{Q}_{dense}^{tot}$ is the target network, and the expectation is
estimated with uniform samples from the same replay buffer ${\mathcal{D}}$. In
the meanwhile, the AUX Loss is given by

(18)
$\mathcal{L}_{aux}\left(\theta_{\pi},\theta_{\rho}\right)=\mathbb{E}_{\mathcal{D}}\left[\left(r+\gamma\max_{\boldsymbol{u}^{\prime}}\bar{Q}_{sparse}^{tot}\left(s^{\prime},\boldsymbol{u}^{\prime}\right)-Q_{sparse}^{tot}(s,\boldsymbol{u})\right)^{2}\right],$

where $\bar{Q}_{tot}^{aux}$ is the auxiliary target network.

(a) 3s5z (b) 3s_vs_5z (c) 5s10z (d) corridor (e) 6h_vs_8z (f) 3s5z_vs_3s6z

Figure 3. Learning curves of our S2RL and baselines on one easy map (3s5z),
one hard map (3s_vs_5z), and 4 super-hard maps (corridor, 5s10z, 6h_vs_8z,
3s5z_vs_3s6z). All experimental results are illustrated with the median
($25-75\%$ percentiles) performance and across 5 runs for a fair comparison.

In our framework, S2RL services as a plug-in module in the agent utility
networks. The outputs of S2RL modules are directly used for subsequent network
computations. Then, each agent is trained by minimizing the total loss

(19)
$\mathcal{L}\left(\theta_{\pi},\theta_{\rho}\right)=\mathcal{L}_{td}(\theta_{\pi},\theta_{\rho})+\lambda\mathcal{L}_{aux}(\theta_{\pi},\theta_{\rho}),$

where $\lambda$ is a regularization parameter that controls the level of
attention to critical states. Obviously, a larger $\lambda$ allows our
algorithm to pay more attention to some critical states, while a smaller
$\lambda$ allows for a more even distribution of attention. The overall
framework is trained in an end-to-end centralized manner. The complete
algorithm is summarized in Algorithm 1.

##  4\. EXPERIMENTS

We conduct experiments on the StarCraft Multi-Agent Challenge (SMAC)111We use
the SC2.4.10 version instead of the older SC2.4.6.2.69232. Performance is not
comparable between different versions. (Samvelyan et al., 2019) to demonstrate
the effectiveness of the proposed sparse state based MARL (S2RL) method. SMAC
has become a standard benchmark for evaluating state-of-the-art MARL methods,
which focuses on micromanagement challenges. The setup of SMAC is that each
ally entity is controlled by an individual learning agent, while the enemy
entities are controlled by a built-in AI. At each time step, agents can move
in four cardinal directions, stop, take no-operation, or choose an enemy to
attack. Thus, if there are $n_{e}$ enemies in the scenario, the action space
for each ally unit consists of $n_{e}+6$ discrete actions. Agents aim to
inflict maximum damage on enemy entities to win the game. Therefore, proper
tactics such as focusing fire and covering attack are required during battles.
Learning these diverse interaction behaviors under partial observation is a
crucial yet challenging task. In what follows, we detail the compared methods
and parameter settings and then present the qualitative and quantitative
performance of different methods.

###  4.1. Comparison Methods and Training Details

Our method is compared with several baseline methods, including IQL, VDN
(Sunehag et al., 2018), QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019),
QPLEX (Wang et al., 2021b), CWQMIX and OWQMIX (Rashid et al., 2020). Our S2RL
implementation uses VDN, QMIX and QPLEX as an integrated architecture to
verify its performance, called S2RL (VDN), S2RL (QMIX) and S2RL (QPLEX). These
three SOTA methods are chosen for their robust performance in different multi-
agent scenarios, while S2RL can also be easily applied to other frameworks.

We adopt the Python MARL framework (PyMARL) (Samvelyan et al., 2019) to run
all experiments. The hyperparameters of the baseline methods are the same as
those in PyMARL to ensure comparability. The regularization parameter in (19)
is set to $\lambda=1$. For all experiments, the optimization is conducted
using RMSprop with a learning rate of $5\times 10^{-4}$, a total timestep of
$2$M, a smoothing constant of $0.99$, and no momentum or weight decay. For
exploration, we use $\epsilon-$greedy with $\epsilon$ annealed linearly from
$1.0$ to $0.05$ over $50K$ time steps and kept constant for the rest of the
training. For four super hard exploration maps (6h_vs_8z, 3s5z_vs_3s6z,
corridor, 5s10z), we extend the epsilon annealing time to $500K$ and the total
timestep to $5M$, and three of them (6h_vs_8z, corridor, 5s10z) optimized with
Adam for both series of S2RL and all the baselines and ablations. Batches of
$32$ episodes are sampled from the replay buffer, and all tested methods are
trained end-to-end on fully unrolled episodes. All experiments on the SMAC
benchmark use the default reward and observation settings of the SMAC
benchmark (Samvelyan et al., 2019). All experiments in this section were
carried out with $5$ different random seeds on NVIDIA GTX V100 GPU.

(a) 3s5z (b) 3s_vs_5z (c) 5s10z (d) corridor (e) 6h_vs_8z (f) 3s5z_vs_3s6z

Figure 4. The performance comparison between the vanilla methods and their
S2RL variants. We integrate the proposed S2RL framework with VDN, QMIX and
QPLEX.

(a) 5s10z (b) 6h_vs_8z (c) 3s5z_vs_3s6z

Figure 5. Ablation studies regarding component of dense attention and
auxiliary sparse attention.

###  4.2. Overall Results

To demonstrate the efficiency of our proposed method, we conduct experiments
on 6 challenging SMAC scenarios, which are classified into Easy (_3s5z_), Hard
(_3s_vs_5z_) and Super-Hard (_6h_vs_8z_ , _3s5z_vs_3s6z_ , _corridor_ ,
_5s10z_). All of these scenarios are heterogeneous, where each army is
composed of more than one entity type. It is worth mentioning that MARL
algorithms are harder to converge on hard and super-hard maps and therefore
need to focus more on important entities to speed up convergence. In this way,
we are more interested in the performance of our method on these maps.

Figure 3 shows the overall performance of the tested algorithms in different
scenarios. The results include the median performance and $25-75\%$
percentiles are shaded to avoid the effect of any outliers as recommended in
(Samvelyan et al., 2019). For the sake of demonstration, here we select the
best plug-in method, referred to as S2RL in the following, to compare with
other baseline algorithms. First of all, we can see that S2RL performs best on
up to all six tasks, which means our proposed method can efficiently enhance
the performance of agents in different scenarios. In the easy map, some
algorithms have achieved good performance, and our S2RL is not significantly
ahead. In contrast, our S2RL significantly improves the learning efficiency
and final performance compared to the baselines in some hard and super-hard
scenarios. Specifically, in _6h_vs_8z_ and _3s5z_vs_3s6z_ , our S2RL
consistently outperforms baselines by a large margin during training. This is
because the number of entities in easy maps is small, all entities are
critical, and the selection gain brought by the sparse attention mechanism is
not apparent. However, when the situation becomes more complex, and the agent
needs to consider which entities are more critical to the decision, the
benefits of the sparse attention mechanism are more pronounced.

In addition, to test the generalization of our method incorporated into
various valued-based algorithms, we incorporate S2RL to VDN, QMIX and QPLEX
respectively, and compare the final performance with vanilla agent utility
networks in Figure 4. In general, most of the learning curves of S2RL (VDN),
S2RL (QMIX) and S2RL (QPLEX) achieve gratifying results superior to VDN, QMIX
and QPLEX. Besides, it is worth mentioning that our method pulls huge margins
on tasks with more severe difficulties, demonstrating the effectiveness of
S2RL. The experimental results show that in the super-hard map _6h_vs_8z_ ,
our proposed S2RL (QPLEX) improves the win rate by almost $55\%$ compared to
the naive QPLEX. Even more encouraging is that S2RL (QMIX) can reach a win
rate of $80\%$ while QMIX basically does not learn any strategy.

Furthermore, the promotion of incorporating S2RL into QMIX and QPLEX is higher
than VDN, which reveals the importance of the mixing network. We hypothesize
that the sparse attention mechanism enables the model to select critical
entities and further clarify their contributions, which may promote the power
of credit assignment. Unlike QMIX and QPLEX, VDN represents the joint action-
value as a summation of individual Q-functions, resulting in this poor
representation of the mixing network challenging to leverage the strengths of
our approach.

###  4.3. Ablation Study

To evaluate the advantage of sparse auxiliary loss on the agent training
process, we conduct ablation studies on three super hard maps (_5s10z_ ,
_6h_vs_8z_ , _3s5z_vs_3s6z_) to test its contribution. Our S2RL mainly
consists of two parts: (A) dense attention, denoted Attn; (B) sparse attention
as an auxiliary, noted as S2RL. We apply these two components to VDN, QMIX and
QPLEX utility networks and compare their performance in Figure 5. The solid
curves indicate that the agents use the dense attention module to calculate
the importance of different entities. The dashed curves indicate that the
agents learn to use the sparse attention module as an auxiliary to teach the
dense attention module.

Generally speaking, the advantages of using S2RL gradually emerge in the
middle and late stages of training. We assume that agents cannot distinguish
which entity is more important at the beginning of training. As training
progresses, agents explore more unknown states and are gradually able to
distinguish which entities are more critical. Finally, the overall performance
of agents is improved when they discard irrelevant entities. Furthermore, we
find that using sparse attention achieves more significant improvements on
_6h_vs_8z_ and _corridor_. On the _6h_vs_8z_ scenario, $6$ Hydralisks face
$24$ enemy Zealots, while on the _corridor_ scenario, $6$ Zealots face $24$
enemy Zerglings. The controllable agents in these scenarios are homogeneous,
making it easier for them to explore cooperative strategies. Moreover, using
the sparse attention module helps simplify the exploration space, making S2RL
more advantageous in these scenarios.

(a) Strategy: Zealots $0$ leave the team separately to attract the attention
of most enemies.

(b) Strategy: Zealots $1$, $2$, $5$ focus fire cooperatively and Zealots $4$
attack the distant enemy to rescue his teammates.

(c) Strategy: Zealots $0$ keep moving to avoid being attacked and others
eliminate the scattered enemies.

Figure 6. A visualization example of the sophisticated strategies adopted by
S2RL (QMIX) in the SMAC corridor scenario. In this super-hard map, ally units
are 6 Zealots labeled by green circle, while enemy units are 24 Zerglings.
Green and red shadows mark enemies attracted by ally units. Green arrows and
red arrows indicate the direction in which ally units and enemy units will
move, respectively. Yellow lines indicate enemies that ally units are
attacking.

###  4.4. Action Representations

Figure 6 visualizes the final trained model S2RL (QMIX) on the SMAC corridor
scenario to better explain why our method performs well. In this super-hard
scenario, 6 friendly Zealots face 24 enemy Zerglings. The disparity in
quantity means that our agents need to learn cooperative strategies such as
moving, pulling and focusing fire. Otherwise, agents are doomed to lose if
they gather together.

As shown in Figure 6(a), the game starts with the Zealots $0$ highlighted in
green as a warrior, leaving the team separately to grab the attention of most
of the enemies in the green oval. Thus other zealots can eliminate a small
number of enemies in the red oval with a high probability of winning. In
Figure 6(b), we can see that Zealots $1$, Zealots $2$ and Zealots $5$ are
focusing fire on the enemy, thus speeding up the eradication of the enemy. In
the meanwhile, Zealots $4$ stands out to attack enemies surrounding their
teammates from a distance. These sophisticated strategies reflect that Zealots
$4$ has a better sense of the situation and knows what it should do to protect
its teammates. In the next time step, we recognize that Zealots $0$ is
constantly moving to avoid being attacked, and the enemy marked by the red
oval is successfully drawn and walking towards our team (see Figure 6(c)).
Although doomed to sacrifice, Zealots $0$ gives teammates plenty of time to
annihilate scattered enemies and rescue Zealots $0$. All in all, S2RL can
effectively allow agents to immediately focus on critical entities and make
decisions, especially in more intricate scenarios.

##  5\. Related works

###  5.1. Value-based Methods in MARL

Recently, value-based methods have been applied to multi-agent scenarios to
solve complex Markov games and have achieved significant algorithmic progress.
VDN (Sunehag et al., 2018) represents the joint action-value as a summation of
individual value functions. Due to its poor expression factorization, QMIX
(Rashid et al., 2018) improves VDN (Sunehag et al., 2018) by using a mixing
network for nonlinear aggregation while maintaining the monotonic relationship
between centralized and individual value functions. Moreover, weighted QMIX
(Rashid et al., 2020) adapts a twin network and encourages underestimated
actions to alleviate the risk of suboptimal outcomes. The monotonic
constraints of QMIX and similar methods lead to provably poor exploratory and
suboptimal properties. To address the structural limitations, QTRAN (Son et
al., 2019) constructs regularizations with linear constraints and relaxes them
with a $\ell_{2}$-norm penalty to improve tractability, but its constraints
are computationally intractable. MAVEN (Mahajan et al., 2019) relaxes QTRAN
(Son et al., 2019) by two penalties and introduces a hierarchical model to
coordinate diverse explorations among agents. In (Wang et al., 2021b), a
duplex dueling network architecture is introduced for factoring joint value
functions, which achieves state-of-the-art on a range of cooperative tasks.
Additionally, some more advanced methods (Wang et al., 2020; Wang et al.,
2021a) introduce role-oriented frameworks to decompose complex MARL tasks. In
general, these methods mainly focus on aggregating local agent utility
networks into a central critic network, while our method improves the
structure of individual agent networks for more robust performance.

###  5.2. Attention Mechanism in MARL

Recently, attention models are increasingly adopted in MARL algorithms
(Vaswani et al., 2017; Hu et al., 2018; Velickovic et al., 2018), since the
attention mechanism is effective in extracting communication channels,
representing relations, and incorporating information in large contexts. ATOC
(Jiang and Lu, 2018) and MAAC (Iqbal and Sha, 2019) process messages from
other agents differently through the attention layer according to their state-
dependent importance. SparseMAAC (Li et al., 2019) extends MAAC (Iqbal and
Sha, 2019) with sparsity by directly replacing the softmax activation function
in the attention mechanism with $\gamma$-sparsemax. In addition, TarMAC (Das
et al., 2019) utilizes a sender-receiver soft attention mechanism and multiple
rounds of cooperative reasoning to allow targeted continuous communication
between agents. Then CollaQ(Zhang et al., 2020) considers the use of attention
mechanisms to handle a variable number of agents to solve the problem of
dynamic reward distribution. Qatten (Yang et al., 2020b) employs an attention
mechanism to compute the weights of local action-value functions and mix them
to approximate the global Q-value. EPC (Long et al., 2020) utilizes an
attention mechanism to combine embeddings from different observation-action
encoders. REFIL (Iqbal et al., 2021) uses attention in QMIX to generate a
random mask group of agents. UPDET (Hu et al., 2021) decouple the policy
distribution from intertwined input observations with the help of a
transformer mechanism. Moreover, G2ANet (Liu et al., 2020) and HAMA (Ryu et
al., 2020) construct the relationship between agents as a graph and utilize
attention mechanisms to learn the relationship between agents. However, most
of these existing attention mechanisms compute the importance weights of all
entities. In this case, all participants are assigned scores according to a
dense fully connected graph, which forces agents to perceive all entities.
SparseMAAC takes sparsity into account, but it ignores that directly applying
the sparse attention mechanism will disrupt sufficient exploration and push
the algorithm towards suboptimal policies. In this paper, agents learn to
perceive more critical entities of observation in the decision-making process
while all observation information is preserved.

##  6\. CONCLUSION

In this work, we investigate how cooperating MARL agents benefit from
extracting significant entities from observations. We design a novel sparse
state based MARL algorithm that utilizes a sparse attention mechanism as an
auxiliary way to select critical entities and ignore extraneous information.
Moreover, S2RL can be easily integrated into various value-based architectures
such as VDN, QMIX, QPLEX, etc. Experimental results on the StarCraft II
micromanagement benchmark and different value-based backbones demonstrate that
our method significantly outperforms existing collaborative MARL algorithms
and achieves state-of-the-art. It is worth mentioning that our method pulls
huge margins on complex tasks, demonstrating the effectiveness of S2RL. It
could be interesting to investigate the grouping between cooperating agents
through sparseness for future work.

###### Acknowledgements.

This work was supported by the National Key Research and Development Project
of China (2021ZD0110400 $\&$ 2018AAA0101900), National Natural Science
Foundation of China (U19B2042), The University Synergy Innovation Program of
Anhui Province (GXXT-2021-004), Zhejiang Lab (2021KE0AC02), Academy Of Social
Governance Zhejiang University, Fundamental Research Funds for the Central
Universities (226-2022-00064 $\&$ 226-2022-00142), Artificial Intelligence
Research Foundation of Baidu Inc., Program of ZJU and Tongdun Joint Research
Lab, Shanghai AI Laboratory (P22KS00111).

## References

  * (1)
  * Cao et al. (2013) Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. 2013\.  An Overview of Recent Progress in the Study of Distributed Multi-Agent Coordination.  _IEEE Trans. Ind. Informat._ 9, 1 (2013), 427–438.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder-decoder approaches.  _arXiv preprint arXiv:1409.1259_ (2014).
  * Das et al. (2019) Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. 2019\.  Tarmac: Targeted multi-agent communication. In _ICML_.
  * Foerster et al. (2016) Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. 2016.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In _NeurIPS_.
  * Foerster et al. (2018) Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018\.  Counterfactual Multi-Agent Policy Gradients. In _AAAI_.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-Excitation Networks. In _CVPR_.
  * Hu et al. (2021) Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. 2021\.  UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers. In _ICLR_.
  * Iqbal et al. (2021) Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. 2021.  Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning. In _ICML_.
  * Iqbal and Sha (2019) Shariq Iqbal and Fei Sha. 2019.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning. In _ICML_.
  * Jiang and Lu (2018) Jiechuan Jiang and Zongqing Lu. 2018.  Learning Attentional Communication for Multi-Agent Cooperation. In _NeurIPS_.
  * Kiran et al. (2022) B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2022.  Deep Reinforcement Learning for Autonomous Driving: A Survey.  _IEEE Trans. Intell. Transp. Syst._ 23, 6 (2022), 4909–4926.
  * Li et al. (2021) Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. 2021.  Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning. In _SIGKDD_.
  * Li et al. (2019) Wenhao Li, Bo Jin, and Xiangfeng Wang. 2019.  SparseMAAC: Sparse attention for multi-agent reinforcement learning. In _DASFAA_.
  * Lillicrap et al. (2016) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016.  Continuous control with deep reinforcement learning. In _ICLR_.
  * Liu et al. (2019) Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. 2019\.  Emergent Coordination Through Competition. In _ICLR_.
  * Liu et al. (2020) Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. 2020\.  Multi-agent game abstraction via graph attention neural network. In _AAAI_.
  * Long et al. (2020) Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. 2020\.  Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning. In _ICLR_.
  * Lowe et al. (2017) Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017\.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In _NeurIPS_.
  * Mahajan et al. (2019) Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. 2019.  MAVEN: Multi-Agent Variational Exploration. In _NeurIPS_.
  * Mao et al. (2020) Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang, Jun Wang, and Zhen Xiao. 2020\.  Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning. In _AAAI_.
  * Martins and Astudillo (2016) André F. T. Martins and Ramón Fernandez Astudillo. 2016\.  From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. In _ICML_.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, et al. 2015\.  Human-level control through deep reinforcement learning.  _Nature_ 518, 7540 (2015), 529–533.
  * Oliehoek and Amato (2016) Frans A. Oliehoek and Christopher Amato. 2016.  _A Concise Introduction to Decentralized POMDPs_.  Springer.
  * Peters et al. (2019) Ben Peters, Vlad Niculae, and André F. T. Martins. 2019\.  Sparse Sequence-to-Sequence Models. In _ACL_.
  * Rashid et al. (2020) Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. 2020.  Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In _NeurIPS_.
  * Rashid et al. (2018) Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. 2018.  QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In _ICML_.
  * Ryu et al. (2020) Heechang Ryu, Hayong Shin, and Jinkyoo Park. 2020\.  Multi-agent actor-critic with hierarchical graph attention network. In _AAAI_.
  * Samvelyan et al. (2019) Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, et al. 2019\.  The StarCraft Multi-Agent Challenge. In _AAMAS_.
  * Son et al. (2019) Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. 2019.  QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. In _ICML_.
  * Sunehag et al. (2018) Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, et al. 2018\.  Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In _AAMAS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NeurIPS_.
  * Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _ICLR_.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, et al. 2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Wang et al. (2021b) Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. 2021b.  QPLEX: Duplex Dueling Multi-Agent Q-Learning. In _ICLR_.
  * Wang et al. (2020) Tonghan Wang, Heng Dong, Victor R. Lesser, and Chongjie Zhang. 2020\.  ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. In _ICML_.
  * Wang et al. (2021a) Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2021a.  RODE: Learning Roles to Decompose Multi-Agent Tasks. In _ICLR_.
  * Wu et al. (2017) Cathy Wu, Aboudy Kreidieh, Eugene Vinitsky, and Alexandre M. Bayen. 2017.  Emergent Behaviors in Mixed-Autonomy Traffic. In _CoRL_.
  * Yang et al. (2020a) Yaodong Yang, Jianye Hao, Guangyong Chen, Hongyao Tang, Yingfeng Chen, Yujing Hu, Changjie Fan, and Zhongyu Wei. 2020a.  Q-value Path Decomposition for Deep Multiagent Reinforcement Learning. In _ICML_.
  * Yang et al. (2020b) Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. 2020b.  Qatten: A general framework for cooperative multiagent reinforcement learning.  _arXiv preprint arXiv:2002.03939_ (2020).
  * Zhang et al. (2020) Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. 2020.  Multi-agent collaboration via reward attribution decomposition.  _arXiv preprint arXiv:2010.08531_ (2020).


# Non-trivial activity dependence of static length scale and critical tests of
active random first-order transition theory

Kallol Paul  TIFR Centre for Interdisciplinary Sciences, Tata Institute of
Fundamental Research, Hyderabad - 500046, India  Saroj Kumar Nandi  TIFR
Centre for Interdisciplinary Sciences, Tata Institute of Fundamental Research,
Hyderabad - 500046, India  Smarajit Karmakar  TIFR Centre for
Interdisciplinary Sciences, Tata Institute of Fundamental Research, Hyderabad
- 500046, India

###### Abstract

Effects of activity on glassy dynamics are fundamental in several biological
processes. Active glasses extend the scope of the equilibrium problem and
provide new control parameters to probe different theoretical aspects. In the
theory of glassy dynamics, different length scales play pivotal roles. Here,
for the first time, we present results for the static length scale, $\xi_{S}$,
in an active glass via large-scale molecular dynamics simulations for model
active glasses in three spatial dimensions. We show that although the
relaxation dynamics are equilibrium-like, activity has non-trivial effects on
$\xi_{S}$. $\xi_{S}$ plays the central role in the random first-order
transition (RFOT) theory. Thus, our work provides critical tests for the
active RFOT theory, a phenomenological extension of its equilibrium
counterpart. We find that the two exponents, $\theta$ and $\psi$, within the
theory, become activity-dependent, exposing the non-trivial effects of
activity on $\xi_{S}$. However, the combination of $\theta$ and $\psi$, which
controls the relaxation dynamics, remains nearly independent of activity
leading to the effectively equilibrium-like behavior. Interestingly, $\xi_{S}$
shows higher growth in an active glass; this should help better comparison of
theories with simulations and experiments.

Glassy dynamics is a hallmark of nearly any liquid; when supercooled below
their melting temperature Berthier and Biroli (2011), for a modest decrease in
temperature, $T$, or increase in density, a glassy system shows an impressive
growth – from picoseconds to hours – of relaxation time, $\tau$. The
microscopic mechanism leading to this tremendous slow down in the supercooled
regime continues to be debated, leading to various theories for the phenomenon
Berthier and Biroli (2011); Lubchenko and Wolynes (2007); Das (2004); Dyre
_et al._ (2009); Biroli and Bouchaud (2012); Tarjus  _et al._ (2005); Karmakar
_et al._ (2014). Most of these theories posit a transition at a lower $T$ that
is inaccessible to simulations or experiments, and thus, definitive tests of
different theories are complex. For example, random first-order transition
(RFOT) theory predicts a thermodynamic transition at $T_{K}$ where the
relaxation time diverges. From this aspect, active glasses, where the systems
consist of particles with a self-propulsion force, $f_{0}$, and a persistence
time, $\tau_{p}$, of their motion Ramaswamy (2010); Marchetti  _et al._
(2013), are significant since they extend the scope and extent of the problem.
Activity can provide an additional probe for glass transition. Furthermore,
they are fundamental in several biological processes, such as wound healing
Poujade  _et al._ (2007); Cochet-Escartin  _et al._ (2014), collective
migration in a cell monolayer Angelini  _et al._ (2011); Park  _et al._
(2015); Malinverno  _et al._ (2017); Garcia  _et al._ (2015), intracellular
dynamics Parry  _et al._ (2014); Nishizawa  _et al._ (2017); Zhou  _et al._
(2009), tumor progression Malmi-Kakkada  _et al._ (2018); Streitberger  _et
al._ (2020). Consequently, various models of such systems have been simulated
Berthier (2014); Paul  _et al._ (2021); Mandal  _et al._ (2016); Flenner  _et
al._ (2016), and theories of equilibrium glasses, such as the mode-coupling
theory Nandi and Gov (2017); Berthier and Kurchan (2013); Feng and Hou (2017);
Szamel (2016) and the random first-order transition (RFOT) theory Nandi  _et
al._ (2018), have been extended for such systems.

How does the presence of nonequilibrium activity in a system affect the
aspects of glassy behavior? Intensive research of the last decade seems to
suggest that apart from some quantitative changes, such as the shift in the
glass transition temperature, $T_{g}$ Ni  _et al._ (2013); Berthier and
Kurchan (2013); Nandi and Gov (2017), the qualitative aspects of glassy
dynamics in an active and passive system remain similar. This similarity
implies that active glasses can be described much like an equilibrium system,
via an effective temperature, $T_{eff}$. Such a similarity, however, is
somewhat surprising and disappointing at the same time. Activity is known to
show nontrivial effects, such as flocking Vicsek  _et al._ (1995), giant
number fluctuation Narayan  _et al._ (2007); Giavazzi  _et al._ (2017), and
long-range velocity correlation Caprini  _et al._ (2020); Szamel (2016);
Berthier  _et al._ (2019) in a dilute system. Some of these effects survive in
the glass regime; however, it would be surprising if the activity had no
nontrivial effects on the glassy properties. In addition, it is disappointing
as activity could provide a worthwhile probe to understand the nature of
equilibrium glasses around $T_{K}$. Specifically, activity allows access to
the system properties close to $T_{K}$, much like random pinning Karmakar  _et
al._ (2012); Cammarota and Biroli (2012); Chakrabarty  _et al._ (2015); Das
_et al._ (2017); Ozawa  _et al._ (2015) that enables one to study the system
properties close to the ideal glass transition in equilibrium systems. Now,
suppose activity merely shifts the transition point. In that case, the problem
remains similar to an equilibrium system: one has to go to the renormalized
transition point to study the properties of the ideal glass. In a recent paper
we showed, that although the relaxation dynamics are qualitatively similar to
an equilibrium system, activity has nontrivial effects on the dynamic
heterogeneity (DH) length scale, $\xi_{D}$ Paul  _et al._ (2021). Our extended
mode-coupling theory (MCT) of active glasses agrees well with these behaviors.
Does then activity also have nontrivial effects on the other length scales,
specifically, the static length scale, $\xi_{S}$, that controls the glassy
dynamics within RFOT? If yes, how can we understand the origin of the
effectively equilibrium-like behavior of the relaxation dynamics?

In this work, we carry out large-scale molecular dynamics simulations of a
well-known, three-dimensional model of active glass, the Kob-Anderson binary
mixture with particle ratio $80:20$, and take $10\%$ of the particles as
active Paul  _et al._ (2021). The motivation to take a fraction of the
particles as active comes from biological systems such as the actomyosin
cortex or cellular cytoplasm, where a fraction of proteins (such as the myosin
molecules) are active Nishizawa  _et al._ (2017); Parry  _et al._ (2014);
Prost  _et al._ (2015). The qualitative results are independent of this
fraction as long as it is non-zero Paul  _et al._ (2021). In our simulations,
we have implemented activity characterized as model-1 in the language of Ref.
Nandi  _et al._ (2018). We have investigated the role of activity by varying
$f_{0}$ in our simulation, keeping $\tau_{p}=1$ fixed. From the perspective of
$f_{0}$, both models, discussed in Ref. Nandi  _et al._ (2018), are similar;
thus, the results presented here are applicable for both models of activity.
We calculated $\xi_{S}$ via two distinct methods, the finite-size scaling
Karmakar  _et al._ (2009) and the block analysis Chakrabarty  _et al._ (2017);
Tah  _et al._ (2021), and find they give nearly the same results. Here, we
report the first measurements of $\xi_{S}$ in active systems and show that
indeed activity has nontrivial effects on $\xi_{S}$. The work presented here
provides critical tests of the active RFOT theory of Ref. Nandi  _et al._
(2018). Specifically, we find that the assumption of the exponents being
independent of activity is inappropriate. However, their combination that
appears within the theory for the relaxation dynamics is independent of
activity, consistent with Ref. Nandi  _et al._ (2018): this leads to the
effectively equilibrium-like behavior of the relaxation dynamics.

RFOT theory is one of the most popular theories of glass transition and has
been remarkably successful in describing the glassy aspects in diverse
systems. Within RFOT theory Lubchenko and Wolynes (2007); Kirkpatrick and
Thirumalai (2015), a glassy system consists of mosaics of different states.
Consider a region of length scale $R$. The state within this region is set by
two competing effects: the surface energy cost of having different states and
the bulk energy. Within RFOT theory, the reconfiguration is entropic in
nature, thus $f=Ts_{c}(\tilde{\Phi},T)$ where$s_{c}$ is the configurational
entropy, $\tilde{\Phi}$ is the effective interatomic interaction, and $T$, the
temperature. Thus, the free energy cost can be written as

$\Delta F=-\Omega_{d}R^{d}Ts_{c}(\tilde{\Phi},T)+S_{d}R^{\theta}\bar{\gamma},$
(1)

where $\Omega_{d}$ and $S_{d}$ are the volume and surface of a hypersphere in
spatial dimension $d$. Minimizing $\Delta F$, we obtain the typical length
scale of the mosaics as
$\xi_{S}\sim[c_{d}\bar{\gamma}/Ts_{c}(\tilde{\Phi},T)]^{1/(d-\theta)}$, where
$c_{d}=\theta S_{d}/d\Omega_{d}$. The energy cost, $\bar{\gamma}$ is
proportional to $T$: $\bar{\gamma}=\Xi(f_{0},\tau_{p})T$, where the activity
dependence is included in $\Xi(f_{0},\tau_{p})$. Vanishing of $s_{c}$
primarily controls the glassy dynamics within RFOT theory; therefore, we
ignore the activity dependence of $\Xi$ and treat it as a constant for
simplicity. It was shown in Ref. Nandi  _et al._ (2018) that $s_{c}(\Phi,T)$
of an active fluid in the small activity regime can be obtained by expanding
the contribution of activity around that for the passive system. Thus
$s_{c}(\tilde{\Phi},T)\sim s_{c}(\Phi,T)+\kappa\delta\Phi$, where $\Phi$
represents the interaction potential for the passive system, $\delta\Phi$ is
the potential coming from activity, and $\kappa$ is a constant. For the
passive system, close to the Kauzmann temperature, $T_{K}$, we have
$s_{c}(\Phi,T)\sim\Delta C_{p}(T-T_{K})/T_{K}$, where $\Delta C_{p}$ is the
difference of specific heats between the crystalline phase to that of the
glassy phase at the same $T$. Then, $\xi_{S}$ becomes

$\xi_{S}\simeq\left(\frac{D}{T-T_{K}+\frac{\kappa T_{K}\delta\Phi}{\Delta
C_{p}}}\right)^{1/(d-\theta)},$ (2)

where $D=c_{d}\Xi T_{K}/\Delta C_{p}$ is a constant. Within the mean-field
approximation, Ref. Nandi  _et al._ (2018) obtained $\delta\Phi$ considering a
single active particle trapped in a potential created by the surrounding
particles. Within the harmonic approximation, it can be analytically shown
that $\delta\Phi=Hf_{0}^{2}$, where $H$ is a constant Nandi  _et al._ (2018).
Therefore, writing $K=H\kappa T_{K}/\Delta C_{p}$, we obtain

$\xi_{S}\simeq\left(\frac{D}{T-T_{K}+Kf_{0}^{2}}\right)^{1/(d-\theta)}.$ (3)

Figure 1: (a) Relaxation time, $\tau$, as a function of $T$ for different
$f_{0}$. Symbols are simulation data and lines are fits with the active RFOT
theory, Eq. (4) with $\psi/(d-\theta)=1$, with $\tau_{0}=0.263$, $E=1.319$,
$T_{K}=0.3$, and $K=0.0155$. (b) $T_{K}$ at different $f_{0}$, symbols
represented the values obtained from fitting simulation data with the VFT
form, and the line is the active RFOT prediction. (c) Static length scale,
$\xi_{S}$, as a function of $T/T_{K}$. Symbols are simulation data, and lines
represent fits with Eq. (3). The exponents, $(d-\theta)$ at different $f_{0}$
are shown in (d) The line is $f(x)=1-0.2x$.

RFOT theory describes relaxation dynamics as the relaxation of individual
mosaics. The energy barrier $\Delta$ associated with a region of length
$\xi_{S}$ is $\Delta\sim\Delta_{0}\xi_{S}^{\psi}$, where $\Delta_{0}$ is an
energy scale, and $\psi$ is another exponent. The precise $T$-dependence of
$\Delta_{0}$ depends on models, following Wolynes et al, we take it as
$\Delta_{0}=k_{B}T$ Lubchenko and Wolynes (2007); Biroli and Bouchaud (2012)
with $k_{B}$ being the Boltzmann constant. Considering a barrier crossing
scenario, we then get the relaxation time, $\tau$ as
$\tau=\tau_{0}\exp[\Delta_{0}\xi_{S}^{\psi}/T]$, where we have set $k_{B}$ to
unity and the high $T$ value of $\tau$, denoted as $\tau_{0}$, is independent
of $T$ but may depend on activity Nandi and Gov (2017). Then, $\tau$ becomes

$\ln\left(\frac{\tau}{\tau_{0}}\right)=\left(\frac{E}{T-T_{K}+Kf_{0}^{2}}\right)^{\psi/(d-\theta)},$
(4)

where $E=k^{(d-\theta)/\psi}D$. Further progress demands knowledge of the
exponents $\theta$ and $\psi$. Note that a first-principle calculation of
$\psi$ does not exist to date even for an equilibrium system Lubchenko and
Wolynes (2007); Biroli and Bouchaud (2012). Reference Nandi  _et al._ (2018)
assumed that they remain the same as in the passive system even in the
presence of activity and found excellent agreement with simulations. However,
this crucial assumption has not yet been tested. Moreover, in the light of our
recent work Paul  _et al._ (2021), where we showed that the exponents of the
DH length scale depend on activity: these results suggest activity dependence
of both $\theta$ and $\psi$ as well, what we indeed find. However, we show
that the activity-dependence of $\theta$ and $\psi$ are such that they cancel
each other in the behavior of $\tau$, Eq. (4).

We now compare the active RFOT theory with our simulation data. Ref. Nandi
_et al._ (2018) assumed that the $\psi/(d-\theta)=1$ remains valid even in the
presence of activity. This assumption was motivated by the observation that
the relaxation dynamics remains equilibrium-like; the resulting theory agrees
well with the simulation data. Although it has not yet been tested, let us
first consider that this assumption holds. For a particular system, the value
of $K$ should be the same. Analysis presented in Ref. Paul  _et al._ (2021)
show that for this particular system, $K=0.0155$. It is also known that
$T_{K}=0.3$ for the passive system. Then, we obtain
$\ln(\tau/\tau_{0})=E/(T-T_{K}^{eff})$, where $T_{K}^{eff}=T_{K}-Kf_{0}^{2}$
is the effective Kauzmann temperature (Eq. 4). Figure 1(a) shows the fits of
the simulation data for $\tau$ with this form. $\tau_{0}$ and $E$ are obtained
by fitting the data for the passive system and are kept the same for the
active systems. We obtain $T_{K}^{eff}$ as a fitting parameter. Figure 1(b)
shows the comparison of $T_{K}^{eff}$, obtained via fitting the simulation
data, with that of the theory. Note that the line is not a fit.

We now discuss the main result of this paper. It is the behavior of the
individual exponents, $\theta$ and $\psi$, in the presence of activity. Note
that the precise values of these parameters, even for the equilibrium systems,
remain controversial. For a passive system, one school of thoughts have argued
Kirkpatrick  _et al._ (1989); Lubchenko and Wolynes (2007); Kirkpatrick and
Thirumalai (2015) that $\theta=\psi=d/2$. However, the other possibilities,
such as $\theta\sim(d-1)$ and $\psi\lesssim 1$, also look reasonable when
fitted with simulation and experimental data Biroli and Bouchaud (2012). For
our system, the passive system data seem to be well-described by the latter,
and we chose these values for the passive system. Using this, fitting Eq. (3)
with simulation data gives $D$, note that $T_{K}^{eff}$ is already known.
Next, we fit Eq. (3) with with the simulation data for active systems using
$(d-\theta)$ as a fitting parameter and show them in Fig. 1(c). The obtained
values of ($d-\theta$) are shown in Fig. 1(d); $\theta$ almost linearly
increases with $f_{0}$. One interesting feature is that $\theta$ becomes
larger than $(d-1)$ as activity increases. In an equilibrium system, $\theta$
has the upper limit of $(d-1)$. This bound is perhaps not applicable for our
system, which is out of equilibrium. What does this progressive increase in
$\theta$ with $f_{0}$ imply? The exponent $\theta$ characterizes the surface
of mosaics. As activity increases, $\theta$ tends closer to $d$, implying that
the surface becomes quite irregular and assumes a fractal nature.

Figure 2: (a) Fit of $\ln\tau=c_{1}+c_{2}\xi_{S}^{\psi}$ for the passive
system, using $\psi=1$ gives the values of $c_{1}=-1.624$ and $c_{2}=2.335$.
Once these parameters are determined, we use $\psi$ as a parameter and fit the
data for the active system. The symbols are our simulation data, and the lines
are the fits. (b) The symbols show the values of $\psi$ as a function of
$f_{0}$ obtained via fitting. The line is $f(x)=1-0.2x$. Inset: Barrier
crossing scenario for an active particle (see text for details).

We next look at the exponent $\psi$. From the discussion preceding Eq. (4), we
obtain $\ln\tau=c_{1}+c_{2}\xi_{S}^{\psi}$, where $c_{1}$ and $c_{2}$ are two
constants. Figure 2(a) shows the fits of this function with the simulation
data. We have used $\psi=1.0$ for the passive system, although note that a
slight variation (0.8 to 1.0) in this value does not affect the fit
noticeably. Fits with the passive system data provides the values of
$c_{1}\simeq-1.624$ and $c_{2}\simeq 2.335$. We then use only $\psi$ as the
fitting parameter in the rest of the fits for the active system data. Figure
2(b) shows the values of $\psi$, obtained from the fits. Note the remarkably
similar trend of $\psi$ with that of $(d-\theta)$ in Fig. 1(d).

To understand the effective barrier and modification of $\psi$, let us
consider a barrier crossing mechanism in one dimension. For concreteness,
consider a two-well potential, $U(q)=-(a/2)q^{2}+(b/4)q^{4}+a^{2}/4b$. The
potential is characterized by the two minima at $q=\pm\sqrt{a/b}$ and a maxima
at $q=0$ with the barrier height $a^{2}/4b$. Consider the barrier crossing of
the system from one minimum to the other in the presence of the active noise,
$\zeta$. Since we are interested in the role of active force, ignoring the
thermal noise, we write the equation of motion of the particle as

$\displaystyle\gamma\dot{q}(t)$ $\displaystyle=F(q)+f_{0}\zeta(t)$ (5)
$\displaystyle\tau_{p}\dot{\zeta}(t)$
$\displaystyle=-\zeta(t)+\sqrt{\tau_{p}}\eta(t)$ (6)

where $F(q)=\partial_{q}U(q)$, $\gamma$ is the friction force, $\eta(t)$ is a
Gaussian white noise with zero mean and
$\langle\eta(t)\eta(t^{\prime})\rangle=\delta(t-t^{\prime})$. With a slight
algebraic manipulation, we can write the above two equations as

$\displaystyle\gamma\tau_{p}\ddot{q}=-[\gamma-\tau_{p}\partial_{q}F(q)]\dot{q}+F(q)+f_{0}\sqrt{\tau_{p}}\eta(t).$
(7)

The above equation essentially describes a barrier crossing scenario with
effective friction $\Gamma=\gamma-\tau_{p}\partial_{q}F(q)$ and the strength
to cross the barrier is provided via active force. $\Gamma$ shows that the
effective friction to cross the barrier reduces when $\tau_{p}$ is appreciably
larger compared to the time-scale of thermal noise. However, it also depends
on the nature of the barrier, in particular on $\partial_{q}^{2}F(q)$. Thus as
the particle approaches the barrier maxima, the second derivative becomes
progressively smaller, and so does the effective friction that becomes
negative beyond a certain point. Thus the barrier effectively exists till this
point. Even in the absence of thermal noise, activity can provide the required
fluctuations to cross the barrier. In our case, when we are varying $f_{0}$
alone, the effective barrier reduction is provided by both $f_{0}$ and
$\xi_{S}$. Since $\xi_{S}$ is not very large in a glassy system, and the
minimum value is 1, we propose the following form: $1-cf_{0}(\xi_{S}-1)\simeq
1-cf_{0}\ln\xi_{S}$. In the limit of small activity, we can write this as
$\xi_{S}^{-cf_{0}}$. Thus the effective barrier height becomes $\xi_{S}^{\psi-
cf_{0}}$; this is the form that we have used to fit the variation in $\psi$,
Fig. 2(b). Note that this simple argument does not work when $\tau_{p}$ is
very small, and the scenario resembles the thermal case. On the other hand,
when $\tau_{p}$ is very large, such that $\Gamma$ remains almost always
negative, other distinct types of phenomena, resembling jammed systems Mandal
_et al._ (2020), are expected to emerge.

Figure 3: (a) For the relaxation dynamics, Eq. (4), the combination
$\psi/(d-\theta)$ is important. We find that although the individual exponents
depend on activity, the combination remains almost independent of activity.
(b) The experimental glass transition, $T_{g}$, as a function of $f_{0}$. (c)
Angell plot for relaxation time, $\tau$, as a function of $T_{g}/T$ at
different activity. (d) Kinetic fragility, $\kappa$, decreases as $f_{0}$
increases. Symbols represent simulation data, and the solid lines are
theoretical predictions (not fits).

The cornerstone of RFOT theory is the mosaic picture with the typical length
scale $\xi_{S}$. The theory provides the expression of $\tau$ via the
relaxation of a typical mosaic. Thus, the exponents $\theta$ and $\psi$ are
pivotal within the theory. We have independently obtained these exponents, as
shown in Figs. (1d) and (2b), and find that activity affects both of them.
Despite this nontrivial activity-dependence of the exponents, how the
relaxation dynamics looks equilibrium-like, and how did the formulation of
Ref. Nandi  _et al._ (2018) provide good agreements with the simulation data?
As evident from Eq. (4), within RFOT theory, the combination,
$\psi/(d-\theta)$, is the important parameter for the relaxation dynamics.
Figure 3(a) shows this combination, together with the individual exponents.
Despite the strong activity dependence of the individual parameters, where
$\psi$ and $\theta$ changes almost linearly with $f_{0}$, the value of
$\psi/(d-\theta)$ remains nearly constant and equal to one; the same value as
Ref. Nandi  _et al._ (2018), which was motivated from the argument that the
relaxation dynamics remains effectively equilibrium-like. This explains the
effective equilibrium-like behavior of the relaxation dynamics.

Finally, we show some further comparisons of the theory with simulation data.
Note that we have already obtained all the constants within the theory. Figure
3(b) shows the glass transition temperature, $T_{g}$, defined via
$\tau(T_{g})=10^{6}$, both in simulation and theory. In our simulations, we
fit the data with the Vogel-Fulcher form $\tau=\tau_{0}\exp[E/(T-T_{K})]$ and
obtain $T_{g}$ using these parameters. The theory agrees quite well with the
simulation data. Figure 3(c) shows the relaxation time data in the Angell plot
representation. Figure 3(d) shows the kinetic fragility obtained via fitting
the data of $\tau$ with the form
$\log(\tau/\tau_{0})=[\kappa(f_{0})(T/T_{K}-1)]^{-1}$ and that from the
theory. Note that none of these theoretical curves are fits; they are plots as
obtained from Eq. (4) with $\psi/(d-\theta)=1$ with the fitting parameters
already fixed previously. These comparisons show that the theory agrees
remarkably well with the simulation results.

The active RFOT theory presented here and in Ref. Nandi  _et al._ (2018) is a
phenomenological extension of the equilibrium theory for active systems. In
principle, both the surface energy and configurational entropy terms should be
affected by activity. However, comparison with simulation data shows the
activity correction to the surface energy term is small, and we have ignored
it for simplicity. The inclusion of this correction does not affect our
qualitative results. We emphasize that the other nontrivial effect of activity
on the surface energy, namely in the exponent $\theta$, is contained within
the theory presented here and forms the basis of one of the main results of
this work. The additional correction in the high-activity regime, proposed in
Ref. Mandal  _et al._ (2021), does not seem to be important in the regime of
our interest in this work. We emphasize that activity leads to much higher
values of $\xi_{S}$, as shown in Fig. 1, compared to equilibrium systems. The
RFOT theory exponents, $\psi$ and $\theta$, depend on activity. The
thermodynamic transition point, $T_{K}^{eff}$, gets shifted to lower $T$. In
particular, one can go to a regime where $T_{K}^{eff}$ becomes negative and
finds sub-Arrhenius behavior of $\tau$. However, this can happen at relatively
higher densities, unlike equilibrium systems Berthier and Witten (2009);
Adhikari  _et al._ (2021). Sub-Arrhenius behavior is readily found in
confluent systems Sadhukan and Nandi (2021). Whether the origins of such
behavior in these distinctively different systems are similar or not remains
unclear.

In conclusion, the results presented here show that although the relaxation
dynamics in an active glass remains equilibrium-like, activity has nontrivial
effects on the glassy properties. These nontrivial effects manifest via the
activity dependence of the RFOT theory exponents, $\theta$ and $\psi$. But,
their combination, $\psi/(d-\theta)$, becomes nearly independent of activity.
This specific combination of the exponents controls the relaxation dynamics
(Eq. 4). This explains the effectively equilibrium-like nature of the
relaxation dynamics in active systems. In addition, we find the static length
scale, $\xi_{S}$, in the presence of activity can grow much higher than in
equilibrium. Given the frustratingly small values of $\xi_{S}$ in equilibrium
systems, the nontrivial effects of activity along with larger $\xi_{S}$ should
encourage devising more rigorous theories whose predictions can be easier to
compare with simulations and experiments of active glasses.

Acknowledgments: We acknowledge support of the Department of Atomic Energy,
Government of India, under Project Identification No. RTI 4007. SK
acknowledges support from Core Research Grant CRG/2019/005373 from Science and
Engineering Research Board (SERB) as well as Swarna Jayanti Fellowship grants
DST/SJF/PSA-01/2018-19 and SB/SFJ/2019-20/05. KP acknowledges financial
support from SB/SFJ/2019-20/05.

## References

  * Berthier and Biroli (2011) L. Berthier and G. Biroli, Rev. Mod. Phys. 83, 587 (2011).
  * Lubchenko and Wolynes (2007) V. Lubchenko and P. G. Wolynes, Annu. Rev. Phys. Chem. 58, 235 (2007).
  * Das (2004) S. P. Das, Rev. Mod. Phys. 76, 785 (2004).
  * Dyre  _et al._ (2009) J. C. Dyre, T. Hechsher, and K. Niss, J. Non-Cryst. Solids 355, 624 (2009).
  * Biroli and Bouchaud (2012) G. Biroli and J. P. Bouchaud, in  _Structural Glasses and Supercooled Liquids: Theory, Experiment, and Applications_ , edited by P. G. Wolynes and V. Lubchenko (John Wiley and Sons, Inc., Hoboken, NJ, USA, 2012).
  * Tarjus  _et al._ (2005) G. Tarjus, S. A. Kivelson, Z. Nussinov, and P. Viot, J. Phys.: Condens. Matter 17, R1143 (2005).
  * Karmakar  _et al._ (2014) S. Karmakar, C. Dasgupta, and S. Sastry, Annual Review of Condensed Matter Physics 5, 255 (2014).
  * Ramaswamy (2010) S. Ramaswamy, Annu. Rev. Condens. Matter Phys. 1, 323 (2010).
  * Marchetti  _et al._ (2013) M. C. Marchetti, J. F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Rev. Mod. Phys. 85, 1143 (2013).
  * Poujade  _et al._ (2007) M. Poujade, E. Grasland-Mongrain, A. Hertzog, J. Jouanneau, P. Chavrier, B. Ladoux, A. Buguin, and P. Silberzan, Proc. Natl. Acad. Sci. (USA) 104, 15988 (2007).
  * Cochet-Escartin  _et al._ (2014) O. Cochet-Escartin, J. Ranft, P. Silberzan, and P. Marcq, Biophys. J. 106, 65 (2014).
  * Angelini  _et al._ (2011) T. E. Angelini, E. Hannezo, X. Trepat, M. Marquez, J. J. Fredberg, and D. A. Weitz, Proc. Natl. Acad. Sci. (USA) 108, 4717 (2011).
  * Park  _et al._ (2015) J.-A. Park, J. H. Kim, D. Bi, J. A. Mitchel, N. T. Qazvini, K. Tantisira, C. Y. Park, M. McGill, S.-H. Kim, B. Gweon, J. Notbohm, R. S. Jr, S. Burger, S. H. Randell, A. T. Kho, D. T. Tambe, C. Hardin, S. A. Shore, E. Israel, D. A. Weitz, D. J. Tschumperlin, E. P. Henske, S. T. Weiss, M. L. Manning, J. P. Butler, J. M. Drazen, and J. J. Fredberg, Nat. Mat. 14, 1040 (2015).
  * Malinverno  _et al._ (2017) C. Malinverno, S. Corallino, F. Giavazzi, M. Bergert, Q. Li, M. Leoni, A. Disanza, E. Frittoli, A. Oldani, E. Martini, T. Lendenmann, G. Deflorian, G. V. Beznoussenko, D. Poulikakos, K. H. Ong, M. Uroz, X. Trepat, D. Parazzoli, P. Maiuri, W. Yu, A. Ferrari, R. Cerbino, and G. Scita, Nat. Mat. 16, 587 (2017).
  * Garcia  _et al._ (2015) S. Garcia, E. Hannezo, J. Elgeti, J. F. Joanny, P. Silberzan, and N. S. Gov, Proc. Natl. Acad. Sci. (USA) 112, 15314 (2015).
  * Parry  _et al._ (2014) B. R. Parry, I. V. Surovtsev, M. T. Cabenn, C. S. O’Hern, E. R. Dufresne, and C. Jacobs-Wagner, Cell 156, 183 (2014).
  * Nishizawa  _et al._ (2017) K. Nishizawa, K. Fujiwara, M. Ikenaga, N. Nakajo, M. Yanagisawa, and D. Mizuno, Sci. Rep. 7, 15143 (2017).
  * Zhou  _et al._ (2009) E. H. Zhou, X. Trepat, C. Y. Park, G. Lenormand, M. N. Oliver, S. M. Mijailovich, C. Hardin, D. A. Weitz, J. P. Butler, and J. J. Fredberg, Proc. Natl. Acad. Sci. (USA) 106, 10632 (2009).
  * Malmi-Kakkada  _et al._ (2018) A. N. Malmi-Kakkada, X. Li, H. S. Samanta, S. Sinha, and D. Thirumalai, Phys. Rev. X 8, 021025 (2018).
  * Streitberger  _et al._ (2020) K.-J. Streitberger, L. Lilaj, F. Schrank, a. K.-T. H. Jürgen Braun, M. Reiss-Zimmermann, J. A. Käs, and I. Sack, Proc. Natl. Acad. Sci. (USA) 117, 128 (2020).
  * Berthier (2014) L. Berthier, Phys. Rev. Lett. 112, 220602 (2014).
  * Paul  _et al._ (2021) K. Paul, S. K. Nandi, and S. Karmakar, arXiv: 2105.12702 (2021).
  * Mandal  _et al._ (2016) R. Mandal, P. J. Bhuyan, M. Rao, and C. Dasgupta, Soft Matter 12, 6268 (2016).
  * Flenner  _et al._ (2016) E. Flenner, G. Szamel, and L. Berthier, Soft Matter 12, 7136 (2016).
  * Nandi and Gov (2017) S. K. Nandi and N. S. Gov, Soft Matter 13, 7609 (2017).
  * Berthier and Kurchan (2013) L. Berthier and J. Kurchan, Nat. Phys. 9, 310 (2013).
  * Feng and Hou (2017) M. Feng and Z. Hou, Soft Matter 13, 4464 (2017).
  * Szamel (2016) G. Szamel, Phys. Rev. E 93, 012603 (2016).
  * Nandi  _et al._ (2018) S. K. Nandi, R. Mandal, P. J. Bhuyan, C. Dasgupta, M. Rao, and N. S. Gov, Proc. Natl. Acad. Sci. (USA) 115, 7688 (2018).
  * Ni  _et al._ (2013) R. Ni, M. A. C. Stuart, and M. Dijkstra, Nat. Commun 4, 2704 (2013).
  * Vicsek  _et al._ (1995) T. Vicsek, A. Czirók, E. Ben-Jacob, I. Cohen, and O. Shochet, Phys. Rev. Lett. 75, 1226 (1995).
  * Narayan  _et al._ (2007) V. Narayan, S. Ramaswamy, and N. Menon, Science 317, 105 (2007).
  * Giavazzi  _et al._ (2017) F. Giavazzi, C. Malinverno, S. Corallino, F. Ginelli, G. Scita, and R. Cerbino, J. Phys. D: Appl. Phys. 50, 384003 (2017).
  * Caprini  _et al._ (2020) L. Caprini, U. M. B. Marconi, C. Maggi, M. Paoluzzi, and A. Puglisi, Phys. Rev. Res. 2, 023321 (2020).
  * Berthier  _et al._ (2019) L. Berthier, E. Flenner, and G. Szamel, J. Chem. Phys. 150, 200901 (2019).
  * Karmakar  _et al._ (2012) S. Karmakar, E. Lerner, and I. Procaccia, Physica A: Statistical Mechanics and its Applications 391, 1001 (2012).
  * Cammarota and Biroli (2012) C. Cammarota and G. Biroli, Proc. Natl. Acad. Sci. (USA) 109, 8850 (2012).
  * Chakrabarty  _et al._ (2015) S. Chakrabarty, S. Karmakar, and C. Dasgupta, Scientific reports 5, 12577 (2015).
  * Das  _et al._ (2017) R. Das, S. Chakrabarty, and S. Karmakar, Soft Matter 13, 6929 (2017).
  * Ozawa  _et al._ (2015) M. Ozawa, W. Kob, A. Ikeda, and K. Miyazaki, Proc. Natl. Acad. Sci. (USA) 112, 6914 (2015).
  * Prost  _et al._ (2015) J. Prost, F. Jülicher, and J. F. Joanny, Nat. Phys. 11, 111 (2015).
  * Karmakar  _et al._ (2009) S. Karmakar, C. Dasgupta, and S. Sastry, Proc. Natl. Acad. Sci. (USA) 106, 3675 (2009).
  * Chakrabarty  _et al._ (2017) S. Chakrabarty, I. Tah, S. Karmakar, and C. Dasgupta, Phys. Rev. Lett. 119, 205502 (2017).
  * Tah  _et al._ (2021) I. Tah, A. Mutneja, and S. Karmakar, ACS Omega 6, 7229–7239 (2021).
  * Kirkpatrick and Thirumalai (2015) T. R. Kirkpatrick and D. Thirumalai, Rev. Mod. Phys. 87, 183 (2015).
  * Kirkpatrick  _et al._ (1989) T. R. Kirkpatrick, D. Thirumalai, and P. G. Wolynes, Phys. Rev. A 40, 1045 (1989).
  * Mandal  _et al._ (2020) R. Mandal, P. J. Bhuyan, P. Chaudhuri, C. Dasgupta, and M. Rao, Nat. Comm. 11, 2581 (2020).
  * Mandal  _et al._ (2021) R. Mandal, S. K. Nandi, C. Dasgupta, P. Sollich, and N. S. Gov, arXiv: 2102.07519 (2021).
  * Berthier and Witten (2009) L. Berthier and T. A. Witten, Phys. Rev. E 80, 021502 (2009).
  * Adhikari  _et al._ (2021) M. Adhikari, S. Karmakar, and S. Sastry, The Journal of Physical Chemistry B 125, 10232 (2021), pMID: 34494429, https://doi.org/10.1021/acs.jpcb.1c03887 .
  * Sadhukan and Nandi (2021) S. Sadhukan and S. K. Nandi, Phys. Rev. E 103, 062403 (2021).