research includes perturbations to various aspects of agents in RL, such as
states, rewards, and actions. One category of methods introduces auxiliary
adversarial attackers, achieving robustness through adversarial training
alternating between the policy in use and the opponent system [432, 433, 434,
435]. Other methods enhance robustness by designing appropriate regularization
terms in the loss function [436, 437, 438]. In comparison to adversarial
training, these methods effectively improve sample efficiency. However, these
approaches lack certifiable robustness guarantees regarding the noise level
and the robustness between policy execution. In response to this issue,
several verifiable robustness methods have been derived [361, 439, 440, 441].

Currently, research on robustness in MARL has started to attract attention
[45]. The main challenges lie in the additional considerations required for
MASs compared to single-agent systems, including non-stationarity arising from
complex interactions among agents [21], trust allocation [22], scalability
[24], etc. Early work aimed to investigate whether cooperative policies
exhibit robustness. For instance, targeting a cooperation policy trained using
the QMIX algorithm, literature [442] trained an attacker against observations
using RL. Subsequently, literature [45] conducted a comprehensive robustness
test on typical MARL algorithms such as QMIX and MAPPO in terms of rewards,
states, and actions. The results further confirmed the vulnerability of MASs
to attacks, emphasizing the necessity and urgency of research on robust MARL.
Recent progress in enhancing robustness in MARL includes work focusing on
learning robust cooperative policies to avoid overfitting to specific
teammates [46] and opponents [443]. Similar to robust SARL, R-MADDPG [47]
addressed the uncertainty of the MAS’s model, establishing the concept of
robust Nash equilibrium under model uncertainty and achieving optimal
robustness in multiple environments. Addressing the issue of perturbed actions
in some agents within MASs, literature [48] introduces a heuristic rule and
relevant equilibrium theory to learn a robust cooperative policy for MASs. The
robustness of multi-agent communication has also received attention in recent
years. Literature [444] designed a filter based on Gaussian processes to
extract valuable information from noisy communication. Literature [445]
studied the robustness of multi-agent communication systems at the neural
network level. Literature [49] modeled multi-agent communication as a two-
player zero-sum game and applies PSRO technology to learn a robust
communication policy. Works such as ARTS [446] and RADAR [447] considered the
resilience of MASs and study the recovery capabilities of cooperative MARL
tasks in the face of environmental changes.

Recently, addressing the challenge of cooperation robustness in dynamically
changing environments, literature [358] proposes the ROMANCE algorithm, an
robust cooperation algorithm against evolutionary auxiliary adversarial
attackers. Literature [169] introduces the MA3C framework for adversarial
robust multi-agent communication training through population adversarial
training. Literature [171] proposes the CroMAC method for verifiable robust
communication with a multi-view information perspective. Research on perturbed
multi-agent observations is discussed in literature [359], and literature
[448] considers learning a robust policy against state attacks in MASs.

####  4.2.6 Multi-Objective (Constrained) Cooperative Multi-Agent
Reinforcement Learning

Multi-objective optimization [449] refers to the existence of multiple
objective functions in an optimization problem, requiring simultaneous
consideration of the optimal solutions for each objective. In multi-objective
optimization problems, conflicts may arise between different objective
functions, meaning that improving one objective function may lead to the
deterioration of another. Therefore, a balance needs to be struck between
different objective functions to find a compromise point, also known as the
Pareto optimal solution set.

In RL, multi-objective optimization is frequently employed. For example, in
multi-objective RL, agents need to learn Pareto optimal policies across
multiple objectives [450, 451, 452, 453]. Similarly, in MARL, some works
introduce multi-objective learning problems, generally modeled as
Multiobjective MASs (MOMAS) [454], where reward functions for different
objectives may conflict. Literature [455] considered the relationship between
individual preferences and shared goals in MASs, modeling it as a multi-
objective problem. The results indicate that a mixed processing approach
achieves better performance than solely considering a single objective.
Literature [456] exploreed how communication and commitment help multi-agents
learn appropriate policies in challenging environments. Literature [457]
addressed the multi-objective opponent modeling problem in general game
problems, accelerating policy learning through multi-objectives.

On another note, recent works focus on single-agent safe RL [458, 459] and
multi-agent safe RL [274]. These constrained problems in RL settings can be
modeled as Constrained Markov Decision Processes (CMDP). In MARL, literature
[274] proposed testing environments, Safe MAMuJoCo, Safe MARobosuite, and Safe
MAIG, for multi-agent tasks. They further propose safe MARL algorithms MACPO
and MAPPO-Lagrangian. Literature [460] studied online secure MARL in
constrained Markov games, where agents compete by maximizing their expected
total utility while constraining the expected total reward. Literature [461]
investigated safe MARL, where agents attempt to jointly maximize the sum of
local objectives while satisfying their individual safety constraints. CAMA
[360] explored safety issues in multi-agent coordination. Literature [462]
considered robustness and safety issues in MASs when states are perturbed.
Additionally, some works consider safety in MARL based on barrier protection
[362, 363, 206], or combine safety in MASs with control techniques [361, 463].

####  4.2.7 Risk-Aware Multi-Agent Reinforcement Learning

Distributional RL has made significant progress in various domains in recent
years [464]. Classical value-based RL methods attempt to model the cumulative
return using expected values, represented as value functions $V(s)$ or action-
value functions $Q(s,a)$. However, complete distribution information is
largely lost in this modeling process. Distributional RL aims to address this
issue by modeling the distribution $Z(s,a)$ of the random variable
representing the cumulative return. Such methods are also applied to multi-
agent cooperation tasks. To alleviate the randomness in the environment caused
by local observability, DFAC [364] extended the reward function of a single
agent from a deterministic variable to a random variable. It models the mixing
function of QMIX-type algorithms as a distributional mixing function,
achieving excellent cooperation results in various challenging tasks.
Furthermore, to mitigate the uncertainty introduced by the stochasticity of
reward functions in multi-agent cooperation tasks, RMIX [365] utilized risk-
based distributional techniques, such as Conditional Value at Risk (CVaR), to
enhance the algorithm’s cooperation ability. The algorithm innovatively
introduces risk assessment based on the similarity of agent trajectories, with
theoretical justification and experimental results verifying its
effectiveness. ROE [160] proposed a risk-based optimistic exploration method
from another perspective. This method selectively samples distributions to
effectively improve the exploration efficiency of MASs.

In addition to distribution-based multi-agent cooperation algorithms mentioned
above, other works explore various aspects, such as reward evaluation based on
value distribution [366], efficient and adaptive multi-agent policy learning
in general game problems [368], risk decoupling learning in the multi-agent
learning process [367], and risk management based on game theory [369].
Although these works have achieved certain results in various environments,
considering the unknown risks in real environments, exploring how to deploy
multi-agent policies in real environments, automatically identifying
environmental risks, and adjusting cooperative policies accordingly is one of
the future research directions.

####  4.2.8 Ad-hoc Teamwork

Ad-hoc teamwork (AHT) [50] aims to endow intelligent agents with the ability
to efficiently collaborate with untrained agents, creating autonomous agents
capable of effective and robust coordination with previously unknown teammates
[370]. Early work assumed knowledge of the cooperative behavior of teammates
for learning agents [370, 465]. Subsequent research gradually relaxed this
assumption, allowing autonomous learning agents to be unaware of teammates’
behaviors during interaction. Some approaches design algorithms to predict
corresponding teammates’ policies by observing their behaviors, promoting
coordination in AHT [466, 467, 468, 469]. Other works attempted to enhance
cooperation among teammates in AHT through effective communication
methods[470]. While these approaches improve coordination performance to some
extent, they assume that cooperating teammates are in a closed environment,
maintaining a constant number and type of teammates in a single trajectory.
Open AHT has been proposed and studied to address this limitation [371], where
GPL [373] tackles changes in teammate types and quantities at different time
points through a graph neural network.

Early AHT work generally considered agents in globally observable
environments. Recent efforts extend this setting to scenarios with locally
observable scenes. ODITS [372] evaluated other teammates’ behaviors using
mutual information-optimized regularization, enabling trained autonomous
agents to infer teammate behavior from local observations. In contrast to
previous studies, a method is proposed to address open-ended temporary
teamwork in partially observable scenarios [471]. TEAMSTER [376] introduced a
method to decouple world model learning from teammate behavior model learning.
Additionally, some work explores various aspects, including AHT problems with
attackers [472], few-shot interaction coordination [52], and teammate
generation coverage in AHT [374, 375] .

####  4.2.9 Zero (Few)-Shot Coordination

Zero-shot coordination (ZSC) is a concept proposed in recent years for
cooperative multi-agent tasks, aiming to train agents to cooperate with unseen
teammates [51]. Self-play [473, 474] is one means of effectively improving
coordination, where agents continuously enhance their coordination abilities
through self-coordination. However, agents generated in this way may lack the
ability to collaborate with unseen teammates. Literature [475] further refined
the problem, involving introducing sequence-independent training methods to
alleviate suboptimality. To address potential overfitting to specific teammate
behavior styles resulting from training with a single teammate, other methods,
such as Fictitious Co-Play (FCP) [476, 377] and co-evolution with teammate
populations [379], have achieved success. Some works leverage few-shot
techniques to cope with multimodal scenarios and have shown effectiveness
[380, 353]. Recent research [477] evaluates and quantifies the capacity of
various ZSC algorithms based on the action preferences of collaborating
teammates.

Beyond these efforts, ZSC research includes diversity metrics [378, 478],
design of training paradigms [475, 377, 379], equivariant network design
[479], coordination enhancement based on policy similarity evaluation [480],
general scenarios of ZSC problems [481], ZSC improvement based on ensemble
techniques [481], studies on human value preferences [382], diverse teammate
generation [381], and policy coevolution in heterogeneous environments [379].
Additionally, few-shot adaptation has found widespread use in single-agent
meta-RL [482, 483, 484], and Few-shot Teamwork (FST) [52] explores generating
agents that can adapt and collaborate in unknown but relevant tasks. CSP [380]
considered a multi-modal cooperation paradigm and develops a few-shot
collaboration paradigm that decouples collaboration and exploration
strategies, it collects a small number of samples during policy execution to
find the optimal policy head. The study in [383] found that current
performance-competitive ZSC algorithms require a considerable number of
samples to adapt to new teammates when facing different learning methods.
Accordingly, they propose a few-shot collaborative method and validate the
algorithm’s effectiveness on Hanabi. Macop [354] considered the adaptability
of strategies under changes in cooperative objects between rounds and proposes
a highly compatible collaboration algorithm for arbitrary teammates,
significantly improving the collaboration algorithm’s generalization ability,
showing astonishing coordination effectiveness.

####  4.2.10 Human-AI Coordination

The capability to enable efficient collaboration between agents (robots) and
humans has been a longstanding goal in artificial intelligence [485, 486]. The
concept of human-AI coordination [258] has a relationship with Human-AI
Interaction (HAI) [487] or Human-Robot Interaction (HRI) [488]. The purpose of
human-AI coordination is to enhance collaboration between human participants
and agents to accomplish specific tasks. Benefiting the strong problem solving
ability, cooperative MARL can be employed to improve human-AI coordination
capabilities for different human participants.

In contrast to the previously mentioned ZSC issue, human-AI coordination
considers human participants as cooperative entities. Although research has
shown that, in some environments, it might be possible for intelligent agents
to collaborate with real humans without training on human data [377], in
scenarios where subtle features of human behavior critically impact the task,
generating effective collaborative strategies is impossible without human
data. For the intelligent agents to be trained, one approach is to directly
encode teammates with human behavioral styles through prior bias [475, 489,
490]. Another approach involves training the agents to varying degrees using
data collected from interactions with real humans. Some methods combine hand-
coding human behaviors based on prior bias with optimizing agents using data
from human interactions [491, 492, 493].

However, these methods make strong assumptions about the patterns of human
behavior during testing, which are often unrealistic. In response to this
issue, a series of methods have emerged to learn models of human behavior and
compute the best responses to them, facilitating human-AI coordination [384].
In the realm of human-aided collaboration, some methods are exploring
alternative perspectives, such as studying task scenarios preferred by humans
[382], promoting human-machine coordination through offline data [385],
developing technology for human-AI mutual cooperation [386], exploring
leadership and following technologies in human-machine coordination [494],
zero-shot human-AI coordination [495], Bayesian optimization-based human-AI
coordination [496], and setting up human-machine collaboration environments
[497]. Although these works have made some progress in human-aided
collaboration, several challenges persist in this direction. For instance,
there is a lack of convenient and effective testing environments; most work is
primarily conducted on Overcooked [258], which has a limited number of agents
and overly simple scenarios. Additionally, some studies mainly validate in
third-party non-open-source environments like custom-made robotic arms, posing
challenges in developing versatile testing environments suitable for various
task requirements and human participants, as well as designing more efficient
algorithms. On the other hand, approaches such as human value alignment [498,
499], and human in-the-loop training [500] could be potential solutions to
address these issues in the future.

####  4.2.11 Cooperative Multi-Agent Reinforcement Learning with Large
Language Models

The development of large models, especially large language models [501], has
gained widespread attention and application in various fields in recent years.
Some recent works are exploring universal decision-making large models [502,
503] and applying them in different contexts. In SARL tasks, research works
like GATO [504], DreamerV3 [505], and DT [506] have achieved surprising
results in many task scenarios. These works leverage the powerful expressive
capabilities of existing technologies such as Transformer [96, 507]. On the
other hand, some recent works are attempting to learn universal decision-
making large models for MASs. For example, MADT [344] promotes research by
providing a large-scale dataset and explores the application of DT in MARL
environments. MAT [32] studies an effective large model method to transform
MARL into a single-agent problem, aiming to map the observation sequences of
agents to optimal action sequences, demonstrating superior performance in
multiple task scenarios compared to traditional methods. Addressing the
information of entities in multi-agent environments, literature [388] proposed
MAGENTA, a study orthogonal to previous time-series modeling. MADiff [389] and
DOM2 [508] introduced generative diffusion models to MARL, promoting
collaboration in various scenarios. SCT [509] accelerates multi-agent online
adaptation using a Transformer model. Literature [510] constructed a humanoid
multi-agent environment, "West World," to simulate and test MASs in large-
scale scenarios.

Additionally, with the development of large language models represented by
ChatGPT, some works are attempting to promote multi-agent collaboration
through language models. For instance, EnDi [511] uses natural language to
enhance the generalization ability of MASs. InstructRL [512] allows humans to
obtain the desired agent strategy through natural language instructions. SAMA
[390] proposes semantically aligned multi-agent collaboration, automatically
assigning goals to MASs using pre-trained language prompts, achieving
impressive collaboration results in various scenarios. HAPLAN [115] utilizes
large language model like ChatGPT to bride the gap between human and AI for
efficient coordination. ProAgent [513] introduces an efficient human-machine
collaboration framework that leverages large language models for teammate
behavior prediction, achieving the best collaborative performance in human-
machine collaboration tasks. On the other hand, some works apply multi-agent
collaboration methods to enhance the capabilities of large language models
[514, 515, 516]. However, due to reasons such as complex interactions, the
field of universal decision-making large models for multi-agents is currently
less explored. Challenges such as learning a universal multi-agent decision-
making large model that can generalize with zero or few shot across various
scenarios or quickly adapt to new tasks through fine-tuning are worth
researching.

##  5 Summary and Prospect

This paper focuses on the development and research of cooperative MARL,
progressing from classical closed environments to open environments that align
with real-world applications. It provides a comprehensive introduction to
reinforcement learning, multi-agent systems, multi-agent reinforcement
learning, and cooperative multi-agent reinforcement learning. Summarizing
different research directions, it extracts and condenses the key research
focuses on MARL in classical environments. Despite the success of many closed-
environment MARL algorithms based on the closed-world assumption, their
application in the real world remains limited. This limitation is largely
attributed to the lack of targeted research on the characteristics of open
environments. There exists a substantial gap between the current methods and
the actual empowerment of daily life. To overcome the challenges posed by the
complexity, dynamism, and numerous constraints in open environments, future
research in cooperative MARL could address the following aspects. This will
trigger more attention and exploration of MARL in open environments, enabling
better application of cooperative MASs in real-world settings to enhance human
life.

  * •

Solutions of Multi-Agent Coordination Issues in Classical Closed Environments:
Algorithms for Cooperative Multi-agent Reinforcement Learning in classical
environments serve as the foundation for transitioning to open environments.
Improving the coordination performance in closed environments enhances the
broader applicability potential of these systems. However, challenges persist
in large-scale scenarios with numerous intelligent agents, such as efficient
policy optimization [517], and balancing the relationship between distributed
and centralized aspects during training and execution. These issues require
careful consideration and resolution in future research.

  * •

Theoretical Analysis and Framework Construction in Open Environments: Open
environments present more stringent conditions and greater challenges compared
to closed ones. While some efforts have utilized heuristic rules to design the
openness of machine learning environments [38], the establishment of a
comprehensive framework, including a well-defined concept of openness in
multi-agent coordination, definitions of environmental openness, and the
performance boundaries of algorithms, is a crucial area for future research.

  * •

Construction of Testing Environments for Cooperative MASs in Open
Environments: Despite ongoing research on robustness in open environments for
cooperative MARL, benchmark testing often involves modifications in classical
closed environments. These approaches lack compatibility with various
challenges posed by different open-ended scenarios. Future research could
focus on constructing testing environments that evaluate eleven identified
aspects, providing a significant boost to the study of cooperative MASs in
open environments.

  * •

Development of General Decision-Making Large Language Models for Multi-agent
Systems in Open Environments: Large models, especially large language models
[501], have garnered attention and applications in various fields. Some works
explore decision-making large language models in multi-agent settings [503,
518], yet there is still a considerable gap in research. Future investigations
could focus on learning universal decision-making large language models for
MASs that generalize across diverse task scenarios, achieving zero or few-shot
generalization, or rapid adaptation to unknown task domains through fine-
tuning.

  * •

Application and Implementation of Cooperative Multi-agent Reinforcement
Learning in Real-world Scenarios: While the efficient cooperative performance
of MARL in classical environments holds great application potential, most
studies are limited to testing in simulators or specific task scenarios [519].
There is still a considerable distance from real-world social applications and
current needs. The primary goal of research in cooperative MARL in open
environments remains the application of algorithms to human life and the
promotion of societal progress. In the future, exploring how to safely and
efficiently apply MARL algorithms in areas such as large-scale autonomous
driving, smart cities, and massive computational resource scheduling is a
topic worthy of discussion.

## Acknowledgments

We would like to thank Rongjun Qin, Fuxiang Zhang, Chenghe Wang, Yichen Li, Ke
Xue, Chengxing Jia, Feng Chen, and Zhichao Wu for their helpful discussions
and support.

## References

  * [1] Richard S Sutton and Andrew G Barto.  Reinforcement Learning: An Introduction.  MIT Press, 2018.
  * [2] Ben Goertzel and Cassio Pennachin.  Artificial General Intelligence.  Springer, 2007.
  * [3] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [4] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  preprint arXiv:1312.5602, 2013.
  * [5] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [6] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, 575(7782):350–354, 2019.
  * [7] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  preprint arXiv:1912.06680, 2019.
  * [8] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, and Hsiao-Wuen Hon.  Suphx: Mastering mahjong with deep reinforcement learning.  preprint arXiv:2003.13590, 2020.
  * [9] Yuxi Li.  Deep reinforcement learning: An overview.  preprint arXiv:1701.07274, 2017.
  * [10] Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty.  Reinforcement learning algorithms: A brief survey.  Expert Systems with Applications, page 120495, 2023.
  * [11] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  preprint arXiv:2304.01852, 2023.
  * [12] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al.  Scientific discovery in the age of artificial intelligence.  Nature, 620(7972):47–60, 2023.
  * [13] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan D. Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin A. Riedmiller.  Magnetic control of tokamak plasmas through deep reinforcement learning.  Nat., 602(7897):414–419, 2022.
  * [14] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al.  Discovering faster matrix multiplication algorithms with reinforcement learning.  Nature, 610(7930):47–53, 2022.
  * [15] Ali Dorri, Salil S Kanhere, and Raja Jurdak.  Multi-agent systems: A survey.  IEEE Access, 6:28573–28593, 2018.
  * [16] Yaodong Yang and Jun Wang.  An overview of multi-agent reinforcement learning from game theoretical perspective.  preprint arXiv:2011.00583, 2020.
  * [17] Afshin Oroojlooy and Davood Hajinezhad.  A review of cooperative multi-agent deep reinforcement learning.  Applied Intelligence, 53(11):13677–13722, 2023.
  * [18] Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer.  Multi-Agent Reinforcement Learning: Foundations and Modern Approaches.  MIT Press, 2023.
  * [19] Sven Gronauer and Klaus Diepold.  Multi-agent deep reinforcement learning: a survey.  Artificial Intelligence Review, 55(2):895–943, 2022.
  * [20] Changxi Zhu, Mehdi Dastani, and Shihan Wang.  A survey of multi-agent reinforcement learning with communication.  preprint arXiv:2203.08975, 2022.
  * [21] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht.  Dealing with non-stationarity in multi-agent deep reinforcement learning.  preprint arXiv:1906.04737, 2019.
  * [22] Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, and Chongjie Zhang.  Towards understanding cooperative multi-agent q-learning with value factorization.  In Advances in Neural Information Processing Systems, pages 29142–29155, 2021.
  * [23] Chongjie Zhang.  Scaling multi-agent learning in complex environments.  PhD thesis, 2011.
  * [24] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht.  Scaling multi-agent reinforcement learning with selective parameter sharing.  In Proceedings of the International Conference on Machine Learning, pages 1989–1998, 2021.
  * [25] Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset.  Primal: Pathfinding via reinforcement and imitation multi-agent learning.  IEEE Robotics and Automation Letters, 4(3):2378–2385, 2019.
  * [26] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green.  Multi-agent reinforcement learning for active voltage control on power distribution networks.  In Advances in Neural Information Processing Systems, pages 3271–3284, 2021.
  * [27] Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu.  Multi-agent dynamic algorithm configuration.  In Advances in Neural Information Processing Systems, pages 20147–20161, 2022.
  * [28] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.
  * [29] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu.  The surprising effectiveness of PPO in cooperative multi-agent games.  In Advances in Neural Information Processing Systems, pages 24611–24624, 2022.
  * [30] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, and Thore Graepel.  Value-decomposition networks for cooperative multi-agent learning based on team reward.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 2085–2087, 2018.
  * [31] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 4295–4304, 2018.
  * [32] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang.  Multi-agent reinforcement learning is a sequence modeling problem.  In Advances in Neural Information Processing Systems, pages 16509–16521, 2022.
  * [33] Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson.  The Starcraft multi-agent challenge.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 2186–2188, 2019.
  * [34] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang.  Discovering generalizable multi-agent coordination skills from multi-task offline data.  In International Conference on Learning Representations, 2023.
  * [35] Xihuai Wang, Zhicheng Zhang, and Weinan Zhang.  Model-based multi-agent reinforcement learning: Recent progress and prospects.  preprint arXiv:2203.10603, 2022.
  * [36] Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato.  Contrasting centralized and decentralized critics in multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 844–852, 2021.
  * [37] Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore.  Open-world machine learning: applications, challenges, and opportunities.  ACM Computing Surveys, 55(10):1–37, 2023.
  * [38] Zhi-Hua Zhou.  Open-environment machine learning.  National Science Review, 9(8), 2022.
  * [39] Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao.  Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability.  preprint arXiv:2209.08025, 2022.
  * [40] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley.  Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions.  In Proceedings of the International Conference on Machine Learning, pages 9940–9951, 2020.
  * [41] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup.  Towards continual reinforcement learning: A review and perspectives.  Journal of Artificial Intelligence Research, 75:1401–1476, 2022\.
  * [42] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.  A survey of zero-shot generalisation in deep reinforcement learning.  Journal of Artificial Intelligence Research, 76:201–264, 2023.
  * [43] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson.  A survey of meta-reinforcement learning.  preprint arXiv:2301.08028, 2023.
  * [44] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund.  Sim-to-real transfer in deep reinforcement learning for robotics: a survey.  In Proceedings of the 2020 IEEE Symposium Series on Computational Intelligence, pages 737–744, 2020.
  * [45] Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, and Simin Li.  Towards comprehensive testing on the robustness of cooperative multi-agent reinforcement learning.  In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 114–121, 2022.
  * [46] Tessa van der Heiden, Christoph Salge, Efstratios Gavves, and Herke van Hoof.  Robust multi-agent reinforcement learning with social empowerment for coordination and communication.  preprint arXiv:2012.08255, 2020.
  * [47] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar.  Robust multi-agent reinforcement learning with model uncertainty.  In Advances in Neural Information Processing Systems, pages 10571–10583, 2020.
  * [48] Yizheng Hu, Kun Shao, Dong Li, Jianye Hao, Wulong Liu, Yaodong Yang, Jun Wang, and Zhanxing Zhu.  Robust multi-agent reinforcement learning driven by correlated equilibrium, 2021.
  * [49] Wanqi Xue, Wei Qiu, Bo An, Zinovi Rabinovich, Svetlana Obraztsova, and Chai Kiat Yeo.  Mis-spoke or mis-lead: Achieving robustness in multi-agent communicative reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1418–1426, 2022.
  * [50] Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht.  A survey of ad hoc teamwork research.  In European Conference on Multi-Agent Systems, pages 275–293, 2022\.
  * [51] Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster.  A new formalism, method and open issues for zero-shot coordination.  In Proceedings of the International Conference on Machine Learning, pages 10413–10423, 2021.
  * [52] Elliot Fosong, Arrasy Rahman, Ignacio Carlucho, and Stefano V Albrecht.  Few-shot teamwork.  preprint arXiv:2207.09300, 2022.
  * [53] Yoav Shoham, Rob Powers, and Trond Grenager.  Multi-agent reinforcement learning: a critical survey.  Technical report, Citeseer, 2003.
  * [54] Yoav Shoham, Rob Powers, and Trond Grenager.  If multi-agent learning is the answer, what is the question?  Artificial intelligence, 171(7):365–377, 2007.
  * [55] Lucian Busoniu, Robert Babuska, and Bart De Schutter.  A comprehensive survey of multiagent reinforcement learning.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008.
  * [56] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor.  A survey and critique of multiagent deep reinforcement learning.  Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019\.
  * [57] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi.  Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications.  IEEE transactions on cybernetics, 50(9):3826–3839, 2020.
  * [58] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  Handbook of reinforcement learning and control, pages 321–384, 2021\.
  * [59] Stefano V Albrecht and Peter Stone.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  Artificial Intelligence, 258:66–95, 2018.
  * [60] Felipe Leno Da Silva and Anna Helena Reali Costa.  A survey on transfer learning for multiagent reinforcement learning systems.  Journal of Artificial Intelligence Research, 64:645–703, 2019.
  * [61] Jim E Doran, SRJN Franklin, Nicholas R Jennings, and Timothy J Norman.  On cooperation in multi-agent systems.  The Knowledge Engineering Review, 12(3):309–314, 1997.
  * [62] Liviu Panait and Sean Luke.  Cooperative multi-agent learning: The state of the art.  Autonomous agents and multi-agent systems, 11:387–434, 2005.
  * [63] St John Grimbly, Jonathan Shock, and Arnu Pretorius.  Causal multi-agent reinforcement learning: Review and open problems.  preprint arXiv:2111.06721, 2021.
  * [64] Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu.  Open-world continual learning: Unifying novelty detection and continual learning.  preprint arXiv:2304.10038, 2023.
  * [65] Christopher JCH Watkins and Peter Dayan.  Q-learning.  Machine learning, 8(3):279–292, 1992.
  * [66] Gerald Tesauro.  Temporal difference learning and td-gammon.  Communications of the ACM, 38(3):58–68, 1995.
  * [67] Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu.  A survey on model-based reinforcement learning.  preprint arXiv:2206.09328, 2022.
  * [68] Steven J Bradtke and Andrew G Barto.  Linear least-squares algorithms for temporal difference learning.  Machine learning, 22(1):33–57, 1996.
  * [69] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [70] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas.  Dueling network architectures for deep reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 1995–2003, 2016.
  * [71] Hado van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2094–2100, 2016.
  * [72] Marc G. Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 449–458, 2017.
  * [73] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  In International Conference on Learning Representations, 2017.
  * [74] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell.  Neural episodic control.  In Proceedings of the International Conference on Machine Learning, pages 2827–2836, 2017.
  * [75] Vijay R. Konda and John N. Tsitsiklis.  Actor-critic algorithms.  In Advances in Neural Information Processing Systems, pages 1008–1014, 1999.
  * [76] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  In Advances in Neural Information Processing Systems, pages 4565–4573, 2016.
  * [77] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller.  Deterministic policy gradient algorithms.  In Proceedings of the International Conference on Machine Learning, pages 387–395, 2014.
  * [78] Ronald J. Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 8:229–256, 1992.
  * [79] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas.  Sample efficient actor-critic with experience replay.  In International Conference on Learning Representations, 2017.
  * [80] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  preprint arXiv:1812.05905, 2018.
  * [81] Gerhard Weiss.  Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence.  MIT press, 1999.
  * [82] Drew Fudenberg and Jean Tirole.  Game theory.  MIT press, 1991.
  * [83] Guillermo Owen.  Game theory.  Emerald Group Publishing, 2013.
  * [84] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein.  The complexity of decentralized control of markov decision processes.  Mathematics of operations research, 27(4):819–840, 2002.
  * [85] Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  In Proceedings of the 2015 AAAI Fall Symposia, pages 29–37, 2015\.
  * [86] Ming Tan.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  In Proceedings of the international conference on machine learning, pages 330–337, 1993.
  * [87] Dean Foster, Dylan J Foster, Noah Golowich, and Alexander Rakhlin.  On the complexity of multi-agent decision making: From learning in games to partial monitoring.  In the Annual Conference on Learning Theory, pages 2678–2792, 2023\.
  * [88] Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Brett Daley, and Christopher Amato.  On centralized critics in multi-agent reinforcement learning.  Journal of Artificial Intelligence Research, 77:295–354, 2023.
  * [89] Yihe Zhou, Shunyu Liu, Yunpeng Qing, Kaixuan Chen, Tongya Zheng, Yanhao Huang, Jie Song, and Mingli Song.  Is centralized training with decentralized execution framework centralized enough for marl?  preprint arXiv:2305.17352, 2023.
  * [90] Martin Lauer and Martin Riedmiller.  An algorithm for distributed reinforcement learning in cooperative multi-agent systems.  In Proceedings of the 17th International Conference on Machine Learning, 2000.
  * [91] Caroline Claus and Craig Boutilier.  The dynamics of reinforcement learning in cooperative multiagent systems.  In Proceedings of the National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, pages 746–752, 1998.
  * [92] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.  Counterfactual multi-agent policy gradients.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2974–2982, 2018.
  * [93] Shariq Iqbal and Fei Sha.  Actor-attention-critic for multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 2961–2970, 2019.
  * [94] Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius.  Towards a standardised performance evaluation protocol for cooperative marl.  In Advances in Neural Information Processing Systems, pages 5510–5521, 2022.
  * [95] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang.  QPLEX: Duplex dueling multi-agent Q-learning.  In International Conference on Learning Representations, 2021.
  * [96] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  * [97] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang.  Dop: Off-policy multi-agent decomposed policy gradients.  In International conference on learning representations, 2020.
  * [98] Jianyu Su, Stephen Adams, and Peter Beling.  Value-decomposition multi-agent actor-critics.  In Proceedings of the AAAI conference on artificial intelligence, pages 11352–11360, 2021.
  * [99] Xiaoteng Ma, Yiqin Yang, Chenghao Li, Yiwen Lu, Qianchuan Zhao, and Jun Yang.  Modeling the interaction between agents in cooperative multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 853–861, 2021.
  * [100] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson.  Facmac: Factored multi-agent centralised policy gradients.  In Advances in Neural Information Processing Systems, pages 12208–12221, 2021.
  * [101] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu.  Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 12491–12500, 2021.
  * [102] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao.  Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning.  preprint arXiv:2102.03479, 2021.
  * [103] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang.  Trust region policy optimisation in multi-agent reinforcement learning.  In International Conference on Learning Representations, 2021.
  * [104] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson.  MAVEN: multi-agent variational exploration.  In Advances in Neural Information Processing Systems, pages 7611–7622, 2019.
  * [105] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang.  Influence-based multi-agent exploration.  In International Conference on Learning Representations, 2019.
  * [106] Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang.  Episodic multi-agent reinforcement learning with curiosity-driven exploration.  In Advances in Neural Information Processing Systems, pages 3757–3769, 2021.
  * [107] Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing.  Cooperative exploration for multi-agent deep reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 6826–6836, 2021.
  * [108] Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson.  Uneven: Universal value exploration for multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 3930–3941, 2021.
  * [109] Shaowei Zhang, Jiahan Cao, Lei Yuan, Yang Yu, and De-Chuan Zhan.  Self-motivated multi-agent exploration.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 476–484, 2023.
  * [110] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson.  Learning to communicate with deep multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 2137–2145, 2016.
  * [111] Sai Qian Zhang, Qi Zhang, and Jieyu Lin.  Efficient communication in multi-agent reinforcement learning via variance based control.  In Advances in Neural Information Processing Systems, pages 3230–3239, 2019.
  * [112] Ziluo Ding, Tiejun Huang, and Zongqing Lu.  Learning individually inferred communication for multi-agent cooperation.  In Advances in Neural Information Processing Systems, pages 22069–22079, 2020.
  * [113] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau.  Tarmac: Targeted multi-agent communication.  In Proceedings of the International Conference on Machine Learning, pages 1538–1546, 2019.
  * [114] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang.  Multi-agent incentive communication via decentralized teammate modeling.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9466–9474, 2022.
  * [115] Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu.  Efficient communication via self-supervised information aggregation for online and offline multi-agent reinforcement learning.  preprint arXiv:2302.09605, 2023.
  * [116] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick.  Machine theory of mind.  In Proceedings of the International conference on machine learning, pages 4218–4227, 2018.
  * [117] Georgios Papoudakis and Stefano V Albrecht.  Variational autoencoders for opponent modeling in multi-agent systems.  preprint arXiv:2001.10829, 2020.
  * [118] Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht.  Agent modelling under partial observability for deep reinforcement learning.  In Advances in Neural Information Processing Systems, pages 19210–19222, 2021.
  * [119] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh.  Learning latent representations to influence multi-agent interaction.  In Conference on robot learning, pages 575–588, 2021.
  * [120] Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu.  Model-based opponent modeling.  In Advances in Neural Information Processing Systems, pages 28208–28221, 2022.
  * [121] Lei Yuan, Chenghe Wang, Jianhao Wang, Fuxiang Zhang, Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu.  Multi-agent concentrative coordination with decentralized task representation.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 599–605, 2022.
  * [122] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon.  Multi-agent generative adversarial imitation learning.  In Advances in Neural Information Processing Systems, pages 7472–7483, 2018.
  * [123] Lantao Yu, Jiaming Song, and Stefano Ermon.  Multi-agent adversarial inverse reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 7194–7201, 2019.
  * [124] Minghuan Liu, Ming Zhou, Weinan Zhang, Yuzheng Zhuang, Jun Wang, Wulong Liu, and Yong Yu.  Multi-agent interactions modeling with correlated policies.  In International Conference on Learning Representations, 2019.
  * [125] Caroline Wang, Ishan Durugkar, Elad Liebman, and Peter Stone.  Distributed multi-agent reinforcement learning for distribution matching.  preprint arXiv:2206.00233, 2022.
  * [126] Daniël Willemsen, Mario Coppola, and Guido CHE de Croon.  Mambpo: Sample-efficient multi-robot reinforcement learning using learned world models.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5635–5640, 2021.
  * [127] Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou.  Model-based multi-agent policy optimization with adaptive opponent-wise rollouts.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 3384–3391, 2021.
  * [128] Zhiwei Xu, Bin Zhang, Yuan Zhan, Yunpeng Baiia, Guoliang Fan, et al.  Mingling foresight with imagination: Model-based cooperative multi-agent reinforcement learning.  Advances in Neural Information Processing Systems, pages 11327–11340, 2022.
  * [129] Vladimir Egorov and Alexei Shpilman.  Scalable multi-agent model-based reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 381–390, 2022.
  * [130] Zhizun Wang and David Meger.  Leveraging world model disentanglement in value-based multi-agent reinforcement learning.  preprint arXiv:2309.04615, 2023.
  * [131] S Ahilan and P Dayan.  Feudal multi-agent hierarchies for cooperative reinforcement learning.  In Workshop on Structure & Priors in Reinforcement Learning (SPiRL 2019) at ICLR 2019, pages 1–11, 2019.
  * [132] Jiachen Yang, Igor Borovikov, and Hongyuan Zha.  Hierarchical cooperative multi-agent reinforcement learning with skill discovery.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1566–1574, 2020.
  * [133] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang.  Rode: Learning roles to decompose multi-agent tasks.  In International Conference on Learning Representations, 2020.
  * [134] Shariq Iqbal, Robby Costales, and Fei Sha.  Alma: Hierarchical learning for composite multi-agent tasks.  In Advances in Neural Information Processing Systems, pages 7155–7166, 2022.
  * [135] Zhiwei Xu, Yunpeng Bai, Bin Zhang, Dapeng Li, and Guoliang Fan.  Haven: hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11735–11743, 2023.
  * [136] Carlos Guestrin, Michail G. Lagoudakis, and Ronald Parr.  Coordinated reinforcement learning.  In Proceedings of the International Conference Machine Learning, pages 227–234, 2002.
  * [137] Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson.  Deep coordination graphs.  In Proceedings of the International Conference on Machine Learning, pages 980–991, 2020.
  * [138] Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer.  Deep implicit coordination graphs for multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 764–772, 2021.
  * [139] Yaru Niu, Rohan Paleja, and Matthew Gombolay.  Multi-agent graph-attention communication and teaming.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 964–973, 2021.
  * [140] Jiechuan Jiang and Zongqing Lu.  Learning attentional communication for multi-agent cooperation.  In Advances in Neural Information Processing Systems, pages 7265–7275, 2018.
  * [141] Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang.  Context-aware sparse deep coordination graphs.  In International Conference on Learning Representations, 2022.
  * [142] Zichuan Liu, Yuanyang Zhu, and Chunlin Chen.  Na2q: Neural attention additive model for interpretable multi-agent q-learning.  preprint arXiv:2304.13383, 2023.
  * [143] Chuming Li, Jie Liu, Yinmin Zhang, Yuhong Wei, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang.  Ace: cooperative multi-agent q-learning with bidirectional action-dependency.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8536–8544, 2023.
  * [144] Jiachen Yang, Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha.  Cm3: Cooperative multi-goal multi-stage multi-agent reinforcement learning.  In International Conference on Learning Representations, 2019.
  * [145] Haotian Fu, Hongyao Tang, Jianye Hao, Zihan Lei, Yingfeng Chen, and Changjie Fan.  Deep multi-agent reinforcement learning with discrete-continuous hybrid action spaces.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2329–2335, 2019.
  * [146] Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu.  Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems.  In Advances in Neural Information Processing Systems, pages 9681–9693, 2021.
  * [147] Felipe Leno Da Silva, Garrett Warnell, Anna Helena Reali Costa, and Peter Stone.  Agents teaching agents: a survey on inter-agent transfer learning.  Autonomous Agents and Multi-Agent Systems, 34:1–17, 2020.
  * [148] Niko A Grupen, Bart Selman, and Daniel D Lee.  Cooperative multi-agent fairness and equivariant policies.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9350–9359, 2022.
  * [149] Felipe Leno Da Silva, Ruben Glatt, and Anna Helena Reali Costa.  Moo-mdp: An object-oriented representation for cooperative multiagent reinforcement learning.  IEEE Transactions on Cybernetics, 49(2):567–579, 2017.
  * [150] Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang.  Exploration in deep reinforcement learning: From single-agent to multiagent domain.  IEEE Transactions on Neural Networks and Learning Systems, 2023\.
  * [151] Woojun Kim and Youngchul Sung.  An adaptive entropy-regularization framework for multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 16829–16852, 2023.
  * [152] Shariq Iqbal and Fei Sha.  Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning.  preprint arXiv:1905.12127, 2019.
  * [153] Alberto Viseras, Thomas Wiedemann, Christoph Manss, Lukas Magel, Joachim Mueller, Dmitriy Shutin, and Luis Merino.  Decentralized multi-agent exploration with online-learning of gaussian processes.  In 2016 IEEE international conference on robotics and automation, pages 4222–4229, 2016.
  * [154] Hans J He, Alec Koppel, Amrit Singh Bedi, Daniel J Stilwell, Mazen Farhood, and Benjamin Biggs.  Decentralized multi-agent exploration with limited inter-agent communications.  In 2023 IEEE International Conference on Robotics and Automation, pages 5530–5536, 2023.
  * [155] M Baglietto, M Paolucci, L Scardovi, and R Zoppoli.  Information-based multi-agent exploration.  In Proceedings of the Third International Workshop on Robot Motion and Control, pages 173–179, 2002.
  * [156] Jingtian Yan, Xingqiao Lin, Zhongqiang Ren, Shiqi Zhao, Jieqiong Yu, Chao Cao, Peng Yin, Ji Zhang, and Sebastian Scherer.  Mui-tare: Cooperative multi-agent exploration with unknown initial position.  IEEE Robotics and Automation Letters, 2023.
  * [157] Mithun Chakraborty, Kai Yee Phoebe Chua, Sanmay Das, and Brendan Juba.  Coordinated versus decentralized exploration in multi-agent multi-armed bandits.  In Proceedings of the International Joint Conferences on Artificial Intelligence, pages 164–170, 2017.
  * [158] Stefanos Leonardos, Georgios Piliouras, and Kelly Spendlove.  Exploration-exploitation in multi-agent competition: Convergence with bounded rationality.  In Advances in Neural Information Processing Systems, pages 26318–26331, 2021.
  * [159] Yonghyeon Jo, Sunwoo Lee, Junghyuk Yum, and Seungyul Han.  Fox: Formation-aware exploration in multi-agent reinforcement learning.  preprint arXiv:2308.11272, 2023.
  * [160] Jihwan Oh, Joonkee Kim, Minchan Jeong, and Se-Young Yun.  Toward risk-based optimistic exploration for cooperative multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1597–1605, 2023.
  * [161] Pei Xu, Junge Zhang, and Kaiqi Huang.  Exploration via joint policy diversity for sparse-reward multi-agent tasks.  In Proceedings of the International Joint Conferences on Artificial Intelligence, pages 326–334, 2023.
  * [162] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus.  Learning multiagent communication with backpropagation.  In Advances in Neural Information Processing Systems, pages 2244–2252, 2016.
  * [163] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar.  Learning when to communicate at scale in multiagent cooperative and competitive tasks.  In International Conference on Learning Representations, 2019.
  * [164] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni.  Learning agent communication under limited bandwidth by message pruning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5142–5149, 2020.
  * [165] Di Xue, Lei Yuan, Zongzhang Zhang, and Yang Yu.  Efficient multi-agent communication via shapley message value.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 578–584, 2022.
  * [166] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni.  Learning multi-agent communication with double attentional deep reinforcement learning.  Autonomous Agents and Multi-Agent Systems, 34(1):1–34, 2020.
  * [167] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang.  Learning nearly decomposable value functions via communication minimization.  In International Conference on Learning Representations, 2020.
  * [168] Sai Qian Zhang, Qi Zhang, and Jieyu Lin.  Succinct and robust multi-agent communication with temporal message control.  In Advances in Neural Information Processing Systems, pages 17271–17282, 2020.
  * [169] Lei Yuan, Tao Jiang, Lihe Li, Feng Chen, Zongzhang Zhang, and Yang Yu.  Robust multi-agent communication via multi-view message certification.  preprint arXiv:2305.13936, 2023.
  * [170] Yanchao Sun, Ruijie Zheng, Parisa Hassanzadeh, Yongyuan Liang, Soheil Feizi, Sumitra Ganesh, and Furong Huang.  Certifiably robust policy learning against adversarial multi-agent communication.  In The International Conference on Learning Representations, 2022\.
  * [171] Lei Yuan, Feng Chen, Zhongzhang Zhang, and Yang Yu.  Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation.  preprint arXiv:2305.05116, 2023.
  * [172] Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu.  Language model self-improvement by reinforcement learning contemplation.  preprint arXiv:2305.14483, 2023.
  * [173] Yuanfei Wang, Jing Xu, Yizhou Wang, et al.  Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind.  In International Conference on Learning Representations, 2021.
  * [174] Michael Shum, Max Kleiman-Weiner, Michael L Littman, and Joshua B Tenenbaum.  Theory of minds: Understanding behavior in groups through inverse planning.  In Proceedings of the AAAI conference on artificial intelligence, pages 6163–6170, 2019.
  * [175] Emre Erdogan, Frank Dignum, Rineke Verbrugge, and Pinar Yolum.  Abstracting minds: Computational theory of mind for human-agent collaboration.  In Proceedings of the First International Conference on Hybrid Human-Artificial Intelligence, Amsterdam,, pages 199–211, 2022.
  * [176] Christelle Langley, Bogdan Ionut Cirstea, Fabio Cuzzolin, and Barbara J Sahakian.  Theory of mind and preference learning at the interface of cognitive science, neuroscience, and ai: A review.  Frontiers in Artificial Intelligence, 5:62, 2022.
  * [177] Jiahan Cao, Lei Yuan, Jianhao Wang, Shaowei Zhang, Chongjie Zhang, Yang Yu, and De-Chuan Zhan.  Linda: Multi-agent local information decomposition for awareness of teammates.  Science China Information Sciences, 66(8):182101, 2023.
  * [178] Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh.  Influencing towards stable multi-agent interactions.  In Conference on robot learning, pages 1132–1143, 2022.
  * [179] Ran Tian, Masayoshi Tomizuka, Anca D Dragan, and Andrea Bajcsy.  Towards modeling and influencing the dynamics of human learning.  In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, pages 350–358, 2023.
  * [180] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus.  Modeling others using oneself in multi-agent reinforcement learning.  In Proceedings of the International conference on machine learning, pages 4257–4266, 2018.
  * [181] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor.  Agent modeling as auxiliary task for deep reinforcement learning.  In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, pages 31–37, 2019.
  * [182] Zhang-Wei Hong, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee.  A deep policy inference q-network for multi-agent systems.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1388–1396, 2018.
  * [183] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, and Zhibo Gong.  Modelling the dynamic joint policy of teammates with attention multi-agent ddpg.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1108–1116, 2019.
  * [184] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  In The International Conference on Learning Representations, 2016\.
  * [185] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan.  Probabilistic recursive reasoning for multi-agent reinforcement learning.  In International Conference on Learning Representations, 2018.
  * [186] Julien Roy, Paul Barde, Félix Harvey, Derek Nowrouzezahrai, and Chris Pal.  Promoting coordination through policy regularization in multi-agent deep reinforcement learning.  In Advances in Neural Information Processing Systems, pages 15774–15785, 2020.
  * [187] Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi.  A survey of imitation learning: Algorithms, recent developments, and challenges.  preprint arXiv:2309.02473, 2023.
  * [188] Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey.  Coordinated multi-agent imitation learning.  In Proceedings of the International Conference on Machine Learning, pages 1995–2003, 2017.
  * [189] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey.  Generating multi-agent trajectories using programmatic weak supervision.  In International Conference on Learning Representations, 2018.
  * [190] Hongwei Wang, Lantao Yu, Zhangjie Cao, and Stefano Ermon.  Multi-agent imitation learning with copulas.  In Machine Learning and Knowledge Discovery in Databases, pages 139–156, 2021.
  * [191] Roger B Nelsen.  An introduction to copulas.  Springer, 2006.
  * [192] Nate Gruver, Jiaming Song, Mykel J Kochenderfer, and Stefano Ermon.  Multi-agent adversarial inverse reinforcement learning with latent variables.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1855–1857, 2020.
  * [193] The Viet Bui, Tien Mai, and Thanh Hong Nguyen.  Inverse factorized q-learning for cooperative multi-agent imitation learning.  preprint arXiv:2310.06801, 2023.
  * [194] Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer.  Multi-agent imitation learning for driving simulation.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1534–1539, 2018.
  * [195] Justin Fu, Andrea Tacchetti, Julien Perolat, and Yoram Bachrach.  Evaluating strategic structures in multi-agent inverse reinforcement learning.  Journal of Artificial Intelligence Research, 71:925–951, 2021.
  * [196] Xingyu Wang and Diego Klabjan.  Competitive multi-agent inverse reinforcement learning with sub-optimal demonstrations.  In Proceedings of the International Conference on Machine Learning, pages 5143–5151, 2018.
  * [197] Yang Chen, Libo Zhang, Jiamou Liu, and Michael Witbrock.  Adversarial inverse reinforcement learning for mean field games.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1088–1096, 2023.
  * [198] Shicheng Liu and Minghui Zhu.  Distributed inverse constrained reinforcement learning for multi-agent systems.  In Advances in Neural Information Processing Systems, pages 33444–33456, 2022.
  * [199] Xin Zhang, Weixiao Huang, Yanhua Li, Renjie Liao, and Ziming Zhang.  Imitation learning from inconcurrent multi-agent interactions.  In Proceedings of the IEEE Annual Conference on Decision and Control, pages 43–48, 2021.
  * [200] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al.  Model-based reinforcement learning: A survey.  Foundations and Trends® in Machine Learning, 16(1):1–118, 2023.
  * [201] Michael Janner, Justin Fu, Marvin Zhang, , and Sergey Levine.  When to trust your model: Model-based policy optimization.  In Advances in Neural Information Processing Systems, pages 12498–12509, 2019.
  * [202] Young Joon Park, Yoon Sang Cho, and Seoung Bum Kim.  Multi-agent reinforcement learning with approximate model learning for competitive games.  PloS one, 14(9):e0222215, 2019.
  * [203] Qizhen Zhang, Chris Lu, Animesh Garg, and Jakob Foerster.  Centralized model and exploration policy for multi-agent rl.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1500–1508, 2022.
  * [204] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  In International Conference on Learning Representations, 2019.
  * [205] Rose Wang, J Chase Kew, Dennis Lee, Tsang-Wei Lee, Tingnan Zhang, Brian Ichter, Jie Tan, and Aleksandra Faust.  Model-based reinforcement learning for decentralized multiagent rendezvous.  In Conference on Robot Learning, pages 711–725, 2021.
  * [206] Wenli Xiao, Yiwei Lyu, and John Dolan.  Model-based dynamic shielding for safe and efficient multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1587–1596, 2023.
  * [207] Anuj Mahajan, Mikayel Samvelyan, Lei Mao, Viktor Makoviychuk, Animesh Garg, Jean Kossaifi, Shimon Whiteson, Yuke Zhu, and Animashree Anandkumar.  Tesseract: Tensorised actors for multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 7301–7312, 2021.
  * [208] Yali Du, Chengdong Ma, Yuchen Liu, Runji Lin, Hao Dong, Jun Wang, and Yaodong Yang.  Scalable model-based policy optimization for decentralized networked systems.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 9019–9026, 2022.
  * [209] Woojun Kim, Jongeui Park, and Youngchul Sung.  Communication in multi-agent reinforcement learning: Intention sharing.  In International Conference on Learning Representations, 2020.
  * [210] Ziluo Ding, Kefan Su, Weixin Hong, Liwen Zhu, Tiejun Huang, and Zongqing Lu.  Multi-agent sequential decision-making via communication.  preprint arXiv:2209.12713, 2022.
  * [211] Shuai Han, Mehdi Dastani, and Shihan Wang.  Model-based sparse communication in multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 439–447, 2023.
  * [212] Barna Pásztor, Andreas Krause, and Ilija Bogunovic.  Efficient model-based multi-agent mean-field reinforcement learning.  Transactions on Machine Learning Research, 2023.
  * [213] Pier Giuseppe Sessa, Maryam Kamgarpour, and Andreas Krause.  Efficient model-based multi-agent reinforcement learning via optimistic equilibrium computation.  In Proceedings of the International Conference on Machine Learning, pages 19580–19597, 2022.
  * [214] Paul Barde, Jakob Foerster, Derek Nowrouzezahrai, and Amy Zhang.  A model-based solution to the offline multi-agent reinforcement learning coordination problem.  preprint arXiv:2305.17198, 2023.
  * [215] Dongge Han, Chris Xiaoxuan Lu, Tomasz Michalak, and Michael Wooldridge.  Multiagent model-based credit assignment for continuous control.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 571–579, 2022.
  * [216] Chao Qian, Yang Yu, and Zhi-Hua Zhou.  Subset selection by pareto optimization.  In Advances in Neural Information Processing Systems, pages 1774–1782, 2015.
  * [217] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek.  Hierarchical reinforcement learning: A comprehensive survey.  ACM Computing Surveys (CSUR), 54(5):1–35, 2021.
  * [218] Rajbala Makar, Sridhar Mahadevan, and Mohammad Ghavamzadeh.  Hierarchical multi-agent reinforcement learning.  In Proceedings of the international conference on Autonomous agents, pages 246–253, 2001.
  * [219] Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Zhaopeng Meng, Changjie Fan, et al.  Hierarchical deep multiagent reinforcement learning with temporal abstraction.  preprint arXiv:1809.09332, 2018.
  * [220] Thomy Phan, Fabian Ritz, Lenz Belzner, Philipp Altmann, Thomas Gabor, and Claudia Linnhoff-Popien.  Vast: Value function factorization with variable agent sub-teams.  In Advances in Neural Information Processing Systems, pages 24018–24032, 2021.
  * [221] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  In International Conference on Learning Representations, 2018.
  * [222] Holger Friedrich, Oliver Rogalla, and Ru¨ diger Dillmann.  Integrating skills into multi-agent systems.  Journal of Intelligent Manufacturing, 9:119–127, 1998.
  * [223] Yuntao Liu, Yuan Li, Xinhai Xu, Yong Dou, and Donghong Liu.  Heterogeneous skill learning for multi-agent tasks.  In Advances in Neural Information Processing Systems, pages 37011–37023, 2022.
  * [224] Shuncheng He, Jianzhun Shao, and Xiangyang Ji.  Skill discovery of coordination in multi-agent reinforcement learning.  preprint arXiv:2006.04021, 2020.
  * [225] Rundong Wang, Longtao Zheng, Wei Qiu, Bowei He, Bo An, Zinovi Rabinovich, Yujing Hu, Yingfeng Chen, Tangjie Lv, and Changjie Fan.  Towards skilled population curriculum for multi-agent reinforcement learning.  preprint arXiv:2302.03429, 2023.
  * [226] Jiayu Chen, Jingdi Chen, Tian Lan, and Vaneet Aggarwal.  Scalable multi-agent covering option discovery based on kronecker graphs.  In Advances in Neural Information Processing Systems, pages 30406–30418, 2022.
  * [227] Jing-Cheng Pang, Xin-Yu Yang, Si-Hang Yang, and Yang Yu.  Natural language-conditioned reinforcement learning with inside-out task language development and translation.  preprint arXiv:2302.09368, 2023.
  * [228] Carlos Guestrin, Shobha Venkataraman, and Daphne Koller.  Context-specific multiagent coordination and planning with factored mdps.  In Proceedings of the Eighteenth National Conference on Artificial Intelligence and Fourteenth Conference on Innovative Applications of Artificial Intelligence, pages 253–259, 2002.
  * [229] Shanjun Cheng.  Coordinating decentralized learning and conflict resolution across agent boundaries.  PhD thesis, The University of North Carolina at Charlotte, 2012.
  * [230] Jelle R Kok and Nikos Vlassis.  Collaborative multiagent reinforcement learning by payoff propagation.  Journal of Machine Learning Research, 7:1789–1828, 2006.
  * [231] Chongjie Zhang and Victor Lesser.  Coordinating multi-agent reinforcement learning with limited communication.  In Proceedings of the international conference on Autonomous agents and multi-agent systems, pages 1101–1108, 2013.
  * [232] Jacopo Castellini, Frans A Oliehoek, Rahul Savani, and Shimon Whiteson.  The representational capacity of action-value networks for multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1862–1864, 2019.
  * [233] Yipeng Kang, Tonghan Wang, Qianlan Yang, Xiaoran Wu, and Chongjie Zhang.  Non-linear coordination graphs.  In Advances in Neural Information Processing Systems, pages 25655–25666, 2022.
  * [234] Qianlan Yang, Weijun Dong, Zhizhou Ren, Jianhao Wang, Tonghan Wang, and Chongjie Zhang.  Self-organized polynomial-time coordination graphs.  In Proceedings of the International Conference on Machine Learning, pages 24963–24979, 2022.
  * [235] Junjie Sheng, Xiangfeng Wang, Bo Jin, Wenhao Li, Jun Wang, Junchi Yan, Tsung-Hui Chang, and Hongyuan Zha.  Learning structured communication for multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 436–438, 2023.
  * [236] Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Haifeng Zhang, David Mguni, Jun Wang, Yaodong Yang, et al.  Settling the variance of multi-agent policy gradients.  In Advances in Neural Information Processing Systems, pages 13458–13470, 2021.
  * [237] Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang.  Order matters: Agent-by-agent policy optimization.  In The International Conference on Learning Representations, 2023\.
  * [238] Zehao Dou, Jakub Grudzien Kuba, and Yaodong Yang.  Understanding value decomposition algorithms in deep cooperative multi-agent reinforcement learning.  preprint arXiv:2202.04868, 2022.
  * [239] Jianghai Hu.  Multi-agent coordination: Theory and applications.  University of California, Berkeley, 2003.
  * [240] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel.  Multi-agent reinforcement learning in sequential social dilemmas.  In Proceedings of the Conference on Autonomous Agents and MultiAgent Systems, pages 464–473, 2017.
  * [241] Yuchen Xiao, Weihao Tan, and Christopher Amato.  Asynchronous actor-critic for multi-agent reinforcement learning.  Advances in Neural Information Processing Systems, 35:4385–4400, 2022.
  * [242] Hancheng Zhang, Guozheng Li, Chi Harold Liu, Guoren Wang, and Jian Tang.  Himacmic: Hierarchical multi-agent deep reinforcement learning with dynamic asynchronous macro strategy.  In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3239–3248, 2023.
  * [243] Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu, and Shiguang Wu.  Concentration network for reinforcement learning of large-scale multi-agent systems.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9341–9349, 2022.
  * [244] Wei Qiu, Weixun Wang, Rundong Wang, Bo An, Yujing Hu, Svetlana Obraztsova, Zinovi Rabinovich, Jianye Hao, Yingfeng Chen, and Changjie Fan.  Off-beat multi-agent reinforcement learning.  preprint arXiv:2205.13718, 2022.
  * [245] Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N Foerster, and Shimon Whiteson.  Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.  preprint arXiv:2212.07489, 2022.
  * [246] Adam Michalski, Filippos Christianos, and Stefano V Albrecht.  Smaclite: A lightweight environment for multi-agent reinforcement learning.  preprint arXiv:2305.05566, 2023.
  * [247] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly.  Google research football: A novel reinforcement learning environment.  In Proceedings of the AAAI conference on artificial intelligence, pages 4501–4510, 2020.
  * [248] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu.  Magent: A many-agent reinforcement learning platform for artificial collective intelligence.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
  * [249] Claude Formanek, Asad Jeewa, Jonathan Shock, and Arnu Pretorius.  Off-the-grid marl: Datasets and baselines for offline multi-agent reinforcement learning.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, page 2442–2444, 2023.
  * [250] Qirui Mi, Siyu Xia, Yan Song, Haifeng Zhang, Shenghao Zhu, and Jun Wang.  Taxai: A dynamic economic simulator and benchmark for multi-agent reinforcement learning.  preprint arXiv:2309.16307, 2023.
  * [251] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht.  Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks.  In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, 2021\.
  * [252] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al.  Pettingzoo: Gym for multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 15032–15043, 2021.
  * [253] Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Zhihui Li, Xiaodan Liang, Xiaojun Chang, and Yaodong Yang.  Marllib: Extending rllib for multi-agent reinforcement learning.  preprint arXiv:2210.13708, 2022.
  * [254] Ming Zhang, Shenghan Zhang, Zhenjie Yang, Lekai Chen, Jinliang Zheng, Chao Yang, Chuming Li, Hang Zhou, Yazhe Niu, and Yu Liu.  Gobigger: A scalable platform for cooperative-competitive multi-agent interactive simulation.  In International Conference on Learning Representations, 2022.
  * [255] Diego Perez-Liebana, Katja Hofmann, Sharada Prasanna Mohanty, Noburu Kuno, Andre Kramer, Sam Devlin, Raluca D Gaina, and Daniel Ionita.  The multi-agent reinforcement learning in malmö (marlö) competition.  preprint arXiv:1901.08129, 2019.
  * [256] Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna.  Pommerman: A multi-agent playground.  preprint arXiv:1809.07124, 2018.
  * [257] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al.  The hanabi challenge: A new frontier for ai research.  Artificial Intelligence, 280:103216, 2020.
  * [258] Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca D. Dragan.  On the utility of learning about humans for human-ai coordination.  In Advances in Neural Information Processing Systems, pages 5175–5186, 2019.
  * [259] Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola.  The neural mmo platform for massively multiagent research.  preprint arXiv:2110.07594, 2021.
  * [260] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch.  Emergent tool use from multi-agent autocurricula.  In International Conference on Learning Representations, 2019.
  * [261] Hangtian Jia, Yujing Hu, Yingfeng Chen, Chunxu Ren, Tangjie Lv, Changjie Fan, and Chongjie Zhang.  Fever basketball: A complex, flexible, and asynchronized sports game environment for multi-agent reinforcement learning.  preprint arXiv:2012.03204, 2020.
  * [262] Daniel Krajzewicz.  Traffic simulation with sumo–simulation of urban mobility.  Fundamentals of traffic simulation, pages 269–293, 2010.
  * [263] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li.  Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario.  In The World Wide Web Conference,, pages 3620–3624, 2019.
  * [264] Roni Stern, Nathan R. Sturtevant, Ariel Felner, Sven Koenig, Hang Ma, Thayne T. Walker, Jiaoyang Li, Dor Atzmon, Liron Cohen, T. K. Satish Kumar, Roman Barták, and Eli Boyarski.  Multi-agent pathfinding: Definitions, variants, and benchmarks.  In Proceedings of the Twelfth International Symposium on Combinatorial Search, pages 151–159, 2019.
  * [265] Sharada Mohanty, Erik Nygren, Florian Laurent, Manuel Schneider, Christian Scheller, Nilabha Bhattacharya, Jeremy Watson, Adrian Egli, Christian Eichenberger, Christian Baumberger, et al.  Flatland-rl: Multi-agent reinforcement learning on trains.  preprint arXiv:2012.05893, 2020.
  * [266] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al.  Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving.  preprint arXiv:2010.09776, 2020.
  * [267] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou.  Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.  IEEE transactions on pattern analysis and machine intelligence, 45(3):3461–3475, 2022.
  * [268] Xuehai Pan, Mickel Liu, Fangwei Zhong, Yaodong Yang, Song-Chun Zhu, and Yizhou Wang.  Mate: Benchmarking multi-agent reinforcement learning in distributed target coverage control.  In Advances in Neural Information Processing Systems, pages 27862–27879, 2022.
  * [269] Reza Torbati, Shubbham Lohiya, Shivika Singh, Meher S Nigam, and Harish Ravichandar.  Marbler: An open platform for standarized evaluation of multi-robot reinforcement learning algorithms.  preprint arXiv:2307.03891, 2023.
  * [270] Xianliang Yang, Zhihao Liu, Wei Jiang, Chuheng Zhang, Li Zhao, Lei Song, and Jiang Bian.  A versatile multi-agent reinforcement learning benchmark for inventory management.  preprint arXiv:2306.07542, 2023.
  * [271] Xiaoteng Ma QihanLiu, Yuhua Jiang.  Light aircraft game: A lightweight, scalable, gym-wrapped aircraft competitive environment with baseline reinforcement learning algorithms.  https://github.com/liuqh16/CloseAirCombat, 2022.
  * [272] Fang Gao, Si Chen, Mingqiang Li, and Bincheng Huang.  Maca: a multi-agent reinforcement learning platform for collective intelligence.  In 2019 IEEE 10th International Conference on Software Engineering and Service Science, pages 108–111, 2019.
  * [273] Yuyu Yuan, Pengqian Zhao, Ting Guo, and Hongpu Jiang.  Counterfactual-based action evaluation algorithm in multi-agent reinforcement learning.  Applied Sciences, 12(7), 2022.
  * [274] Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang.  Safe multi-agent reinforcement learning for multi-robot control.  Artificial Intelligence, 319:103905, 2023.
  * [275] F Bahrpeyma and D Reichelt.  A review of the applications of multi-agent reinforcement learning in smart factories.  Frontiers in Robotics and AI, 9:1027340–1027340, 2022.
  * [276] Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò.  Multi-agent reinforcement learning: A review of challenges and applications.  Applied Sciences, 11(11):4948, 2021.
  * [277] Tianxu Li, Kun Zhu, Nguyen Cong Luong, Dusit Niyato, Qihui Wu, Yang Zhang, and Bing Chen.  Applications of multi-agent reinforcement learning in future internet: A comprehensive survey.  IEEE Communications Surveys & Tutorials, 24(2):1240–1279, 2022\.
  * [278] Ziyuan Zhou, Guanjun Liu, and Ying Tang.  Multi-agent reinforcement learning: Methods, applications, visionary prospects, and challenges.  preprint arXiv:2305.10091, 2023.
  * [279] Zun Li, Marc Lanctot, Kevin R McKee, Luke Marris, Ian Gemp, Daniel Hennes, Paul Muller, Kate Larson, Yoram Bachrach, and Michael P Wellman.  Combining tree-search, generative models, and nash bargaining concepts in game-theoretic reinforcement learning.  preprint arXiv:2302.00797, 2023.
  * [280] Zhijian Zhang, Haozheng Li, Luo Zhang, Tianyin Zheng, Ting Zhang, Xiong Hao, Xiaoxin Chen, Min Chen, Fangxu Xiao, and Wei Zhou.  Hierarchical reinforcement learning for multi-agent moba game.  preprint arXiv:1901.08004, 2019.
  * [281] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al.  Mastering the game of stratego with model-free multiagent reinforcement learning.  Science, 378(6623):990–996, 2022.
  * [282] Yang Li, Kun Xiong, Yingping Zhang, Jiangcheng Zhu, Stephen Mcaleer, Wei Pan, Jun Wang, Zonghong Dai, and Yaodong Yang.  Jiangjun: Mastering xiangqi by tackling non-transitivity in two-player zero-sum games.  preprint arXiv:2308.04719, 2023.
  * [283] Xiangyu Zhao and Sean B Holden.  Towards a competitive 3-player mahjong ai using deep reinforcement learning.  In 2022 IEEE Conference on Games, pages 524–527, 2022.
  * [284] Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu.  Douzero: Mastering doudizhu with self-play deep reinforcement learning.  In international conference on machine learning, pages 12333–12344, 2021.
  * [285] Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy.  Diverse generation for multi-agent sports games.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4610–4619, 2019.
  * [286] Dimitrios Troullinos, Georgios Chalkiadakis, Ioannis Papamichail, and Markos Papageorgiou.  Collaborative multiagent decision making for lane-free autonomous driving.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1335–1343, 2021.
  * [287] Tong Wang, Jiahua Cao, and Azhar Hussain.  Adaptive traffic signal control for large-scale scenario with cooperative group-based multi-agent reinforcement learning.  Transportation research part C: emerging technologies, 125:103046, 2021.
  * [288] Siyuan Chen, Meiling Wang, Wenjie Song, Yi Yang, and Mengyin Fu.  Multi-agent reinforcement learning-based twin-vehicle fair cooperative driving in dynamic highway scenarios.  In 2022 IEEE International Conference on Intelligent Transportation Systems, pages 730–736, 2022.
  * [289] Sangwoo Jeon, Hoeun Lee, Vishnu Kumar Kaliappan, Tuan Anh Nguyen, Hyungeun Jo, Hyeonseo Cho, and Dugki Min.  Multiagent reinforcement learning based on fusion-multiactor-attention-critic for multiple-unmanned-aerial-vehicle navigation control.  Energies, 15(19):7426, 2022.
  * [290] Shutong Chen, Guanjun Liu, Ziyuan Zhou, Kaiwen Zhang, and Jiacun Wang.  Robust multi-agent reinforcement learning method based on adversarial domain randomization for real-world dual-uav cooperation.  IEEE Transactions on Intelligent Vehicles, 2023.
  * [291] Ho-Bin Choi, Ju-Bong Kim, Youn-Hee Han, Se-Won Oh, and Kwihoon Kim.  Marl-based cooperative multi-agv control in warehouse systems.  IEEE Access, 10:100478–100488, 2022.
  * [292] Yanchang Liang, Xiaowei Zhao, and Li Sun.  A multiagent reinforcement learning approach for wind farm frequency control.  IEEE Transactions on Industrial Informatics, 19(2):1725–1734, 2022\.
  * [293] Zhenhan Huang and Fumihide Tanaka.  Correction: Mspm: A modularized and scalable multi-agent reinforcement learning-based system for financial portfolio management.  PLOS ONE, 17(3):e0265924, 2022.
  * [294] Ali Shavandi and Majid Khedmati.  A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets.  Expert Systems with Applications, 208:118124, 2022.
  * [295] Yuling Huang, Chujin Zhou, Kai Cui, and Xiaoping Lu.  A multi-agent reinforcement learning framework for optimizing financial trading strategies based on timesnet.  Expert Systems with Applications, page 121502, 2023.
  * [296] Chao Wen, Miao Xu, Zhilin Zhang, Zhenzhe Zheng, Yuhui Wang, Xiangyu Liu, Yu Rong, Dong Xie, Xiaoyang Tan, Chuan Yu, et al.  A cooperative-competitive multi-agent framework for auto-bidding in online advertising.  In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1129–1139, 2022.
  * [297] Yuchen Fang, Zhenggang Tang, Kan Ren, Weiqing Liu, Li Zhao, Jiang Bian, Dongsheng Li, Weinan Zhang, Yong Yu, and Tie-Yan Liu.  Learning multi-agent intention-aware communication for optimal multi-order execution in finance.  In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4003–4012, 2023.
  * [298] Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, Andrea Ricci, and Sergio Spano.  An fpga-based multi-agent reinforcement learning timing synchronizer.  Computers and Electrical Engineering, 99:107749, 2022.
  * [299] Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, and An-Xiang Zeng.  Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4902–4909, 2019.
  * [300] Zool Hilmi Ismail, Nohaidda Sariff, and E Gorrostieta Hurtado.  A survey and analysis of cooperative multi-agent robot systems: challenges and directions.  Applications of Mobile Robots, pages 8–14, 2018.
  * [301] Ammar Abdul Ameer Rasheed, Mohammed Najm Abdullah, and Ahmed Sabah Al-Araji.  A review of multi-agent mobile robot systems applications.  International Journal of Electrical & Computer Engineering (2088-8708), 12(4), 2022.
  * [302] Abhinav Dahiya, Alexander M Aroyo, Kerstin Dautenhahn, and Stephen L Smith.  A survey of multi-agent human–robot interaction systems.  Robotics and Autonomous Systems, 161:104335, 2023.
  * [303] Paul Maria Scheikl, Balázs Gyenes, Tornike Davitashvili, Rayan Younis, André Schulze, Beat P Müller-Stich, Gerhard Neumann, Martin Wagner, and Franziska Mathis-Ullrich.  Cooperative assistance in robotic surgery through multi-agent reinforcement learning.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1859–1864, 2021.
  * [304] Zhaozhi Wang, Kefan Su, Jian Zhang, Huizhu Jia, Qixiang Ye, Xiaodong Xie, and Zongqing Lu.  Multi-agent automated machine learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11960–11969, 2023.
  * [305] Haiyang Wang, Wenguan Wang, Xizhou Zhu, Jifeng Dai, and Liwei Wang.  Collaborative visual navigation.  preprint arXiv:2107.01151, 2021.
  * [306] Zeyu Fang, Jian Zhao, Mingyu Yang, Wengang Zhou, Zhenbo Lu, and Houqiang Li.  Coordinate-aligned multi-camera collaboration for active multi-object tracking.  preprint arXiv:2202.10881, 2022.
  * [307] Shiqi Lin, Tao Yu, Ruoyu Feng, Xin Li, Xiaoyuan Yu, Lei Xiao, and Zhibo Chen.  Local patch autoaugment with multi-agent collaboration.  IEEE Transactions on Multimedia, 2023.
  * [308] Xinyuan Zhang, Cong Zhao, Feixiong Liao, Xinghua Li, and Yuchuan Du.  Online parking assignment in an environment of partially connected vehicles: A multi-agent deep reinforcement learning approach.  Transportation Research Part C: Emerging Technologies, 138:103624, 2022.
  * [309] Yu Sui and Shiming Song.  A multi-agent reinforcement learning framework for lithium-ion battery scheduling problems.  Energies, 13(8):1982, 2020.
  * [310] Xiaohan Wang, Lin Zhang, Tingyu Lin, Chun Zhao, Kunyu Wang, and Zhen Chen.  Solving job scheduling problems in a resource preemption environment with multi-agent reinforcement learning.  Robotics and Computer-Integrated Manufacturing, 77:102324, 2022\.
  * [311] Tong Zhou, Dunbing Tang, Haihua Zhu, and Zequn Zhang.  Multi-agent reinforcement learning for online scheduling in smart factories.  Robotics and Computer-Integrated Manufacturing, 72:102202, 2021\.
  * [312] Elie Kadoche, Sébastien Gourvénec, Maxime Pallud, and Tanguy Levent.  Marlyc: Multi-agent reinforcement learning yaw control.  Renewable Energy, 217:119129, 2023.
  * [313] Yinfeng Yu, Changan Chen, Lele Cao, Fangkai Yang, and Fuchun Sun.  Measuring acoustics with collaborative multiple agents.  In Proceedings of the International Joint Conference on Artificial Intelligence, pages 335–343, 2023.
  * [314] Xu Xu, Youwei Jia, Yan Xu, Zhao Xu, Songjian Chai, and Chun Sing Lai.  A multi-agent reinforcement learning-based data-driven method for home energy management.  IEEE Transactions on Smart Grid, 11(4):3201–3211, 2020.
  * [315] Haoyu Zhou, Haifeng Zhang, Yushan Zhou, Xinchao Wang, and Wenxin Li.  Botzone: an online multi-agent competitive platform for ai education.  In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, pages 33–38, 2018.
  * [316] Yinda Chen, Wei Huang, Shenglong Zhou, Qi Chen, and Zhiwei Xiong.  Self-supervised neuron segmentation with multi-agent reinforcement learning.  pages 609–617, 2023.
  * [317] Hanane Allioui, Mazin Abed Mohammed, Narjes Benameur, Belal Al-Khateeb, Karrar Hameed Abdulkareem, Begonya Garcia-Zapirain, Robertas Damaševičius, and Rytis Maskeliūnas.  A multi-agent deep reinforcement learning approach for enhancement of covid-19 ct image segmentation.  Journal of personalized medicine, 12(2):309, 2022.
  * [318] Zihao Gong, Yang Xu, and Delin Luo.  Uav cooperative air combat maneuvering confrontation based on multi-agent reinforcement learning.  Unmanned Systems, 11(03):273–286, 2023.
  * [319] Shaowei Li, Yongchao Wang, Yaoming Zhou, Yuhong Jia, Hanyue Shi, Fan Yang, and Chaoyue Zhang.  Multi-uav cooperative air combat decision-making based on multi-agent double-soft actor-critic.  Aerospace, 10(7):574, 2023.
  * [320] Lixing Liu, Nikolos Gurney, Kyle McCullough, and Volkan Ustun.  Graph neural network based behavior prediction to support multi-agent reinforcement learning in military training simulations.  In 2021 Winter Simulation Conference, pages 1–12. IEEE, 2021.
  * [321] Anjon Basak, Erin G Zaroukian, Kevin Corder, Rolando Fernandez, Christopher D Hsu, Piyush K Sharma, Nicholas R Waytowich, and Derrik E Asher.  Utility of doctrine with multi-agent rl for military engagements.  In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV, volume 12113, pages 609–628. SPIE, 2022.
  * [322] KONG Weiren, ZHOU Deyun, Kai Zhang, and YANG Zhen.  Air combat autonomous maneuver decision for one-on-one within visual range engagement base on robust multi-agent reinforcement learning.  In 2020 IEEE 16th International Conference on Control & Automation (ICCA), pages 506–512. IEEE, 2020.
  * [323] Haiyin Piao, Yue Han, Shaoming He, Chao Yu, Songyuan Fan, Yaqing Hou, Chengchao Bai, and Li Mo.  Spatio-temporal relationship cognitive learning for multi-robot air combat.  IEEE Transactions on Cognitive and Developmental Systems, 2023.
  * [324] Tianrui Jiang, Dongye Zhuang, and Haibin Xie.  Anti-drone policy learning based on self-attention multi-agent deterministic policy gradient.  In International Conference on Autonomous Unmanned Systems, pages 2277–2289, 2021.
  * [325] Longfei Yue, Rennong Yang, Jialiang Zuo, Ying Zhang, Qiuni Li, and Yijie Zhang.  Unmanned aerial vehicle swarm cooperative decision-making for sead mission: A hierarchical multiagent reinforcement learning approach.  IEEE Access, 10:92177–92191, 2022.
  * [326] Zhang Jiandong, YANG Qiming, SHI Guoqing, LU Yi, and WU Yong.  Uav cooperative air combat maneuver decision based on multi-agent reinforcement learning.  Journal of Systems Engineering and Electronics, 32(6):1421–1438, 2021.
  * [327] Wei-ren Kong, De-yun Zhou, Yong-jie Du, Ying Zhou, and Yi-yang Zhao.  Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat.  IET Control Theory & Applications, 17(13):1840–1862, 2023.
  * [328] Zhixiao Sun, Haiyin Piao, Zhen Yang, Yiyang Zhao, Guang Zhan, Deyun Zhou, Guanglei Meng, Hechang Chen, Xing Chen, Bohao Qu, et al.  Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play.  Engineering Applications of Artificial Intelligence, 98:104112, 2021\.
  * [329] Zhixiao Sun, Huahua Wu, Yandong Shi, Xiangchao Yu, Yifan Gao, Wenbin Pei, Zhen Yang, Haiyin Piao, and Yaqing Hou.  Multi-agent air combat with two-stage graph-attention communication.  Neural Computing and Applications, 35(27):19765–19781, 2023.
  * [330] Gráinne Conole.  Designing for Learning in An Open World, volume 4.  Springer Science & Business Media, 2012.
  * [331] Asiiah Song.  A little taxonomy of open-endedness.  In ICLR Workshop on Agent Learning in Open-Endedness, 2022.
  * [332] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen.  Recent advances in open set recognition: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(10):3614–3631, 2020.
  * [333] Bo-Jian Hou, Lijun Zhang, and Zhi-Hua Zhou.  Learning with feature evolvable streams.  IEEE Transactions on Knowledge and Data Engineering, 33(6):2602–2615, 2019.
  * [334] Djordje Grbic, Rasmus Berg Palm, Elias Najarro, Claire Glanois, and Sebastian Risi.  Evocraft: A new challenge for open-endedness.  In Applications of Evolutionary Computation, pages 325–340. Springer, 2021.
  * [335] David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel.  Open-ended learning in symmetric zero-sum games.  In Proceedings of the International Conference on Machine Learning, pages 434–443, 2019.
  * [336] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu.  Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks.  preprint arXiv:2303.16563, 2023.
  * [337] Robert Meier and Asier Mujika.  Open-ended reinforcement learning with neural reward functions.  In Advances in Neural Information Processing Systems, pages 2465–2479, 2022.
  * [338] Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward Grefenstette, and Tim Rocktäschel.  Skillhack: A benchmark for skill transfer in open-ended reinforcement learning.  In ICLR Workshop on Agent Learning in Open-Endedness, 2022.
  * [339] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al.  Open-ended learning leads to generally capable agents.  preprint arXiv:2107.12808, 2021.
  * [340] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al.  Human-timescale adaptation in an open-ended task space.  preprint arXiv:2301.07608, 2023.
  * [341] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao.  Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 10299–10312, 2021.
  * [342] Jiechuan Jiang and Zongqing Lu.  Offline decentralized multi-agent reinforcement learning.  preprint arXiv:2108.01832, 2021.
  * [343] Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang.  Learning from good trajectories in offline multi-agent reinforcement learning.  preprint arXiv:2211.15612, 2022.
  * [344] Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu.  Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.  preprint arXiv:2112.02845, 2021.
  * [345] Xiangsen Wang and Xianyuan Zhan.  Offline multi-agent reinforcement learning with coupled value factorization.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 2781–2783, 2023.
  * [346] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji.  Counterfactual conservative q learning for offline multi-agent reinforcement learning.  preprint arXiv:2309.12696, 2023.
  * [347] Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P. How.  Learning to teach in cooperative multiagent reinforcement learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6128–6136, 2019.
  * [348] Tianpei Yang, Weixun Wang, Hongyao Tang, Jianye Hao, Zhaopeng Meng, Hangyu Mao, Dong Li, Wulong Liu, Yingfeng Chen, Yujing Hu, et al.  An efficient transfer learning framework for multiagent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 17037–17048, 2021.
  * [349] Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang.  Evolutionary population curriculum for scaling multi-agent reinforcement learning.  In International Conference on Learning Representations, 2019.
  * [350] Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rocktäschel, and Shimon Whiteson.  Generalization in cooperative multi-agent systems.  preprint arXiv:2202.00104, 2022.
  * [351] Rongjun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Zongzhang Zhang, Chongjie Zhang, and Yang Yu.  Multi-agent policy transfer via task relationship modeling.  preprint arXiv:2203.04482, 2022.
  * [352] Hadi Nekoei, Akilesh Badrinaaraayanan, Aaron Courville, and Sarath Chandar.  Continuous coordination as a realistic scenario for lifelong learning.  In Proceedings of the International Conference on Machine Learning, pages 8016–8024, 2021.
  * [353] Lei Yuan, Lihe Li, Ziqian Zhang, Fuxiang Zhang, Cong Guan, and Yang Yu.  Multi-agent continual coordination via progressive task contextualization.  preprint arXiv:2305.13937, 2023.
  * [354] Lei Yuan, Lihe Li, Ziqian Zhang, Feng Chen, Tianyi Zhang, Cong Guan, Yang Yu, and Zhi-Hua Zhou.  Learning to coordinate with anyone.  preprint arXiv:2309.12633, 2023.
  * [355] Somdeb Majumdar, Shauharda Khadka, Santiago Miret, Stephen McAleer, and Kagan Tumer.  Evolutionary reinforcement learning for sample-efficient multiagent coordination.  In Proceedings of the International Conference on Machine Learning, pages 6651–6660, 2020.
  * [356] Gaurav Dixit and Kagan Tumer.  Balancing teams with quality-diversity for heterogeneous multiagent coordination.  In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 236–239, 2022.
  * [357] Gaurav Dixit, Everardo Gonzalez, and Kagan Tumer.  Diversifying behaviors for learning in asymmetric multiagent systems.  In Proceedings of the Genetic and Evolutionary Computation Conference, pages 350–358, 2022.
  * [358] Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu.  Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11753–11762, 2023.
  * [359] Chenghe Wang, Yuhang Ran, Lei Yuan, Yang Yu, and Zongzhang Zhang.  Robust multi-agent reinforcement learning against adversaries on observation.  2022\.
  * [360] Ziyan Wang, Yali Du, Aivar Sootla, Haitham Bou Ammar, and Jun Wang.  CAMA: A new framework for safe multi-agent reinforcement learning using constraint augmentation, 2023.
  * [361] Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan.  Learning safe multi-agent control with decentralized neural barrier certificates.  In International Conference on Learning Representations, 2020.
  * [362] Ingy ElSayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng.  Safe multi-agent reinforcement learning via shielding.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 483–491, 2021.
  * [363] Daniel Melcer, Christopher Amato, and Stavros Tripakis.  Shield decentralization for safe multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 13367–13379, 2022.
  * [364] Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Lee.  Dfac framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning.  In Proceedings of the International Conference on Machine Learning, pages 9945–9954, 2021.
  * [365] Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich.  Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents.  In Advances in Neural Information Processing Systems, pages 23049–23062, 2021.
  * [366] Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Yi Chang, Lichao Sun, et al.  Distributional reward estimation for effective multi-agent deep reinforcement learning.  In Advances in Neural Information Processing Systems, pages 12619–12632, 2022.
  * [367] Kyunghwan Son, Junsu Kim, Sungsoo Ahn, Roben D Delos Reyes, Yung Yi, and Jinwoo Shin.  Disentangling sources of risk for distributional multi-agent reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 20347–20368, 2022.
  * [368] Ziyi Liu and Yongchun Fang.  Learning adaptable risk-sensitive policies to coordinate in multi-agent general-sum games.  arXiv preprint arXiv:2303.07850, 2023.
  * [369] Oliver Slumbers, David Henry Mguni, Stefano B Blumberg, Stephen Marcus Mcaleer, Yaodong Yang, and Jun Wang.  A game-theoretic framework for managing risk in multi-agent systems.  In Proceedings of the International Conference on Machine Learning, pages 32059–32087, 2023.
  * [370] Peter Stone, Gal A. Kaminka, Sarit Kraus, and Jeffrey S. Rosenschein.  Ad hoc autonomous agent teams: Collaboration without pre-coordination.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1504–1509, 2010.
  * [371] Muthukumaran Chandrasekaran, A. Eck, Prashant Doshi, and Leen-Kiat Soh.  Individual planning in open and typed agent systems.  In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 82–91, 2016.
  * [372] Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An.  Online ad hoc teamwork under partial observability.  In International Conference on Learning Representations, 2022.
  * [373] Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrecht.  Towards open ad hoc teamwork using graph-based policy learning.  In Proceedings of the International Conference on Machine Learning, pages 8776–8786, 2021.
  * [374] Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht.  Generating teammates for training robust ad hoc teamwork agents via best-response diversity.  Transactions on Machine Learning Research, 2023.
  * [375] Arrasy Rahman, Jiaxun Cui, and Peter Stone.  Minimum coverage sets for training robust ad hoc teamwork agents.  preprint arXiv:2308.09595, 2023.
  * [376] João G Ribeiro, Gonçalo Rodrigues, Alberto Sardinha, and Francisco S Melo.  Teamster: Model-based reinforcement learning for ad hoc teamwork.  Artificial Intelligence, page 104013, 2023.
  * [377] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett.  Collaborating with humans without human data.  In Advances in Neural Information Processing Systems, pages 14502–14515, 2021.
  * [378] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster.  Trajectory diversity for zero-shot coordination.  In Proceedings of the International Conference on Machine Learning, pages 7204–7213, 2021.
  * [379] Ke Xue, Yutong Wang, Lei Yuan, Cong Guan, Chao Qian, and Yang Yu.  Heterogeneous multi-agent zero-shot coordination by coevolution.  preprint arXiv:2208.04957, 2022.
  * [380] Hao Ding, Chengxing Jia, Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu.  Coordination scheme probing for generalizable multi-agent reinforcement learning, 2023.
  * [381] Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul.  Generating diverse cooperative agents by learning incompatible policies.  In International Conference on Learning Representations, 2023.
  * [382] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu.  Learning zero-shot cooperation with humans, assuming humans are biased.  In International Conference on Learning Representations, 2023.
  * [383] Hadi Nekoei, Xutong Zhao, Janarthanan Rajendran, Miao Liu, and Sarath Chandar.  Towards few-shot coordination: Revisiting ad-hoc teamplay challenge in the game of hanabi.  preprint arXiv:2308.10284, 2023.
  * [384] Hengyuan Hu, David J Wu, Adam Lerer, Jakob Foerster, and Noam Brown.  Human-ai coordination via human-regularized search and learning.  preprint arXiv:2210.05125, 2022.
  * [385] Joey Hong, Anca Dragan, and Sergey Levine.  Learning to influence human behavior with offline reinforcement learning.  preprint arXiv:2303.02265, 2023.
  * [386] Sagar Parekh and Dylan P Losey.  Learning latent representations to co-adapt to humans.  Autonomous Robots, pages 1–26, 2023.
  * [387] Xingzhou Lou, Jiaxian Guo, Junge Zhang, Jun Wang, Kaiqi Huang, and Yali Du.  Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 679–688, 2023.
  * [388] Rundong Wang, Weixuan Wang, Xianhan Zeng, Liang Wang, Zhenjie Lian, Yiming Gao, Feiyu Liu, Siqin Li, Xianliang Wang, QIANG FU, Yang Wei, Lanxiao Huang, Longtao Zheng, Zinovi Rabinovich, and Bo An.  Multi-agent multi-game entity transformer, 2023.
  * [389] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang.  Madiff: Offline multi-agent learning with diffusion models.  preprint arXiv:2305.17330, 2023.
  * [390] Wenhao Li, Dan Qiao, Baoxiang Wang, Xiangfeng Wang, Bo Jin, and Hongyuan Zha.  Semantically aligned task decomposition in multi-agent reinforcement learning.  preprint arXiv:2305.10865, 2023.
  * [391] Julien M Hendrickx and Samuel Martin.  Open multi-agent systems: Gossiping with random arrivals and departures.  In Proceedings of the IEEE Annual Conference on Decision and Control, pages 763–768, 2017.
  * [392] Jonathan Cohen, Jilles Steeve Dibangoye, and Abdel-Illah Mouaddib.  Open decentralized pomdps.  In Proceedings of the IEEE International Conference on Tools with Artificial Intelligence, pages 977–984, 2017.
  * [393] Jonathan Cohen and Abdel-Illah Mouaddib.  Monte-carlo planning for team re-formation under uncertainty: Model and properties.  In Proceedings of the IEEE International Conference on Tools with Artificial Intelligence, pages 458–465, 2018.
  * [394] Jonathan Cohen and Abdel-Illah Mouaddib.  Power indices for team reformation planning under uncertainty.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1886–1888, 2019.
  * [395] A. Eck, Maulik Shah, Prashant Doshi, and Leen-Kiat Soh.  Scalable decision-theoretic planning in open and typed multiagent systems.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7127–7134, 2020.
  * [396] Anirudh Kakarlapudi, Gayathri Anil, Adam Eck, Prashant Doshi, and Leen-Kiat Soh.  Decision-theoretic planning with communication in open multiagent systems.  In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 938–948, 2022.
  * [397] Adam Eck, Leen-Kiat Soh, and Prashant Doshi.  Decision making in open agent systems.  AI Magazine, 2023.
  * [398] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  preprint arXiv:2005.01643, 2020.
  * [399] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini.  A survey on offline reinforcement learning: Taxonomy, review, and open problems.  IEEE Transactions on Neural Networks and Learning Systems, 2023\.
  * [400] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In Proceedings of the International Conference on Machine Learning, pages 2052–2062, 2019.
  * [401] Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning.  preprint arXiv:1911.11361, 2019.
  * [402] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy Q-learning via bootstrapping error reduction.  In Advances in Neural Information Processing Systems, pages 11761–11771, 2019.
  * [403] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative Q-learning for offline reinforcement learning.  In Advances in Neural Information Processing Systems, pages 1179–1191, 2020.
  * [404] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar.  Finite-sample analysis for decentralized batch multiagent reinforcement learning with networked agents.  IEEE Transactions on Automatic Control, 66(12):5925–5940, 2021\.
  * [405] Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu.  Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.  In Proceedings of the International Conference on Machine Learning, pages 17221–17237, 2022.
  * [406] Wei-Cheng Tseng, Tsun-Hsuan Wang, Yen-Chen Lin, and Phillip Isola.  Offline multi-agent reinforcement learning with knowledge distillation.  In Advances in Neural Information Processing Systems, pages 226–237, 2022.
  * [407] Linghui Meng, Jingqing Ruan, Xuantang Xiong, Xiyun Li, Xi Zhang, Dengpeng Xing, and Bo Xu.  M3: Modularization for multi-task and multi-agent offline pre-training.  In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 1624–1633, 2023.
  * [408] Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou.  Transfer learning in deep reinforcement learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [409] Samir Wadhwania, Dong-Ki Kim, Shayegan Omidshafiei, and Jonathan P. How.  Policy distillation and value matching in multiagent reinforcement learning.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 8193–8200, 2019.
  * [410] Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao.  From few to more: Large-scale dynamic multiagent curriculum learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7293–7300, 2020.
  * [411] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang.  UPDeT: Universal multi-agent reinforcement learning via policy decoupling with transformers.  In International Conference on Learning Representations, 2021.
  * [412] Tianze Zhou, Fubiao Zhang, Kun Shao, Kai Li, Wenhan Huang, Jun Luo, Weixun Wang, Yaodong Yang, Hangyu Mao, Bin Wang, et al.  Cooperative multi-agent transfer learning with level-adaptive credit assignment.  preprint arXiv:2106.00517, 2021.
  * [413] Haobin Shi, Jingchen Li, Jiahui Mao, and Kao-Shing Hwang.  Lateral transfer learning for multiagent reinforcement learning.  IEEE transactions on cybernetics, 53(3):1699–1711, 2023.
  * [414] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al.  Biological underpinnings for lifelong learning machines.  Nature Machine Intelligence, 4(3):196–210, 2022.
  * [415] David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh.  A definition of continual reinforcement learning.  preprint arXiv:2307.11046, 2023.
  * [416] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71, 2019.
  * [417] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [418] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne.  Experience replay for continual learning.  In Advances in Neural Information Processing Systems, pages 348–358, 2018.
  * [419] Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti.  Continual model-based reinforcement learning with hypernetworks.  In Proceedings of the IEEE International Conference on Robotics and Automation, pages 799–805, 2021.
  * [420] Samuel Kessler, Piotr Miłoś, Jack Parker-Holder, and Stephen J Roberts.  The surprising effectiveness of latent world models for continual reinforcement learning.  preprint arXiv:2211.15944, 2022.
  * [421] Zhi Wang, Chunlin Chen, and Daoyi Dong.  Lifelong incremental reinforcement learning with online bayesian inference.  IEEE Transactions on Neural Networks and Learning Systems, 33(8):4003–4016, 2022.
  * [422] Samuel Kessler, Jack Parker-Holder, Philip J. Ball, Stefan Zohren, and Stephen J. Roberts.  Same state, different task: Continual reinforcement learning without interference.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7143–7151, 2022.
  * [423] Jean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu.  Building a subspace of policies for scalable continual learning.  In International Conference on Learning Representations, 2023.
  * [424] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [425] Zhi-Hua Zhou, Yang Yu, and Chao Qian.  Evolutionary learning: Advances in theories and algorithms.  Springer, 2019.
  * [426] Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers.  Evolutionary dynamics of multi-agent learning: A survey.  Journal of Artificial Intelligence Research, 53:659–697, 2015.
  * [427] Shayegan Omidshafiei, Christos Papadimitriou, Georgios Piliouras, Karl Tuyls, Mark Rowland, Jean-Baptiste Lespiau, Wojciech M Czarnecki, Marc Lanctot, Julien Perolat, and Remi Munos.  $\alpha$-rank: Multi-agent evaluation by evolution.  Scientific reports, 9(1):9937, 2019.
  * [428] Takanori Shibata and Toshio Fukuda.  Coordination in evolutionary multi-agent-robotic system using fuzzy and genetic algorithm.  Control Engineering Practice, 2(1):103–111, 1994.
  * [429] Pengyi Li, Jianye Hao, Hongyao Tang, Yan Zheng, and Xian Fu.  Race: Improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution.  In Proceedings of the International Conference on Machine Learning, pages 19490–19503, 2023.
  * [430] Gaurav Dixit and Kagan Tumer.  Learning synergies for multi-objective optimization in asymmetric multiagent systems.  In Proceedings of the Genetic and Evolutionary Computation Conference, pages 447–455, 2023.
  * [431] Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters.  Robust reinforcement learning: A review of foundations and recent advances.  Machine Learning and Knowledge Extraction, 4(1):276–315, 2022.
  * [432] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta.  Robust adversarial reinforcement learning.  In Proceedings of the International Conference on Machine Learning, pages 2817–2826, 2017.
  * [433] Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen.  Robust reinforcement learning using adversarial populations.  preprint arXiv:2008.01825, 2020.
  * [434] Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh.  Robust reinforcement learning on state observations with learned optimal adversary.  In International Conference on Learning Representations, 2020.
  * [435] Yeeho Song and Jeff Schneider.  Robust reinforcement learning via genetic curriculum.  In Proceedings of the International Conference on Robotics and Automation, pages 5560–5566, 2022.
  * [436] Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui-Wei Weng.  Robust deep reinforcement learning through adversarial loss.  In Advances in Neural Information Processing Systems, pages 26156–26167, 2021.
  * [437] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang.  Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl.  In International Conference on Learning Representations, 2021.
  * [438] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang.  Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning.  In Advances in Neural Information Processing Systems, pages 22547–22561, 2022.
  * [439] Michael Everett, Björn Lütjens, and Jonathan P. How.  Certifiable robustness to adversarial state uncertainty in deep reinforcement learning.  IEEE Transactions on Neural Networks and Learning Systems, 33(9):4184–4198, 2022.
  * [440] Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li.  Crop: Certifying robust policies for reinforcement learning through functional smoothing.  In International Conference on Learning Representations, 2022.
  * [441] Fan Wu, Linyi Li, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li.  Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks.  In International Conference on Learning Representations, 2021.
  * [442] Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot.  On the robustness of cooperative multi-agent reinforcement learning.  In 2020 IEEE Security and Privacy Workshops, pages 62–68, 2020\.
  * [443] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell.  Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4213–4220, 2019.
  * [444] Rupert Mitchell, Jan Blumenkamp, and Amanda Prorok.  Gaussian process based message filtering for robust multi-agent cooperation in the presence of adversarial communication.  preprint arXiv:2012.00508, 2020.
  * [445] James Tu, Tsunhsuan Wang, Jingkang Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun.  Adversarial attacks on multi-agent communication.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7768–7777, 2021.
  * [446] Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Marc Zeller, et al.  Learning and testing resilience in cooperative multi-agent systems.  In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems, pages 1055–1063, 2020.
  * [447] Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff-Popien.  Resilient multi-agent reinforcement learning with adversarial value decomposition.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11308–11316, 2021.
  * [448] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao.  Robust multi-agent reinforcement learning with state uncertainty.  Transactions on Machine Learning Research, 2023.
  * [449] Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen.  Multi-objective optimization.  In Decision sciences, pages 161–200. CRC Press, 2016.
  * [450] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley.  A survey of multi-objective sequential decision-making.  Journal of Artificial Intelligence Research, 48(1):67–113, 2013\.
  * [451] Chunming Liu, Xin Xu, and Dewen Hu.  Multiobjective reinforcement learning: A comprehensive overview.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385–398, 2014.
  * [452] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan.  A generalized algorithm for multi-objective reinforcement learning and policy adaptation.  In Advances in Neural Information Processing Systems, pages 14610–14621, 2019.
  * [453] Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al.  A practical guide to multi-objective reinforcement learning and planning.  Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
  * [454] Roxana Rădulescu, Patrick Mannion, Diederik M Roijers, and Ann Nowé.  Multi-objective multi-agent decision making: a utility-based analysis and survey.  Autonomous Agents and Multi-Agent Systems, 34(1):10, 2020.