# On Multi-Agent Deep Deterministic Policy Gradients and their Explainability
for SMARTS Environment

Ansh Mittal  USC Viterbi School of Engineering)
University of Southern California
Los Angeles, CA, USA
<EMAIL_ADDRESS>Aditya Malte  USC Viterbi School of Engineering)
University of Southern California
Los Angeles, CA, USA
<EMAIL_ADDRESS>
###### Abstract

Multi-Agent RL or MARL is one of the complex problems in Autonomous Driving
literature that hampers the release of fully-autonomous vehicles today.
Several simulators have been in iteration after their inception to mitigate
the problem of complex scenarios with multiple agents in Autonomous Driving.
One such simulator–SMARTS, discusses the importance of cooperative multi-agent
learning. For this problem, we discuss two approaches–MAPPO and MADDPG, which
are based on-policy and off-policy RL approaches. We compare our results with
the state-of-the-art results for this challenge and discuss the potential
areas of improvement while discussing the explainability of these approaches
in conjunction with waypoints in the SMARTS environment.

###### Index Terms:

Multi-Agent RL, SMARTS, MAPPO, MADDPG, Explainability, Post-Hoc Explainability

##  I Introduction

We are interested in solving a multi-agent learning (MAL) problem related to
the SMARTS environment [1]. The main motivation for choosing this project was
that despite the immense amount of deep learning architecture, the Autonomous
Vehicle industry has yet to bring more Autonomous Vehicles on road. This
majorly stems from the problems that arise when different agents interact with
each other. This can further have a corresponding effect of the agents being
non-cooperative with each other. Further, interpretability issues arise due to
both of these which leads market stakeholders to invest less in the Autonomous
Vehicle industry. This has been represented in fig. 1. Hence, till today there
has been no fully automated vehicle on road. Further, only 54.2 million semi-
Automated Vehicles 111Statistics taken from the website
https://www.statista.com/statistics/1230664/projected-number-autonomous-cars-
worldwide/ are projected to be there in 2024 which makes this a bleak future
for the AV industry. Hence, we explore two different types of policies in the
field of Multi-Agent Reinforcement Learning (MARL) and propose a way to work
their explainability for future use-cases. We plan to open-source the code
after extending and getting this research published. The simulations will be
made available at that time as well.

Figure 1: Motivation

The SMARTS (abb. Scalable Multi-Agent RL Training School) environment actually
creates several scenarios such as the ones given in fig 2. It also defines
different levels for different types of RL agents in Autonomous Vehicle
literature as given below in table I.

Figure 2: Scenarios Level | Description | Possible RL Approaches
---|---|---
M0 | Rule-based Planning | Not Applicable
M1 | Single-Agent Learning w/o coordinated learning | Agent implicitly learns to anticipate other agent’s actions according to our car and then takes the next steps
M2 | Multi-Agent Modelling w Opponent Modelling | MARL to model learning of our agent using other agents, e.g., “how likely to our car is to yield if their lane starts to change”
M3 | Coordinated MAL w/ independent execution | Coordinated Learning of what to expect of each other even if no explicit coordination, e.g., some cars will leave some gap
M4 | Local Equilibrium oriented MAL | Learn as a local group towards a certain equilibrium for double merge scenarios, etc.
M5 | Social Welfare-oriented MAL | Learning broader repercussions of the car’s actions
TABLE I: Different levels of Multi-Agent Learning in Autonomous Driving
discussed in the SMARTS environment [1]

The contributions of this project are three-fold:

  1. 1.

We implement a Multi-Agent algorithm for Proximal Policy Optimization and Deep
Deterministic Policy Gradients (using Prioritized Replay Buffers)

  2. 2.

We compare their performance on the 4 evaluation metrics given in the SMARTS
environment

  3. 3.

Further, we discuss the explainability of the better algorithm for MARL

The rest of the report is structured as follows. Section II discusses the main
problem statement of this research project effort. Section III discusses the
problems and challenges faced when working on this project. We discuss our
solutions to this problem formulation in IV. We further list our results in
section V and compare them with the results obtained in the official challenge
that NeurIPS organized. Then, we briefly discuss the explainability aspect
that this problem can take from here on in section VI. Conclusively, section
VII discusses the conclusions and reasons why our method falls short of the
state-of-the-art results.

##  II Problem Definition

This project aimed to develop methods using different simulation environment
data from SMARTS [1] to innovate autonomous driving multi-agent that drive to
their destination as safely and quickly as possible while following all the
rules of the traffic system. For this formulation, the algorithms were
evaluated on 4 different metrics, namely Completion, Time, Humanness, and
Rules. All these metrics are discussed in detail in Section V. This project
leverages the SMARTS docker containers for
simulations222https://github.com/huawei-noah/SMARTS/tree/comp-1 (accessed:
October 20, 2022). Further, a lot of different methods of environment setup
(Docker, Singularity, and WSL direct setup) were tried for different reasons
that have been mentioned in the next section.

##  III Technical Challenges

During working on this project, we faced several challenges which have been
mentioned below.

  1. 1.

We were faced with the most striking requirement of GPU compute as the
computation involved in any RL algorithm can take days to train on a normal
CPU. For this, we had to further divide our approach into three steps.

     * •

We first trained for 1000 episodes on the general system and ran a simulation
while the model was training using Envision Sumo.

     * •

Then, we took the model used for our approach to the Singularity on HPC.

     * •

The model weights were then transferred to our system to check the model
performance on the envision server since a GUI wasn’t available on the HPC
client.

  2. 2.

Since there are multiple setups that can help us run the Envision Server, we
were able to successfully run the competition environment but there were some
files missing and version inconsistencies in different branches in the GitHub
repository (till November 3, 2022; the competition end date ended up being
postponed by a week due to this) and hence we weren’t able to efficiently work
at that time.

  3. 3.

With the baseline (Single Agent) that SMARTS proposes, there were issues with
finding which approach works well. Further, the Ray package and TensorFlow
packages can have conflicting dependencies for some versions. Hence, Version
Control was vital for this research project.

Figure 3: Schematic Diagram for MAPPO using Ray and TensorFlow environments
(taken from the blog in the footnotes) Figure 4: Schematic Diagram for MADDPG
taken from the research here [14]

##  IV Proposed Solution and Methodology

This section discusses the solution that the authors of this research project
proposed for training the policies for the SMARTS environment. The authors
used 2 different types of algorithms, namely, MAPPO (Multi-Agent Proximal
Policy Optimization) and MADDPG (Multi-Agent Deep Deterministic Policy
Gradient) for MAL and learning the policies. These have been discussed briefly
below for the two algorithms mentioned in table II.

###  IV-A Multi-Agent Proximal Policy Optimization

Proximal Policy Optimization is a class of on-policy Reinforcement Learning
algorithms that uses some of the advantages of Trust Region Policy
Optimization (TRPO) [2, 3, 4]. This set of algorithms alternates between
sampling data from the environment through interactions, and optimizing a
“surrogate” objective function that enables multiple epochs of minibatch
updates. For more information about the algorithm, it is advised to read the
original paper by OpenAI [4]. Further, an adaptive clipping approach for PPO
[5] was developed later by building on prior works. After the inception of
PPO, there have been various cooperative and Multi-Agent Proximal Policy
Optimization implementations for various use cases such as targeted
localization [6], Online scheduling for Production [7], Health-care [8].
Moreover, there has been a combination of Convolutions with RL approaches such
as CMAPPO (Convolutional Multi-Agent PPO) [9] which learn an objective based
on learning and exploring a new environment most effectively by combining
various domains such as Convolutions (for RGBD+ information), Curriculum-based
Learning, and Motivation-based Reinforcement Learning. Finally, there have
been approaches that use the replay buffers approach from Deep Deterministic
Policy Gradients (off-policy method) [11] for the task of cooperative
MARL333Github source code can be found here
https://github.com/marlbenchmark/on-policy [12]. For implementing MAPPO, we
referred to various sources to implement an architecture as given in the fig 3
444Blog read: https://medium.com/analytics-vidhya/deep-multi-agent-
reinforcement-learning-with-tensorflow-agents-1d4a91734d1f and GitHub
repository: https://github.com/rmsander/marl_ppo.git. Further, OpenAI’s Gym
was used during the experiments for testing and tutorial purposes for
understanding the various aspects of Multi-Agent Training and Self-Training
for Open World Models [13]555Github Page for the World Model which leverages
MDN, CNN-VAE, and RL techniques for various RL-based simulations:
https://worldmodels.github.io/.

Apart from the approaches discussed above, there has been recent work on
integrating the Actor-Critic method with the PPO RL paradigm that are beyond
the scope of this project but should be able to guide this project in several
future directions.

###  IV-B Multi-Agent Deep Deterministic Policy Gradients (with Priority-based
Replay buffers)

MADDPG is composed of various agents interacting cooperatively for MARL
scenarios. It is composed of DDPG [11] for multiple agents which interact with
each other in a cooperative environment. It is an off-policy RL approach as
opposed to MAPPO as it learns the value of the optimal policy independent of
the agent’s actions666Difference given here:
https://stats.stackexchange.com/questions/184657/what-is-the-difference-
between-off-policy-and-on-policy-learning. Before discussing the term MAPPO,
it should be interesting to get some background for Q-learning since DDPG is
based on Q-learning 777Blog Post for introduction to Q-Learning:
https://medium.com/intro-to-artificial-intelligence/q-learning-a-value-based-
reinforcement-learning-algorithm-272706d835cf. Since it’s a model-free off-
policy actor-critic algorithm that integrates the benefits of Deterministic
Policy Gradients [15] with the Deep Q-Networks (DQNs). Being an off-policy
algorithm, it separately learns stochastic behavior policy for exploration and
deterministic policy for target updates [16]. Further, according to the
original article, it maintains a replay buffer based on the temporal
difference. But in our use case, we keep a priority list for this temporal
replay buffer based on the number of accidents, rule violations, completion
time, and jerks much like what had been done for MAPPO [17].
There has been a considerable extension to the MADDPG paradigm where recurrent
DPGs were used for complex environments like Cognitive Electronic Warfare [18]
and Partially Observable Environments for Communication systems [19]. A mixed
environment approach was taken for complex environments using MADDPG [20],
whereas Decomposed Approach was introduced for learning multi-agent policies
for UAV clusters to build a connected communication network [21]. Further,
these set of algorithms were also discussed in the context of working with
smart grids for edge technology [22] and have shown to perform considerably
well when compared to the other state-of-the-art.

##  V Results and Comparative Analysis

For the evaluation of the algorithms introduced in section IV, we discuss the
evaluation metrics we calculated using the evaluation subdirectory on the
branch comp-1 under track-1 of the NeurIPS competition. Please note that these
approaches are discussed in the context of only online learning (which is
listed in task 1 of the NeurIPS competition). So, the evaluation metrics are
as given below.

  1. 1.

Completion This represents the completion of the total scenarios and hence can
be calculated as the total number of crashes for each episode

  2. 2.

Time This metric discusses the total number of steps used by each agent (red
cars) for each episode

  3. 3.

Humannness It refers to how close are our agents to imitating human-level
driving scenarios. It is the average of the total distance to obstacles,
angular jerk, linear jerk, and lane center offset for each episode

  4. 4.

Rules It refers to the total number of rules violated such as lane changing,
Wrong Way, Speed Overlimit for each episode.

It is of importance to note that since we are taking the average for each
episode, there can be multiple agents (red cars) in a single episode. This
leads to higher values in our evaluation and similar is the case with all
other participating teams in the competition. We list our results with both
the algorithms in table II.

Algorithms | Completion | Time | Humanness | Rules
---|---|---|---|---
Speed-Direction Meta Controller Policies (TJUDRL lab) | 0.24 (1) | 385.43 (1) | 8161.24 (16) | 1.03 (12)
Waypoints-Trajectory Planning (CNN for Birdview) w/ Safety Score Learning | 0.31 (2) | 467.30 (4) | 4292.02 (12) | 16.96 (17)
Interaction-Aware Transformers | 0.32 (3) | 386.43 (2) | 641.86 (4) | 0.43 (10)
MAPPO (w Replay Buffer and fine-tuning) | 0.72 (16*) | 723.1 (13*) | 4865.07 (13*) | 0.72 (12*)
MADDPG (with Priority-based Replay Buffers & fine-tuning) | 0.64 (10*) | 746.29 (15*) | 1362.7 (9*) | 0.22 (9*)
TABLE II: Algorithms used and their comparison with the state-of-the-art
winner solutions. The numbers in brackets are the ranks of the algorithm. The
(*) represents projected ranks as we weren’t able to submit our models to the
official competition.

It can be seen that MADDPG considerably outperforms MAPPO. This is because the
priority experience buffers we explore for this problem statement take the
gradients of speed, acceleration, and completion rate to define the priority
queues. We further add a few screenshots for simulations for MADDPG in fig 5.

##  VI On explainability of MADDPG

We go through several research [23, 24, 25] to comment on the explainability
of these deep reinforcement learning approaches to understand the decisions
made by MARL policy networks. In our approach, we have used various metric
parameters to design the priority experience replay buffer which can help the
MADDPG learn the prioritized experience for better results. This is visible in
the table II, where we see that our MADDPG approach is able to outperform the
top 3 winners of the SMARTS challenge. Further, we also observe better values
for Humanness when compared to Top-2 winners. This happens as the priority
experience replay buffer encodes the information for implicit explainability
(transparent explainability) of the algorithm and post-hoc explainability of
the model can be depicted by the blue waypoints as depicted in fig 5. This can
help appeal to various stakeholders about how the car is taking action in the
evaluation scenarios over different periods.

##  VII Conclusion and Discussion

As noted above the results depict that our model falls short of the state-of-
the-art approaches. This might be due to several reasons as discussed below.

  1. 1.

The training for our model didn’t converge when we stopped the training. We
had to stop the training as the amount of time that we could query an HPC node
was about to be exhausted and we had to save the results and get the
evaluation scores for the same.

  2. 2.

The top-2 approaches efficiently leverage Birdeyes view images from the
simulation as feedback to change the directions of the car, which we haven’t
integrated into our model. This can lead to suboptimal performances.

  3. 3.

Finally, suboptimal hyperparameters of the approaches that we leverage might
lead to local minima rather than getting global minima.

We plan to work on this project further and extend its scope twofold. First,
we plan to implement a CV algorithm to reinforce the RL strategies with
additional heuristics (left, right, keep) as is done in the best approach.
Secondly, after getting feedback from the professor, we’d like to work on
evolutionary computing for this approach. For example, these approaches can
include Parameter-Exploring Policy Gradients with or without CV reinforcement
[26, 27]. We can also tune the hyperparameter search using evolutionary
techniques like Genetic Algorithms, Particle Swarm Optimization, CMAES, Gray
Wolf Optimization, etc (given enough GPU compute).

(a) Highway Merge scenario

(b) Zoo Intersection Scenario

Figure 5: Simulation results for MADDPG (w Priority Experience Replay Buffers)

## Acknowledgment

We thank Prof Jyotirmoy V. Deshmukh for his continuous support during the
duration of the course CSCI513. We also thank the course TAs–Xin Qin and
Aniruddh Puranic, for their continuous support and accommodation during the
duration of the course CSCI513. We thank USC for providing the services for
High-Performance Computing for the purpose of this project.

## References

  * [1] Zhou, Ming, et al. "Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving." arXiv preprint arXiv:2010.09776 (2020).
  * [2] Schulman, John, et al. "Trust region policy optimization." International conference on machine learning. PMLR, 2015.
  * [3] Kurutach, Thanard, et al. "Model-ensemble trust-region policy optimization." arXiv preprint arXiv:1802.10592 (2018).
  * [4] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
  * [5] Chen, Gang, Yiming Peng, and Mengjie Zhang. "An adaptive clipping approach for proximal policy optimization." arXiv preprint arXiv:1804.06461 (2018).
  * [6] Alagha, Ahmed, et al. "Target localization using multi-agent deep reinforcement learning with proximal policy optimization." Future Generation Computer Systems 136 (2022): 342-357.
  * [7] Lohse, Oliver, Noah Pütz, and Korbinian Hörmann. "Implementing an Online Scheduling Approach for Production with Multi Agent Proximal Policy Optimization (MAPPO)." IFIP International Conference on Advances in Production Management Systems. Springer, Cham, 2021.
  * [8] Allen, Ross E., et al. "Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning." arXiv preprint arXiv:1908.01022 (2019).
  * [9] Sander, Ryan. "EMERGENT AUTONOMOUS RACING VIA MULTI-AGENT PROXIMAL POLICY OPTIMIZATION."
  * [10] Chen, Zichen, Budhitama Subagdja, and Ah-Hwee Tan. "End-to-end deep reinforcement learning for multi-agent collaborative exploration." 2019 IEEE International Conference on Agents (ICA). IEEE, 2019.
  * [11] Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).
  * [12] Yu, Chao, et al. "The surprising effectiveness of ppo in cooperative, multi-agent games." arXiv preprint arXiv:2103.01955 (2021).
  * [13] Ha, David, and Jürgen Schmidhuber. "World models." arXiv preprint arXiv:1803.10122 (2018).
  * [14] Qie, Han, et al. "Joint optimization of multi-UAV target assignment and path planning based on multi-agent reinforcement learning." IEEE access 7 (2019): 146264-146272.
  * [15] Silver, David, et al. "Deterministic policy gradient algorithms." International conference on machine learning. PMLR, 2014.
  * [16] Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).
  * [17] Hou, Yuenan, and Yi Zhang. "Improving DDPG via prioritized experience replay." no. May (2019).
  * [18] Wei, Xiaolong, et al. "Recurrent MADDPG for object detection and assignment in combat tasks." IEEE Access 8 (2020): 163334-163343.
  * [19] Wang, Rose E., Michael Everett, and Jonathan P. How. "R-MADDPG for partially observable environments and limited communication." arXiv preprint arXiv:2002.06684 (2020).
  * [20] Wan, Kaifang, et al. "ME-MADDPG: An efficient learning-based motion planning method for multiple agents in complex environments." International Journal of Intelligent Systems 37.3 (2022): 2393-2427.
  * [21] Zhu, Zixiong, et al. "Building a Connected Communication Network for UAV Clusters Using DE-MADDPG." Symmetry 13.8 (2021): 1537.
  * [22] Lei, Wenxin, et al. "MADDPG-based security situational awareness for smart grid with intelligent edge." Applied Sciences 11.7 (2021): 3101.
  * [23] Krajna, Agneza, et al. "Explainability in reinforcement learning: perspective and position." arXiv preprint arXiv:2203.11547 (2022).
  * [24] Wells, Lindsay, and Tomasz Bednarz. "Explainable ai and reinforcement learning—a systematic review of current approaches and trends." Frontiers in artificial intelligence 4 (2021): 550030.
  * [25] Heuillet, Alexandre, Fabien Couthouis, and Natalia Díaz-Rodríguez. "Explainability in deep reinforcement learning." Knowledge-Based Systems 214 (2021): 106685.
  * [26] Sehnke, Frank, et al. "Parameter-exploring policy gradients." Neural Networks 23.4 (2010): 551-559.
  * [27] Sehnke, Frank, et al. "Multimodal parameter-exploring policy gradients." 2010 Ninth International Conference on Machine Learning and Applications. IEEE, 2010.