# Demystifying Deep Reinforcement Learning-Based Autonomous Vehicle Decision-
Making

Hanxi Wan, Pei Li, and Arpan Kusari H.Wan is with the Department of Robotics,
University of Michigan (e-mail:wanhanxi@umich.edu).P. Li is with the
Department of Civil & Environmental Engineering, University of Wisconsin-
Madison (e-mail:pei.li@wisc.edu).A. Kusari is with the University of Michigan
Transportation Research Institute, University of Michigan
(e-mail:kusari@umich.edu). Manuscript received April 19, 2021; revised August
16, 2021.

###### Abstract

With the advent of universal function approximators in the domain of
reinforcement learning, the number of practical applications leveraging deep
reinforcement learning (DRL) has exploded. Decision-making in automated
driving tasks has emerged as a chief application among them, taking the sensor
data or the higher-order kinematic variables as the input and providing a
discrete choice or continuous control output. However, the black-box nature of
the models presents an overwhelming limitation that restricts the real-world
deployment of DRL in autonomous vehicles (AVs). Therefore, in this research
work, we focus on the interpretability of an attention-based DRL framework. We
use a continuous proximal policy optimization-based DRL algorithm as the
baseline model and add a multi-head attention framework in an open-source AV
simulation environment. We provide some analytical techniques for discussing
the interpretability of the trained models in terms of explainability and
causality for spatial and temporal correlations. We show that the weights in
the first head encode the positions of the neighboring vehicles while the
second head focuses on the leader vehicle exclusively. Also, the ego vehicle’s
action is causally dependent on the vehicles in the target lane spatially and
temporally. Through these findings, we reliably show that these techniques can
help practitioners decipher the results of the DRL algorithms.

###### Index Terms:

Autonomous vehicles, deep reinforcement learning, attention, explainability,
causality.

##  I Introduction

The objective of autonomous vehicles (AVs) is to safely navigate the uncertain
dynamic environment using onboard sensors and some prior knowledge about the
surroundings. To do so, the AVs utilize the paradigm of “sense-plan-act” that
has been central to robotics for the past half-a-century [1] -

  * •

sensing and perception module uses the high-dimensional sensing data to
determine the static objects for localizing themselves and the dynamic objects
and their trajectories for obstacle avoidance,

  * •

decision-making module utilizes the localization information, kinematics of
the dynamic objects, and the goal of the AV to take the optimal action, and

  * •

control module executes the action in the form of path planning and path
following.

Of these three high-level modules, the perception and control modules have
reached a certain stage of high precision based on the deep learning
architectures and the model-based methods respectively. However, in the realm
of decision-making, the algorithms need to reason about the uncertainties
present in the sensor data along with the incompleteness of the data (due to
occlusion) among other factors, and provide an optimal action under these
constraints. Thus, a stochastic cost function needs to be optimized for a
high-dimensional, sequential process. Typically, a finite state machine (FSM)
has been used in practical AV applications for decision-making. FSM uses
discrete decision choices based on the current state and predicted future
states of the ego and other vehicles based mostly on hand-tuned parameters
[2]. As can be easily seen, even though the solution is practical, it is
neither scalable nor it can probabilistically handle the uncertainties.

Reinforcement learning (RL) is a probabilistic formalism of sequential
decision-making, where an agent learns the policy (mapping of states to
actions) which maximizes the expected rewards in an environment [3]. For
continuous state spaces and discrete/continuous action spaces, universal
function approximators such as neural networks have been proposed with
outstanding achievements - AlphaGO defeated the reigning GO champion [4] and
the Atari games were played at superhuman abilities by the deep reinforcement
learning (DRL) agents [5]. Researchers have reported great gains in AV
decision-making using DRL as well, with the learned policy adapting to
uncertainty in the future state space [6].

So, a central question arises - if DRL has shown such exceptional performance
in decision-making tasks including in AVs, why are state-based decision models
such as FSM still being used primarily?

The answer lies in the black-box nature of the DRL and the difficulty that
arises in understanding the intention behind the action. For a safety-critical
application such as AVs, a sub-optimal but explainable solution is clearly
preferred. In this paper, we hope to provide some scientific insights into
that black box using some interpretation tools. Specifically, we focus on the
attention-based mechanism which provides weights on the states (in this case,
the neighboring vehicles) to interpret the policies. Our novel contributions
are as follows:

  * •

we provide a benchmarking for the attention-based model with respect to a
baseline proximal policy optimization (PPO) model in terms of kinematic
variables, including average velocity, distance traveled, and time-to-
collision (TTC).

  * •

we look at the explainability of the attention-based model in terms of spatial
and temporal correlations and show that the specific discrete events (going
straight to lane change decision) can be mapped by a change in attention
weights.

  * •

we also look at the causality of the attention-based model using Iterative
Causal Discovery (ICD) [7] for the spatial changes and the temporal ICD [8]
for the temporal changes. We show that the particular lane change has a
specific causal marker in the leader and follower in the target lane (i.e.
left leader and left follower in the left lane). Also, temporal causality can
be ascertained by the ego vehicle’s influence on the previous time step along
with the leading vehicle’s influence.

The paper is organized as follows: Section II provides the background in AV
decision-making and where explainability and causality are applied in this
process. Section III provides the state and action representation, the
baseline and multi-head attention-based models used in this study, the reward
function employed, the training process utilized, and the methods used in
providing explainability and causality. Section IV compares the results for
the baseline and the attention-based models and showcases results in
explainability and causality using spatial and temporal correlations. Finally,
in Section V, we provide some discussion, conclusions and future work.

##  II Background

###  II-A AV Decision-making

Two kinds of methods are generally utilized for AV decision-making, classical
and learning-based methods [9]. Classical methods, such as FSM [10, 11], Model
Predictive Control (MPC) [12, 13, 14], game theory, etc., have been widely
used for AV decision-making tasks including path tracking, car-following, and
lane-changing. The advantages of classical methods are their simplicity and
strong interpretability. Moreover, methods such as MPC and game theory are
able to model the interactions between different traffic participants.
However, rule-based methods in general have poor performance in complex
driving scenarios as their parameters are usually hand-tuned after simulation
and field testing [15].

Learning-based methods, in particular, deep learning, offer several benefits
for vehicle control compared to classical methods. Deep learning methods have
shown great promise in not only providing excellent performance for complex
and non-linear control problems but also in generalizing previously learned
rules to new scenarios [15]. Among existing deep-learning methods, model-free
DRL methods have become more and more popular for various AV control tasks,
including lane-change, car-following, path-tracking, and platooning [16, 17,
18, 19]. DRL methods treat AVs as agents, which interact with the environments
and learn the consequences of their actions through experience via algorithms
such as policy gradient, Q-Learning, etc. As they are model-free, they do not
require specific models to measure the environments, and the agents can learn
directly from experience. This makes them well-suited for environments where
the dynamics are unknown or complicated to model, such as various driving
scenarios.

###  II-B Explainability and Causality in AV Decision-making

In addition to achieving safer and more efficient AV decision-making,
explainability and causality are two crucial considerations for evaluating AV
decision-making systems, especially for deep learning methods. The need for
them is due to the increasing concerns for transparency and accountability of
AVs [20]. Specifically, explainability refers to the ability to explain and
justify the underlying rationale of AVs’ decisions. Differently, causality
explores the cause-and-effect relationships between variables, such as the
actions of neighbor vehicles and the actions of AVs. It aims to determine
whether changes in one variable directly lead to changes in another variable,
rather than just identifying similarities. Although DRL-based models have made
significant progress and achieved superior performance in AV control compared
to classical models, one major limitation of DRL-based models is the lack of
explainability and causality. To solve this problem, explainable reinforcement
learning, known as XRL, has been proposed and has gained a lot of attention in
recent years. The main focus of XRL is on the reasoning behind the decisions
or predictions made by the RL model [21]. Specifically, the majority of XRL
methods function by mimicking and simplifying a complex model using a more
explainable model, such as tree-based models [22, 23, 24]. Specifically, a DRL
policy is first trained for controlling the AV. Then, a decision tree model is
developed to explain the model using the observations of the agent as inputs
and the decisions generated by the policy as outputs. This approach is easy to
implement and interpret, however, the explainability may not reveal the actual
knowledge learned by the agent. To further improve the explainability of DRL-
based AV control models, additional research needs to be done on improving the
inherent explainability of the DRL model rather than using a simple model to
imitate it. Moreover, little research has been done to understand the
causality behind deep learning-based methods for AV decision-making. [25]
attempts to utilize a causality test for understanding the causality of a
vehicle’s actions using image data. A visual attention model is trained end-
to-end from images to the steering angle. The causality is then decided by
iteratively masking attention regions in images and evaluating the impact on
the model’s performance. This casual filtering technique produces a simpler
and more accurate map of visual saliency by filtering out spurious attention
regions. However, causality was not further analyzed but served as a technique
to improve the interpretation of the attention maps.

##  III Methodology

From the background work, it is abundantly clear that while there have been
efforts made to understand the DRL models used in AV decision-making, to the
best of the authors’ knowledge, there has been no concerted effort made to
decode the state space and actions produced by DRL with respect to the
kinematics. Similarly, there has been some work done in marrying the field of
causality with RL, but it has not been applied to the spatio-temporal nature
of AV decision-making. In this section, we present our method to train a
reinforcement learning-based AV decision-making in a simulated highway
environment task and examine the attention mechanism to infer
interpretability.

###  III-A State and action representation

We choose the observation space of the environment
$o_{t}\in\mathbb{R}^{7\times 5}$ as a vector representation of the scene.
Referred to as affordance features in the literature [26], it provides
information on the ego vehicle as well as the six neighboring vehicles
(leading and following vehicles in the left/same/right lane of the ego
vehicle). For each vehicle, there are 5 features: presence showing if the
corresponding vehicle exists, longitudinal/lateral position, and
longitudinal/lateral velocity. The position of the neighboring vehicle is
represented as the distance to the ego vehicle, and all the distances and
velocities are normalized. The action space of the environment
$a_{t}\in\mathbb{R}^{2}$ is a two-dimensional vector that directly controls
the throttle and steering angle of the vehicle. The vehicle dynamic is
simulated using the Kinematic Bicycle Model ([27]).

Figure 1: Architecture of the attention-based feature extractor.

###  III-B Model architectures

The model is separated into two main parts: the feature extractor for
extracting features from observations, and an action predictor for mapping
features to actions. We propose two agents, a baseline agent and an attention-
based agent. They have different feature extractor architectures, but they
both use fully connected networks for the action predictor.

####  III-B1 Baseline Model

The baseline model is a multi-layer perceptron (MLP) model that inputs the
observation vector and outputs the action predictor. We use the baseline model
as an expert for the attention-based agent. Therefore, the action predictor
for the baseline agent is scaled so that the baseline agent and attention-
based agent have a comparable number of trainable parameters.

####  III-B2 Multi-Head Attention-Based Model

To provide importance weights of the input features, we use a multi-head
attention-based model inspired by [28] to extract features. However, different
from the original idea, we vary the number of attention heads to find the
optimal number of heads required for optimal use. Figure 1 shows an overview
of the attention-based model.

Firstly the observation goes through an ego-attention feature extractor. The
ego vector $o_{e}\in\mathbb{R}^{1\times 5}$ and neighboring vectors
$o_{n}\in\mathbb{R}^{6\times 5}$ are encoded by an MLP layer to get the
embeddings of size $d_{k}$. Then the embeddings are fed into a multi-head ego-
attention layer to extract features of the observation. Different from the
self-attention mechanism which produces output for each input element, the
ego-attention layer only produces one single output corresponding to the ego
vehicle. Each vehicle’s embeddings go through a linear layer
$L_{k}\in\mathbb{R}^{d_{x}\times d_{k}}$ to get its key $k_{i}$, and a linear
layer $L_{v}\in\mathbb{R}^{d_{x}\times d_{v}}$ to get its value $v_{i}$. The
ego embedding alone goes through a query linear layer
$L_{q}\in\mathbb{R}^{d_{x}\times d_{k}}$ to get its query
$q_{0}\in\mathbb{R}^{1\times d_{k}}$. The attention matrix
$attn\in\mathbb{R}^{1\times 7}$ is the modified similarities between the query
from ego vehicle $q_{0}$ and the keys from neighboring vehicles $k_{i}$, shown
in Equation 1. The inner product of $Q=q_{0}$ and
$K_{T}=[k_{0},k_{1},...,k_{6}]^{T}$ is the similarity, which is then scaled by
the square-root-dimension and normalized with a softmax function $\sigma$.
After that, the attention matrix is multiplied by the value
$V=[v_{0},v_{1},..,v_{6}]$ to get the final output for one head. The outputs
of all head $out_{i}\in\mathbb{R}^{1\times d_{v}}$ are fed through a linear
layer to form the combined feature representation from different heads
$feat\in\mathbb{R}^{1\times 64}$.

$attn=\sigma(\frac{QK^{T}}{\sqrt{d_{k}}})$ (1)

###  III-C Reward Function

For any RL algorithm, the reward function (referred to as the objective
function or the utility function in other optimization literature) is the most
crucial step - the agent optimizes its expected sum of rewards based on the
reward function. However, determining the reward function is not
straightforward in the best of situations: in many real-world applications,
rewards are delayed or sparse. In the case of AVs, a certain acceleration or
steering angle can lead to a crash in the large time horizon which makes it
difficult to assign credit/blame on a particular action choice. Therefore, in
RL literature, several studies have used reward shaping [29] to provide
additional expert knowledge, to improve learning speed and convergence.

Inspired by the reward function in [30], we design our reward function as a
vector, as shown in Eq. 2. This reward function utilizes multiobjective RL
(MORL) where tradeoffs between conflicting objective functions. Our reward
function takes into consideration the velocity, the lateral position in the
lane, the action, and the distance from the leading vehicle by creating a
scalarization function [31]:

$r=\begin{cases}0,\quad\quad\text{ if TTC }<0.2\text{ or }v_{x}<20\\\
w_{v}\cdot r_{v}+w_{y}\cdot r_{y}+w_{a}\cdot r_{a}+w_{x}\cdot r_{x},&\text{
otherwise }\end{cases}$ (2)

$\displaystyle r_{v}$
$\displaystyle=e^{-\frac{((v_{y}-v_{des})/10)^{2}}{10}},r_{y}=e^{-\frac{(y-y_{des})^{2}}{30}},r_{a}=e^{-(3*a_{steering})^{2}},$
(3) $\displaystyle r_{x}$
$\displaystyle=e^{-\frac{(d_{lead}-d_{safe})^{2}}{10d_{safe}}}\text{, if
}d_{lead}<d_{safe},$

where $v_{des}$ is the desired speed, $y_{des}$ is the center of the desired
lane determined by the previous series of actions and the current lane,
$d_{lead}$ is the distance to leading vehicle, $d_{safe}$ is the safe
longitudinal distance, and $w_{v},w_{y},w_{a},w_{x}$ are the weights for
different rewards. We performed a set of controlled experiments to determine
the weights of the various factors and chose a set of weights that balanced
the driving behavior with the safety of the vehicle.

###  III-D Training Process

####  III-D1 Reinforcement learning

Reinforcement learning is a general probabilistic framework for sequential
decision-making under uncertainty. Based on the feature we extracted, we train
an action predictor in reinforcement learning using the proximal policy
optimization (PPO) algorithm [32]. We train our agent in highway-env using PPO
as the reinforcement learning algorithm that takes the $feat$ from the feature
extractor as input and outputs an action $a\in\mathbb{R}^{2}$.

PPO is an RL architecture that improves training stability by avoiding large
policy updates. The intuition is that performing smaller policy updates during
training converges to an optimal solution more often. Therefore, the change is
clipped to $\epsilon$ over the current policy. Its loss function is

$\displaystyle
L_{t}^{CLIP+VF+S}(\theta)=\hat{\mathbb{E}_{t}}[L_{t}^{CLIP}(\theta)-c_{1}L_{t}^{VF}(\theta)+c_{2}S[\pi_{\theta}](s_{t})],$
(4) $\displaystyle
L_{t}^{CLIP}(\theta)=\hat{\mathbb{E}_{t}}[\min(r_{t}(\theta)\hat{A_{t}},clip(r_{t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A_{t}}],$

where $\hat{\mathbb{E}_{t}}$ is the expectation operator, $L_{t}^{CLIP}$ is
the clipped surrogate objective, $L_{t}^{VF}$ is the squared error loss of the
value function $(V_{\theta}(S_{t})-V_{t}^{targ})^{2}$,
$S[\pi_{\theta}](s_{t})$ is an entropy bonus to ensure sufficient
exploitation. In $L_{t}^{CLIP}$, $\varepsilon$ is a hyperparameter,
$r_{t}(\theta)=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta old}(a_{t}|s_{t})$ is the
probability ratio. The clipped loss function effectively avoids taking a large
policy update.

####  III-D2 Imitation Pre-Training for Attention-based framework

To decrease the overall training time and increase the sample efficiency of
the attention-based algorithm, we devise a two-stage process to train our
model. In the pre-training stage, imitation learning enables the model to
learn policy efficiently based on the sampled trajectories from the baseline
agent. In imitation learning, the agent learns from the trajectories and the
corresponding actions provided by an expert. The primary objective of the pre-
training phase is to accelerate the process of policy learning by imitating
the behavior of the baseline agent.

During the pre-training phase, we generate a set of observation-action pairs
using a baseline model. Then we train both the feature extractor and action
predictor as a whole network by minimizing the minimum squared error loss of
action and value between the model’s prediction and the baseline model’s
reference. However, since imitation uses a sampling of observation and action
pairs, the trained agent might get exposed to states not encountered in
training. We then utilize the second reinforcement learning stage that enables
the model to adapt to the stochastic nature of the environment as well as
optimize its policy.

###  III-E Explainability using the attention mechanism

Explainability in machine learning models aims towards “making sense” to a
human being at an acceptable level. Attention mechanisms have been used
successfully towards providing explainability of deep learning models [33, 34,
35].

They do so by generating an attention matrix, which essentially encodes the
importance correlation - the magnitude of the explanation weights that
precisely reflect the importance of the input data. High attention weights
indicate that the corresponding features are important in the current
observation and decisive in generating the policy. This, therefore, allows us
to visualize and interpret how the model focuses on different parts of the
input data and assigns varying levels of importance to different features.

We would like to mention that while some researchers provide the notion that
attention does not automatically encode explainability [36], other researchers
have countered their claims based on the meaning of “explanation” which needs
to take into account all the elements of the model [37].

In our case, similar to [28], we learn the attention matrices corresponding to
the heads of the model. Essentially, for a state matrix comprised of the
kinematics of the vehicles around a given ego vehicle, the DRL model should
pay attention to the closest vehicles and also be attentive to the vehicle in
direct conflict with the ego vehicle’s future trajectory. Specifically, we
look at the correlations between distances and weights across the heads to
look at the spatial explainability. We also plot the weights over the entire
run to look at the temporal explainability of the DRL agent based on the
attention weights.

###  III-F Causality using Iterative Causal Discovery

While explainability based on the attention mechanism has been touted as
sufficient, it doesn’t suffice in discovering the reason why the model
produces certain outputs given the observation. Also, the attention-based
models suffer from a lack of explanation faithfulness [38] where there is a
polarity inconsistency [39], enhancing or suppressing the input. We,
therefore, look towards applying causality in inferring causal relations using
the attention matrix through the Iterative Causal Discovery (ICD) [7]. In ICD,
session-specific causal graphs are learned from attention to the possible
presence of latent confounders. The authors utilize the graphs to learn neural
recommenders.

Causal discovery uses certain assumptions - chief among them are the causal
Markov and faithfulness, with no parametric assumptions placed on the
distribution of the data. Given these assumptions, tests of conditional
independence (CI-tests) can be performed to learn the causal structure. The
CI-test checks whether the two nodes are independent given a conditioning set
of nodes using partial-correlation CI testing after evaluating the correlation
matrix from the attention matrix $A$. The covariance matrix is defined as
$K=AA^{T}$ and the correlation coefficients are defined as
$\rho_{i,j}=K_{i,j}/\sqrt{K_{i,i}.K_{j,j}}$.

Based on running the CI-tests, a partial ancestral graph (PAG) can be
constructed, which defines a set of causal graphs where the causal relations
definitely exist. In a PAG, there are three types of edge-marks at a given
node X - an arrow-head ’$\rightarrow$X ’, a tail ’—–X ’, and a circle
‘—$\circ$ X’ which represents an edge-mark that cannot be determined given the
data.

We look at ICD to provide causal relations between the vehicles in the event
of a lane change. Therefore, we separate the lane changes based not only on
left and right lane changes but also, on the origin lane. We aggregate the
attention matrices from the two heads for each type of lane change and the
source lane for 10 seconds before the lane change. We then conduct the CI-
tests and construct PAG for each lane change subtype.

Similar to the previous subsection, we look at the temporal nature of the
causal relationships for the neighboring vehicles using the temporal ICD [8].

##  IV Results

Figure 2: Comparing rewards over iterations for different heads Figure 3:
Baseline model vs the attention-based model using common driving metrics

In this section, we aim to provide interpretations of the actions of the DRL
given the state. To do this, we use an open-source AV simulation environment,
highway-env [40], developed for highway driving tasks. The agent is placed on
a straight three-lane highway with a user-defined number of other vehicles
spawned randomly in every episode. The other vehicles have IDM (Intelligent
Driver Model)-based longitudinal control [41] with a discrete lane change
decision based on the MOBIL model (Minimizing Overall Braking Induced by Lane
change) [42]. This environment provides the agent with a kinematic
representation of ego and neighboring vehicles, and the agent can take either
discrete lane change decisions or continuous acceleration and steering angle
information at each step.

We provide results first by benchmarking the multi-head attention-based DRL
against the baseline model setup in terms of common driving metrics. One of
the first questions we aim to answer is the number of heads that can provide
the optimal output. In order to answer this question, we attempt to vary the
number of heads to be between 1, 2, and 4. We also interpret the results of
the model in two adjoining functionalities: explainability and causality;
using both spatial and temporal correlations.

###  IV-A Benchmarking DRL agent performance

We train the baseline model and the 1-, 2- and 4-head attention models over
the 3M global steps. Figure 2 shows the reward over the iterations for the
different heads. We find that the 2-head fares better compared to others in
collecting rewards over the course of the training.

We also compare the performance of the trained baseline and the multi-head
attention-based models on a few common driving metrics-

  * •

number of times in an episode that the TTC is below 1 sec as a ratio of the
total episode length,

  * •

minimum TTC in seconds during an episode,

  * •

average velocity in an episode in meter/sec and,

  * •

total distance traveled over an episode in m.

We run the trained models for 100 episodes in highway-env. As mentioned in
section III, we use the baseline model as the expert and then train attention-
based models additionally. From the upper left subfigure of Figure 3, we find
that the attention-based models generally get into more unsafe situations
(given by the number of times the TTC goes below 1 sec) than the baseline
model. The 1-sec threshold is based on the fact that for a vehicle at $35m/s$
speed, reaction time of 100 ms, and deceleration of $10~{}m/s^{2}$, it takes
the vehicle about 3 seconds to stop completely. The upper right subfigure of
Figure 3 provides the minimum TTC in the episode which is highest for the
baseline model. The low TTC indicates that the attention-based models are more
likely to demonstrate risk-taking behaviors. For the average velocity given in
the lower left subfigure, the baseline model fares slightly better than the 2-
and 4-head attention-based models while 1-head is the fastest consistently.
The lower right subfigure indicates the distance traveled over an episode
without having a crash. Interestingly, the 2-head model performs the best
consistently over the baseline and other attention-based models. Overall, our
experiments show that adding attention weights skews the model towards being
riskier drivers, even though its not linear. The 2-head model, for example, is
more conservative than the 1- and 4-head models in terms of TTC and average
speed. Therefore, we choose the 2-head model for further analysis.

Figure 4: Change in attention weights and distance during left lane change and
right lane change.

###  IV-B Explainability Using Attention Interpretation

We interpret the attention in two different ways. Firstly, we analyze the
relation between the distance to the leading vehicle and the corresponding
attention weights. Then, we analyze the attention pattern in a specific lane
change scenario.

Figure 5: The top figure shows the lane change of the ego vehicle shown in
yellow with the neighboring vehicles with the evolution into the future. The
bottom plot shows the attention weights of the two heads for the lead vehicles
in the three lanes mentioned in the legend.

####  IV-B1 Spatial explainability of AV DRL

We plot the attention weight of the leading vehicle in the target lane against
the normalized distance from the AV to the leading vehicle for 10 seconds
leading up to the lane change point. Both left lane changes and right lane
changes are analyzed, and they are further divided based on the initial lane
position of the AV. For example, two types of left lane changes are generated
from the results, with the AV in the middle and right lane, respectively.

The relationships between the attention weights and the normalized distance
are visualized using 2D histograms as shown in Figure 4. We can see that the
agent pays more attention to the leading vehicle when it’s closer, which is
reasonable since the closer distance indicates a higher risk of collision,
thus this feature should be taken seriously. Moreover, taking the upper left
figure as an example, the variations of the weights get smaller when the
distance gets longer. This corresponds to the reality as the distance is
related to the interaction between the leading vehicle and the AV, the
interaction would be stronger for a shorter distance, which would result in a
larger variation of attention weights. Lastly, the attention weight has a
relatively left-skewed distribution when the distance is small, suggesting
that the agent pays more attention when it is close to the leading vehicle.

####  IV-B2 Temporal explainability of AV DRL

We investigate the temporal explainability by plotting the changes in
attention weights when the AV performs a left lane change from the right lane
to the middle lane as shown in Figure 5111The full video including the weights
are given in
https://drive.google.com/file/d/1niiYw1GgUFWJcuzsVfSiKKdABtB6aZkb/view?usp=drive_link.
The weights of the left leader from the two attention heads gradually increase
as the AV starts the preparation of the lane change. They then reach the
maximum values when the AV performs the lane change. This result corresponds
to the reality as the AV would pay more attention to the left leader since it
needs to find the appropriate gap to change the lane. On the contrary, the
weights of the leader gradually decrease as the AV starts performing the lane
change. The AV pays more attention to the leader in the beginning as it needs
to maintain a safe distance to the leader. However, it focuses less on the
leader while performing the lane change as this action does not affect the
leader directly.

Figure 6: ICD graphs of lane changes divided by the source and target lane -
(a) left lane change from right to middle lane, (b) left lane change from
middle to left lane, (c) right lane change from left to middle lane, and (d)
right lane change from middle to right lane

###  IV-C Causality

In this section, we perform a causal analysis of the attention weights using
ICD graphs [43]. Similar to the previous results, we collect the duration of
the lane changes and classify them based on the type of lane change and the
source lane.

####  IV-C1 Spatial causality

To determine the spatial causality, we attempt to understand whether there is
a causal relation between the ego-vehicle and certain neighboring vehicles
during the lane change. We choose the first head since it focuses on all the
neighboring vehicles instead of the second which focuses on the leader. As
mentioned in Section III-F, there are three different types of edge-marks
depending on the type of causal relationship: arrow-head for a causal
relationship to the node, tail for causal relation from the node, and circle
mark for no causal relation. Based on these edge-marks, we can get three edge
colors - blue if both nodes affect each other, green if one node affects
another, and red if there is not enough data to determine the relation.

Figure 6 shows the results of the first attention head during the lane changes
to understand which vehicle movement has a causal effect on the others. In the
left lane changes (shown in the top row of the figure), the ego vehicle is
causally related to the left leader and left follower, which influences how
the ego vehicle changes lanes. Similarly, for right lane changes (bottom row),
the ego vehicle is primarily influenced by the right leader. While these plots
confirm our intuition that causal relationships can be formulated among the
vehicles, we also find spurious correlations between other vehicles since lane
change is inherently a spatiotemporal process. Therefore, we look at temporal
measures of causality to infer a more meaningful relationship.

####  IV-C2 Temporal causality

To determine the temporal causal structures present in our data, we utilize
the temporal ICD [8]. We divide the time series into three discrete steps:
lane change point is given as t-1, while t is 5 sec after the lane change
point and t-2 is defined as 5 sec before the lane change point. From Figure 7,
we can see that the ego vehicle is primarily influenced by the leaders and
followers in the target lane at t-1 step. Specifically, in the upper row of
the figure, the ego vehicle is influenced by the left leader and left follower
in the t-1 step while in the bottom row, the ego vehicle is influenced by the
right leader and right follower in the t-1 step. This result clearly shows
that a temporal causal relationship can be firmly established between the
leader and the follower in the target lane.

Figure 7: Time-series ICD graph of the lane changes with t-1 step signifying
lane change point; t-2 step is 5 sec before and t is 5 sec after with - a)
left lane change from right to middle lane, (b) left lane change from middle
to left lane, (c) right lane change from left to middle lane, and (d) right
lane change from middle to right lane

##  V Discussion and conclusions

In this paper, we examine the attention-based mechanism to provide answers
regarding the decision-making used in DRL. While the attention-based DRL model
does not substantially increase the performance of the model, it provides
substantial benefits in interpreting the decision-making process. We show that
explainability can be provided through the weights of the attention both in
the spatial and the temporal domains. We also look at the causality of the
neighboring vehicles - which shows that the particular lane change is
triggered by the lead and the following vehicles. Our goal through this paper
is to provide researchers with the analysis to provide some scientific
interpretation of the actions provided by DRL and make the black-box neural
network interpretable. Our future work would be in using the physically
informed driving models to provide constraints on the reward function.

##  VI References Section

## References

  * [1] A. E. Hartavi, T. Kabbani, P. Sarhadi, A. Shah Alias Sangani, J. Zaya, S. Maroun, M. Saleh, S. Shanmugam, A. Krishna, J. Ou _et al._ , “Reliable sense-plan-act approaches for trustvehicles,” _Enhanced Trustworthiness and End User Acceptance of Conditionally Automated Vehicles in the Transition Period_ , pp. 69–85, 2021.
  * [2] M. Zhang, N. Li, A. Girard, and I. Kolmanovsky, “A finite state machine based automated driving controller and its stochastic optimization,” in _Dynamic Systems and Control Conference_ , vol. 58288. American Society of Mechanical Engineers, 2017, p. V002T07A002.
  * [3] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [4] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * [5] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [6] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 6, pp. 4909–4926, 2021.
  * [7] R. Y. Rohekar, S. Nisimov, Y. Gurwicz, and G. Novik, “Iterative causal discovery in the possible presence of latent confounders and selection bias,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 2454–2465, 2021.
  * [8] R. Rohekar, S. Nisimov, Y. Gurwicz, and G. Novik, “From temporal to contemporaneous iterative causal discovery in the presence of latent confounders,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 39 939–39 950.
  * [9] Q. Liu, X. Li, S. Yuan, and Z. Li, “Decision-making technology for autonomous vehicles: Learning-based methods, applications and future outlook,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2021, pp. 30–37.
  * [10] J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli, A. Huang, S. Karaman _et al._ , “A perception-driven autonomous urban vehicle,” _Journal of Field Robotics_ , vol. 25, no. 10, pp. 727–774, 2008.
  * [11] J. Ziegler, P. Bender, M. Schreiber, H. Lategahn, T. Strauss, C. Stiller, T. Dang, U. Franke, N. Appenrodt, C. G. Keller _et al._ , “Making bertha drive—an autonomous journey on a historic route,” _IEEE Intelligent transportation systems magazine_ , vol. 6, no. 2, pp. 8–20, 2014.
  * [12] H. Wang, B. Liu, X. Ping, and Q. An, “Path tracking control for autonomous vehicles based on an improved mpc,” _IEEE Access_ , vol. 7, pp. 161 064–161 073, 2019.
  * [13] S. Cheng, L. Li, H.-Q. Guo, Z.-G. Chen, and P. Song, “Longitudinal collision avoidance and lateral stability adaptive control system based on mpc of autonomous vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 6, pp. 2376–2385, 2019.
  * [14] C. Yu, Y. Zheng, B. Shyrokau, and V. Ivanov, “Mpc-based path following design for automated vehicles with rear wheel steering,” in _2021 IEEE International Conference on Mechatronics (ICM)_. IEEE, 2021, pp. 1–6.
  * [15] S. Kuutti, R. Bowden, Y. Jin, P. Barber, and S. Fallah, “A survey of deep learning applications to autonomous vehicle control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 2, pp. 712–733, 2020.
  * [16] M. Zhu, X. Wang, and Y. Wang, “Human-like autonomous car-following model with deep reinforcement learning,” _Transportation research part C: emerging technologies_ , vol. 97, pp. 348–368, 2018.
  * [17] S. B. Prathiba, G. Raja, K. Dev, N. Kumar, and M. Guizani, “A hybrid deep reinforcement learning for autonomous vehicles smart-platooning,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 12, pp. 13 340–13 350, 2021\.
  * [18] X. He, H. Yang, Z. Hu, and C. Lv, “Robust lane change decision making for autonomous vehicles: An observation adversarial reinforcement learning approach,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 1, pp. 184–193, 2022.
  * [19] Y. Chen, C. Dong, P. Palanisamy, P. Mudalige, K. Muelling, and J. M. Dolan, “Attention-based hierarchical deep reinforcement learning for lane change behaviors in autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [20] D. Omeiza, H. Webb, M. Jirotka, and L. Kunze, “Explanations in autonomous driving: A survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 8, pp. 10 142–10 162, 2021.
  * [21] P. J. Phillips, C. A. Hahn, P. C. Fontana, D. A. Broniatowski, and M. A. Przybocki, “Four principles of explainable artificial intelligence,” _Gaithersburg, Maryland_ , vol. 18, 2020.
  * [22] E. Puiutta and E. M. Veith, “Explainable reinforcement learning: A survey,” in _International cross-domain conference for machine learning and knowledge extraction_. Springer, 2020, pp. 77–95.
  * [23] L. M. Schmidt, G. Kontes, A. Plinge, and C. Mutschler, “Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning,” in _2021 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2021, pp. 171–178.
  * [24] Z. Cui, M. Li, Y. Huang, Y. Wang, and H. Chen, “An interpretation framework for autonomous vehicles decision-making via shap and rf,” in _2022 6th CAA International Conference on Vehicular Control and Intelligence (CVCI)_. IEEE, 2022, pp. 1–7.
  * [25] J. Kim and J. Canny, “Interpretable learning for self-driving cars by visualizing causal attention,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2942–2950.
  * [26] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2722–2730.
  * [27] P. Polack, F. Altché, B. d’Andréa Novel, and A. de La Fortelle, “The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?” in _2017 IEEE Intelligent Vehicles Symposium (IV)_ , 2017, pp. 812–818.
  * [28] E. Leurent and J. Mercat, “Social attention for autonomous decision-making in dense traffic,” _arXiv preprint arXiv:1911.12250_ , 2019.
  * [29] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _Icml_ , vol. 99. Citeseer, 1999, pp. 278–287.
  * [30] A. Baheri, S. Nageshrao, H. E. Tseng, I. Kolmanovsky, A. Girard, and D. Filev, “Deep reinforcement learning with enhanced safety for autonomous highway driving,” 2020.
  * [31] A. Kusari and J. P. How, “Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 7484–7490.
  * [32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  * [33] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 3319–3328.
  * [34] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” _arXiv preprint arXiv:1706.03825_ , 2017\.
  * [35] G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui, “Attention is not only a weight: Analyzing transformers with vector norms,” _arXiv preprint arXiv:2004.10102_ , 2020.
  * [36] S. Jain and B. C. Wallace, “Attention is not explanation,” _arXiv preprint arXiv:1902.10186_ , 2019.
  * [37] S. Wiegreffe and Y. Pinter, “Attention is not not explanation,” _arXiv preprint arXiv:1908.04626_ , 2019.
  * [38] A. Jacovi and Y. Goldberg, “Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?” _arXiv preprint arXiv:2004.03685_ , 2020.
  * [39] Y. Liu, H. Li, Y. Guo, C. Kong, J. Li, and S. Wang, “Rethinking attention-model explainability through faithfulness violation test,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 13 807–13 824.
  * [40] E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018.
  * [41] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” _Physical review E_ , vol. 62, no. 2, p. 1805, 2000.
  * [42] A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,” _Transportation Research Record_ , vol. 1999, no. 1, pp. 86–94, 2007.
  * [43] I. C. Lab, “Intel causality lab github repository,” https://github.com/IntelLabs/causality-lab, 2022, accessed:01-25-2023.

|  Hanxi Wan is a Master’s student at the University of Michigan. He received
his Bachelor’s degree in Computer Engineering from the University of Michigan
and Bachelor’s degree in Electrical and Computer Engineering from Shanghai
Jiao Tong University. His research interests include reinforcement learning,
robot planning, and robot perception.
---|---
|  Pei Li is a Scientist at the University of Wisconsin-Madison. He received
his Ph.D. in Civil Engineering from the University of Central Florida,
Orlando, FL, USA. His research interests include traffic safety, deep
learning, connected vehicles, and traffic simulation.
---|---
|  Arpan Kusari is an Assistant Research Scientist in the Engineering Systems
Group at University of Michigan Transportation Research Institute. His primary
interest is in Advanced Driver Assistance Systems (ADAS) and Automated Driving
Systems (ADS) where he works on enhancing robustness of the software pipeline.
He has been part of a number of federal and industry based projects in
researching safety, sensor calibration and testing individual ADS systems such
as localization. Previously, he spent five years at Ford Motor Company working
collaboratively with researchers from MIT and University of Michigan;
formulating research problems in autonomous vehicles. His doctoral research
was in LiDAR mapping in the areas of sensor calibration, precise estimation of
earthquake displacement and uncertainty quantification in the point cloud. Dr.
Kusari is a member of technical committees in SAE and ISPRS.
---|---