# A Survey of Explainable Reinforcement Learning

Stephanie Milani1 Equal contribution Nicholay Topin11footnotemark: 11 Manuela
Veloso1&Fei Fang1 1Carnegie Mellon University

###### Abstract

Explainable reinforcement learning (XRL) is an emerging subfield of
explainable machine learning that has attracted considerable attention in
recent years. The goal of XRL is to elucidate the decision-making process of
learning agents in sequential decision-making settings. In this survey, we
propose a novel taxonomy for organizing the XRL literature that prioritizes
the RL setting. We overview techniques according to this taxonomy. We point
out gaps in the literature, which we use to motivate and outline a roadmap for
future work.

##  1 Introduction

In reinforcement learning (RL), an agent learns to take actions in an
environment to maximize the accumulated reward through trial and error. This
framework has been successfully applied to many sequential decision-making
problems, like games Mnih et al. (2013), robotics Hayes and Shah (2017), and
more. However, the difficulty of verifying and predicting the behavior of RL
agents often hampers their real-world deployment. This problem is exacerbated
when RL is combined with the generalization and representational power of deep
neural networks (NNs). Without an understanding of how the agent works, it is
hard to intervene when necessary or trust that the agent will act reasonably
and safely.

Recently, there has been increasing interest in explaining RL models to gain
insight into the agent’s decision-making process. The catalyzation of this
interest can be traced to the DARPA XAI project Gunning (2017), which evolved
from a narrower focus on supervised learning to broader AI tasks, including
RL. Explainable RL (XRL) offers unique research opportunities: we can apply
techniques from explainable supervised learning and also develop methods that
account for and utilize the sequential nature of RL problems.

We present a comprehensive survey of XRL literature, organized based on our
novel proposed taxonomy. Existing surveys on XRL focus on delving deeper into
a more limited set of papers Heuillet et al. (2021); Puiutta and Veith (2020);
Wells and Bednarz (2021) or assessing the need for XRL in different domains
instead of summarizing techniques Zelvelder et al. (2021). In contrast, we
present a novel taxonomy for XRL research and cover a larger and more up-to-
date set of papers.

Figure 1: XRL taxonomy and its relationship to the RL process.

#### Scope.

We focus on papers in which the authors explicitly identify explainability or
interpretability as a goal of the work. Since we mainly care about the
learning and the behavior of the RL agent, we omit papers that focus only on
explaining the environment in which the agent acts. We only consider
techniques developed for a single agent. We limit our survey to published
work. After curating a smaller set of 92 candidate papers that satisfy the
inclusion criteria, we produce a representative sample of 49 papers in this
area.

Our contributions are summarized as follows:
$\boldsymbol{\cdot}$ New taxonomy: We present a new taxonomy for understanding
XRL research. We propose to organize techniques according to the part of the
RL agent they explain: the immediate context for single actions (termed
feature importance or FI), influential experiences from training (termed
Learning Process and MDP or LPM), and long-term behavior (termed policy-level
or PL). Figure 1 depicts the high-level categories of the taxonomy and their
relationship to the RL framework.
$\boldsymbol{\cdot}$ Comprehensive review: We overview the XRL literature,
categorizing representative approaches according to our novel taxonomy. We
compare and contrast the different types of explanation-generation techniques.
$\boldsymbol{\cdot}$ Future directions: We highlight gaps in the literature
and outline a roadmap for future work, covering: investigating properties of
explanations, developing benchmarks for standardized comparisons, and creating
RL-specific explanations.

Category | Subcategory | Publications
---|---|---
| Learn Intrinsically |  Hein et al. Hein et al. (2017, 2019), Silva et al. Silva et al. (2020), Landajuela et al. Landajuela et al. (2021),
| Interpretable Policy |  Topin et al. Topin et al. (2021), Zhang et al. Zhang et al. (2021a)
Feature | Interpretable Format |  Jhunjhunwala Jhunjhunwala (2019), Zhang et al. Zhang et al. (2020), Bewley and Lawry Bewley and Lawry (2021)
Importance |  |  Hayes and Shah Hayes and Shah (2017), Ehsan et al. Ehsan et al. (2018), Goel et al. Goel et al. (2018),
(FI) | Directly Generate |  Greydanus et al. Greydanus et al. (2018), Iyer et al. Iyer et al. (2018), Weitkamp et al. Weitkamp et al. (2018),
| Explanations |  Annasamy and Sycara Annasamy and Sycara (2019), Huber et al. Huber et al. (2019), Mott et al. Mott et al. (2019),
|  |  Wang et al. Wang et al. (2019), Atrey et al. Atrey et al. (2020), Shi et al. Shi et al. (2020), Tang et al. Tang et al. (2020),
|  |  Itaya et al. Itaya et al. (2021), Zhang et al. Zhang et al. (2021b)
Learning | Model Domain |  Cruz et al. Cruz et al. (2019, 2021), Chen et al. Chen et al. (2020), Madumal et al. Madumal et al. (2020),
Process | Information |  Yau et al. Yau et al. (2020), Lin et al. Lin et al. (2021), Volodin Volodin (2021)
and MDP | Decompose Reward Function |  Anderson et al. Anderson et al. (2019), Beyret et al. Beyret et al. (2019), Bica et al. Bica et al. (2021), Guo et al. Guo et al. (2021)
(LPM) | Identify Training Points |  Dao et al. Dao et al. (2018), Gottesman et al. Gottesman et al. (2020)
| Summarize Using Transitions |  Amir and Amir Amir and Amir (2018), Huang et al. Huang et al. (2018), Lage et al. Lage et al. (2019)
Policy-Level | Convert RNN |  Koul et al. Koul et al. (2019), Danesh et al. Danesh et al. (2021), Hasanbeig et al. Hasanbeig et al. (2021)
(PL) | Extract Clusters or Abstract States |  Zahavy et al. Zahavy et al. (2016), Topin and Veloso Topin and Veloso (2019), Sreedharan et al. Sreedharan et al. (2020)
Table 1: Taxonomy and corresponding representative publications in XRL.
Category color corresponds to color used in Figure 1.

##  2 Background and Preliminaries

###  2.1 Reinforcement Learning

In RL, an agent learns how to behave in an environment to maximize a reward
signal Sutton and Barto (2018). The environment is defined by a Markov
decision process (MDP) $M=(\mathcal{S},\mathcal{A},T,R,\gamma)$, where
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is the
state-transition function dictating the environment’s transition dynamics,
$R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the
reward function, and $\gamma\in(0,1]$ is the scalar discount factor that
governs the importance of future rewards. In RL, the model (i.e., $R$ and $T)$
is unknown. At each time step $t$, an agent observes a state $s_{t}$ and
chooses an action $a_{t}$ according to its policy $\pi$. The agent executes
$a_{t}$, receives reward $r_{t}$ according to $R$, and observes a new state
$s_{t+1}^{\prime}$ according to $T$. The goal of RL is to find an optimal
policy $\pi^{*}$, specifying which action to take in each state, which results
in the highest expected discounted future return:
$\pi^{*}=\operatorname*{arg\,max}_{\pi}Q^{\pi}(s,a)$, where $Q^{\pi}(s,a)$ is
the action-value function.

Large or continuous state spaces prevent exactly storing $Q^{\pi}$ or $\pi$.
Instead, many RL methods leverage function approximators: most commonly, NNs.
The input to these networks is often features describing the state. Agents
that use a non-interpretable function approximator require additional steps
for a person to understand.

###  2.2 Explainable Machine Learning

Most work in explainable machine learning focuses on supervised learning,
which learns from a labeled data set to predict a label $y$ given input data
$x$. These methods are typically grouped into two categories: intrinsic and
post-hoc. Intrinsic interpretability refers to the direct construction of
understandable models, such as small decision trees (DTs) Lipton (2018). Post-
hoc interpretability refers to creating an additional model to explain an
existing one. Explanations can also be local or global. Local explanations
explain the prediction for a specific data point; global explanations
holistically view the machine learning model and its predictions.

We focus on a few key metrics for evaluating explanations. We refer an
interested reader to a more comprehensive overview of metrics Molnar (2019).
Fidelity measures how faithful the model explanation is to the true
prediction. Performance refers to the standard evaluation metric for the task.
Relevancy refers to the ability of the explanation to produce insight for a
particular audience in a domain Murdoch et al. (2019). Cognitive load refers
to the cognitive resources required for performing a mental task Abdul et al.
(2020). Relevancy and cognitive load are challenging to operationalize and
quantitatively measure, but assessing them is invaluable for producing useful
and actionable explanations.

##  3 XRL Taxonomy

We propose a novel taxonomy that prioritizes the RL setting. We primarily
distinguish between the surveyed literature according to the central goals of
explanations for RL. Figure 1 depicts how these categorizations correspond to
parts of the RL framework. Feature importance (FI) explanations identify the
features that affect an agent’s action choice $a_{t}$ for the input state
$s_{t}$. Learning process and MDP (LPM) explanations show the past experiences
or the components of the MDP that led to the current behavior. Policy-level
(PL) explanations illustrate the long-term behavior of the agent.

These explanations communicate different aspects of the RL agent. FI
explanations provide an action-level look at the agent’s behavior: for each
action, one can query for the immediate context that was critical for making
that decision. LPM explanations provide additional information about the
effects of the training process or the MDP. Some LPM explanations enable
people to understand the influential experiences that led to the agent’s
current behavior. Others describe how the agent acts in terms of other MDP
components, like reward. PL explanations present summaries of long-term
behavior through abstraction or representative examples. They are critical for
understanding an agent’s behavior to evaluate its overall competency. See
Table 1 for the taxonomy and representative XRL publications.

##  4 Feature Importance (FI)

Many FI techniques extend those from the explainable supervised learning
literature. Viewed in the supervised learning framework, the input data is the
state, and the label is the action. Some FI techniques directly learn an
intrinsically interpretable policy (Section 4.1); others produce an
intrinsically interpretable policy through post-hoc conversion of a non-
interpretable one (Section 4.2). Other methods do not learn an interpretable
policy but instead generate action explanations in various forms, e.g.,
natural language (Section 4.3). Figure 2 depicts an example of different types
of explanations.

###  4.1 Learn an Intrinsically Interpretable Policy

We overview methods that directly learn intrinsically interpretable policies.
We distinguish between those that learn DT policies and those that learn other
policy representations.

DTs are not differentiable, so Silva et al. Silva et al. (2020) propose to
learn a soft DT, in which sigmoid activation functions replace the Boolean
decisions in classic DTs. This tree can then be discretized (at the cost of
performance) to form a DT-form policy approximation. In contrast, CUSTARD
Topin et al. (2021) constructs an augmented MDP where the action set contains
actions for constructing a DT, i.e., choosing a feature to branch a node. By
training the agent with the augmented MDP, the agent directly learns a DT
policy for the original environment while still using NNs during training.
However, DTs have drawbacks: they produce only axis-parallel partitions,
cannot handle approximate reasoning, and are non-differentiable. Alternative
representations mitigate these issues: trees with algebraic expressions
represented by nodes Hein et al. (2019); Landajuela et al. (2021) can
represent complex functions, fuzzy controllers Hein et al. (2017) enable
approximate reasoning, and certain logic formulations Zhang et al. (2021a)
permit differentiable learning.

These techniques enjoy the benefit of producing an intrinsically interpretable
policy. However, some representations risk more initial cognitive load from a
user to understand, so they are more applicable when their long-term benefit
outweighs the cost of the initial scaffolding for understanding explanations
Dodge et al. (2021).

###  4.2 Convert Policy to Interpretable Format

Some methods approximate an existing, performant NN policy $\pi$ with an
interpretable surrogate model $\hat{\pi}$. To learn $\hat{\pi}$, we can use
imitation learning (IL) Abbeel and Ng (2004) — also known as learning from
demonstration Argall et al. (2009) — where the RL policy $\pi$ is the expert,
and the IL policy $\hat{\pi}$ is the learner. The problem becomes a
classification problem: the learner aims to correctly classify which action
(label) to take from a state, given examples from the expert.

These techniques augment the IL process, expand the data used for training, or
propose a new type of surrogate model. VIPER Bastani et al. (2018) trains a
sequence of DTs on data points sampled based on how “critical” they are,
measured by $V^{\pi}(s)-\min_{a\in\mathcal{A}}Q^{\pi}(s,a)$. PIRL Verma et al.
(2018) locally searches over interpretable programmatic policies by minimizing
the distance between $\pi$’s outputs and $\hat{\pi}$’s outputs on
heuristically-chosen states. Jhunjhunwala Jhunjhunwala (2019) distills deep RL
policies into Q-value-based trees with nodes that encode more complex boundary
expressions. Zhang et al. Zhang et al. (2020) derive $\hat{\pi}$ from $\pi$
with genetic programming. TripleTree Bewley and Lawry (2021) expands the
information used in the DT-learning algorithm. Liu et al. Liu et al. (2018)
introduces DTs with linear models at the leaves and proposes to use stochastic
gradient descent to update their values.

An IL policy is necessarily based only on the states and actions encountered
during the example executions from the expert, so it may not be robust to
novel situations. As robust IL techniques advance and become incorporated with
RL, we expect even more developments in post-hoc XRL. Furthermore, these
papers show that it is possible to achieve reasonable performance and
fidelity, but the explanation is still the approximated model, not the
original policy.

(a)

(b)

(c)

(d)

Figure 2: Example object saliency map (b), natural language explanation (c),
and DT policy (d), given domain state (a).

###  4.3 Directly Generate Explanation

Action explanations generated from a non-interpretable policy most commonly
take the forms of natural language and saliency maps. Natural-language
explanations linguistically describe the agent’s decision-making process.
Saliency maps are images that topographically highlight regions that are
important for an agent’s decision. We also discuss explanations that cannot be
categorized as one of the previous types.

#### Natural Language.

We distinguish natural-language explanations by their means of production.
Some work leverages templates for the agent to fill in Hayes and Shah (2017);
other work allows the agent to generate free-form explanations. The former
type typically enjoys more clarity but requires the explanation to adhere to a
specified template. Furthermore, there is some question about the validity of
these explanations. Ehsan et al. Ehsan et al. (2018) and Wang et al. Wang et
al. (2019) leverage human-provided action-explanation pairs to rationalize
agent behavior. We caution against post-hoc rationalizations that produce
plausible but not valid explanations: they may increase agent trust, but the
trust is unfounded without grounding to the agent’s decision-making process.

Type | Publications
---|---
|  Zahavy et al. Zahavy et al. (2016), Weitkamp et al. Weitkamp et al. (2018),
Gradient |  Huber et al. Huber et al. (2019)
Perturbation |  Greydanus et al. Greydanus et al. (2018), Rupprecht et al. Rupprecht et al. (2020)
|  Iyer et al. Iyer et al. (2018), Goel et al. Goel et al. (2018),
Object |  Annasamy and Sycara Annasamy and Sycara (2019)
|  Mott et al. Mott et al. (2019), Shi et al. Shi et al. (2020),
Attention |  Tang et al. Tang et al. (2020), Itaya et al. Itaya et al. (2021),
|  Zhang et al. Zhang et al. (2021b)
Table 2: Saliency maps and their associated categorization.

#### Saliency Maps.

We categorize saliency map techniques using the decomposition in Atrey et al.
Atrey et al. (2020) (see Table 2). For example Greydanus et al. Greydanus et
al. (2018) construct saliency maps by perturbing the input image describing
the state with Gaussian blur, then measuring changes in the policy after
removing information from an area. For more details, we refer the reader to
Atrey et al. Atrey et al. (2020). Saliency is viewed as insufficient for
explaining RL, as the conclusions drawn from it are highly subjective Atrey et
al. (2020). Indeed, the majority of claims in the surveyed papers either
propose an ad hoc explanation of agent behavior after observing saliency or
develop an a priori explanation of behavior evaluated using saliency. In real-
world settings, we desire more reliable explanations.

#### Other.

There exist other forms of explanation, including counterfactual states Olson
et al. (2021), which illustrate the minimal change needed to induce a
different action selection $a_{t}^{\prime}$. This area is ripe for
exploration, especially given the recent interest in multimodal machine
learning Baltrušaitis et al. (2018). In this setting, we may want custom local
explanations that reflect different modalities.

##  5 Learning Process and MDP (LPM)

LPM explanations reveal the influence of other parts of the MDP (e.g., $R$ and
$T$) or the interactions with the MDP during learning on the agent’s behavior.
Most commonly, the agent reveals its learned transition dynamics $\hat{T}$
(Section 5.1). Alternatively, techniques decompose the reward function to
produce explanations in terms of the agent’s objectives (Section 5.2). Other
techniques determine training points that influenced the learned policy
(Section 5.3).

###  5.1 Model Domain Information

Approaches in this category learn models $\hat{T}$ that approximate $T$ and
generate explanations based on $\hat{T}$. They differ in terms of the
questions these generated explanations answer.

To answer “what-if” questions, Rupprecht et al. Rupprecht et al. (2020) learn
a generative model of the environment, which enables the probing of agent
behavior in novel states created by an optimization scheme that induces
certain actions in agents. For example, one can optimize for states where the
agent expects to perform poorly, then visualize such states to observe the
agent’s behavior. To contrast between a selected action $a_{t}$ and
alternative ones $a_{t}^{\prime}$, Cruz et al. Cruz et al. (2019, 2021) use
episodic memory to explain decisions with success probability, i.e., “$a_{t}$
gives an 85% probability of task success compared to 38 % of
$a_{t}^{\prime}$”. Lin et al. Lin et al. (2021) provide action preference
explanations in terms of human-provided features. Yau et al. Yau et al. (2020)
provide explanations in terms of intended outcome: the agent projects
predicted future trajectories from its current state and proposed action. To
answer “why?” and “why not?” questions, Madumal et al. Madumal et al. (2020)
leverage an action influence graph, which is a structural causal model (with
pre-specified, potentially learned Volodin (2021) structure) that encodes
causal action-based relationships between state features of interest. To
understand correlative context for failure cases, Chen et al. Chen et al.
(2020) learn a sequential latent environment model that can be used to
generate a semantic mask to understand the surrounding context.

Explanations can address a plethora of potential questions a user may have.
Clearly defining these questions and the ability of the techniques to address
them is important for users to understand an explanation’s capabilities. For
example, we do not want users to assume causal relationships when the
explanation only provides correlations.

###  5.2 Decompose Reward Function

By conveying aspects of the reward that contributed to the agent’s behavior,
an explanation provides information in terms of the agent’s objectives. Most
work in this area focuses on presenting interpretable views of $Q^{\pi}(s,a)$.

Before the agent starts learning, the explanation designer can use reward
decomposition charts (RDCs) Juozapaitis et al. (2019); Anderson et al. (2019)
to split $R(s,a)$ into a set of additive terms with semantic meaning:
$R(s,a)=\sum_{c\in C}R_{c}(s,a)$, where $C$ is the set of reward types.
$Q(s,a)$ is similarly decomposed as $Q^{\pi}(s,a)=\sum_{c\in
C}Q^{\pi}_{c}(s,a)$. With this decomposition, one can inspect actions in terms
of trade-offs between the types. Bica et al. Bica et al. (2021) model an
expert’s reward function in terms of preferences over proposed alternative
outcomes to produce explanations of the trade-offs associated with actions.
Beyret et al. Beyret et al. (2019) decompose the agent’s task into subtasks to
visualize per-subtask Q-values as a heatmap over the environment. EDGE Guo et
al. (2021) learns to identify which timesteps contributed to the final reward
to identify policy errors and weaknesses.

Reward-based explanations may help laypeople understand how reward-based
objectives influence the decision-making process of an RL policy Anderson et
al. (2019). However, we require further explanatory steps to produce
actionable explanations. For example, RDCs could be combined with an approach
from Section 5.3 to illustrate which experiences impact each of the
$Q^{\pi}_{c}$ estimates. Ideally, this combination also reveals the
experiences to remove to affect the estimates of these different components.

###  5.3 Identify Training Points

These techniques identify important training points, or transition tuples,
that influence the agent’s learned behavior. Influence refers to a change in
some function value (e.g., $Q(s,a)$).

Dao et al. Dao et al. (2018) select and record states with associated
importance weights that correspond to different decisions in a learned policy.
Equipped with this knowledge, one can understand the most influential
transition tuples for a policy’s individual decisions. However, it is unclear
how to amend the training process to remove undesirable behaviors based on
this information alone. Gottesman et al. Gottesman et al. (2020) identify
training points that are influential for the estimation of Q-values (i.e.,
whose removal causes a large change in Q-value estimates for initial states).
This technique enables users to determine if there is sufficient data for
learning the desired behavior and what type of data is lacking.

##  6 Policy-Level (PL)

A less-studied but important category of explanations is those which describe
an agent’s longer-term behavior. We discuss three sub-categories: summarizing
behavior in the context of experiences encountered by the agent during
training (Section 6.1), converting recurrent neural networks (RNNs) to
interpretable forms (Section 6.2), and summarizing behavior according to
similar states (Section 6.3).

###  6.1 Summarize Using Transitions

We can explain an agent’s behavior in terms of the experiences encountered by
the agent during training. With these explanations, human operators know which
experiences influence the agent’s behavior, providing more information to
adjust the behavior if desired. These methods are distinct from those
described in Section 5.3 because the goal is to describe the agent’s long-term
behavior based on influential trajectories, not individual transition tuples.
These trajectories are combined to create a summary of agent behavior.

These methods use different selection criteria for choosing the trajectories
or transition tuples. Amir and Amir Amir and Amir (2018) select trajectories
based on the state importance $I(s)=\max_{a}Q^{\pi}(s,a)-\min_{a}Q^{\pi}(s,a)$
and diversity, which is considered by identifying the most similar state
$s_{s}$ to a state $s$ currently included in the summary and comparing
$I(s_{s})$ with $I(s)$. The more important state and its corresponding
trajectory is used in the summary. Similarly, Huang et al. Huang et al. (2018)
select states where the chosen action has a much higher Q-value than another
action. To investigate the effects of different summary-extraction models on
policy reconstruction, Lage et al. Lage et al. (2019) compare approaches for
choosing subsets of $s,a$ pairs that characterize the agent’s behavior. They
motivate personalizing user models for summarization, underscoring the
importance of relevancy as an evaluation metric.

These metrics for importance tend to highlight extreme policy behavior: states
in which the agent risks performing very poorly compared to its best action.
This property is useful for high-stakes, real-world scenarios. However, these
techniques are not helpful for understanding how these behaviors fit together.
This problem is exacerbated when scaling the environment complexity, as the
trajectories may be so widely temporally spaced that the interpretation is not
useful.

###  6.2 Convert RNN to Interpretable Format

RNNs commonly represent policies because their internal memory encodes
features of the observation history critical for decision making. One way to
understand these structures is through learning more compact memory
representations by converting the RNN to a finite state machine (FSM) Koul et
al. (2019). However, this minimization process can merge semantically-distinct
states, obfuscating a deeper understanding of the FSM. As a result, Danesh et
al. Danesh et al. (2021) propose a more interpretable reduction technique to
preserve key decision points of the policy. This family of techniques only
applies in cases where the agent contains a RNN. In contrast, Hasanbeig et al.
Hasanbeig et al. (2021) construct a deterministic finite automaton to capture
these sequential dependencies in an intrepretable representation without
reliance on an RNN policy.

Completely removing the RNN from the agent permits a more interpretable
structure. However, the agent is changed through the minimization process, yet
it is treated as equivalent to the original agent. In other words, the agent
acts like it is following the resulting FSM; however, it is not internally
behaving this way.

###  6.3 Extract Clusters or Abstract States

Sometimes it is invaluable to understand the overall behavior of the agent in
terms of how it will act when it encounters similar states. One advantage of
this approach is that people only need to understand the similarity metric for
aggregating states and the model for summarizing the policy. XRL methods that
produce summarizations using these clustered or abstract states are similar to
those for generating abstractions for RL. We refer an interested reader to
recent work Abel and Littman (2020) on abstraction-generation methods.

TLdR Sreedharan et al. (2020) identifies landmarks, which are defined as
propositional formulas that must be satisfied for an agent to complete a goal.
The possible transitions between landmarks are displayed as a graph depicting
how an agent would reach a goal state from the starting state. Abstract policy
graphs Topin and Veloso (2019) are Markov chains of abstract states that
summarize a policy in terms of expected future transitions. Zahavy et al.
Zahavy et al. (2016) do not directly construct graphs; instead, states are
embedded into a space, where the agent treats states located close to one
another in this space similarly. A human operator can identify clusters and
relationships between these clusters through manual inspection.

These techniques rely on different assumptions, such as the binarization of
features Topin and Veloso (2019). Future work could investigate relaxing
assumptions to produce more general techniques. Furthermore, we may desire
different semantic graph structures, such as hierarchical graphs, where the
high-level graph describes the subtask transitions and the lower-level graphs
describe the lower-level control policies.

##  7 Discussion

We compare and contrast FI, LPM, and PL explanations, focusing specifically on
what these techniques do and do not enable. With these comparisons, we aim to
demonstrate the trade-offs between the different types of explanations.

FI explanations communicate the local behavior, or individual action choice,
of an agent in a particular situation. Some techniques guarantee that the
explanations capture the information used in the agent’s decisions, but others
do not. For example, techniques that directly produce DT policies necessarily
elucidate the features and corresponding values that influence the agent’s
decision. In contrast, techniques that generate post-hoc rationalizations,
like some natural language explanations, produce explanations based on a
human’s decision-making process when solving a task. Furthermore, FI
explanations do not provide global behavioral summaries that convey subtask or
plan information: they only explain individual action choices. PL
explanations, however, provide this information by producing summaries using
transitions, model approximations, or similar states.

LPM explanations enable understanding of the influence of the MDP or training
process on the agent. However, they generally require extra information or
learning during training, so we cannot generate them post-hoc. For example,
RDCs require the explanation designer to use domain information to decompose
the reward function before any learning occurs. In contrast, many FI and PL
explanations do not require additional information and/or learning, so we can
generate them post-hoc.

PL explanations permit the understanding of long-term behavior. Because these
explanations are summaries, they necessarily exclude details for the sake of
interpretability. For example, techniques from Section 6.3 exclude information
in the state aggregation techniques by definition of abstraction. In contrast,
intrinsically-interpretable FI explanations explicitly include all of the
information available about how the agent makes choices.

##  8 A Roadmap for XRL

Here we synthesize the gaps in the XRL literature to discuss opportunities for
future research. We focus on the following directions: investigating the
properties of explanations for RL, standardizing evaluation through
benchmarks, and focusing on techniques that leverage the unique facets of the
RL problem to provide correct and coherent explanations.

###  8.1 Properties

We believe it is important to investigate properties of explanations for RL
and to standardize these evaluations through operationalized metrics. Some
metrics, such as cognitive load and relevancy, necessitate user studies to
properly evaluate; however, only 12 of the 49 surveyed papers include user
studies. Unfortunately, some studies give an incomplete picture of the utility
of the proposed explanations. For example, it is not enough to evaluate
whether people believe explanations are useful; we must also evaluate whether
they are useful and accurate in explaining the agent’s behavior.

When proposing a new form of explanation, we recommend performing a user study
to evaluate if the explanation is practical and understandable. Measuring
comprehensibility tells us whether the explanation is understandable at all,
whereas measuring the associated cognitive load provides insight into how
challenging the explanation is to interpret. In these studies, we also suggest
accounting for influential characteristics of the users, such as domain
expertise, to evaluate the relevance of the explanations Ehsan and Riedl
(2020).

We caution against substituting explanations with ad-hoc rationalizations.
Instead, explanations ought to reflect the agent’s underlying decision-making
process, rather than a post-hoc justification of behavior based on subjective
interpretation Atrey et al. (2020). With these more accurate explanations, we
can not only increase user trust in the system but also ensure that this trust
is well-founded.

Finally, we encourage researchers to provide concrete descriptions of what
their methods enable. For example, it can be misleading to provide
explanations phrased as causal relationships for techniques that only explain
correlations. Transparency surrounding the affordances of explanations is
critical to prevent user misunderstanding.

###  8.2 Benchmarks

Domains used in the literature to evaluate explanations vary widely, from toy
control Zhang et al. (2021a) to games Olson et al. (2021) to real healthcare
data Bica et al. (2021). We believe it is critical to evaluate the quality of
explanations using benchmarks with standardized environments and metrics.
Benchmarks, and competitions based off of these benchmarks Cobbe et al.
(2020), can propel research forward by providing appropriate scope and focus
to an important problem. To that end, we propose two types of benchmarks for
XRL research: type-based and objective-based.

Type-based benchmarks aim to produce high-quality explanations of a certain
type, e.g., DT policies, which are evaluated in a standardized set of
environments with common metrics. If possible, these environments should
permit evaluation against some ground-truth. For example, when comparing DTs,
it should be possible to represent $\pi^{*}$ with a DT in (at least) the
simple environments (e.g., Cartpole) to ensure that the techniques can
reliably learn the optimal DT policy. In an objective-based benchmark, the
goal is to produce high-quality explanations of any type for a particular
task. The goal is to produce explanations that score the highest on a set of
evaluation metrics, which can include comprehensibility and predictability of
the system by users.

In both types of benchmarks, we require human-in-the-loop evaluations to
properly compare techniques against one another. To our knowledge, only one
benchmark provides a principled means of doing such an evaluation Shah et al.
(2021); however, it is done in the context of learning from hard-to-define
reward functions, not interpretability. As a result, another critical research
direction is developing a principled approach for human-in-the-loop
evaluations Dodge et al. (2021) of XRL techniques.

###  8.3 RL-Specific Explanations

As shown in Table 1, a large portion of XRL techniques stem from the
supervised-learning literature. In particular, FI techniques mirror the
standard interpretable classification setup by providing local explanations of
single actions (the label) in the context of the immediate state (the input
data to be classified). Although these explanations are useful for elucidating
the immediate context of local decisions, we can leverage important components
of the RL problem to produce more actionable and informative explanations. As
a result, we believe a fruitful direction of research is one which leverages
these critical components, such as reward to communicate agent objectives
(Section 5.2), causal models of the agent’s interactions with the environment
to understand the agent’s impact (Section 5.1), and long-term behavior to
depict how an agent will behave over time (Section 6).

##  9 Conclusion

We presented a comprehensive summary of XRL techniques in the context of our
novel taxonomy. We compared representative approaches of each type and
discussed their applicability and critiques. We identified gaps in the current
literature and concluded by suggesting three major avenues of research to
drive the field forward.

## Acknowledgments

This research was sponsored by the U.S. Army Combat Capabilities Development
Command Army Research Laboratory and was accomplished under Cooperative
Agreement Number W911NF-13-2-0045 (ARL Cyber Security CRA). The views and
conclusions contained in this document are those of the authors and should not
be interpreted as representing the official policies, either expressed or
implied, of the Combat Capabilities Development Command Army Research
Laboratory or the U.S. Government. The U.S. Government is authorized to
reproduce and distribute reprints for Government purposes notwithstanding any
copyright notation here on. This material is based upon work supported by the
Department of Defense (DoD) through the National Defense Science & Engineering
Graduate (NDSEG) Fellowship Program. Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the Department of Defense, Defense
Advanced Research Projects Agency or the Air Force Research Laboratory.

## References

  * Abbeel and Ng [2004] P Abbeel and AY Ng.  Apprenticeship learning via inverse reinforcement learning.  In ICML, 2004.
  * Abdul et al. [2020] A Abdul, C von der Weth, et al.  Cogam: measuring and moderating cognitive load in machine learning model explanations.  In CHI, 2020.
  * Abel and Littman [2020] D Abel and M Littman.  A theory of abstraction in reinforcement learning.  Brown Univ., 2020.
  * Amir and Amir [2018] D Amir and O Amir.  Highlights: Summarizing agent behavior to people.  In AAMAS, 2018.
  * Anderson et al. [2019] A Anderson, J Dodge, et al.  Explaining reinforcement learning to mere mortals: An empirical study.  In IJCAI, 2019.
  * Annasamy and Sycara [2019] RM Annasamy and K Sycara.  Towards better interpretability in deep q-networks.  In AAAI, 2019.
  * Argall et al. [2009] BD Argall, S Chernova, et al.  A survey of robot learning from demonstration.  Robot. Auton. Syst., 2009.
  * Atrey et al. [2020] A Atrey, K Clary, and D Jensen.  Exploratory not explanatory: Counterfactual analysis of saliency maps for deep rl.  In ICLR, 2020.
  * Baltrušaitis et al. [2018] T Baltrušaitis, C Ahuja, and LP Morency.  Multimodal machine learning: A survey and taxonomy.  TPAMI, 2018.
  * Bastani et al. [2018] O Bastani, Y Pu, and A Solar-Lezama.  Verifiable reinforcement learning via policy extraction.  In NeurIPS, 2018.
  * Bewley and Lawry [2021] T Bewley and J Lawry.  Tripletree: A versatile interpretable representation of black box agents and their environments.  In AAAI, 2021.
  * Beyret et al. [2019] B Beyret, A Shafti, and AA Faisal.  Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation.  In IROS, 2019.
  * Bica et al. [2021] I Bica, D Jarrett, et al.  Learning “what-if” explanations for sequential decision-making.  In ICLR, 2021.
  * Chen et al. [2020] J Chen, S Eben Li, and M Tomizuka.  Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning.  T-ITS, 2020.
  * Cobbe et al. [2020] K Cobbe, C Hesse, et al.  Leveraging procedural generation to benchmark reinforcement learning.  In ICML, 2020.
  * Cruz et al. [2019] F Cruz, R Dazeley, and P Vamplew.  Memory-based explainable reinforcement learning.  AI*IA, 2019.
  * Cruz et al. [2021] F Cruz, R Dazeley, and P Vamplew.  Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario.  Neural. Comput. Appl., 2021.
  * Danesh et al. [2021] MH Danesh, A Koul, et al.  Re-understanding finite-state representations of recurrent policy networks.  In ICML, 2021.
  * Dao et al. [2018] G Dao, I Mishra, and M Lee.  Deep reinforcement learning monitor for snapshot recording.  In ICMLA, 2018.
  * Dodge et al. [2021] J Dodge, A Anderson, et al.  From “no clear winner” to an effective explainable artificial intelligence process: An empirical journey.  Appl. AI Letters, 2021.
  * Ehsan and Riedl [2020] U Ehsan and MO Riedl.  Human-centered explainable ai: Towards a reflective sociotechnical approach.  In HCII, 2020.
  * Ehsan et al. [2018] U Ehsan, B Harrison, et al.  Rationalization: A neural machine translation approach to generating natural language explanations.  AIES, 2018.
  * Goel et al. [2018] V Goel, J Weng, and P Poupart.  Unsupervised video object segmentation for deep reinforcement learning.  In NeurIPS, 2018.
  * Gottesman et al. [2020] O Gottesman, J Futoma, et al.  Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions.  In ICML, 2020.
  * Greydanus et al. [2018] S Greydanus, A Koul, et al.  Visualizing and understanding atari agents.  In ICML, 2018.
  * Gunning [2017] D Gunning.  Explainable artificial intelligence.  DARPA, 2017.
  * Guo et al. [2021] W Guo, X Wu, et al.  Edge: Explaining deep reinforcement learning policies.  NeurIPS, 2021.
  * Hasanbeig et al. [2021] M Hasanbeig, NY Jeppu, et al.  Deepsynth: Automata synthesis for automatic task segmentation in deep reinforcement learning.  In AAAI, 2021.
  * Hayes and Shah [2017] B Hayes and JA Shah.  Improving robot controller transparency through autonomous policy explanation.  In HRI, 2017.
  * Hein et al. [2017] D Hein, A Hentschel, et al.  Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies.  Eng. App. AI, 2017.
  * Hein et al. [2019] D Hein, S Udluft, and TA Runkler.  Interpretable policies for reinforcement learning by genetic programming.  In GeCCO, 2019.
  * Heuillet et al. [2021] A Heuillet, F Couthouis, and N Díaz-Rodríguez.  Explainability in deep reinforcement learning.  Knowl. Based Syst., 2021.
  * Huang et al. [2018] SH Huang, K Bhatia, et al.  Establishing appropriate trust via critical states.  In IROS, 2018.
  * Huber et al. [2019] T Huber, D Schiller, and E André.  Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation.  In KI, 2019.
  * Itaya et al. [2021] H Itaya, T Hirakawa, et al.  Visual explanation using attention mechanism in actor-critic-based deep reinforcement learning.  In IJCNN, 2021.
  * Iyer et al. [2018] R Iyer, Y Li, et al.  Transparency and explanation in deep reinforcement learning neural networks.  In AIES, 2018.
  * Jhunjhunwala [2019] A Jhunjhunwala.  Policy extraction via online q-value distillation.  M. Thesis, Univ. Waterloo, 2019.
  * Juozapaitis et al. [2019] Z Juozapaitis, A Koul, et al.  Explainable reinforcement learning via reward decomposition.  Tech. Report, 2019.
  * Koul et al. [2019] A Koul, S Greydanus, and A Fern.  Learning finite state representations of recurrent policy networks.  In ICLR, 2019.
  * Lage et al. [2019] I Lage, D Lifschitz, et al.  Exploring computational user models for agent policy summarization.  In IJCAI, 2019.
  * Landajuela et al. [2021] M Landajuela, BK Petersen, et al.  Discovering symbolic policies with deep reinforcement learning.  In ICML, 2021.
  * Lin et al. [2021] Z Lin, KH Lam, and A Fern.  Contrastive explanations for reinforcement learning via embedded self predictions.  In ICLR, 2021.
  * Lipton [2018] ZC Lipton.  The mythos of model interpretability.  ACM Queue, 16(3), 2018.
  * Liu et al. [2018] G Liu, O Schulte, et al.  Toward interpretable deep reinforcement learning with linear model u-trees.  In ECML-KDD, 2018.
  * Madumal et al. [2020] P Madumal, T Miller, et al.  Explainable reinforcement learning through a causal lens.  In AAAI, 2020.
  * Mnih et al. [2013] V Mnih, K Kavukcuoglu, et al.  Playing Atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * Molnar [2019] C Molnar.  Interpretable Machine Learning.  2019\.
  * Mott et al. [2019] A Mott, D Zoran, et al.  Towards interpretable reinforcement learning using attention augmented agents.  In NeurIPS, 2019.
  * Murdoch et al. [2019] WJ Murdoch, C Singh, et al.  Definitions, methods, and applications in interpretable machine learning.  PNAS, 116(44), 2019.
  * Olson et al. [2021] ML Olson, R Khanna, et al.  Counterfactual state explanations for reinforcement learning agents via generative deep learning.  AI, 2021.
  * Puiutta and Veith [2020] E Puiutta and E Veith.  Explainable reinforcement learning: A survey.  CD-MAKE, 2020.
  * Rupprecht et al. [2020] C Rupprecht, C Ibrahim, and CJ Pal.  Finding and visualizing weaknesses of deep reinforcement learning agents.  In ICLR, 2020.
  * Shah et al. [2021] R Shah, C Wild, et al.  The MineRL basalt competition on learning from human feedback.  NeurIPS Compet. Track, 2021.
  * Shi et al. [2020] W Shi, Z Wang, et al.  Self-supervised discovering of causal features: Towards interpretable reinforcement learning.  TPAMI, 2020.
  * Silva et al. [2020] A Silva, TW Killian, et al.  Optimization methods for interpretable differentiable decision trees in reinforcement learning.  In AISTATS, 2020.
  * Sreedharan et al. [2020] S Sreedharan, S Srivastava, and S Kambhampati.  Tldr: Policy summarization for factored ssp problems using temporal abstractions.  In ICAPS, 2020.
  * Sutton and Barto [2018] RS Sutton and AG Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * Tang et al. [2020] Y Tang, D Nguyen, and D Ha.  Neuroevolution of self-interpretable agents.  In GeCCO, 2020.
  * Topin and Veloso [2019] N Topin and M Veloso.  Generation of policy-level explanations for reinforcement learning.  In AAAI, 2019.
  * Topin et al. [2021] N Topin, S Milani, et al.  Iterative bounding mdps: Learning interpretable policies via non-interpretable methods.  In AAAI, 2021.
  * Verma et al. [2018] A Verma, V Murali, et al.  Programmatically interpretable reinforcement learning.  In ICML, 2018.
  * Volodin [2021] S Volodin.  Causeoccam: Learning interpretable abstract representations in reinforcement learning environments via model sparsity.  Tech. Report, 2021.
  * Wang et al. [2019] X Wang, Y Schengcheng, et al.  Verbal explanations for deep reinforcement learning neural networks with attention on extracted features.  In RO-MAN, 2019.
  * Weitkamp et al. [2018] L Weitkamp, E van der Pol, and Z Akata.  Visual rationalizations in deep reinforcement learning for Atari games.  In BNAIC, 2018.
  * Wells and Bednarz [2021] L Wells and T Bednarz.  Explainable ai and reinforcement learning—a systematic review of current approaches and trends.  Frontiers in AI, 2021.
  * Yau et al. [2020] H Yau, C Russell, and S Hadfield.  What did you think would happen? explaining agent behaviour through intended outcomes.  In NeurIPS, 2020.
  * Zahavy et al. [2016] T Zahavy, N Ben-Zrihem, and S Mannor.  Graying the black box: Understanding dqns.  In ICML, 2016.
  * Zelvelder et al. [2021] AE Zelvelder, M Westberg, and K Främling.  Assessing explainability in reinforcement learning.  In EXTRAAMAS, 2021.
  * Zhang et al. [2020] H Zhang, A Zhou, and X Lin.  Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis.  Complex Intell. Syst., 2020.
  * Zhang et al. [2021a] L Zhang, X Li, et al.  Off-policy differentiable logic reinforcement learning.  In ECML-KDD, 2021.
  * Zhang et al. [2021b] Q Zhang, X Ma, et al.  Learning to discover task-relevant features for interpretable reinforcement learning.  I-RAL, 2021.