#  View & Perspective, Frontiers of Physics
Tasting Nuclear Pasta Made with Classical Molecular Dynamics Simulations

Bao-An Li  Department of Physics and Astronomy, Texas A&M University-Commerce,
TX 75429-3011, USA

###### Abstract

Nuclear clusters or voids in the inner crust of neutron stars were predicted
to have various shapes collectively nicknamed nuclear pasta. The recent review
in Ref. Lopez1  by López, Dorso and Frank summarized their systematic
investigations into properties especially the morphological and
thermodynamical phase transitions of the nuclear pasta within a Classical
Molecular Dynamics model, providing further stimuli to find more observational
evidences of the predicted nuclear pasta in neutron stars.

Known as the densest visible object in the Universe, neutron stars have many
mysterious properties to be understood. From the external magnetic field
through the surface and crust to the core of neutron stars, there are many
fundamental questions to be addressed NAP2011 ; NAP2012 . Thanks to the recent
advances especially in radio, X-ray and gravitational wave observations of
both isolated neutron stars and their mergers, much progresses have been made
in recent years in understanding the maximum mass, radius and tidal
deformation of neutron stars. These global observables provided some of the
much needed constraints on various theories about the equation of state,
internal structure and composition of neutron stars. However, because of the
scarcity of data available especially those from the un-distorted messengers
directly from the interior of neutron stars and the model dependence of their
interpretations, many mysteries associated with neutron stars remain to be
resolved. In fact, to understand the nature of neutron stars and dense
neutron-rich matter has been a long-standing and shared goal of both
astrophysics and nuclear physics communities LRP2015 ; NuPECC .

Among the main challenges, one important task is to understand the structure,
content and formation mechanism of neutron stars. People have imagined that
neutron stars have several layers with distinct features from the atmosphere,
envelope, crust to the outer and inner core in an analogy to the structure of
earth. The outer core of neutron stars is speculated to be some kind of fluid
consisting of neutron, protons, electrons, muons, hyperons, etc. As the
density of this fluid decreases towards the crust, the system is expected to
enter the so-called spinodal region where the matter (either pure nucleonic
matter of neutrons and protons Siemens83  or neutron star matter Lat00 ) is
dynamically unstable against the growth of small density fluctuations when the
temperature is sufficiently low. Consequently, bubbles/voids or
droplets/clusters are expected to develop depending on the trajectory of the
system before entering the spinodal region, marking a transition between the
uniform core and the inhomogeneous crust. This transition is expected to
happen in the inner crust and the exact transition density has been found to
depend sensitively on the characteristics of nuclear symmetry energy encoding
the energy necessary to make nuclear matter more neutron rich, see, e.g.,
Refs. Li19 ; JXu ; Newton12 . Very interestingly, because of the competition
between the surface tension and long-range Coulomb force, the bubbles and
nuclear clusters may have various shapes: from the spherical void near the
core, through the cylindrical void to the plates, cylinders and spheres of
nucleons towards the surface of neutron stars. Because of their similarity to
meatballs, spaghetti, lasagna, macaroni, and Swiss cheese, respectively, these
structures were collectively nicknamed nuclear pasta. For an earlier review of
the physics of nuclear pasta, see, Ref.Pethick . Since the pioneering works of
Ravenhall _et al._ Rav  and Hashimoto _et al._ Has , besides the above five
classical shapes, several other shapes, such as the gyroid, double-diamond Ken
, sponge-like Dorso  and parking-garage like Hor1  structures have been
predicted using various approaches, see, e.g., Refs. Wil85 ; Oya93 ; Lor93 ;
Che97 ; Wat00 ; Wat02 ; Wat03 ; Mar98 ; Kid00 ; Hor04 . Interestingly, similar
shapes have been found in nano materials and/or biological systems.

The various shapes of the nuclear voids and clusters correspond to the local
minima of potential energies separated by energy barriers. Often, the energy
minima are very close to each other. The realization of various shapes are
thus somewhat model dependent and the transition from one shape to another
depends sensitively on the nuclear physics inputs, such as the nuclear
equation of state especially its symmetry energy term Newton12 ; Newton09 ;
Bao ; Kaz20 ; Xia . Both static and dynamic approaches using various nuclear
interactions and assumptions about the composition of neutron stars have been
used in studying the formation, properties and phase transition of nuclear
pasta by many groups over the last three decades. The recent review in Ref.
Lopez1  by López, Dorso and Frank summarized their systematic investigations
into properties especially the morphological and thermodynamical phase
transitions of the nuclear pasta within a Classical Molecular Dynamics model.
Using several morphologic and thermodynamic tools, they characterized the
morphology of the emerging structures in neutron star crust by varying the
temperature, density, neutron to proton ratio with and without considering
electrons. They constructed the phase diagrams for both nucleonic matter and
neutron star matter. They also investigated the isospin (neutron to proton
ratio) dependence of the critical points and morphologic phase transition as
well as the symmetry energy of clustered matter. Bearing in mind the possible
model dependence, effects of some poorly known but necessary nuclear physics
inputs, finite size effects in simulating infinite matter using finite-sized
unit cells with periodic boundary condition and the lack of quantum effects,
their results obtained with the Classical Molecular Dynamics simulations are
certainly stimulating and useful for further exploring the interesting physics
of nuclear pasta.

As the nuclear pasta may actually exist inside the core of supernovae and the
crust of neutron stars, the physics of nuclear pasta is important for
understanding some astrophysical observations Cham ; Newton14 ; Chuck . For
example, supernova explosions, protoneutron star cooling mechanism and the
associated neutrino transport Hor04 ; Chuck ; Rog ; All ; Newton13 ; Sc ,
pulsar glitches Wat17 ; Josh , quadrupole deformation or formation of
mountains on neutron stars and the associated gravitational wave emission Cap
; Pet20 ; Bis ; And , frequencies of torsional oscillations of neutron stars
and the associated mechanism for generating quasi-periodic oscillations in the
tails of light curves of giant flares from soft gamma-ray repeaters Gea ; Sot
, all depend strongly on properties of neutron star crust especially whether
the nuclear pasta is considered or not. Moreover, the crustal properties are
also important for determining the r-mode stability window of super-fast
pulsars Wen ; Isaac . They are also critical for determine whether GW190814’s
second component of mass (2.5-2.67)M⊙ discovered by LIGO/VIRGO GW  very
recently is the lightest black hole or the most massive and fastest rotating
neutron star observed so far Most ; Zhang  as pointed out in Ref. rmode . Of
course, the main challenge is to find unique signatures of the nuclear pasta
as most of the astrophysical phenomena mentioned above have alternative
explanations without considering the nuclear pasta in the inner crust of
neutron stars. Hopefully, the review in Ref. Lopez1  by López, Dorso and Frank
will stimulate more work in this direction.

BALI is supported in part by the U.S. Department of Energy, Office of Science,
under Award Number DE-SC0013702 and the CUSTIPEN (China-U.S. Theory Institute
for Physics with Exotic Nuclei) under the US Department of Energy Grant No.
DE-SC0009971.

## References

  * (1) J. A. López, C. O. Dorso and G. Frank, Frontiers of Physics, 16 (2), 24 (2021).
  * (2) The National Academies Press, New Worlds, New Horizons in Astronomy and Astrophysics, 2011,
https://www.nap.edu/catalog/12951/new-worlds-new-horizons-in-astronomy-and-
astrophysics

  * (3) The National Academies Press, Nuclear Physics: Exploring the Heart of Matter, Report of the Committee on the Assessment of and Outlook for Nuclear Physics, 2012,
https://www.nap.edu/catalog/13438/nuclear-physics-exploring-the-heart-of-
matter

  * (4) The 2015 U.S. Long Range Plan for Nuclear Science, Reaching for the Horizon,
https://science.energy.gov/~/media/np/nsac/pdf/2015LRP/2015_LRPNS_091815.pdf

  * (5) The Nuclear Physics European Collaboration Committee (NuPECC) Long Range Plan 2017, Perspectives in Nuclear Physics,
http://www.esf.org/fileadmin/user_upload/esf/Nupecc-LRP2017.pdf.

  * (6) P.J. Siemens, Nature 305, 29 (1983).
  * (7) J.M. Lattimer, M. Prakash, Phys. Rep. 333, 121 (2000).
  * (8) B.A. Li, P.G. Krastev, D.H. Wen and N.B. Zhang, Eur. Phys. J. A 55, 39 (2019).
  * (9) J. Xu, L.W. Chen, B.A. Li, H.R. Ma, Astrophys. J. 697, 1549 (2009).
  * (10) W.G. Newton, M. Gearheart, B.A. Li, Astrophys. J. Supplement Series 204, 9 (2013).
  * (11) C.J. Pethick, D.G. Ravenhall, Ann. Rev. Nucl. Part. Sci. 45, 429 (1995).
  * (12) D. G. Ravenhall, C. J. Pethick and J. R. Wilson, Phys. Rev. Lett. 50, 2066 (1983).
  * (13) M. Hashimoto, H. Seki and M. Yamada, Prog. Theor. Phys. 71, 320 (1984).
  * (14) K.I. Nakazato, K. Oyamatsu and S. Yamada, Phys. Rev. Lett. 103 132501 (2009)
  * (15) C.O. Dorso, P.A. Giménez Molinelli and J.A. López, Phys. Rev. C86, 055805 (2012).
  * (16) C. J. Horowitz, D. K. Berry, M. E. Caplan, Greg Huber, A. S. Schneider, Phys. Rev. C 94, 055801 (2016).
  * (17) W.G. Newton, J.R. Stone, Phys. Rev. C 79, 055801 (2009).
  * (18) S. S. Bao and H. Shen, Phys. Rev. C 91, 015807 (2015).
  * (19) K. Oyamatsu, K. Iida, and H. Sotani, J. Phys.: Conf. Ser. 1643, 012059 (2020)
  * (20) C.J. Xia, T. Maruyama, N. Yasutake, T. Tatsumi and J.X. Zhang, arXiv:2012.01218
  * (21) R. D. Williams and S. E. Koonin, Nucl. Phys. A435, 844 (1985).
  * (22) K. Oyamatsu, Nucl. Phys. A561, 431 (1993).
  * (23) C. P. Lorenz, D. G. Ravenhall and C. J. Pethick, Phys.Rev. Lett. 70, 379 (1993).
  * (24) K. S. Cheng, C. C. Yao and Z. G. Dai, Phys. Rev. C55, 2092 (1997).
  * (25) G. Watanabe, K. Iida and K. Sato, Nucl. Phys. A676, 445 (2000).
  * (26) G. Watanabe, K. Sato, K. Yasuoka and T. Ebisuzaki, Phys. Rev. C66, 012801 (2002).
  * (27) G. Watanabe and K. Iida, Phys. Rev. C68, 045801 (2003).
  * (28) T. Maruyama, K. Niita, K. Oyamatsu, T. Maruyama, S. Chiba and A. Iwamoto, Phys. Rev. C57, 655 (1998).
  * (29) T. Kido, Toshiki Maruyama, K. Niita and S. Chiba, Nucl. Phys. A663-664, 877 (2000).
  * (30) C. J. Horowitz, M. A. Pérez-Garcia, J. Carriere, D. K. Berry, and J. Piekarewicz, Phys. Rev. C70, 065806 (2004).
  * (31) N. Chamel and P. Haensel, Living Rev. Relativ. 11, 10 (2008).
  * (32) W. G. Newton, J. Hooker, M. Gearheart, K. Murphy, D.H. Wen, F. Fattoyev and B.A. Li, The Euro. Phys. Jou. A, 50, 41 (2014).
  * (33) M. E. Caplan and C. J. Horowitz, Rev. Mod. Phys. 89, 041002 (2017).
  * (34) M. D. Alloy and D. P. Menezes, Phys. Rev. C 83, 035803 (2011).
  * (35) W. G. Newton, K. Murphy, J. Hooker and B.A. Li, The Astrophysical Journal Letters 779, L4 (2013).
  * (36) A. Roggero, J. Margueron, L.F. Roberts, and S. Reddy, Phys. Rev. C 97, 045804 (2018)
  * (37) B. Schuetrumpf, G. Martinez-Pinedo, and P.-G. Reinhard, Phys. Rev. C 101, 055804 (2020).
  * (38) G. Watanabe and C.J. Pethick, Phys. Rev. Lett. 119, 062701 (2017).
  * (39) J. Hooker, W.G. Newton and B.A. Li, Mon. Not. R. Astron. Soc., 449 (4): 3559 (2015).
  * (40) M.E. Caplan, A.S. Schneider, and C.J. Horowitz, Phys. Rev. Lett. 121, 132701 (2018).
  * (41) C.J. Pethick, Z.-W. Zhang, and D. N. Kobyakov, Phys. Rev. C 101, 055802 (2020).
  * (42) B. Biswas, R. Nandi, P. Char, and S. Bose, Phys. Rev. D 100, 044056 (2019).
  * (43) F. Gittins, N. Andersson, and J. P. Pereira, Phys. Rev. D 101, 103025 (2020).
  * (44) M. Gearheart, W. G. Newton, J. Hooker and B.A. Li, Mon. Not. R. Astron. Soc., 418, 2343 (2011)
  * (45) H. Sotani, K. Iida, and K. Oyamatsu, Mon. Not. R. Astron. Soc. 489, 3022 (2019).
  * (46) D.H. Wen, W.G. Newton and B.A. Li, Phys. Rev. C 85, 025801 (2012).
  * (47) Isaac Vidaña, Physical Review C 85, 045808 (2012).
  * (48) R. Abbott et al. The Astrophysical Journal Letters 896, L44 (2020).
  * (49) E. R. Most, L. Jens Papenfort, L. R. Weih and L. Rezzolla, Mon. Not. R. Astron. Soc. Letters 499, L82 (2020).
  * (50) N.B. Zhang and B.A. Li, The Astrophysical Journal 902, 38 (2020).
  * (51) X. Zhou, A. Li and B.A. Li, arXiv:2011.11934


# Counterfactual Explainer Framework for Deep Reinforcement Learning Models
Using Policy Distillation

Amir Samadi1, Konstantinos Koufos1, Kurt Debattista1 and Mehrdad
Dianati${}^{1,2},\emph{Senior Member, IEEE}$ 1The authors are with the Warwick
Manufacturing Group (WMG), The University of Warwick, Coventry CV4 7AL.
(e-mail:amir.samadi, konstantinos.koufos, k.debattista and
m.dianati@warwick.ac.uk). This research is sponsored by Centre for Doctoral
Training to Advance the Deployment of Future Mobility Technologies (CDT FMT)
at the University of Warwick.2The author is also with the School of
Electronics, Electrical Engineering and Computer Science (EEECS), Queen’s
University of Belfast (e-mail: m.dianati@qub.ac.uk).

###### Abstract

Deep Reinforcement Learning (DRL) has demonstrated promising capability in
solving complex control problems. However, DRL applications in safety-critical
systems are hindered by the inherent lack of robust verification techniques to
assure their performance in such applications. One of the key requirements of
the verification process is the development of effective techniques to explain
the system functionality, providing why the system produces specific results
in given circumstances. Recently, interpretation methods based on the
Counterfactual (CF) explanation approach have been proposed to address the
problem of explanation in DRLs. This paper proposes a novel CF explanation
framework to interpret the decisions made by a black-box DRL. To evaluate the
efficacy of the proposed explanation framework, we carried out several
experiments in the domains of automated driving systems and Atari Pong game.
Our analysis demonstrates that the proposed framework generates plausible and
meaningful explanations for various decisions made by deep underlying DRLs.
Source codes are available at: https://github.com/Amir-Samadi/Counterfactual-
Explanation.

##  I Introduction

Recent breakthroughs in solving various challenging problems by Deep
Reinforcement Learning (DRL) models have been demonstrated in the literature
[1, 2]. However, deploying DRL models to safety-critical applications, such as
automated driving systems [3] and healthcare [4], is impeded due to lack of
ability to ensure the trustworthiness of such models [5]. To address this
challenge, the concept of Interpretable Artificial Intelligence (IAI) has
recently emerged as a promising solution [6, 7]. Within the realm of IAI,
interpretability refers to the endeavour of generating human-understandable
explanations for the functionality of black-box AI systems, benefiting various
stakeholders, including end-users, engineers and legal authorities. To this
end, in this paper, we investigate the use of Counterfactual (CF) reasoning, a
promising IAI paradigm, for explaining the behaviour of DRL-based models
particularly in the context of Automated Driving System (ADS).

The literature offers various CF reasoning methodologies to interpret the
decisions made by black-box Deep Learning (DL) models. Pioneering work,
Wachter et al. [8] introduced CF explanation as the minimum changes required
in the input of the black-box model to alter a given output to its complement,
highlighting the most crucial features in the original input-output pairs. In
principle, finding such necessary and minimum changes shapes the CF
explanation as an optimisation problem. A range of optimisation methods have
been proposed for CF explanations, including gradient-based optimisation
(WachterCF) [8], genetic algorithms [9, 10, 11], game theory [12, 13] and
monte-carlo methods [14]. While these techniques have demonstrated promise in
low-dimensional-input systems, such as tabular data, they are ineffective in
systems with high-dimensional inputs, as in computer vision applications where
high-resolution images must be processed [15](see section II-B for further
discussion).

To overcome this challenge, deep generative CF methods [15, 16, 17, 18] have
been proposed, which generate CF explanations directly from specified input
attributes using Generative Adversarial Networks (GANs). However, it’s worth
noting that these methods cannot guarantee the plausibility and meaningfulness
of the generated CFs; This stems from the fact that GANs generate synthetic
data rather than modifying the original data specifications, such as the
localisation of Participant Vehicles (PVs) in the context of ADSs [19]. The
term “plausibility” underscores the necessity of generated CF examples being
both legitimate and actionable. For instance, in ADS, a fundamental rule
dictates that all PVs must remain on the appropriate side of the road, making
a CF state with an off-road vehicle implausible. On the other hand,
“meaningfulness” implies that any alternation should be reasonable and
sensible. For example, a minor change in the color of a leading PV is not
comprehensible.

Generating high-dimensional CF examples poses a greater challenge in the realm
of DRL agents due to the temporal dependencies introduced by processing a
history length of observations at every time step [20]. For instance, consider
a DRL-based ADS which receives the latest four stacked states observations of
the driving environment in each time step to provide an action. To generate a
CF explanation for such a DRL agent, firstly, four CF states must be provided,
which results in multiplying the high-dimensional input by the history
observation length. Secondly, all four generated CF states should demonstrate
internal consistency to represent temporal dependencies accurately. This means
that in a highway driving scenario, the locations of vehicles in subsequent CF
state images must mirror realistic driving manoeuvres, showcasing
spatiotemporal correlations while adhering to speed limits and road geometry.
Due to these challenges, to the best of our knowledge, there are currently
only two studies in the literature designed to generate CF examples for a
given DRL agent handling high-dimensional visual data. The first, by Olson et
al. [19], relies on an intricate combination of deep generative models to
provide CF states for altering the DRL’s action. However, the user cannot
specify the desirable alternate action of the DRL agent, which limits its
applicability. Besides, Olson’s model fails to generate CF state in complex
reinforcement learning environments such as ADS, proved in section IV. The
second model, developed by Huber et al. [20] and known as GANterfactual-RL,
leverages a well-establised image-to-image translation model, namely StarGAN
[21], to transfer the original state-action pairs to CF states associated with
complement actions. Despite shedding some light on the behaviour of the DRL
agents, both models often produce CFs that are implausible or lack meaningful
interpretation (see sections II-B and IV). This limitation stems from the
reliance of the utilized deep generative models solely on the observed states,
lacking further direction on how to narrow down the solution space. This can
result in suboptimal CF solutions, especially within the expansive visual
domain. For instance, they fail to consider the inherent environmental
limitations that would filter out impractical CF states, like generating
vehicles off-road. Consequently, the existing literature lacks a proficient
model capable of generating CF examples that are plausible and offer
meaningful insights for DRL agents processing visual input states.

The work in this paper extends the existing literature by proposing an
explanation framework (hereinafter called PDCF) that utilises policy
distillation to address the challenge of generating meaningful and plausible
CFs for DRL-based models with high-dimensional input. The proposed PDCF
framework comprises (i) training a DRL-based control system with a high-
dimensional input state, such as bird’s eye view image of a highway, hereafter
referred to as the teacher model, (ii) distilling the teacher’s learned
policies to a student network with a low-dimensional input in the form of
human-understandable features, such as the objectives list of the PVs in the
highway (iii) providing candidate CFs for the student network, and finally
(iv) verifying whether the teacher and student networks provide similar
outputs for the candidate CFs. This approach alleviates the need to solve
optimisation problems in high-dimensional input spaces. Furthermore, finding
the CF explanations by altering human-understandable low-dimensional features,
such as the positions of vehicles (belonging to the objective list
attributes), ensures the meaningfulness and plausibility of the generated CFs.
To the best of our knowledge, the proposed explanation framework is the first
method that guarantees the plausibility of generated CF explanations (see
section II-B).

The main contributions of this paper are also summarised in the list below:

  1. 1.

A novel CF explainer framework named PDCF is proposed, which can be applied to
any DRL model, using policy distillation.

  2. 2.

We discuss how stakeholders, such as end-users and engineers can utilise the
CF explanations. In particular, using the proposed explanation framework, we
extract human-understandable if-then rules from black-box DRL models to
explain the learned policies.

  3. 3.

A comprehensive analysis of the proposed framework is carried out in the
context of three DRL-based systems, namely, the deep Q-network (DQN) [2], the
asynchronous actor-critic (A2C) algorithm [22] and the proximal policy
optimization (PPO) [23] acting in Atari Pong game [24], highway and roundabout
driving [25].

The rest of this paper is organised as follows. We summarise the related works
in the field of counterfactual explanation and review the policy distillation
method in Section II. Section III provides technical aspects of the proposed
method. The conducted experiments, further discussion and research findings
are described in Sections IV and V and VI, respectively.

##  II Background and Related Work

This section presents a brief overview of the background and related work in
the realm of interpretability, focusing on the use of counterfactual (CF)
explanations as a promising approach to achieve interpretability. In addition,
we review the policy distillation method as the key technique employed in this
study, which simplifies Deep Reinforcement Learning (DRL) models, thereby
enabling the generation of CF explanation.

###  II-A Interpretability

Intending to open the AI black-box, we have recently witnessed considerable
research on interpretability methods. The family of interpretable models can
be divided into two main streams: Intrinsic and post-hoc techniques. While
intrinsic interpretable algorithms refer to techniques that could generate
explanations during the training phase [26], post-hoc methods employ auxiliary
models to extract an explanation from already trained models [27]. Intrinsic
models are also known as transparent models, implying that several models
belonging to this category, such as decision trees, linear models, logistic
models, and the k-nearest neighbours algorithm, are intrinsically
interpretable due to their straightforward structure.

The post-hoc interpretable algorithms can be categorised as model-specific or
model-agnostic [7]. Model-specific interpretability approaches consider the
model as a ”grey box” in which model internals, such as weights and features,
are accessible, and explanations can be produced around these features and
weights. On the other hand, model-agnostic methods treat the model as a
”black-box” whose input and output can only be probed. In this manner, a
surrogate model is widely used to learn a behaviour similar to that of the
black-box model, and this surrogate (or proxy) model is then interpreted. The
CF paradigms, including our study, are mainly classified as model-agnostic.

Figure 1: Overview of the CF explanation method. Conventional CF methods could
be able to generate CFs for (low-dimensional) tabular data; However, for
(high-dimensional) image input data, they generate noise (adversarial attack
examples). The boxes outlined in solid and dashed lines indicate fixed model
parameters and training model parameters, respectively.

###  II-B Counterfactual explanations

Counterfactual (CF) explanations, as described in the study in [8], attempt to
interpret why a model $M_{\omega}$ classifies/regresses an input $x$ into
class/value $y\in{\mathbb{R}}$ rather than a counter class/value
$y^{\prime}\in\mathbb{R}\backslash y$ (see the upper row in Fig. 1). Towards
this, a CF example called $x^{\prime}$ that is near $x$ is generated, but the
model $M_{\omega}$ classifies/regresses it as $y^{\prime}$. The main challenge
lies in generating a CF $x^{\prime}$ in the input-dimension space by
calculating the gradient of an assumed loss function between the
classes/values $y$ and $y^{\prime}$ in the output-dimension space. That is why
the gradient-based CF explanation method was initially designed for low-
dimensional input space problems, such as credit scoring tasks [8]. Applying
the same method to high-dimensional input space, such as a high-resolution
image, leads to the problem of generating adversarial attacks [28, 29, 30, 31,
32], i.e., imperceptible alternations to the input image that deceive the
classifier.

Although the main idea behind adversarial and CF examples are the same, they
are applied in two distinct ways. While the input changes in the CF method are
meaningful and plausible, they are not noticeable in the adversarial attack
and are typically perceived only as additive noise at the output (see the
lower row in Fig.1). The first attempts to provide CF explanations for high-
dimensional input models were to compare an image $x$ to other _real
instances_ $x^{\prime}$ labelled as $y^{\prime}$ to explain a decision [33,
34, 35], rather than varying the original image’s ($x$) features. Afterwards,
generative model-based CF methods were proposed to shift the model’s
classification by providing a CF image $x^{\prime}$ similar to the original
image $x$, but slightly different in a few high-level attributes. Progressive
Exaggeration (PE)[17], DiVE [16], STEEX [15] and Counterfactual State (CFS)
[19] are remarkable studies that used the generative model to generate CFs.
However, as has been emphasised in [19], the generative models cannot
guarantee that the generated image is meaningful, as they produce a synthetic
image rather than modifying the original image specifications. To demonstrate
this in practice, we have implemented the two state-of-the-art generative-CF
explainers proposed by Olson et al. [19] and Huber et al. [20] to generate CFs
for the Atari Pong and Highway environments. Fig. 2 illustrates implausible CF
explanations (right frames) generated from the original input images (left
frames), generating two paddles for the right player or locating participant
vehicles on the offside of the road.

In our approach, we propose a framework that uses the original data and
applies sparse changes through the gradient-based method to ensure that all
generated CFs are meaningful and plausible. This is made possible by using
policy distillation and transferring the high-dimensional input space to a
low-dimensional input space in order to mitigate the problem of adversarial
attack.

Figure 2: Generated CF examples for the Pong and Highway environments by the
Generative-CF explainer method proposed by Olson et. al [19] and Huber et. al
[20], respectively. For a given initial state (left images), the methods
generated implausible CF states on the right (right images). In the Pong
environment, the CF state contains two green paddles for the player, and in
the Highway environment, two participant vehicles are generated offside of the
road, which are out of the rules.

###  II-C Policy Distillation

Some RL methods such as PPO [23] and A2C [22] employ Policy Gradient (PG)
techniques that directly optimise the policy by taking the gradient of the
expected returns. Another RL method is the Q-Learning algorithm that learns an
intermediate quality function, namely the Q-function, for all the state-action
pairs $(s,a)$ [2]. Finally, there are methods that lie between policy
optimisation and Q-learning, such as DDPG [36], TD3 [37] and SAC [38]. Due to
the high dimensionalities of the state and/or action spaces in complex
environments, deep neural networks (DNNs) are used in RL algorithms to map
states to values or state-action pairs to Q-values, known as DRL approaches.
While advanced end-to-end DRL algorithms, which map high-dimensional states to
actions, offer superior performance, their deployment is impeded by the
challenge of interpreting their black-box policies. Interpreting such policies
is a vital step to establish trust and ensure effective debugging. To address
this challenge, we distil the teacher’s policy to a shallow neural network
with a lower input dimension to interpret the black-box policy while avoiding
the adversarial attack illustrated in Fig. 1. Noteworthy, any DRL algorithm
can be used in the proposed framework.

In machine learning, Knowledge Distillation (KD) is the transfer of knowledge
from a complex model (teacher) to a simpler model (student). It has been
proposed in [39, 40, 41], and in the context of DRL is often called policy
distillation[42]. In order to transfer knowledge, the teacher’s soft labels
(softmax of Q-values in the case of DQN) are passed onto the student network
using a KD loss function during supervised learning (see Fig. 3). The Mean
Square Error (MSE), the Negative Log likelihood (NLL), and the Kullback-
Leibler divergence (KL) [43] are the most popular KD loss functions. Also,
some studies in the literature use a linear combination of KL divergence and
NLL loss functions to shape the KD loss function [44]. In the literature, the
policy distillation paradigm has been used in various applications, including
improved multitask learning [45, 46], deep neural network size compression
[42, 47], improved learning generalisation [48] and policy pruning [49].

Figure 3: Overview of the policy distillation method. The boxes outlined in
solid and dashed lines indicate fixed model parameters and training model
parameters, respectively

##  III Generating CFs Using Policy Distillation

This section describes the proposed framework for generating CF explanations
for the actions of a DRL agent using a policy distillation approach
implemented into four steps. The aim is to find CFs that can correctly
interpret the actions of the trained, black box DRL agent.

The first step (see Step 1 in Fig. 4 for a block diagram) is referred to as
teacher learning, in which the agent learns to act using a high-dimensional
input space to describe the state of its environment. Considering that it is
intractable to find the CFs in high-dimensional input space, in the second
step (see Step 2 in Fig. 4), the agent’s policy is distilled to a
straightforward neural network called the student network that employs low-
dimensional inputs. The outcome of the second step is that the student network
learns to imitate the teacher’s policy within the same environment upon the
reception of low-dimensional inputs instead of the teacher’s high-dimensional
input, e.g., state object lists instead of raw state images. Fortunately, this
is possible because most environments provide both high- and low-dimensional
states, e.g., in the case of automated driving systems (ADS), a bird’s eye
view (BEV) image of a highway can be seen as extensive information, while the
positions and velocities of vehicles in the scene can be considered as the
human-understandable information.

In the third step of the proposed framework (see Step 3 in Fig. 4), the
potential CFs are generated by calculating gradients of the output (actions)
of the student network with respect to the low-dimensional input. In other
words, the minimum required changes of the input to obtain the desired
system’s output are computed. Finally, in the last step (see Step 4 in Fig. 4
step 4), we evaluate whether the obtained candidate CFs can alter the DRL
agent’s output (teacher network). In the subsequent paragraphs, each step is
detailed in a separate subsection.

Figure 4: The proposed CF explainer framework in a highway driving
environment. The states $s_{t}$ and $\sigma_{t}$ are the states of the
environment at time $t$ in the high- and low-dimensional representation,
respectively, hence $s_{t}\in S$ and $\sigma_{t}\in\Sigma$. Also, $r_{t}\in R$
denotes the received reward at time $t$ which is used to train the teacher
network. The boxes outlined in solid and dashed lines indicate fixed model
parameters and training model parameters, respectively.

###  III-A Teacher Model Training

In the first step of the policy distillation framework, we train a DRL model
to learn how to act within its environment. We refer to the trained model as
the teacher model $M^{T}$ since, in the next step, a shallow neural network
model (the student model $M^{S}$) will also be trained to imitate the
teacher’s policy based on the labels that the teacher’s model provides.

In RL algorithms, an agent learns to act by maximising its expected cumulative
future reward obtained through its actions within an environment. The agent
first observes the current state of the environment and the gained reward from
its previous action. Based on that it takes another action, and the
environment returns its new state and reward to the agent. Through this
iterative process, the agent learns an optimised policy, $\pi$, of mapping
states onto actions that maximises the cumulative received reward over a time
horizon.

Mathematically, an RL problem is formulated as a Markov Decision Process (MDP)
of $(S,A,T,R,\gamma)$, where $S$ and $A$ stand for the states of the
environment and the available actions, respectively, $T$ represents the
learned policy as a conditional probability distribution function, i.e.
${T:S\times A\times S\rightarrow[0,1]}$ assigns the transition probabilities
from the current state $s_{t}\in S$ to the next state $s_{t+1}\in S$ by taking
action $a_{t}\in A$, and finally, $R$ and $\gamma\in[0,1]$ are the
environment’s reward and the discount factor, respectively. The discount
factor sets the agent’s preference for acquiring long-term rewards ($\gamma$
nearer to unity) over immediate rewards ($\gamma$ nearer to zero). Maximising
the expected discounted future reward leads to an estimation of the optimal
policy $\pi$.

###  III-B Student Model Training

Policy distillation is a technique of policy transferring from a teacher model
$M^{T}$ to a student model $M^{S}$ in order to learn to imitate the teacher’s
policy $\pi^{T}$ [42]. To this end, the soft labels of the teacher network
(applying a softmax function to the last network layer’s weighted sums) are
passed to the student model through the distillation loss function. Three loss
functions, including the MSE, $\mathcal{L}_{MSE}$, the NLL,
$\mathcal{L}_{NLL}$ and the KL divergence, $\mathcal{L}_{KL}$, are commonly
utilised to measure the distillation loss given the hyper-parameters $\theta$
of the student network as follows:

$\begin{split}&\mathcal{L}_{MSE}(\mathcal{D},\theta)=\sum_{i=1}^{N}\|v_{i}^{T}-v_{i}^{S}\|_{2}^{2},\\\
&\mathcal{L}_{NLL}(\mathcal{D},\theta)=-\sum_{i=1}^{N}\log
P(a_{i}^{S}=a_{i,best}^{T}|\sigma_{i},\theta),\\\
&\mathcal{L}_{KL}(\mathcal{D},\theta)=\sum_{i=1}^{N}\mathrm{softmax}(\frac{v_{i}^{T}}{\tau})\ln\frac{\mathrm{softmax}(\frac{v_{i}^{T}}{\tau})}{\mathrm{softmax}(v_{i}^{S})}\\\
\end{split}$ (1)

With reference to equation (1), the proposed framework provides to the student
network an $N$-pair dataset, $\mathcal{D}$, of low-dimensional object lists
(converted from the high-dimensional state offered by the environment)
alongside the labels generated by the teacher model. Let us denote by
$\sigma_{i}$ the low-dimensional state associated with the state $s_{i}$ in
the teacher network at the $i$-th time step. Furthermore, we employed
$v_{i}^{T}$ notation to introduce the soft action of the teacher network,
which will be used to train the student model. Therefore, the dataset provided
to the student network can be denoted as
$\mathcal{D}=\\{(\sigma_{i},v_{i}^{T})\\}^{N}_{i=1}$. The parameter
$v_{i}^{S}$ in equation (1) is the soft action generated by the student model,
$\tau>0$ is the temperature hyper-parameter and $P(\cdot|\cdot)$ denotes
conditional probability distribution function. Finally, the parameter
$a_{i}^{S}$ is the action taken by the student network and
${a_{i,best}^{T}=\arg\max_{i}v_{i}^{T}}$ is the optimal action of the teacher
network at the same time step. Note that the actions of the teacher and the
student networks belong to the same action space, i.e., $a^{T}\in A$ and
$a^{S}\in A$.

Considering the loss functions in equation (1), it is intuitive to deduce that
selecting the MSE loss function maintains the action values $v^{S}$ generated
by the student model close to the teacher’s action values $v^{T}$ by
minimising the Euclidean distance between the vectors $v^{S}$ and $v^{T}$. The
NLL loss function maintains the same indices for the maximum element of the
vectors $v^{S}$ and $v^{T}$. Finally, the most frequently employed knowledge
distillation (KD) loss function is the KL divergence loss between the softened
probability distributions of the two models. Intuitively, swinging the
hyperparameter $\tau$ to be greater/less than one results in
softening/sharpening the models’ output probability distribution that leads to
the KL divergence loss function attempting to match the models’ logits/labels.
In this paper, we will conduct a comparison among the aforementioned loss
functions in Section IV.

###  III-C Counterfactual Explanation Generation

In this step, the proposed framework generates a CF example for the student
model that is able to alter its action for a given state. In the next and last
step, we will also have to ensure that the provided CF can alter the teacher
model’s decision. In the student network, $\Sigma\subseteq\mathbb{R}$
represents the input feature space and the actions $A$ as the output space,
which can be either continuous, $A\subseteq\mathbb{R}$, or discrete
$A\in\\{1,\ldots,K\\}$. The student model is a mapping from the input vector
state $\sigma\in\Sigma$ to the output action $a^{S}\in A$, i.e.,
$M^{S}_{\theta}:\Sigma\mapsto A$. Given an input state $\sigma$ leading to the
action ${a^{S}=M^{S}_{\theta}(\sigma)}$, a CF example is a modified state
$\tilde{\sigma}\neq\sigma$ that leads to another action
${\tilde{a}^{S}=M^{S}_{\theta}(\tilde{\sigma})\neq M^{S}_{\theta}(\sigma)}$,
where ${\tilde{a}^{S}\in A\backslash a^{S}}$. In the case of continuous output
(action), the validity of CF state $\tilde{\sigma}$ should satisfy the
boundary condition of
$|{M^{S}_{\theta}(\sigma)-M^{S}_{\theta}(\tilde{\sigma})}|>\delta$ where
$\delta\in\mathbb{R}\backslash\\{0\\}$ [50].

Intuitively, there could exist several CFs $\tilde{\sigma}$ for a given input
state $\sigma$. To select among them, various metrics are considered to be
maximised or minimised in the literature, including CF and original state
distance, sparsity and plausibility of CFs. The most common metric is the
distance between the CF and the original state, i.e., distance functions. In
this paper, we will employ the Wachter method, which uses the Euclidean
distance between the states $\sigma$ and $\tilde{\sigma}$ [8]. Let us denote
the selected (optimal) CF by $\tilde{\sigma}^{*}$. The Wachter method uses the
following loss function to find $\tilde{\sigma}^{*}$:

$\mathcal{L}_{CF}(\tilde{\sigma},\lambda;\sigma,a^{S},M^{S}_{\theta})=\lambda(M^{S}_{\theta}(\tilde{\sigma})-\tilde{a}^{S})^{2}+\textnormal{dist}(\sigma,\tilde{\sigma})$
(2)

with adjustable variables the CF $\tilde{\sigma}$ and the hyperparameter
$\lambda$, and fixed inputs the original output $a^{S}$ and state $\sigma$,
and the model $M^{S}_{\theta}$. The function
$\textnormal{dist}(\sigma,\tilde{\sigma})$ stands for Euclidean distance
between the states $\sigma$ and $\tilde{\sigma}$. Since we prefer to apply
sparse changes to the original state, we calculate the closest possible CF
$\tilde{\sigma}$ to the original state $\sigma$ such that
$M^{S}_{\theta}(\tilde{\sigma})$ is equal to the new output $\tilde{a}^{S}$.
To this end, the optimal CF $\tilde{\sigma}^{*}$ could be identified by
minimising the loss function in equation (2) with respect to the
hyperparameter $\lambda$. Algorithmically, the $\lambda$ is maximised under an
iterative procedure, such that we reach to the optimal CF $\tilde{\sigma}^{*}$
associated with the minimum Euclidean distance between the original and CF
states, i.e.
${\textnormal{dist}(\sigma,\tilde{\sigma}^{*})=\min_{\tilde{\sigma}}\textnormal{dist}(\sigma,\tilde{\sigma})}$.
This can be written as:

$\tilde{\sigma}^{*}=\arg\min_{\tilde{\sigma}}\max_{\lambda}\lambda(M^{S}_{\theta}(\tilde{\sigma})-\tilde{a}^{S})^{2}+\textnormal{dist}(\sigma,\tilde{\sigma}).$
(3)

###  III-D Validation

In the final step, the proposed framework checks whether the candidate CF can
alter the teacher’s model output as well as student model’s output in the
previous step. To do that, the states $\tilde{\sigma}^{*}$ and $\sigma$ from
the student input space (low-dimensional) are first transferred to the teacher
input space (high-dimensional). Let us denote them by $\tilde{s}^{*}$ and $s$,
respectively. Next, the framework validates the candidate CF by feeding the
teacher model with both states. While, according to the definition of CF
explanation, the inequality of the teacher’s model output for both the
original and CF states indicates the validity of the generated CF, this
framework aims to assess the performance of policy distillation by conducting
an additional check. Specifically, the validity of a given CF is determined by
verifying if both the teacher and student models produce the same action for
that CF. The two possible outcomes of the validation process are

$\begin{cases}M^{T}({s})\neq
M^{T}(\tilde{s}^{*})=M^{S}(\tilde{\sigma}^{*})&{validity}=1\\\
\textnormal{else}&{validity}=0\end{cases}$ (4)

where ${validity}=1$ indicates that a CF example for the teacher model has
been successfully found.

##  IV Experiments and Evaluations

This section conducts a comprehensive evaluation of the proposed
counterfactual (CF)-based explainer method for three black-box DRL agents:
DQN[2], A2C[22] and PPO[23]. These are used in two applications: 1)
automated/autonomous driving in highway and roundabout driving scenarios [25]
and 2) Atari Pong game [24]. Initially, we outline the implementation details
and descriptions of the RL environments employed in the experiments.
Subsequently, a comparative analysis of the rewards obtained by both the
teacher and student models in these environments demonstrates the successful
distillation of the teacher’s policy into the student model. Finally, we
provide a comprehensive quantitative and qualitative comparison of the CFs
generated by our proposed PDCF framework against state-of-the-art models,
including Olson et al. [19] and Huber et al. [20].

###  IV-A Implementation Details

In our experiments, we employed the DQN, A2C and PPO agents implemented from
the stable-baselines3 [51] PyTorch package to serve as the teacher models. For
training the student models, we utilised the Adam optimiser with a learning
rate of $10^{-3}$. The KL loss coefficient $\tau$ (refer to Eq. (1)) was
determined heuristically and set to $0.01$ for the DQN model, and $0.7$ for
the PPO and A2C models. To ensure the robustness of the results, we conducted
each experiment five times with randomly generated seeds, and subsequently
presented the mean values of the extracted metrics. The experiments are
executed utilising the PyTorch framework, harnessing the computational power
of an NVIDIA RTX-3090 GPU.

###  IV-B Automated Driving and Atari Pong Game Environments

The PDCF framework is evaluated in the domains of ADS and the Atari Pong game.
The following paragraphs provide descriptions of the Highway and Roundabout
environments, the two driving scenarios implemented as the ADS application,
and Atari pong game. In the automated driving scenarios, a DRL agent controls
the EGO vehicle which is:

  1. 1.

traveling on a multi-lane highway with left-to-right traffic flow, in the
highway environment (Fig. 5)

  2. 2.

approaching a roundabout with anticlockwise flowing traffic, in the roundabout
environment (Fig. 6).

The agent’s objective is to drive at the fastest possible pace while avoiding
collisions with Participant Vehicles (PV). Therefore, the corresponding reward
for taking action $a$ at state $s$ can be shaped as follows:

$R(s,a)=\frac{u-u_{min}}{u_{max}-u_{min}}-\beta\times collision,$ (5)

where $R(.,.)$ is the reward function of the environment, $\beta$ is a
regulation coefficient, and $u,u_{min}$ and $u_{max}$ are the EGO vehicle’s
current, minimum and maximum velocities, respectively. In highway and
roundabout driving, there are five actions in total comprising “Left” lane
change, “Right” lane change, and keep the lane with “Faster”, “Idle” or
“Slower” velocity. According to the proposed framework, the DRL teacher model
predicts the action by feeding four stacked-state-image observations, while
the student network imitates the teacher’s policy by receiving object lists,
including the vertical/lateral positions and velocities of each vehicle in the
environment.

Using Gym Atari Pong game[24], the DRL agent controls the right paddle to play
Pong against the built-in opponent of the environment (as shown in Fig. 7).
The agent’s objective is to deflect the ball away from the right side of the
game board (right goal) into the opponent’s left goal. The available actions
for the agent are “NOOP”, “Up”, and “Down”, which refer to the paddle being
moved without change, up, and down, respectively. The agent receives one
positive/negative point each time the ball passes the opponent’s/agents’s
goal. Therefore, after 21 rounds of play, the accumulated reward for the agent
is $R\in\\{-21,\dots,21\\}$. According to the proposed framework, the teacher
network predicts the action based on the four stacked-state-image
observations, whereas the student network imitates the teacher’s policy by
receiving only the lateral position of the paddles as well as the vertical and
lateral position of the ball as the object lists.

TABLE I: Teacher and Student model architectures for the DQN, PPO and A2C
models.

Teacher Model Student Model Layers (Filter, Kernel, Stride) Output (Highway &
Roundabout) Output (Atari Pong) Layers Output (Highway & Roundabout) Output
(Atari Pong) Input Image - 128$\times$64$\times$4 84$\times$84$\times$4 Input
Features 24 16 ReLu(Conv2D) $($4,8,4$)$ 31$\times$15$\times$4
20$\times$20$\times$4 ReLu(Linear) 256 256 ReLu(Conv2D) $($32,4,2$)$
14$\times$6$\times$32 9$\times$9$\times$32 ReLu(Linear) 256 256 ReLu(Conv2D)
$($64,3,1$)$ 12$\times$4$\times$64 7$\times$7$\times$64 Linear 5 12
Relu(Linear) 3072 512 512 - - - Linear 512 5 3 - - - Total parameters 88686246
highway:23674534,47348042 pong: Total parameters 72453

TABLE II: Comparison of performance of DQN, PPO, and A2C teachers and their
students with identical network architectures for two tasks of highway and
roundabout driving.

DRL Model loss func Highway Roundabout Atari Pong $T_{s}$ $S_{s}$ RS% $T_{s}$
$S_{s}$ RS% $T_{s}$ $S_{s}$ RS% A2C KL 13.61 13.17 96.80 9.06 9.00 99.33 18.70
16.06 85.87 NLL 13.08 96.14 8.90 98.23 14.78 79.03 MSE 13.06 96.03 8.81 97.24
17.88 95.61 DQN KL 14.34 13.25 92.36 9.31 8.30 89.15 19.26 21.00 109.03 NLL
13.55 94.45 8.52 91.51 10.11 52.49 MSE 13.47 93.93 8.71 93.55 5.55 28.81 PPO
KL 12.75 12.62 98.94 9.23 8.79 95.23 19.16 17.76 92.69 NLL 12.72 99.76 7.63
82.66 17.59 91.80 MSE 12.96 101.64 7.90 85.59 17.76 92.69

TABLE III: Evaluation of SAFE-RL in comparison to baseline models. The desired
directions of metrics improvement are indicated by arrows ($\downarrow$ low or
high $\uparrow$). The most favorable results are highlighted in bold, and the
second-bests are underscored. LPIPS, proximity, sparsity, and validity metrics
are represented by LPS, Prx, Spr, and Vld, respectively. The methods that were
unable to converge are denoted by NC.

DRL Model CF Method Highway Roundabout Pong $\uparrow$Vld% $\downarrow$Spr%
$\downarrow$Prx $\downarrow$FID $\downarrow$LPS $\uparrow$IS $\uparrow$Vld%
$\downarrow$Spr% $\downarrow$Prx $\downarrow$FID $\downarrow$LPS $\uparrow$IS
$\uparrow$Vld% $\downarrow$Spr% $\downarrow$Prx $\downarrow$FID
$\downarrow$LPS $\uparrow$IS A2C Olson [19] 88.38 89.28 2.413 0.088 0.164
1.112 NC NC NC NC NC NC 95.64 92.76 5.560 5.065 0.417 1.230 Huber [20] 78.05
59.25 3.636 1.447 0.130 1.290 28.08 100 0.073 4.778 0.362 1.964 67.87 80.09
1.371 0.266 0.120 1.390 PDCF 97.38 1.51 1.067 0.002 0.270 1.021 68.00 0.35
0.253 0.003 0.253 1.364 76.00 0.15 0.140 7e-6 0.103 1.072 DQN Olson [19] 99.82
99.27 12.99 2.049 0.405 1.486 NC NC NC NC NC NC 91.81 95.58 4.369 0.178 0.163
1.157 Huber [20] 73.88 79.50 2.739 1.353 0.127 1.205 81.65 100 0.014 3.505
0.150 1.981 66.25 56.80 1.012 0.181 0.087 1.443 PDCF 87.38 1.78 1.255 0.002
0.258 1.022 60.63 0.51 0.365 7e-4 0.271 1.481 89.27 0.11 0.107 1e-5 0.094 1.07
PPO Olson [19] 80.59 88.60 2.331 0.866 0.159 1.136 NC NC NC NC NC NC 71.28
91.97 3.978 0.114 0.164 1.338 Huber [20] 75.19 73.38 2.426 1.466 0.122 1.262
73.14 100 0.006 2.150 0.103 1.715 85.04 82.33 1.409 0.130 0.136 1.382 PDCF
97.38 1.94 1.369 0.002 0.266 1.024 86.75 0.43 0.312 7e-3 0.255 1.606 91.77
0.14 0.131 8e-6 0.102 1.072

TABLE IV: Evaluating the impact of $\lambda$ on the quality of generated CF
states using CF metrics. Desired directions of improvement are indicated by
arrows ($\downarrow$ low or high $\uparrow$). Optimal results are highlighted
in bold. Validity, sparsity and proximity metrics are represented by Vld, Spr
and Prx, respectively.

DRL Model $\lambda$ Highway Roundabout Pong $\uparrow$Vld% $\downarrow$Spr%
$\downarrow$Prx $\uparrow$Vld% $\downarrow$Spr% $\downarrow$Prx $\uparrow$Vld%
$\downarrow$Spr% $\downarrow$Prx A2C 0.99 97.38 1.51 1.067 68.0 0.35 0.253
76.00 0.15 0.140 0.5 93.75 1.33 0.946 55.88 0.30 0.220 35.15 0.08 0.079 0.1
47.88 1.02 0.732 26.38 0.23 0.168 33.43 0.09 0.081 DQN 0.99 87.38 1.78 1.255
60.63 0.51 0.365 89.27 0.11 0.107 0.5 56.38 0.92 0.659 29.63 0.17 0.130 33.91
0.08 0.075 0.1 23.00 0.36 0.266 20.125 0.03 0.027 33.91 0.08 0.075 PPO 0.99
97.38 1.94 1.369 86.75 0.434 0.312 91.77 0.14 0.131 0.5 84.00 1.43 1.021 73.50
0.27 0.200 55.52 0.11 0.110 0.1 22.25 0.43 0.316 30.62 0.15 0.097 33.54 0.008
0.078

###  IV-C Quantitative Results: Counterfactual Explanation Performance

To start the evaluation process of the CF-based explainer framework, we first
train the three teacher-DRL models considered in this study using high-
dimensional state images. Following this, the policies of these teachers are
distilled into a shallow student model using low-dimensional objective lists
as input (translated from the state images), the teacher’s soft labels as
targets, and employing KL divergence, NLL, and MSE as loss functions. The
architectures of both the teacher and student models are detailed in Table I,
highlighting the considerable reduction in the number of parameters in the
student model compared to the teacher model. Table II compares the efficacy of
policy distillation in terms of generalisation performance in three
environments, measured by the average rewards earned by the models throughout
$10$ trials, each comprising $15$, $10$, $2000$ episodes for the Highway,
Roundabout automated/autonomous driving scenarios and the Atari Pong game,
respectively. The relative student score percentage
($\frac{S_{s}}{T_{s}}\times 100$) is reported for ease of comparison with
teacher and student model scores. The proximity between teacher and student
model scores suggests successful policy distillation. Subsequently, candidate
CF explanations are generated for the student model using the Wachter method
[8] (Eq. (2)). In the final step, to validate the candidate CFs, we verify
whether both the teacher and student models yield the same action for a given
CF example input, i.e., we check whether
$M^{T}(\tilde{s}^{*})=M^{S}(\tilde{\sigma}^{*})$.

To assess the effectiveness of the generated CF explanations, we employ three
standard CF metrics: validity, proximity, and sparsity, as presented in Table
III. “Validity” indicates the percentage of CF candidates that successfully
induce a shift in the model’s output towards the desired target action.
Intuitively, a higher validity percentage shows that the framework has not
only generated more valid CFs, but also has been more successful in distilling
the teacher’s policy to the student network. “Proximity” and “sparsity” are
metrics that assess the resemblance between the original and CF states.
Proximity measures this similarity by computing the average value of the
modified pixels, while sparsity evaluates it by counting the mean number of
pixels that have been altered. Such metrics provide insights into the extent
of changes made to the original state during the CF generation procedure.
Intuitively, sparse CFs may lead to more traceable and human-understandable
explanations, as they alter fewer original input features, thereby
highlighting the most fundamental features attended to by the black-box model.
However, there is a trade-off between generating sparse CFs (low in proximity
and sparsity metrics) and more valid CFs (high in validity metric), where
allowing more perturbed features increases the chances of obtaining a valid CF
[7]. This trade-off can be controlled by hyperparameter $\lambda$ in Equation
(2), examined in Table IV. Higher values in the regulator parameter prioritise
providing more valid CFs, the first term in Equation (2), while lower values
lead to less distance between original states and CF states, the second term
in Equation (2). This phenomenon is evident in Table IV, where the validity is
decreased $75.13\%$ by changing $\lambda$ from 0.99 to 0.1 for the PPO agent
in the Highway environment.

In addition to standard CF metrics, Table III also compares generative metrics
used to assess the realism of the generated CF examples, such as Fréchet
Inception Distance (FID), Learned Perceptual Image Patch Similarity (LPIPS)
and Inception Score (IS). FID measures the similarity distance between
generated and real states based on their feature representations, LPIPS
evaluates perceptual differences between real and fake state images using
high-level visual features. IS provides a measure of the quality and diversity
exhibited by the counterfactual examples. Such metrics offer insights into the
visual fidelity and realism of the provided CF examples. Expectantly,
generating CF object-list states and re-rendering them into CF image enabled
PDCF framework to provide more realistic CF examples, such that in FID, LPIPS
and IS metrics the PDCF outperformed the SOA methods, as shown in Table III.

Figure 5:  A visual evaluation of the performance comparison between the SOTA
methods and PDCF framework across DRL agents: DQN, PPO, and A2C. The original
action is denoted as $a$, and the query action is marked as $a^{\prime}$. The
EV and PVs are depicted with green and blue boxes, respectively. Figure 6:  A
visual assessment comparing the performance of SOTA methods with the PDCF
framework across three distinct DRL agents: DQN, PPO, and A2C. Notably, the
traffic flow in the roundabout is anti-clockwise. The initial action is
labelled as $a$, and the target action is denoted by $a^{\prime}$. The EV and
PVs are represented by green and blue boxes, respectively. Figure 7:  A visual
evaluation comparing the performance of the SOTA methods with PDCF framework
across the DRL agents, including DQN, PPO, and A2C. Here, the initial action
is denoted as $a$, and the desired action is marked as $a`$. The modified
sections of the query states are highlighted by dashed white lines. The ball’s
movement trajectory is illustrated by white arrows over the specified time
period. Figure 8:  CF explanations generated by the PDCF framework for the A2C
agent operating in the Highway environment, covering all possible alternative
actions in response to the initial ”Faster” decision. The lateral velocity
direction of PVs is interpreted as their lane change intention, visually
represented by arrows.

###  IV-D Qualitative Results: Counterfactual Explanation Examples

Fig. 5\- 7 offer visual comparisons among the PDCF and the SOA CF explainer
models, for all mentioned DRL models and environments. The First row frames in
Fig. 5 present an example illustrations of the query state for the three DRL
agents, while the subsequent three rows showcase CF examples generated with
baseline methods including Olson [19] and Huber [20] as well as the proposed
PDCF framework. These scenarios offer valuable insights into the effectiveness
of the models in generating informative and plausible CF explanations.

For instance, when aiming to alter the decision of the A2C agent from ”Faster”
to ”Left” (Fig. 5, first column), the PDCF framework successfully relocates
the front PV (blue box) closer to the EV (green box). This adjustment compels
the agent to change its lane to the left, providing a meaningful explanation.
In contrast, the method by Huberet al. [20] produces an unrealistic
modification by moving a PV off-road to the left, introducing implausible
artifacts. The model proposed by Olsonet al. [19] performs better than that of
Huberet al., but it is evident from Figure 5 that the DQN agent does not
entirely prevent the possibility of a lane change as the PDCF does.

The disparity in quality is also evident in the Roundabout Automated Driving
scenarios, as shown in Figure 6. When attempting to change the A2C decision
from ”Left” to ”Slower”, the PDCF framework successfully repositions the PV in
the left lane, altering the lane position and compelling the EV to maintain
its current lane while driving at a slower speed. However, in the case of the
Huber et al. method, while it attempted to achieve a similar CF, the PV in the
generated CF state does not conform to the shape of the other PVs.
Additionally, in the case of PPO (Figure 6, third column, second row), it
produced a PV on the off-side of the road, which is an incorrect
representation. Notably, the method proposed by Olsonet al. [19] was unable to
converge in this environment due to the high-dimensional pixel size of the
state images.

The PDCF method consistently demonstrates high-quality CF explanations, even
in the challenging Atari Pong game environment. For example, when considering
an initial state classified as ”Down” by the PPO model (Figure 7, third
column), the PDCF model effectively generates a modified version where a new
ball is positioned further away from our paddle, as indicated by the dashed
blue lines. This alteration leads to a decision change to ”NOOP” (Figure 7,
fourth row). In contrast, the state-of-the-art (SOTA) models face difficulties
in achieving such distinct adjustments. Olson et al.’s method yields two
paddle balls, failing to properly obscure the original ball. Huber et al.’s
method provides two paddles, multiple balls, and introduces unrelated
alterations. The SAFE-RL model consistently introduces sparser changes,
underscoring its superior performance in generating substantively meaningful
CF explanations when compared to the baseline methods.

##  V Discussion: counterfactuals Explanations for Stakeholders Insight

This section explores the potential benefits of generating Counterfactual
Examples (CFs) in elucidating the rationale behind learned policies for
various stakeholders. Existing literature suggests that CF explanations serve
several crucial functions: Firstly, they facilitate the debugging of failure
cases by providing researchers and engineers with additional insights into
corner cases. Secondly, they enhance trustworthiness for end-users by making
the behavior and decision-making process of the DNN system transparent.
Lastly, they offer justifications to comply with legislation, such as the
”right to explanation” regulation outlined in the General Data Protection
Regulation (GDPR) applicable across the European Union. In Figure 8, we
showcase the CF explanations generated by the PDCF framework for all potential
counter-actions. This demonstration aims to clarify how CF examples can offer
comprehensible insights for both end-users and developers. Commencing with the
initial state associated with the specified action ”Faster” by the A2C agent
in the top row, the PDCF framework has generated four distinct CF states for
each desired action, involving the following modifications:

  * •

“Left”: The front PV is moved closer, indicating an intention for a right lane
change.

  * •

“Idle”: The PVs in the left and front lanes are brought closer, indicating an
intention for a left lane change.

  * •

“Right”: The left and front lanes are closed with PVs.

  * •

“Slower”: A PV is introduced in the back, indicating an intention for a right
lane change, and another PV is generated in front, indicating a left lane
change.

These clear and informative changes provided by the PDCF framework indicate
that the underlying policy of the A2C agent is rational, as it necessitates
reasonable adjustments for decision alterations.

In addition to visually justifying the usage of CF explanations to gain
insight into DRL models, we offer two straightforward yet informative
techniques, namely “Time-to-Collision Explanation” and “Sample-based DRL
Agent’s Rule Extraction” for automated driving systems.

###  V-A Time-to-Collision Explanation

Generating multiple CFs can be employed to estimate the decision boundaries of
black-box DRL models, enabling engineers to analyse their boundary conditions
and interpret their decision-making. For instance, consider the top row in
Fig. 9, which illustrates a driving scene marked with the time-to-collision
(TTC) values between the EGO Vehicle (EV) and the left, forward, and right
Participant Vehicles (PVs). While all DRL models select the “Faster” action in
this scene, the DQN agent maintains a greater TTC to the leading vehicle
($TTC_{Forward}$) compared to the A2C and PPO agents. This suggests that the
DQN agent exercises more caution than the other agents. The result
demonstrates that the A2C agent will switch lanes only when the
$TTC_{Forward}$ is extremely small. Generating these CFs enables engineers to
analyse and, if necessary, tweak their model’s behavior. For instance, in this
case, the A2C’s discount factor $\gamma$ could be adjusted closer to unity to
prevent the agent from adopting a myopic policy (maintaining its velocity
unchanged until the last moment). As the DRL environment is configurable,
engineers could also impose stricter penalties for collisions on the DQN
agent.

Figure 9:  A driving scene for analysing and comparing the EGO vehicle’s
behavior for two DRL models.

###  V-B Sample-based DRL Agent’s Rule Extraction

Employing the generated CFs to construct the decision tree of agent decisions
with respect to the TTC can provide a comprehensive estimation of the agent’s
behavior. Coppens et al. [52] collected a dataset from a DRL agent operating
in an environment to feed a rule mining algorithm, distilling the DRL policy
to a rule mining system. Although the extracted rules can provide
interpretability to some degree for the end-users, in complex environments,
the agent may not necessarily act near its decision boundaries; hence, the
rules do not indicate the boundary conditions. Instead, in this paper, we
generate CFs, which are naturally near the decision boundary, as explained by
Chou et al. [7]. In this example, we have gathered pairs of the generated CFs
(transferred them to TTC values, which is a more human-understandable feature)
and actions and compiled them as a dataset. Subsequently, by fitting a
decision tree to this dataset, we extracted if-then rules. For instance here
we have provided two extracted rules for the PPO agent in the highway driving
environment example as follows:

Rule 1: $\displaystyle{\scriptstyle\text{IF }TTC_{\text{left}}<7.34\text{ AND
}TTC_{\text{Forward}}<5.96\text{ AND }TTC_{\text{right}}<9.64,}$
$\displaystyle{\scriptstyle\text{THEN Action}=\text{``Right''}}$ Rule 2:
$\displaystyle{\scriptstyle\text{IF }TTC_{\text{left}}<10.48\text{ AND
}TTC_{\text{Forward}}<4.23\text{ AND }TTC_{\text{right}}<3.5,}$
$\displaystyle{\scriptstyle\text{THEN Action}=\text{``Left''}}$

The extracted if-then rules are straightforward and transparent enough for the
end-users to comprehend the reasoning underlying the agent’s policy. Although
the derived rules include a considerable level of simplification, yet they
remain a viable source for representing the agent’s behaviour. For legal
authorities, future work could lead to the formation of standards, compliant
with legislation, by collecting all extracted rules and CF explanation images
as templates.

##  VI Conclusion

This paper developed a framework that can generate human-understandable
explanations for the decisions of black-box DRL-based systems, which is a
prerequisite for their adoption in safety-critical applications. This is
achieved by offering counterfactual (CF) examples, which can capture the
essential features in the input data. The key contribution as compared to the
state-of-the-art techniques is that the proposed framework can handle high-
dimensional input data, e.g., bird’s eye view images of motorway traffic,
while securing the plausibility of the generated CF explanations. To do that
the framework alters human-understandable features such as vehicles’
velocities and positions, rather than generating synthetic input images from
scratch as with a deep generative approach. In the end, high validity and low
sparsity performance are attained allowing us to accurately represent the
decision boundary of the DRL models. We carried out several evaluations of the
proposed explainer framework to justify, firstly, its ability to outperform
alternative approaches from the literature by generating plausible
explanations for several DRL methods (Section IV), and secondly, its potential
benefits to various stakeholders (Section V). As a part of future research, it
is important to expand the developed framework to include cases where the
environment cannot offer both high-dimensional and low-dimensional observation
spaces. [53, 54].

## References

  * [1] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * [2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [3] A. Tampuu, T. Matiisen, M. Semikin, D. Fishman, and N. Muhammad, “A survey of end-to-end driving: Architectures and training methods,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020.
  * [4] M. Mahmud, M. S. Kaiser, A. Hussain, and S. Vassanelli, “Applications of deep learning and reinforcement learning to biological data,” _IEEE transactions on neural networks and learning systems_ , vol. 29, no. 6, pp. 2063–2079, 2018.
  * [5] E. Tjoa and C. Guan, “A survey on explainable artificial intelligence (xai): Toward medical xai,” _IEEE transactions on neural networks and learning systems_ , vol. 32, no. 11, pp. 4793–4813, 2020.
  * [6] D. Bacciu and D. Numeroso, “Explaining deep graph networks via input perturbation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [7] Y.-L. Chou, C. Moreira, P. Bruza, C. Ouyang, and J. Jorge, “Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications,” _Information Fusion_ , vol. 81, pp. 59–83, 2022.
  * [8] S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual explanations without opening the black box: Automated decisions and the gdpr,” _Harv. JL & Tech._, vol. 31, p. 841, 2017.
  * [9] R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, and F. Giannotti, “Local rule-based explanations of black box decision systems,” _arXiv preprint arXiv:1805.10820_ , 2018.
  * [10] S. Sharma, J. Henderson, and J. Ghosh, “Certifai: Counterfactual explanations for robustness, transparency, interpretability, and fairness of artificial intelligence models,” _arXiv preprint arXiv:1905.07857_ , 2019.
  * [11] S. Dandl, C. Molnar, M. Binder, and B. Bischl, “Multi-objective counterfactual explanations,” in _International Conference on Parallel Problem Solving from Nature_. Springer, 2020, pp. 448–469.
  * [12] Y. Ramon, D. Martens, F. Provost, and T. Evgeniou, “A comparison of instance-level counterfactual explanation algorithms for behavioral and textual data: Sedc, lime-c and shap-c,” _Advances in Data Analysis and Classification_ , vol. 14, no. 4, pp. 801–819, 2020.
  * [13] S. Rathi, “Generating counterfactual and contrastive explanations using shap,” _arXiv preprint arXiv:1906.09293_ , 2019.
  * [14] A. Lucic, H. Haned, and M. de Rijke, “Why does my model fail? contrastive local explanations for retail forecasting,” in _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , 2020, pp. 90–98.
  * [15] P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Pérez, and M. Cord, “Steex: Steering counterfactual explanations with semantics,” _arXiv preprint arXiv:2111.09094_ , 2021.
  * [16] P. Rodríguez, M. Caccia, A. Lacoste, L. Zamparo, I. Laradji, L. Charlin, and D. Vazquez, “Beyond trivial counterfactual explanations with diverse valuable explanations,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1056–1065.
  * [17] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in _International conference on machine learning_. PMLR, 2017, pp. 3145–3153.
  * [18] A. Samadi, A. Shirian, K. Koufos, K. Debattista, and M. Dianati, “Safe: Saliency-aware counterfactual explanations for dnn-based automated driving systems,” in _26th IEEE International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2023\.
  * [19] M. L. Olson, R. Khanna, L. Neal, F. Li, and W.-K. Wong, “Counterfactual state explanations for reinforcement learning agents via generative deep learning,” _Artificial Intelligence_ , vol. 295, p. 103455, 2021.
  * [20] T. Huber, M. Demmler, S. Mertes, M. L. Olson, and E. André, “Ganterfactual-rl: Understanding reinforcement learning agents’ strategies through visual counterfactual explanations,” _arXiv preprint arXiv:2302.12689_ , 2023.
  * [21] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _CVPR_ , 2018, pp. 8789–8797.
  * [22] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1928–1937.
  * [23] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [24] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [25] E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018.
  * [26] J. Kim, T. Misu, Y.-T. Chen, A. Tawari, and J. Canny, “Grounding human-to-vehicle advice for self-driving vehicles,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 591–10 599.
  * [27] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang _et al._ , “End to end learning for self-driving cars,” _arXiv preprint arXiv:1604.07316_ , 2016.
  * [28] L. Shao, D. Wu, and X. Li, “Learning deep and wide: A spectral method for learning deep networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 25, no. 12, pp. 2303–2308, 2014.
  * [29] K. Browne and B. Swift, “Semantics and explanation: why counterfactual explanations produce adversarial examples in deep neural networks,” _arXiv preprint arXiv:2012.10076_ , 2020.
  * [30] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [31] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2574–2582.
  * [32] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [33] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, “Counterfactual visual explanations,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2376–2384.
  * [34] L. A. Hendricks, R. Hu, T. Darrell, and Z. Akata, “Grounding visual explanations,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 264–279.
  * [35] P. Wang and N. Vasconcelos, “Scout: Self-aware discriminant counterfactual explanations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8981–8990.
  * [36] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [37] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International conference on machine learning_. PMLR, 2018, pp. 1587–1596.
  * [38] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International conference on machine learning_. PMLR, 2018, pp. 1861–1870.
  * [39] J. Ba and R. Caruana, “Do deep nets really need to be deep?” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [40] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2006, pp. 535–541.
  * [41] A. Henelius, K. Puolamäki, H. Boström, L. Asker, and P. Papapetrou, “A peek into the black box: exploring classifiers by randomization,” _Data mining and knowledge discovery_ , vol. 28, no. 5, pp. 1503–1529, 2014.
  * [42] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” _arXiv preprint arXiv:1511.06295_ , 2015.
  * [43] G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _Deep Learning and Representation Learning Workshop, Neural Information Processing Systems (NIPS)_ , 2015.
  * [44] T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun, “Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,” _arXiv preprint arXiv:2105.08919_ , 2021.
  * [45] Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu, “Distral: Robust multitask reinforcement learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [46] R. Traoré, H. Caselles-Dupré, T. Lesort, T. Sun, G. Cai, N. Díaz-Rodríguez, and D. Filliat, “Discorl: Continual reinforcement learning via policy distillation,” _arXiv preprint arXiv:1907.05855_ , 2019.
  * [47] J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, “Progress & compress: A scalable framework for continual learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4528–4537.
  * [48] M. Igl, G. Farquhar, J. Luketina, W. Boehmer, and S. Whiteson, “Transient non-stationarity and generalisation in deep reinforcement learning,” _arXiv preprint arXiv:2006.05826_ , 2020.
  * [49] R. Xu, S. Luan, Z. Gu, Q. Zhao, and G. Chen, “Lrp-based policy pruning and distillation of reinforcement learning agents for embedded systems,” in _2022 IEEE 25th International Symposium On Real-Time Distributed Computing (ISORC)_. IEEE, 2022, pp. 1–8.
  * [50] Z. Chen, F. Silvestri, J. Wang, H. Zhu, H. Ahn, and G. Tolomei, “Relax: Reinforcement learning agent explainer for arbitrary predictive models,” in _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, 2022, pp. 252–261.
  * [51] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , vol. 22, no. 268, pp. 1–8, 2021\. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html
  * [52] Y. Coppens, D. Steckelmacher, C. M. Jonker, and A. Nowé, “Synthesising reinforcement learning policies through set-valued inductive rule learning,” in _International Workshop on the Foundations of Trustworthy AI Integrating Learning, Optimization and Reasoning_. Springer, 2020, pp. 163–179.
  * [53] R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [54] E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh, “Disentangling disentanglement in variational autoencoders,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 4402–4412.