# Counterfactual Explanations for Misclassified Images:
How Human and Machine Explanations Differ

Eoin Delaney  Arjun Pakrashi  Derek Greene  Mark T. Keane

###### Abstract

Counterfactual explanations have emerged as a popular solution for the
eXplainable AI (XAI) problem of elucidating the predictions of black-box deep-
learning systems due to their psychological validity, flexibility across
problem domains and proposed legal compliance. While over 100 counterfactual
methods exist, claiming to generate plausible explanations akin to those
preferred by people, few have actually been tested on users ($\sim$7%). So,
the psychological validity of these counterfactual algorithms for effective
XAI for image data is not established. This issue is addressed here using a
novel methodology that (i) gathers ground truth human-generated counterfactual
explanations for misclassified images, in two user studies and, then, (ii)
compares these human-generated ground-truth explanations to computationally-
generated explanations for the same misclassifications. Results indicate that
humans do not “minimally edit” images when generating counterfactual
explanations. Instead, they make larger, “meaningful” edits that better
approximate prototypes in the counterfactual class.

###### keywords:

XAI , Counterfactual Explanation , User Testing

††journal: Artificial Intelligence

[label1]organization=School of Computer Science, University College
Dublin,addressline=Belfield, city=Dublin, country=Ireland

[label2]organization=Insight Centre for Data Analytics,addressline=Belfield,
city=Dublin, country=Ireland

[label3]organization=VistaMilk SFI Research Centre,addressline=Belfield,
city=Dublin, country=Ireland

##  1 Introduction

As Artificial Intelligence (AI) is increasingly used in everyday life for
high-stakes decision-making, many new roles have emerged for eXplainable AI
(XAI) [1, 29, 37, 93]. For instance, in computer vision systems, explanations
can help to debug black-box models (e.g., showing why images were
misclassified) [85, 7], to audit system safety (e.g., why a self-driving car
misidentified a postbox as a red light [38]), to assess fairness and bias
(e.g., why one person’s face was cropped from an image over another’s [9])
and, even, to provide novel domain insights (e.g., identifying mass legions in
digital mammography [5]).

In computer vision systems, many diverse strategies have been advanced to
explain model predictions [74, 1, 41] using, for instance, saliency maps [106,
90], feature importance [84, 76], prototypes [57, 86], and factual [94, 51],
counterfactual [78, 13] or semifactual examples [55]. Saliency maps have been
extensively used in image classification to highlight “important regions” in
the input image by using back-propagation and up-sampling to generate an
activation map [106, 90]. In a similar fashion, explanations using model-
agnostic feature importance (e.g., LIME [84] and SHAP [76]) aim to show the
features or super-pixels of an instance that contributed most to a prediction.
Indeed, recently, to achieve better fidelity to the underlying black-box model
[2], there has been a move away from input-level features (pixels or segments)
to more high-level concepts (e.g., stripes in an image of a zebra) based on
extracted latent features (e.g., as concept activation vectors [59, 35, 18,
104]). By the same token, example-based explanations try to leverage
meaningful, concept-level units (e.g., as prototypes, factual, counterfactual
or semifactual examples). Here, we focus on the psychological validity of
counterfactual methods, although our results are also shown to have
significant implications for prototype techniques.

Counterfactual explanations have received significant attention in the XAI
literature (for reviews see [41, 97, 49, 52]) as they provide “what if”
explanations that use a contrasting case to show how a prediction would change
if the input features had been different [38, 40, 78, 49, 52]. For instance,
when a two-year loan application for $5k is refused by an automated system and
the user asks “Why?”, the counterfactual explanation might suggest “If you had
asked for a $4.5k loan over a one-year term, then you would have been granted
the loan”. In this case it can be seen that changing the loan-amount and loan-
duration features can flip the decision to the user’s desired outcome (i.e.,
loan-granted). In the image domain, a corresponding counterfactual could be
used to audit a misclassification by a black-box deep learner; for instance,
when a self-driving car misidentifies a postbox as a red light and the user
asks “Why?”, the counterfactual explanation might suggest ”if the round-red-
postbox had been a square-red-postbox, it would not have been misidentified as
a traffic light”. More formally, given a black-box classifier $b$ and $I$ as
some to-be-explained query image with the predicted class $b(I)=y$, then
$I^{\prime}$ is a candidate counterfactual explanation when
$b(I^{\prime})=y^{\prime}$, where $y$ and $y^{\prime}$ are contrasting classes
(see e.g., [38]).

The current AI interest in counterfactual methods has been boosted by
philosophical proposals about their centrality in causality [71, 103],
psychological findings that they are important to people’s understanding of
causes [78, 13, 12, 82, 66] and legal analyses suggesting they are GDPR
compliant [99]. Indeed, there are now 100+ counterfactual methods in the XAI
literature, all of which claim to generate the plausible counterfactual
explanations people need to understand AI systems [52]. However, most of these
plausibility claims are based on intuition rather than hard psychological
evidence [6, 69]. As with much of the XAI literature [51, 4], the user testing
of methods remains uncommon; Keane et al. [52] found that only $\sim$7% of
counterfactual methods specifically user-tested proposed functionalities. So,
we really do not know which, if any, of these counterfactual methods really
generates explanations that people find plausible.

Accordingly, in this paper, we advance a novel methodology to address this
significant gap in the counterfactual XAI literature. Stated simply, we ask
people to counterfactually explain images misclassified by an AI system and
then we compare these explanations to those produced by several benchmark
counterfactual methods for the same misclassifications. These ground-truth
human explanations allow us to assess definitively whether the machine-
generated counterfactuals are, indeed, plausible. To presage our results, the
evidence shows that they are not, that people actually prefer counterfactual
explanations that rely more on prototypes from a contrasting class, rather
than minimally perturbed instances close to decision boundaries.

###  1.1 Contributions & Outline of Paper

This paper aims to make significant progress in providing novel results and a
better psychological grounding for the use of counterfactual explanations in
XAI. As such, we make several novel contributions to the field:

  * 1.

We provide a critical analysis of the notion of plausibility in the
counterfactual XAI literature, showing how different intuitive claims for it
can be mapped to evaluative metrics measuring proximity, representativeness,
and prototypicality.

  * 2.

We provide an up-to-date survey of the main user-study findings on
counterfactual explanations in XAI including a critical analysis that reveals
the system-centered nature of this work.

  * 3.

We advance a novel user-centered methodology for collecting counterfactual
explanations used by people and show how this ground truth can be related to
matched outputs from computational counterfactual methods.

  * 4.

We report new results on the specific divergences that occur between human and
machine explanations when evaluation metrics for proximity,
representativeness, and prototypicality are applied; detailing the major
implications they have for current methods in the field.

In the next section, we review the related work on counterfactual methods in
XAI, their plausibility claims, and the evaluation metrics used to test these
claims (see section 2). Then, we survey the main findings emerging from the
scarce user tests that have been performed on the efficacy of counterfactual
explanations methods, introducing the proposal that they are predominantly
system-centered (see section 3). We then sketch our novel methodology for
collecting a ground truth of human counterfactual explanations for
misclassifications in two benchmark image datasets (MNIST [70] and QuickDraw
Doodles [14]; see sections 4). Section 5 presents our comparative study of
counterfactual explanation, detailing the correspondences between human-
generate and machine-generated counterfactuals. Finally, we discuss the
implications of these results for counterfactual explanations, in particular
and, more generally, for he explanation of AI systems.

##  2 Related Work I: Plausibility of Counterfactual Explanations

In the current section, we do not attempt to survey the 100+ counterfactual
methods that exist in the XAI literature (see [48, 97, 52] for good surveys).
Rather, we focus on the literature of most relevance to the current study to
contextualise profile, and detail the methods tested here. Most counterfactual
methods claim, typically on intuitive grounds, to produce plausible
counterfactuals; that is, counterfactuals that people would find realistic,
comprehensible and appropriate for explanatory purposes. However, different
methods differ in what they see as the basis for this plausibility, proposing
that it relies on some deeper aspect of the counterfactual-computations made
by the method. For instance, many counterfactual methods, since Wachter et
al.’s seminal paper [99], argue that proximity is the key to plausibility;
that counterfactuals that are close to the query and to the decision boundary
of the counterfactual class lead to the plausible counterfactuals that people
require. Focusing on proximity also induces the generation of sparse
counterfactuals that modify few aspects of the original query instance, thus
making them easier for people to comprehend [53]. Other methods have proposed
that plausibility owes more to representativeness, that the counterfactual
explanation needs to be representative of the domain to be plausible (e.g.,
[68]). Still others have argued that plausibility hinges on using
semantically-meaningful features that represent the central tendency of
instances in the class, specifically, those captured by prototypes (e.g.,
[96]). In this section, we introduce and critically review the methods tested
in the current study, from these different plausibility perspectives (i.e.,
proximity, representativeness, prototypicality) in terms of (i) the
perspective on plausibility taken, (ii) the details of the methods tested,
(iii) the evaluation metrics used to assess this type of plausibility.

###  2.1 Proximity: Plausibility as Counterfactual Closeness

David Lewis’ influential philosophical analysis [71] proposed that
counterfactuals captured the closest possible world in which some target event
minimally differed. In the last few years, this view that plausible
counterfactual explanations depend on balancing the distance from the query
and decision boundary has, perhaps, been the dominant paradigm in XAI. This
idea underlies so-called proximity methods that aim to find the minimal
changes or Min-Edits to a query-instance’s features to generate one with a
contrasting predictive outcome [78, 99, 80, 53, 96].

The earliest methods in this literature relied on proximity to find Nearest
Unlike Neighbours (NUNs) in the dataset as counterfactual explanations [83,
77], though this solution makes the explanation process overly dependent on
the availability of suitably-close instances in the dataset. Wachter et al.’s
[99] seminal contribution, the so-called Min-Edit method, introduced the major
innovation of generating synthetic counterfactual instances to provide much
better explanatory coverage, though generating such instances does lead to
other issues (e.g., invalid data-points being proposed as explanations). In
passing it should be said that in Psychology identifying plausibility with
similarity has not been commonly proposed (for a critique see [52], though see
also [21]).

####  2.1.1 Testing Proximity: Using Min-Edit

The Min-Edit method searches a space of perturbations of the query-instance
using gradient descent, applying a loss function that balances the closeness
of the counterfactual to the query against making minimal feature-changes
required to deliver a prediction change [99]. So, this method aims to generate
a counterfactual explanation by minimizing:

$(b_{t}(I^{\prime})-p_{t})^{2}+\lambda\|I-I^{\prime}\|_{1}$ (1)

The first loss term pushes the predicted class probability of the candidate
counterfactual $b_{t}(I^{\prime})$ towards a target $p_{t}$, while the second
term minimizes the Manhattan distance between the query and counterfactual to
promote proximate and sparse solutions. The Lagrangian multiplier, $\lambda$,
acts as a balancing term.

This method has been hugely influential in the recent counterfactual
literature and is the recognised baseline for many algorithmic comparisons.
Furthermore, many subsequent counterfactual methods have extended this
approach with additional constraints to improve sparsity [23, 39], diversity
[81], and to include causal models [49]. Other methods have tried to tackle
the invalid-data-point issue by including representativeness ideas, using
auto-encoders or generative models to ensure that the counterfactual lies
close to the data manifold [46, 96, 56, 91, 27].

In the present comparative study, we use Min-Edit as a baseline for this
approach to counterfactual XAI and implement it using [61]. The standard
evaluation metrics for assessing plausibility-as-proximity are the L1 and L2
norms, applied to the query-explanation pairings produced by a given
explanation method, where lower distances are taken to indicate the success of
the method.

###  2.2 Representativeness: Plausible Explanations Are Within-Distribution

Part of the motivation behind early counterfactual methods using Nearest
Unlike Neighbours (NUNs) was the recognition that explanations would be
plausible if they were representative and within the domain. By definition,
these methods cannot generate an out-of-domain counterfactual explanation, as
NUNs are typically within the training distribution. However, as we said
earlier, if queries have few or no close NUNs, this can still lead to low-
quality or failed explanations [53, 92]. Recent perturbation methods (such as
Min-Edit) deal with this coverage issue by generating synthetic
counterfactuals. However, these methods can risk generating instances that are
either unrealistic, or not perceptibly different from the query instance [52,
46, 27, 87, 26]. Indeed, Laugel et al. [68] showed that for some domains these
methods could result in 30% of explanations being out-of-distribution. Such
concerns led to a re-emphasis on representativeness, to ensure that plausible
explanations are within-domain and, if possible, not out-of-distribution.
Indeed, this representativeness perspective on plausibility is much closer to
proposals made in psychology, where it is often cast as coherence or
consistency with prior knowledge in a domain [33, 19, 20, 21, 62].

####  2.2.1 Testing Representativeness: CEM-PN, VLK & Revise

Many counterfactual techniques have supplemented proximity-based methods to
take representativeness into account, often by using auto-encoders or
generative models. The goal is to ensure that plausible counterfactuals that
belong to the data distribution are preferred [56, 27, 46]. In the current
study, we selected three methods taking this approach, based on their
recognition in the area, their ability to handle image data, and open-source
availability of their code (see [27, 96, 46]). Here we briefly describe each
in turn.

CEM-PN [27] computes pertinent negatives using an objective function that
contains an elastic net $(\beta L_{1}+L{2})$ regulariser to select features to
alter via perturbation whilst keeping the perturbations sparse. An autoencoder
is leveraged to ensure that the generated explanations lie close to the data
manifold through minimizing the $L_{2}$ reconstruction error. This method was
implemented using [61].

VLK [96] generates a counterfactual by minimising a multi-objective loss
function defined by

$Loss=cL_{pred}+\beta L_{1}+L_{2}+L_{AE}+L_{proto}$ (2)

where the first term encourages the perturbed instance to belong to the
counterfactual class. The elastic net regulariser $\beta L_{1}+L_{2}$ aims to
ensure sparsity and proximity in the generated instance. $L_{AE}$ is the
reconstruction error of the candidate counterfactual instance, which is
minimized to encourage the counterfactual to belong to the training data
distribution. To guide the counterfactual-instance towards the distribution of
the perturbed class, the $L_{2}$ distance between it and the counterfactual
class prototype is minimised in the $L_{proto}$ term. Following [96], the
encoder from $L_{AE}$ is used to retrieve class prototypes. Specifically, the
counterfactual class prototype is defined as the average encoding of the five
nearest instances (according to Euclidean distance) in the latent space with
the same counterfactual class label.

Revise [46] relies on a generative model that is a decoder of a variational
autoencoder (VAE) trained on the training data. The overall idea is to
minimise the function

$\ell(b(G(\textbf{z'})),t)+\lambda\|G(\textbf{z'})-\textbf{I}\|_{1}$ (3)

where $b$ is the classifier, $t$ is the target, $\ell$ is some loss function,
and $G$ is the generative model. To find a $z^{\prime}$ that minimises the
loss, $z$ is initialised to the encoding of the original input $I$. Then the
gradient of the loss in the latent space is computed and the algorithm
iteratively takes small steps in that space until the prediction changes to
the target. Since the resulting counterfactual $I^{\prime}=G(z^{\prime})$ is
produced by the generative model, it can be dissimilar to $I$. Although the L1
norm is known to encourage sparsity, the algorithm may not necessarily
generate sparse solutions, as the changes occur in the latent space rather
than the input space. The method may also fail to generate explanations in
some cases. For instance, [45] reported a 22% failure rate on MNIST images.
This method is implemented using code from [45], with the recommended
hyperparameters of $\lambda$ = 1 and gradient step $\delta$ = $10^{-5}$. As in
[46, 45], we use cross entropy loss for $\ell$.

In the present study, these three methods are used as benchmark approaches for
this representativeness approach to counterfactual XAI, with all being
implemented using [61]. Several different evaluation metrics, typically out-
of-distribution measures, have been used to assess plausibility-as-
representativeness, applied to the counterfactual-instances produced by a
given explanation method. We use MC-Dropout, IM1, 10-LOF and R%Sub to perform
these evaluations (see Section 4.3.2).

###  2.3 Testing Prototypicality: MMD-Critic & Grad-Cos

A final plausibility claim on counterfactual explanations is the idea that
they should be within-class representative, which can be reflected in
prototypicality measures; another consideration that arises from issues with
perturbation methods in image domains. As proximity methods try to find
counterfactuals which lie just over the decision boundary (minimising distract
from the query), this often means that they are not particularly
representative of the counterfactual class, or indeed the original query-class
either. Indeed, as has been discovered in adversarial learning [36], small
pixel-changes to a query can sometimes result in a change of class, even when
these pixel-changes are actually imperceptible to humans [55, 36] (see Figure
3 for examples). Clearly, such counterfactuals are not plausible explanations
for people.

One solution to this problem is to generate the counterfactuals in a latent
space using higher-level features (e.g., conceptual [38] or exceptional
features [55]). Indeed, some have gone further, arguing that it is better to
instead leverage instances that are maximally representative of a class (i.e.,
class prototypes) [57, 17, 58, 72, 5], or to interpret a model’s prediction
using human-friendly concepts [59, 35, 18, 104]).

Accordingly, we also measure the similarity of generated counterfactuals to
computed prototypes from the counterfactual class (using Grad-Cos [16] on
prototypes from MMD-Critic [57]). MMD-critic [57] is implemented to compute
prototypes by minimizing the maximum mean discrepancy between the prototype
distribution and the data distribution using a kernel density function (full
technical details can be found in Appendix B). This evaluation aims to
determine whether generated counterfactuals are actually close to prototypes
for the counterfactual class. Recall, Min-Edit methods try to find instances
that are close to the query but just over the decision boundary in a
contrasting class; hence, they are much less likely to be representative
members of this counterfactual class. To evaluate prototypicality we used the
Grad-Cos metric to determine whether machine-generated and human-generated
counterfactuals are close to prototypes (for details see Section 4.3.3).

##  3 Related Work II: User Tests of Counterfactual XAI

The recent rapid algorithmic advances in counterfactual XAI have not been
paralleled by a significant program of user testing. One survey of user
studies reports that only 31% (36 out of 127 papers) of counterfactual papers
conducted any form of user evaluation and only 7% competitively test
alternative algorithms [52]. Furthermore, many of these studies are vitiated
by poor/non-reproducible experimental designs, imbalanced material sets,
low/inappropriate numbers of participants, and inappropriate/absent
statistical analyses. So, very few studies report solid evidence on the
efficacy of counterfactual explanations in XAI. Indeed, as we shall see, most
of these user studies adopt a very system-centered approach, in which users
are cast as passive recipients of machine-generated explanations. In the
following subsections, we critically assess the main findings from this
literature under the headings of findings on (i) the general efficacy of
counterfactual explanations, (ii) specific comparative tests of methods, (iii)
tests of image-based methods.

###  3.1 Counterfactual Explanations: General Efficacy

Most user studies on counterfactual XAI focus on tabular datasets and, simply,
attempt to establish in broad terms whether or not explanations impact user
behaviour [28, 73, 100, 75]. Methodologically, they test whether the provision
of a counterfactual explanation to an algorithmic decision has some/any effect
on user behaviour by comparison to no-explanation controls and/or some other
explanatory method (e.g., causal explanations or case-based explanations [51,
100]).

For instance, an early study by Lim et al. [73] gave separate groups of people
different explanation-types – What-if, Why-Not, How-to and Why explanations –
and found that that all of the explanation-interventions improved performance
relative to no-explanation controls. However, the Why-Not counterfactual
explanations did no better than the other explanation-types. In a similar
vein, Dodge et al. [28] assessed four different explanation strategies (e.g.,
case based, counterfactual, and two forms of global explanation) for
biased/unbiased classifiers and found that counterfactual explanations had the
greatest impact on user responding (although these tests used quite a small
set of materials). Similarly, using a simulated drink-driving-advice app,
Warren et al. [100] found that counterfactual explanations improved people’s
understanding of the app’s predictions, over causal explanations and no-
explanation controls; they had participants predict outcomes for unseen
instances in a testing phase after a training phase in which they were shown
the app’s decisions.

However, other studies have found less evidence for the efficacy of
counterfactual explanations. In a carefully-controlled study on a simulated
app to advise diabetics about insulin treatments, van der Waa et al. [95]
found that counterfactual and example-based explanations did not improve
people’s knowledge of the domain over no-explanation controls (as measured by
users’ predictive accuracy on test cases). Worryingly, this study also found
that people followed the app’s advice even when it was incorrect. The latter
findings raises the worrying prospect that people can be increasingly
satisfied with a system when it is explained to them, without gaining any
insight into how the system works (which has been raised as an issue for XAI
[11, 100]). Other work [65] that asked people to predict/simulate what a model
might do under different inputs found that tasks using counterfactuals
elicited longer response times, were rated as being more difficult, and showed
reduced accuracy in users. Lucic et al. [75] found that people were less
accurate on tasks that asked them to introduce a counterfactual change for an
instance than when they were asked to predict outcomes from the input features
of the instance. These latter studies point to an added cognitive overhead in
processing counterfactuals that has been long-noted in psychology [12, 13].

So, overall, this literature shows mixed support for the efficacy of
counterfactual explanations. Of course, part of the problem here may arise
from paucity of good studies in the literature. In time, as more studies are
done, we may be able to reach a better understanding of what it is that makes
counterfactual explanations work in some contexts and not in others. However,
it could also be the case that people are not being properly tested in these
studies, that the orientation taken is too much from the machine-perspective
than that of users.

###  3.2 Comparative Tests of Counterfactual Methods

Apart from user tests on the general efficacy of counterfactual XAI, given the
differnet proposals on the plausibility of different algorithms, we also
require comparative tests between methods. However, even fewer user studies
have done such tests [3, 33, 32, 60, 64]. Forster and colleagues [33, 32] have
performed a series of unique studies that asked participants to assess
counterfactuals produced by different algorithms or parametric-variations of a
given algorithm [33, 32]. They compared a gradient-free optimisation method
against a popular gradient-based method [99] asking people to rate the
typicality, realism and suitability of generated counterfactual explanations
(i.e., proxy metrics for representativeness). The results showed that their
optimisation method, did better on all metrics than the popular Wachter et al.
[99] perturbation method. Kuhl et al. [64] also compared methods that differed
in how they selected counterfactual instances (i.e., perturbation methods with
and without density computations) but found no evidence that people could
distiguish them. Akula et al.’s [3] report a user study that compared their
own CoCoX counterfactual method to two other counterfactual methods (i.e., CEM
[27] and CVE [38]) showing that their method performed best, though several
methodological issues undermine these tests (see next subsection). Finally,
Kirfel & Liefgreen [60] specifically tested for effects of “actionable
features” on counterfactual explanation (i.e., features that the user is known
to be able to change). They found that people’s perception of the quality and
comprehensibility of explanations was affected the involvement of actionable
and mutable features, as opposed to immutable ones. However, they also
suggested that the actionable/mutability distinctions made in the methods were
not clear-cut in the human domain and were often hard to define. So, in
summary, the few comparative tests that have been carried out perhaps show
less support for benchmark Min-Edit methods in favor of algorithms that are
more concerned with computing within-domain explanations, though again the
evidence is patchy.

###  3.3 User Testing on Image Datasets

Only a handful of papers consider user-tests of counterfacual explanations for
image datasets, most focus on tabular datasets. Yet, many counterfactual
methods have specifically been proposed for image datasets (e.g.,[38, 27, 15,
98, 43]). Unfortunately, most of the few papers testing images have
significant issues around their experimental designs, statistical analyses
and/or the statistical significance of the results [38, 67, 105, 91]. This
leaves us with three core papers that report anything indicative on the topic
[38, 14, 3].

Goyal et al. [38] reported an influential method, Counterfactual Visual
Explanation (CVE), that highlights key regions in an image (e.g., the beak
colour of a bird) as feature differences behind counterfactual class changes
(e.g., classifying a bird image as an auklet or a cormorant). They performed a
user study (N=26)111For appropriate statistical power this design requires an
N$>$100 and confidence levels should be 95% or higher. with three conditions
testing a no-explanation control against two explanation conditions (i.e., a
non-counterfactual feature-region explanation and counterfactual-region
explanation). They found the counterfactual-region explanation elicited the
highest accuracy (77.8%), followed by the feature-region explanation (74.3%),
followed by the no-explanation controls (71.1%), differences that were only
significant at lower-than-usual confidence levels (i.e., 87% and 51%). So, at
best, these results are indicative rather than conclusive.

Cai et al. [14] used QuickDraw Doodles (one of the datasets we use here) to
reveal more conclusive results in a design that elicited better user
interaction. They had participants (N=1150) generate QuickDraw Doodles of
common objects (e.g., draw a helicopter or an avocado) and then had a
classifier identify the object using a dataset of labelled drawings. The
classifications produced were accompanied by normative explanations (i.e.,
similar examples from the same class, such as other doodles of avocados) or
comparative explanations (i.e., counterfactuals or similar example from other
classes, such as doodles of a pear or potato) with participants being asked to
rate how well they understood the system and their views on the system’s
capability. The results showed that explanations only impacted
misclassifications by the system (i.e., no effects for correct
classifications) and that the example-based, normative explanations improved
people’s understanding and assessments of system capability. Unfortunately,
these effects did not extend to the counterfactual-based comparative
explanations. Cai et al. considered this failure to find counterfactual
effects as being due to the “surprisingness” of the counterfactual examples,
but the failure could also have been due to the presentation of items from
multiple contrasting classes (i.e., people could be shown three different
contrasting-classes instances). In passing, it should be said that this study
is quite close to the current one, in that it pioneered the user-generation of
QuickDraw images in an XAI user-study; however, it differs from the current
study in that it asked users to generate training-instances rather than
specifically asking them to generate the counterfactual explanations for test-
instances222To labor this point, Cai et al.’s system retrieved the labelled-
drawings of users and used them as counterfactual explanations. So,
technically speaking, the counterfactuals were user-generated. But it was
their system that decided which of these drawings were used as counterfactual
examples, rather than the participants in their study. So, they were not user-
generated explanations..

Finally, Akula et al.’s user tested their CoCoX method, which uses a “fault-
lines” technique that feature-highlights counterfactual explanations akin to
those advanced by CVE [38]. CoCoX was compared against CEM [27] and CVE [38]
alongside seven other non-counterfactuals methods333The training phase in
these experiments appears show people the model’s class-selection for 15 test-
instances and then asks them for a counterfactual-class for that prediction,
with no feedback. This is an unusual training task that probably hampers
transfer-of-training to non-counterfactual methods in the test phase of the
study, perhaps explaining why these methods do so badly. (e.g., LIME, GradCAM,
CAM) using two measures: (i) a measure of people’s agreement with the model’s
predictions for test-instances (they call this Justified Trust), and (ii) some
of the satisfaction questions proposed by the DARPA group [44]. The results
showed that CoCoX does best on both measures with CEM and CVE competing for
second position. Furthermore, these explanation conditions do markedly better
than no-explanation controls (i.e., 30%-40% better). Although these authors
are to be commended for their user-testing efforts, unfortunately this study
has several serious design flaws. It appears to be designed as two separate
10-group between-subjects experiments, one for ML experts (N=20) and one for
non-experts (N=60), neither of which are appropriately powered (i.e., a
10-condition experiment of this type would require several 100 participants).
So, for instance, in the expert experiment this design means that the positive
results found for CoCoX are based on just two participants seeing 5 test-items
(i.e., 10 data-points), which could by-chance just happen to provide positive
results. Furthermore, five test-materials seem too few; assuming all groups
received the same five test-items, we do not know whether these particular
test-instances just happened resonate well with training on CoCoX’s
explanations over other methods (i.e., a larger set of training and test-
instances or randomly selected training- and test-instances should have been
used). So, again, these results on image-datasets are indicative rather than
conclusive.

###  3.4 Interim Conclusions on User Testing

In the previous sections, we have seen how the extant user-testing literature
on counterfactual explanations has produced mixed and, sometimes, inconclusive
results. Indeed, there are further recent results that seem to show that other
exogenous factors (i.e., variables independent of counterfactual
manipulations) have greater impacts; factors such as (i) expertise or
familiarity with the domain [31], (ii) whether explanation tasks are framed as
predictions or diagnoses [22], and (iii) whether instances involve categorical
or continuous features [100, 101]. To us, these findings suggest that the
community may not be taking the right approach to users, that current user
studies has not sufficiently human-centered. Also, that taking such a
perspective may shed a very different light on our current understanding of
how people understand counterfactual explanation. In the next section, we
consider a new methodology which attempts to rectify these failings.

Figure 1: System-centered user-tests of counterfactual XAI present people with
the outputs from an AI-model and XAI method to evaluate them in different
ways. User-centered tests try to capture the user’s perspective on
explanation. Here an XAI-method’s explanation of model outputs (e.g.,
misclassified images) are evaluated by comparing them to human explanations of
the same model outputs (e.g., misclassifications) which are collected as a
ground-truth. Most, if not all, of the current user-tests of counterfactual
XAI are system- rather than user-centred.

##  4 A User-Centered Methodology for Counterfactual XAI

All the user studies that have been reviewed thus far have been decidedly
system-centered ones, in which users are cast as passive recipients of
machine-generated explanations. In these studies, XAI methods are used to
generate explanations for AI-model outputs, that are then fed to people to be
evaluated in different tasks (e.g., for correctness, acceptability,
helpfulness, trustworthiness). Such studies lack a reality-check on whether
these machine-generated explanations are actually the ones that people really
require; for instance, we do not even know if these machine-explanations are
representative of human counterfactual explanations. In contrast, a more user-
centered approach would focus on the user, on their conception of
counterfactual explanation. So, arguably, the user-testing of counterfactual
XAI requires a Copernican reversal from being overly system-centered to being
more user-centered (see e.g., Figure 1). However, to date, the XAI community
has clearly found it hard to design such user-centered studies444In many ways,
[50] is very different to the current work in its focus on data augmentation
in NLP, but conceptually it tries to be user-centered in the same way, as it
uses a human-in-the-loop task to elicit people’s counterfactual understanding
of sentiment in texts. Here, we advance a novel user-centered methodology that
starts with gathering a user-generated ground-truth of counterfactual
explanations for a CNN’s misclassifications, before comparing these human-
generated explanations to the machine-generated ones from several benchmark
counterfactual methods. This user-centred study allows us to verify the
plausibility claims made for different counterfactual XAI algorithms. In the
following sections, we introduce (i) this new two-step methodology, (ii) the
editing tool used to collect the ground-truth explanations, (iii) the
evaluation metrics used to compare the human-generated and machine-generated
explanations.

###  4.1 A User-Centered Two-Step Methodology

The current methodology realises a human-centered approach in two steps: (i)
ground truth data collection, followed by (ii) comparative evaluations of
human- and machine-generated explanations. Two datasets are used: the
benchmark MNIST images of written Arabic numbers [70] and QuickDraw Doodle
images [14]. The QuickDraw dataset is arguably more complex than the MNIST
one; notably, it involves images with parts that people can readily name
(e.g., the toppings on a pizza slice). We train CNNs for each of these
datasets and randomly select a sample of misclassifications made by the
models. These misclassified instances are then presented to (i) human
participants in a psychological experiment and (ii) to each counterfactual
method to collect the explanations generated. The two main steps in the
methodology are as follows:

  * 1.

Ground-Truth Collection. People were provided with a simple editing tool to
create their own counterfactual explanations for misclassified images from the
CNN, to collect “ground-truth explanations” for each dataset. This collection
was done in two separate experiments, one using the MNIST items (N=42) and a
separate pilot using the QuickDraw items (N=5).

  * 2.

Human-Machine Comparative Evaluation. The same misclassified images were then
presented to each of four counterfactual methods – Min-Edit, CEM-PN, VLK and
Revise – to produce parallel sets of machine-generated explanations, before
doing a human-machine comparative evaluation of the explanation sets. We use
benchmark evaluation metrics that have previously been used in computational
evaluations of counterfactual methods to assess plausibility claims (i.e., on
proximity, representativeness, prototypicality).

This methodology definitively tests whether the explanations generated by
people correspond in any way to those generated by these counterfactual
methods. As such, to the best of our knowledge, this is the first true user-
centered assessment of counterfactual algorithms. In the following sub-
sections, we provide more details on each of the steps in this methodology.

Figure 2: Screenshot of the editing tool used for collecting ground-truth
explanations, showing a misclassified MNIST image of a “5”, along with the
instructions to participants. The interface allows pixels to be added or
removed using the cursor as a pen or eraser, after clicking the “Draw” or
“Erase” buttons, respectively. The “Reset” button removes all edits, resetting
the image to its original form.

###  4.2 The Ground-Truth Step: Task & Tool

For the first step in the methodology, a simple software tool was developed to
present the misclassified images to participants. The tool allowed images to
be edited via a custom interactive GUI implemented using the tkinter Python
package (see Figure 2). The ground-truth collection was carried out by
presenting the CNN’s misclassifications to people and asking them to edit the
query-image to correct the incorrect prediction. For each misclassification,
they were told the model’s label and its correct label (e.g., 3 and 5,
respectively) and that it was misclassified (i.e., “This is an image of a 5
that was incorrectly labelled by the program as being a 3”). They were then
invited to edit the image using the editing tool, to explain how the
misclassification would have to change to be correctly labelled (i.e., “Your
task is to make changes or edits to the image, to help the program correctly
label the image as a 5”). This task requires people to create a counterfactual
instance that shows how the image would have to change to be correctly
classified.

In the user test involving the MNIST dataset, two separate groups of
participants were given slightly different instructions, the so-called
“Normal” and “Min-Edit” groups. The Normal group was given the instructions
discussed above, asking participants to “…make changes or edits to help the
program correctly label the image…”. In contrast, the Min-Edit Group was asked
to “…make the smallest possible changes needed, to help the program correctly
label the image…”. As we saw in Section 2, a key assumption of many algorithms
is that the counterfactual should make minimal changes or “minimal edits” to
the query. This instructional manipulation was designed to determine whether
instructions to uses to act in accordance with the Min-Edit method changed
responding relative to the “normal” non-directive instructions.

###  4.3 The Comparative-Evaluation Step: Three Perspectives

For the second step in the methodology, the comparative-evaluation step, the
counterfactual explanations produced by human and machine were systematically
compared using key evaluation metrics that are commonly used in this area.
This section describes each of the evaluation metrics used for these
comparisons grouped by (i) proximity tests comparing distances between query
and counterfactual instances (using L1 and L2 norms), (ii) representativeness
tests assessing generated counterfactual instances (using Monte Carlo Dropout,
IM1, 10-LOF, and R% Sub, see Section 4.3.2), and (iii) prototypicality tests
comparing distances generated counterfactual instances to class prototypes
(using Grad-Cos on prototypes retrieved from MMD-Critic). Taken together,
these evaluations provide a definitive test of the extent to which current XAI
methods produce plausible, counterfactual explanations and the extent to which
they diverge from those produced by people. In the following subsections, we
describe the specific metrics used.

####  4.3.1 Proximity Metrics: L1 & L2 Norms

The distance metrics – L1 (Eqn. 4) and L2 (Eqn. 5) norms – have typically been
used to evaluate counterfactual methods, measuring the closeness of the
counterfactual image, $I^{\prime}$, to the query image, $I$, where lower
distance-scores are assumed to be a proxy for explanation quality and
plausibility (but see [52] for criticism). Here, we compare distance scores
for machine-generated query-counterfactual pairs to the corresponding ground-
truth pairs produced by people in two user studies (see the MNIST and
QuickDraw experiments in Sections 5.1 and 5.2.4).

$\sum_{i}|I_{i}-I^{\prime}_{i}|$ (4)

$\sqrt{(\sum_{i}|(I_{i}-I^{\prime}_{i})|^{2})}$ (5)

####  4.3.2 Representativeness Metrics: Out-of-Distribution (OOD) Measures

Point representativeness and out-of-distribution metrics – Monte Carlo Dropout
[34, 56, 26], R%-Substitutability [88, 56], LOF [10, 47, 25], IM1 [96, 56] –
have often been used to evaluate counterfactual methods. These measure the
representativeness of explanations to the data-distribution or classes found
in the distribution. Here, we briefly describe these metrics that are used to
compare the machine-generated counterfactuals and those generated by people.

##### Monte Carlo Dropout

Originally proposed by [34], MC-Dropout enables one to quickly estimate the
uncertainty associated with a prediction by inspecting the variance of a
predictive distribution [24]. Multiple stochastic forward passes with
different dropout configurations can yield this predictive distribution.
Following [56, 26] we leverage MC-dropout to evaluate counterfactual
explanations by estimating the posterior mean of the predictive distribution
MC-Mean (higher is better) and the posterior standard deviation MC-Std (lower
is better). The intuition is that explanations with lower uncertainty scores
should be more representative of the counterfactual class as they are better
grounded in the data distribution. Full technical details of MC-Dropout and
the implementation can be found in Appendix A.

##### R%-Substitutability

Inspired by [88, 56], the generated counterfactuals are used as training data
to fit to a $k$-NN classifier (in pixel space) which then predicts the full
test-set. For MNIST, as we are using 50 instances we compare to an MMD
Prototype 1-NN classifier [57] that achieves 75.57% accuracy on the full MNIST
test set (10,000 images), using only 50 prototypical instances and a Euclidean
distance function. A method that achieves half this accuracy would achieve an
R% - Substitutability score of 50%.

##### IM1

Originally presented by [96] as an interpretability metric, a convolutional
autoencoder is trained on the predicted class $c$, $(AE_{c})$ and on the
counterfactual class $c^{\prime}$, $(AE_{c^{\prime}})$ and the $l_{2}$
reconstruction error is monitored to compute IM1. Specifically:

$IM1=\frac{\||I^{\prime}-AE_{c^{\prime}}(I^{\prime})\||_{2}^{2}}{\||I^{\prime}-AE_{c}(I^{\prime})\||_{2}^{2}}$
(6)

A lower value of IM1 implies that the candidate counterfactual image
$I^{\prime}$ can be better reconstructed by autoencoders that have seen
instances of the counterfactual class, relative to an autoencoder that has
seen instances in the original class, implying that $I^{\prime}$ lies closer
to the data manifold of $c^{\prime}$.

##### 10-LOF

Originally presented by [10], the local outlier factor (LOF) algorithm is a
distance-based technique that determines whether an instance is out of
distribution by computing the local density deviation with respect to its
neighbours in the pixel space. Following [47], the 10-LOF can be used to
determine if a counterfactual explanation is plausible according to the data
distribution. The decision-score metric is centred on zero, with higher values
indicating that a sample is more within the distribution according to 10-LOF.

####  4.3.3 Prototypicality Metric: MMD-Critic & Grad-Cos

MMD-critic [57] is implemented to compute prototypes by minimizing the maximum
mean discrepancy between the prototype distribution and the data distribution
using a kernel density function, full technical details can be found in
Appendix B. This evaluation aims to determine whether generated
counterfactuals are actually close to prototypes for the counterfactual class.
Recall, Min-Edit methods try to find instances that are close to the query but
just over the decision boundary in a contrasting class; hence, they are much
less likely to be representative members of this counterfactual class. The
Grad-Cos metric allows us to determine whether machine-generated and human-
generated counterfactuals are close to prototypes and is briefly described
below.

##### Latent Space Similarity: Grad-Cos

Given some labelled input image $I_{A}=(x,y)$ and a black-box neural network,
$b_{\theta}(I)$ that is parameterized by $\theta$, with loss $\ell(I;\theta)$
and gradient $\nabla_{\theta}\ell(I;\theta)$; Grad-Cos [16] is a gradient
based similarity metric that quantifies the degree to which the loss will
change when a small update to the model is made using some candidate training
instance, $I_{B}$, (e.g., a class prototype). If these two images are very
similar from the neural network’s perspective, this change will be large.
Formally, the cosine similarity of gradients kernel can be expressed as:

$k_{\theta}(I_{A},I_{B})=\frac{{\nabla_{\theta}\ell(I_{A})\cdot\nabla_{\theta}\ell(I_{B})}}{\||\nabla_{\theta}\ell(I_{A})\||\||\nabla_{\theta}\ell(I_{B})\||}$
(7)

Recent research independently evaluated several popular relevance metrics
(e.g., Influence Functions [63]) and found Grad-Cos to be the recommended
choice in practice [42]. Grad-Cos passed the weight randomization tests of
[2], indicating that it is faithful to the underlying machine learning model.

##  5 A Comparative Study: Human & Machine Explanations

As described in Section 4, a two-step, user-centered methodology was applied
to the explanation of a CNN’s misclassifications after training on two
datasets (the MNIST and QuickDraw Doodles datasets). Two user experiments, one
for each dataset, were carried out to gather ground-truth explanations for
each of the selected misclassifications. Machine-generated explanations were
generated for each of the same misclassifications using four benchmark
counterfactual XAI methods – Min-Edit, CEM-PN, VLK and Revise (see 2). These
user-generated and machine-generated explanations where then compared using
benchmark evaluation metrics for proximity, representativeness and
prototypicality. The following sub-sections lay out the method for this
overarching comparative study and specific methods used in the two user-tests
that collected the ground-truth data.

###  5.1 Method: Comparative Study

####  5.1.1 Model Setup: CNN Classifier & Datasets

The to-be-explained, black-box model was a convolutional neural network (CNN)
trained using a well-known architecture [96]. Two image datasets were used:
the MNIST [70] and Google QuickDraw [14] datasets. The MNIST dataset contains
images of written numbers, with 70,000 images covering 10 classes (i.e., the
digits 0–9). These images were scaled to $[-0.5,0.5]$ and the default training
and test sets were used. Dropout layers were implemented for regularization
and to facilitate uncertainty computations, using MC-Dropout [34]. The CNN was
trained with an Adam optimiser for 10 epochs using a batch size of 256, to
achieve an accuracy of 98.93% on the test set, resulting in 107 to-be-
explained images being misclassified by the model. Pretrained model weights
are provided to aid reproducibility. The Google QuickDraw dataset contains
images gathered from studies that presented people with common objects, asking
them to draw the object in 20 seconds as a “doodle”. It has 50 million doodle
images covering 345 classes (i.e., common object categories such as “bicycle”
or “helicopter”). The architecture of the classifier was the same as that used
on the MNIST dataset. This CNN was trained on a sample of 35,000 images from 5
categories (i.e., “bicycle”, “giraffe”, “helicopter”, “mushroom”, and “pizza”)
using an Adam optimiser for 10 epochs with a batch size of 256. The model
achieved an accuracy of 97.02% on the test set, resulting in 447 to-be-
explained images which were misclassified.

####  5.1.2 Materials: Comparative Study

The same misclassifications were presented to people in the user-tests and to
the counterfactual methods for the comparative study. For the MNIST dataset,
50 misclassified images were randomly selected from 107 MNIST images
misclassified by the CNN. For the QuickDraw dataset, 30 misclassified
QuickDraw Doodles were randomly selected from 447 QuickDraw images
misclassified by the CNN.

####  5.1.3 Explanation Methods: Comparative Study

Four state-of-the-art counterfactual methods were selected from the literature
on counterfactual XAI [48, 52] based on their (i) popularity as benchmark
methods (i.e., according to citations), (ii) their availability as maintained
open-sourced code (e.g., on GitHub), and (iii) their ability to handle image
data. The selected benchmark methods, previously outlined in Section 2, are:
Min-Edit [99], CEM [27], VLK [96] and Revise [46].

####  5.1.4 Procedure: Comparative Study

The two user tests were run as independent experiments on each of the
datasets, one for MNIST and one for the QuickDraw data. The responses from
these user studies were post-processed to find the mean group responses for
each misclassified item. The same misclassifications were presented to the
four counterfactuals methods and the generated explanations recorded. Pairwise
comparisons between the human explanation for a given misclassification and
the method-generated explanation for that misclassification made by each of
the four counterfactual methods. These pairings were then evaluated with
respect to the six metrics for assessing different aspects of plausibility
(see next subsection). Finally, statistical summaries of the results from the
metrics were collated and analysed using statistical tests where appropriate.

####  5.1.5 Evaluation Measures: Comparative Study

Six evaluation metrics were used to process the pairings of human and machine
explanations (for detailed descriptions see Section 4.3). The metrics were (i)
L1, (ii) L2, (iii) MC-Mean, (iv) MC-Std, (v) IMI, (vi) 10-LOF, (vii) R% Sub
and (viii) Grad-Cos Similarity.

###  5.2 Method: User Tests

####  5.2.1 Participants and Design: User Tests

Forty-seven participants took part in the two user studies: the MNIST Study
(N=42) and QuickDraw Study (N=5). In the MNIST Study, participants were
randomly assigned to two independent groups, the Normal and Min-Edit Groups
(both N=21). This sample size was based on a power analysis designed to
balance the probability of Type I and Type II errors. Using GPOWER [30], for a
two separate one-way, t-tests design, with the assumption of a large effect
size for each (d = .0.8), the power analysis showed that an N of 42 ensured an
alpha of .05 and power of .80. The QuickDraw Study was designed as single-
group pilot to determine whether the findings for the MNIST data generalised
to the, arguably, more-complex QuickDraw dataset. Both studies were reviewed
by the university’s ethics board (ref. LS-E-21-215-Delaney-Keane).
Participants were Computer Science students at UCD and were paid an hourly
rate of €13.00 in accordance with the living wage in the jurisdiction.

####  5.2.2 Apparatus: User Tests

The software tool was developed that allowed images to be edited via a custom
interactive GUI implemented using the tkinter Python package (see Figure 2 for
a screenshot). The presented image was up-sampled to a $600\times 600$ canvas
where it could be edited and the final image was down-sampled to the original
$28\times 28$ size. Participants had the option to add pixels, remove pixels
or reset the image to its original form if they made a mistake. A log of the
stroke information carried out by the user and the final edited image for each
presented image was recorded and saved for later analysis.

####  5.2.3 Procedure: User Tests

In both of the user experiments, all participants were tested in a face-to-
face experiment with a single experimenter (ED) who presented them with task
instructions and the editing tool. After reading the instructions,
participants were given three practice trials to learn how to use the tool.
The software tool presented one misclassified image at a time, along with (i)
a statement about the label it was given by the program and its correct label
given by humans, (ii) a statement about how the image was incorrectly labelled
by the program, and (iii) the explanation-task instructions to edit the image
to help the program correctly label the image. This task asks the user to
counterfactually explain the program’s misclassification to help improve its
subsequent learning. In the MNIST Study, the Normal Group were instructed to
“…make changes or edits to the image, to help the program correctly label the
image…”, whereas the Min-Edit Group were instructed to “…make the smallest
possible changes needed, to help the program correctly label the image…”. The
latter instructions encouraged this group to conform to a key assumption made
by many XAI methods (e.g., the Min-Edit assumption), even though this
requirement seems somewhat artificial in this context. In the QuickDraw Study,
all participants were given the Normal-Group instructions to “…make changes or
edits to the image, to help the program correctly label the image…”. In both
studies, after receiving the instructions and practice trials, participants
proceeded through all the presented images at their own pace. The presented
set of images was randomly shuffled anew for each participant to control for
possible order effects. Each experimental session took $\sim$15-30 minutes
(typically, $\sim$20 min in MNIST Study and $\sim$15 min in QuickDraw Study),
including the final de-briefing on the rationale for the study. The logs of
participant’s stroke information and final edited image for each item were all
recorded and saved after being suitably anonymised.

####  5.2.4 Response Post-Processing: User Tests

In each of the user studies, for a given misclassified item a response from
each participant in the experiment is recorded; so, for MNIST experiment we
have 42 explanations for the first misclassification, 42 for the second and so
on. So, overall 2,250 human explanations were gathered: 42 people x 50 items
for the MNIST experiment and 5 people x 30 items for the QuickDraw experiment.
However, for each of the counterfactual methods we have just one explanation
per misclassification; so, 320 explanations (80 items x 4 methods). So, to
compare the human and machine explanations in a one-to-one fashion, we
computed the centroid of human responses to a given item. This group-level
response was then used in the explanation-to-explanation comparison for each
of the metrics.

###  5.3 Results & Discussion: Comparative Study

Figure 3: Using the MNIST dataset, four misclassified query images and their
corresponding counterfactual explanations generated by four XAI methods (Min-
Edit, VLK, CEM-PN and Revise) and by humans (from the MNIST User Study).

Figure 3 shows some representative data on the types of explanations generated
by people and the fours methods examined; even a cursory glance at these items
shows that the human explanations tend to be more complete and identifiable
instances of the counterfactual class for the misclassified number. These
explanations collected from the two user-tests and those generated by each of
the four methods were comparatively analysed using the six evaluation
measures. The evaluation results are grouped and reported under the headings
of proximity, representativeness and prototypicality. Significantly, these
evaluations do not support the intuition-based claims made for many current
counterfactual methods. In summary, they show:

  * 1.

Proximity Evaluations: the distance measures for machine-generated query-
explanation pairs were significantly closer to one another than the ground
truth, human-generated pairs; human counterfactual explanations are not Min-
Edits of queries, instead humans make large edits to the query when generating
counterfactuals (see Section 5.4).

  * 2.

Representativeness Evaluations: representativeness tests reveal that people’s
counterfactuals are much more representative of the counterfactual class than
the machine-generated explanations; indeed, they are further from the decision
boundary and closer to the centre of the counterfactual class distribution
(see Section 5.5).

  * 3.

Prototype Evaluations: human counterfactual explanations are much closer to
prototypes, computed in the latent space; they tend to sparsely modify high-
level semantic features, in ways that counterfactual methods do not mimic well
(see Section 5.6).

In the following sub-sections we elaborate on the detailed evidence for these
summary findings.

###  5.4 Proximity: Human Counterfactuals are Not Min-Edits

Popular proximity-based counterfactual methods aim to produce Min-Edit
counterfactuals that minimise the distance between the query and explanation
instances, achieving a class change between their predictions [99, 102, 80,
96, 23]. Hence, proponents of this view argue that plausible counterfactual
explanations are ones that deliver lower L1 or L2 distances between the query
and counterfactual. However, this is a proxy evaluation in the absence of a
human ground-truth; here, counterfactual goodness can be directly tested by
comparing query-explanation distance profiles to human ones. Figure 4 shows
that the distance profiles using L1 and L2 norms for the automated methods,
all differ from the human profiles in tests using the MNIST and QuickDraw
data.

(a)

(b)

(c)

(d)

Figure 4: Proximity Evaluations. Mean L1 and L2 distance scores for query-
explanation pairs produced by four counterfactual XAI methods – Min-Edit, CEM-
PN, VLK, Revise – (left of the dotted red line) compared to the human ground-
truth (right of dotted red line), for (a, b) MNIST and (c, d) QuickDraw
datasets (Error bars show standard error of the mean). *Note, Revise failed to
generate counterfactuals for many instances in the QuickDraw dataset (a
coverage deficit also found by [45]); so, Revise’s results only reflect
instances where explanations were found, making its quite poor performance
look better.

For the MNIST data, a statistical analysis, using a one-way ANOVA, of the
distance metrics found a reliable main effect of Group for L1, $F$(4,45)$=$
331.81, $p<0.001$, and L2, $F(4,45)=314.14$, $p<0.001$ (see Figure 4a and 4b).
Pairwise comparisons between the groups shows that the L1 and L2 scores for
three methods (Min-Edit, CEM-PN, VLK) were all significantly lower that those
for humans (all $p<0.001$; using t-tests and a Bonferroni-Holm correction). In
contrast, the Revise method is much closer to the human ground-truth; on L1
its distance scores are higher than human ones ($p<0.001$) but on L2 it is not
reliably different ($p>.05$).

Figure 3 provides a representative sample of data from the MNIST user-test,
revealing how the Revise and human counterfactuals tend to be well-formed
examples of the counterfactual class (intuitively, they appear almost
prototypical). In contrast, the other three methods produce counterfactuals
that are minimal changes to the original query, that are not representative
members of the counterfactual class. In short, when people generate
counterfactuals they do not produce counterfactual explanations that are Min-
Edits of the query, contradicting a key assumption of most popular XAI
methods.

Notably, this pattern of results does not change, even when participants were
specifically instructed to use a Min-Edit strategy. Recall, in the MNIST user-
test, half of the participants were specifically instructed to minimally-edit
the images in contrast to more “normal” instructions (Min-Edit versus Normal
groups). Although, these instructions significantly reduced the distance
scores in the Min-Edit group relative to the Normal group (t-tests, all
$p<0.001$), the human distance scores for query-explanation pairs were all
still significantly different and higher than the machine-generated ones (see
Appendix C for details). So, even when we instruct people to act in a Min-Edit
way, they do not Min-Edit the images to the same degree as the methods do.

For the QuickDraw data, the L1 and L2 distance in the pixel space, show
essentially the same patterns between groups; a one-way ANOVA found a reliable
main effect of Group for L1, $F$(3,26) $=$ 107.03, $p<0.001$, and L2,
$F(3,26)=123.62$, $p<0.001$ (see Figure 4c and 4d). However, as we shall see
later, exploring these distances in the latent space may more appropriate and
informative.

Table 1: Representativeness Evaluations. Four out-of-distribution measures for
the XAI methods (Min-Edit, VLK, CEM-PN and Revise) compared to human responses
for A - MNIST and B - QuickDraw (bold indicates best score in each case).

| MC-Mean | MC-Std | IM1 | 10-LOF | R%-Sub
---|---|---|---|---|---
CF-Method | A | B | A | B | A | B | A | B | A | B
Min-Edit | 0.62 | 0.34 | 0.33 | 0.21 | 1.01 | 1.06 | 0.04 | 0.00 | 42.72 | 41.29
CEM | 0.59 | 0.19 | 0.33 | 0.13 | 1.00 | 1.10 | 0.04 | 0.00 | 43.17 | 41.46
VLK | 0.66 | 0.31 | 0.30 | 0.21 | 1.01 | 1.03 | 0.08 | 0.06 | 49.25 | 45.85
Revise | 0.33 | 0.16 | 0.23 | 0.03 | 1.04 | 0.99 | 0.32 | 0.12 | 45.76 | 49.42
Human | 0.94 | 0.71 | 0.11 | 0.15 | 0.98 | 1.02 | 0.06 | 0.05 | 50.05 | 55.98

###  5.5 Representativeness: Human Explanations are Within Distribution (the
Counterfactual One)

Counterfactual explanations should be within distribution and, to some degree,
representative of the counterfactual class to function as useful explanations.
Hence, methods with better within-distribution scores are to be preferred.
But, how do the within-distribution properties of human explanations compare
to those of machine explanations?

The Monte Carlo Dropout (MC-Mean, MC-Std) [34, 8] metric which measures the
uncertainty in a model’s prediction confidence [56, 89, 26, 8], shows that
human counterfactuals are the least uncertain with respect to the model’s
classification [56, 34], whereas all four XAI methods have lower certainty
scores. Notably, Revise, which was closest to the human counterfactuals on
distance, diverges more than any other method on this measure, indicating that
its explanations are distributionally quite different to the human ground-
truth. In short, humans do not create visual explanations that are close to
the model’s decision boundary (i.e., ones with high aleatoric uncertainty
[89]).

Furthermore, the R%-sub metric [88] shows that human counterfactuals are
prototypical with respect to the counterfactual class; they have the highest
R%-sub scores showing that they are the most representative of the
counterfactual class, relative to class prototypes retrieved using MMD-critic
[57], in contrast to the scores seen for the XAI methods. Finally, the IM1 and
LOF metrics confirm this interpretation of where human counterfactuals sit,
class-wise. IM1 [96], which uses an autoencoder to estimate the reconstruction
error for the counterfactual class, shows that human counterfactuals have the
lowest error relative to all four XAI methods for MNIST. 10-LOF [10], which is
a proximity based out-of-distribution measure in the pixel space [47, 52]
demonstrates that human explanations are more well grounded in the
counterfactual class relative to min-edit counterfactuals.

In short, none of the current XAI methods, whether they be constraint
optimisers, autoencoders or generative models [46, 96, 56, 91, 27], do a good
job of corresponding to the human ground-truth. Human explanations reveal a
tendency to produce counterfactuals that can be quite distant from the query,
while being close to the prototype(s) of the counterfactual class.

Figure 5: Prototype Evaluations. From the QuickDraw data, a query (a
“helicopter” misclassified as a “mushroom”) and the explanations generated by
four XAI methods (as four counterfactual “helicopters”) compared to those
generated by users. Note, how people add “rotor blades”, a semantic-feature,
whereas the automated methods perform minimal pixel changes. Revise fails to
generate an explanation, confirming findings by [45].

###  5.6 Prototypicality: Human Counterfactuals Are More Prototypical

On the face of it, people’s counterfactual explanations for misclassified
QuickDraw Doodles show semantic-features being added, informed by prototypes
in the counterfactual class (i.e., latent features in many CNNs [59, 35, 18,
104]). Figure 5 shows an image of a “helicopter” that was misclassified as a
“mushroom”, to which people add “rotor blades” to identify it as a
“helicopter”555Interestingly, in both user studies people could add pixels or
erase them when producing their explanations, but they tended to add pixels
more often than remove them, echoing known psychological findings. Byrne [13]
noted that “people tend to create counterfactuals about how things could have
been different, that add something new to what they already know about the
situation, rather than ones that remove something from it”. In contrast, the
XAI methods make small changes to a few pixels that imperceptibly modify the
image.

This role of prototypes in counterfactual explanations can be evaluated more
directly by analysing similarities in the latent space. We used the Grad-Cos
similarity metric [16, 42] to compare human counterfactuals to class
prototypes from the counterfactual class (retrieved using MMD-Critic [57]). As
Figure 6 shows, in this latent space, human counterfactuals are more similar
than all four XAI methods, to the prototypes of the counterfactual class.
These results confirm the intuition that people modify the semantic-features
of images in producing counterfactual explanations, shaping these explanations
relative to the prototypes of the counterfactual class.

(a) MNIST

(b) QuickDraw

Figure 6: Prototype Evaluations. Mean Grad-Cos Similarity scores for
counterfactual-prototype and query-prototype pairs (prototypes retrieved using
MMD-critic) from XAI methods compared to the human ground-truth, for (a) MNIST
and (b) QuickDraw datasets (Error bars show standard error of the mean).

Humans generate counterfactual explanations that are more similar to class
prototypes in the latent space, relative to other computational methods for
both MNIST and QuickDraw data-sets. In the case of the QuickDraw dataset, the
original QuickDraw image doodles were produced under a strict time constraint
of 20 seconds, which meant that some doodles were left unfinished. These
unfinished, noisy doodles present significant challenges for the XAI methods,
as they are degraded inputs (and possibly corrupt the ability of the Revise
and VLK generative models to reconstruct the class distribution). However,
people manage these degraded inputs using their knowledge of the semantic
features of the represented objects. Figure 7 shows the case of an unfinished
doodle of a pizza, which the model predicts to be a mushroom. All the XAI
methods fail to generate plausible counterfactuals to this image, yet people
simply add significant missing semantic features, leveraging their knowledge
of class prototypes.

Figure 7: As the original QuickDraw images were created by humans under a 20
second time constraint, some are unfinished, presenting interpretation and
reconstruction difficulties for many XAI models (see insert). In contrast,
people readily fill in missing portions of the image, presumably using their
knowledge of the class prototype (e.g., see similarities to MMD-Critic
prototypes)

##  6 Conclusion

Although many different methods for counterfactual explanation have been
developed in the XAI literature (e.g., [48, 52]), the paucity of user studies
in this field has left most claims of psychological validity untested. The
present work addresses this deficit by gathering human ground-truths on
counterfactual explanation for two benchmark image datasets (i.e., MNIST,
QuickDraw Doodles). It then used benchmark metrics (e.g., on distance,
representativeness and latent similarity) to compare machine-generated
explanations to those that people actually use when explaining
counterfactually. Hence, for the first time, the extent of the divergence
between XAI methods and human, ground-truth performance can be accurately
assessed. The results show that, unlike machine methods, human explanations
focus more on semantic features and are closer to prototypes in the
counterfactual class, than those of the query class. As such, this work
establishes key, novel human benchmarks for assessing the plausibility of XAI
methods. Lewis’ [71] philosophical analysis of counterfactuals argued that
they were the closest possible world to the current one based on minimal
differences. The present work shows that people compute those minimal
differences in a semantic space, rather than a pixel space, and do so with an
eye to the counterfactual world, rather than the current one.

##  7 Data and Code Availability

The MNIST dataset is openly available666http://yann.lecun.com/exdb/mnist/ and
the Google QuickDraw dataset is made available by Google, Inc. under the
Creative Commons Attribution 4.0 International
license777https://github.com/googlecreativelab/quickdraw-dataset. All data
from the user study is made available
888https://github.com/e-delaney/cfe_images_how_people_differ_from_machines/tree/main/User_data
and the code used to detail hyperparamaters and produce our results is also
provided999https://github.com/e-delaney/cfe_images_how_people_differ_from_machines.

##  8 Acknowledgements

This publication has emanated from research conducted with the financial
support of (i) Science Foundation Ireland (SFI) to the Insight Centre for Data
Analytics under Grant Number 12/RC/2289 P2 and (ii) SFI and the Department of
Agriculture, Food and Marine on behalf of the Government of Ireland under
Grant Number 16/RC/3835 (VistaMilk). Thanks also to Greta Warren for the Power
analysis in the literature review of Goyal, and the ideas on subjective and
objective measures.

##  Appendix A Monte Carlo Dropout Technical Details

Following the description of [24, 26], we provide an overview of how MC-
Dropout can be applied. When a predictive distribution $p(y|x,D)$ is obtained,
the corresponding uncertainty can be uncovered by analysing the variance.

From a Bayesian perspective, total predictive uncertainty, ${\mathbb{V}(y\mid
x)}$, can be decomposed into a sum of two components, namely epistemic (model)
uncertainty and aleatoric (data) uncertainty [54, 24]. Let $x$ represent some
input, $y$ the target variable and $\Theta$ the random parameters of the
model, then

${\mathbb{V}(y\mid x)}=\underbrace{\mathbb{V}(\mathbb{E}(y\mid
x,\Theta))}_{Epistemic}+\underbrace{{\mathbb{E}(\mathbb{V}(y\mid
x,\Theta))}}_{Aleatoric}$ (8)

To learn this distribution we learn the distribution over the model parameters
p($\Theta|D$) (i.e., the parametric posterior distribution).

Work in [34] showed that, by randomly switching off neurons in a neural
network using different dropout configurations, one could approximate the
parametric posterior distribution without the need to retrain the network.
Each dropout configuration $\Theta_{t}$ corresponds to a sample from the
approximate parametric posterior distribution $q(\Theta|D)$ s.t.
$\Theta_{t}\sim q(\Theta|D)$.

Sampling from the approximate posterior enables us to uncover the predictive
distribution $p(y\mid x)$:

$p(y\mid x,D)\approx\int_{\Omega}\underbrace{p(y\mid
x,\Theta)}_{likelihood}\underbrace{q(\Theta\mid D}_{posterior})\,d\Theta$ (9)

The above integral can be approximated through Monte Carlo methods, giving;

$p(y\mid x,D)\underbrace{\approx}_{MC}\frac{1}{T}\sum_{t=1}^{T}p(y\mid
x,\Theta_{t})$ (10)

Multiple forward passes with different dropout configurations allow one to
uncover the predictive distribution. Under the assumption that the likelihood
is Gaussian distributed, the mean $f(x,\theta)$ and variance $s^{2}(x,\Theta)$
parameters of the Gaussian function are determined from Monte Carlo simulation
such that $f(x,\theta),s^{2}(x,\Theta)\sim$ MC-Dropout(x), and can yield
useful information about the predictive uncertainty through connecting back
with Equation A.2.

##  Appendix B Prototype Evaluations: Prototypes & Similarity

To determine the closeness of generated counterfactuals to the prototype(s) of
the counterfactual class, MMD-Critic [57] was used to generate prototypes for
the class and then Grad-Cos was used to measure the latent similarity between
explanations and prototypes in order to determine if explanations generated by
humans are more similar to class prototypes relative to explanations that are
automatically generated. MMD-critic is briefly described below.

##### Prototype Retrieval: MMD-Critic

Introduced by Kim et al. [57], this approach computes prototypes by minimizing
the maximum mean discrepancy between the prototype distribution and the data
distribution. These densities are estimated using a kernel density function,
$k$. Following [79, 57], let $m$ represent the number of individual prototypes
$z$ and $n$ represent the number of data-points $x$ in the dataset. Then the
$MMD^{2}$ can be represented by:

$MMD^{2}=\frac{1}{m^{2}}\sum_{i,j=1}^{m}{}k(z_{i},z_{j})-\frac{2}{mn}\sum_{i,j=1}^{m,n}k(z_{i},x_{j})+\frac{1}{n^{2}}\sum_{i,j=1}^{n}{}k(x_{i},x_{j})$
(11)

The first term calculates the average proximity of the prototypes to each
other, while the last term calculates the average proximity of the data-points
to each other. The middle term calculates the average proximity between the
prototypes and the other data-points (multiplied by 2). In our implementation
we use a standard radial basis function as our choice for the kernel $k$,
defined by:

$k(x,x^{\prime})=exp\left(-\gamma\||x-x^{\prime}\||_{2}\right)$ (12)

The $MMD^{2}$ measure, kernel function and greedy search are combined in an
algorithm to find prototypes [79]. Starting with an empty list of prototypes,
each point in the class are evaluated using $MMD^{2}$, and the point that
minimizes $MMD^{2}$ to the largest degree is added to the list.

##  Appendix C Additional User Study Details and Results

Unsurprisingly, users who were instructed to minimally edit the query to
generate a counterfactual explanation, made significantly smaller edits
(according to L1 and L2 distance between the query and generated
counterfactual, two-sample t-test, $p<0.001$) relative to users who were not
given any specific instructions on how to modify the image (See Figure 8).

Figure 8: Proximity Evaluations. Mean L1 and L2 distances between the
misclassified query and the corresponding counterfactual. Explanations
produced by the two human groups – Normal (people who weren’t given any
specific instructions on how to edit the image) & Min-Edit (people who were
specifically instructed to make the smallest edit possible) – for MNIST (error
bars show standard error of the mean).

## References

  * Adadi and Berrada [2018] A. Adadi, M. Berrada, Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access 6 (2018) 52138–52160.
  * Adebayo et al. [2018] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, B. Kim, Sanity checks for saliency maps, in: Advances in Neural Information Processing Systems, 9505–9515, 2018\.
  * Akula et al. [2020] A. Akula, S. Wang, S.-C. Zhu, Cocox: Generating conceptual and counterfactual explanations via fault-lines, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2594–2601, 2020.
  * Anjomshoae et al. [2019] S. Anjomshoae, A. Najjar, D. Calvaresi, K. Främling, Explainable agents and robots: Results from a systematic literature review, in: 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13–17, 2019, International Foundation for Autonomous Agents and Multiagent Systems, 1078–1088, 2019\.
  * Barnett et al. [2021] A. J. Barnett, F. R. Schwartz, C. Tao, C. Chen, Y. Ren, J. Y. Lo, C. Rudin, A case-based interpretable deep learning model for classification of mass lesions in digital mammography, Nature Machine Intelligence 3 (12) (2021) 1061–1070.
  * Barocas et al. [2020] S. Barocas, A. D. Selbst, M. Raghavan, The hidden assumptions behind counterfactual explanations and principal reasons, in: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 80–89, 2020\.
  * Bäuerle et al. [2018] A. Bäuerle, H. Neumann, T. Ropinski, Training De-Confusion: An Interactive, Network-Supported Visual Analysis System for Resolving Errors in Image Classification Training Data, arXiv preprint arXiv:1808.03114 .
  * Bhatt et al. [2021] U. Bhatt, J. Antorán, Y. Zhang, Q. V. Liao, P. Sattigeri, R. Fogliato, G. Melançon, R. Krishnan, J. Stanley, O. Tickoo, et al., Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 401–413, 2021.
  * Birhane et al. [2022] A. Birhane, V. U. Prabhu, J. Whaley, Auditing Saliency Cropping Algorithms, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 4051–4059, 2022\.
  * Breunig et al. [2000] M. M. Breunig, H.-P. Kriegel, R. T. Ng, J. Sander, LOF: identifying density-based local outliers, in: ACM SIGMOD, 93–104, 2000.
  * Buçinca et al. [2020] Z. Buçinca, P. Lin, K. Z. Gajos, E. L. Glassman, Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems, in: Proceedings of the 25th International Conference on Intelligent User Interfaces, 454–464, 2020.
  * Byrne [2007] R. M. Byrne, The rational imagination: How people create alternatives to reality, MIT press, 2007\.
  * Byrne [2019] R. M. Byrne, Counterfactuals in explainable artificial intelligence (XAI): Evidence from human reasoning, in: IJCAI-19, 6276–6282, 2019\.
  * Cai et al. [2019] C. J. Cai, J. Jongejan, J. Holbrook, The effects of example-based explanations in a machine learning interface, in: Proceedings of the 24th international conference on intelligent user interfaces, 258–262, 2019\.
  * Chang et al. [2018] C.-H. Chang, E. Creager, A. Goldenberg, D. Duvenaud, Explaining image classifiers by counterfactual generation, arXiv preprint arXiv:1807.08024 .
  * Charpiat et al. [2019] G. Charpiat, N. Girard, L. Felardos, Y. Tarabalka, Input similarity from the neural network perspective, Advances in Neural Information Processing Systems 32\.
  * Chen et al. [2018] C. Chen, O. Li, A. Barnett, J. Su, C. Rudin, This looks like that: deep learning for interpretable image recognition, arXiv preprint arXiv:1806.10574 .
  * Chen et al. [2020] Z. Chen, Y. Bei, C. Rudin, Concept whitening for interpretable image recognition, Nature Machine Intelligence 2 (12) (2020) 772–782.
  * Connell and Keane [2004] L. Connell, M. T. Keane, What plausibly affects plausibility? Concept coherence and distributional word coherence as factors influencing plausibility judgments, Memory & cognition 32 (2) (2004) 185–197.
  * Connell and Keane [2006] L. Connell, M. T. Keane, A model of plausibility, Cognitive Science 30 (1) (2006) 95–120.
  * Costello and Keane [2000] F. J. Costello, M. T. Keane, Efficient creativity: Constraint-guided conceptual combination, Cognitive Science 24 (2) (2000) 299–349.
  * Dai et al. [2022] X. Dai, M. T. Keane, L. Shalloo, E. Ruelle, R. M. Byrne, Counterfactual explanations for prediction and diagnosis in xai, in: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 215–226, 2022.
  * Dandl et al. [2020] S. Dandl, C. Molnar, M. Binder, B. Bischl, Multi-Objective Counterfactual Explanations, arXiv preprint arXiv:2004.11165 .
  * Davis et al. [2020] J. Davis, J. Zhu, J. Oldfather, S. MacDonald, M. Trzaskowski, Quantifying uncertainty in deep learning systems - An Amazon Web Services Prospective, in: AWS Prespictive Guidance Report, URL https://docs.aws.amazon.com/prescriptive-guidance/latest/ml-quantifying-uncertainty/welcome.html, 2020\.
  * Delaney et al. [2021a] E. Delaney, D. Greene, M. T. Keane, Instance-based counterfactual explanations for time series classification, in: International Conference on Case-Based Reasoning, Springer, 32–47, 2021a.
  * Delaney et al. [2021b] E. Delaney, D. Greene, M. T. Keane, Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions, arXiv preprint arXiv:2107.09734 .
  * Dhurandhar et al. [2018] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, P. Das, Explanations based on the missing: Towards contrastive explanations with pertinent negatives, in: Advances in Neural Information Processing Systems, 592–603, 2018.
  * Dodge et al. [2019] J. Dodge, Q. V. Liao, Y. Zhang, R. K. Bellamy, C. Dugan, Explaining models: an empirical study of how explanations impact fairness judgment, in: Proceedings of the 24th International Conference on Intelligent User Interfaces, 275–285, 2019\.
  * Doshi-Velez and Kim [2017] F. Doshi-Velez, B. Kim, Towards a rigorous science of interpretable machine learning, arXiv preprint arXiv:1702.08608 .
  * Erdfelder et al. [1996] E. Erdfelder, F. Faul, A. Buchner, GPOWER: A general power analysis program, Behavior research methods, instruments, & computers 28 (1) (1996) 1–11.
  * Ford and Keane [2022] C. Ford, M. Keane, Explaining Classifications to Non-Experts: An XAI User Study of Post-Hoc Explanations for a Classifier When People Lack Expertise, in: 26th International Conference on Pattern Recognition, 2022\.
  * Förster et al. [2021] M. Förster, P. Hühn, M. Klier, K. Kluge, Capturing Users’ Reality: A Novel Approach to Generate Coherent Counterfactual Explanations, in: Proceedings of the 54th Hawaii International Conference on System Sciences, 1274, 2021.
  * Förster et al. [2020] M. Förster, M. Klier, K. Kluge, I. Sigler, Evaluating explainable Artifical intelligence–What users really appreciate .
  * Gal and Ghahramani [2016] Y. Gal, Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: international conference on machine learning, 1050–1059, 2016\.
  * Ghorbani et al. [2019] A. Ghorbani, J. Wexler, J. Y. Zou, B. Kim, Towards automatic concept-based explanations, Advances in Neural Information Processing Systems 32\.
  * Goodfellow et al. [2015] I. Goodfellow, J. Shlens, C. Szegedy, Explaining and Harnessing Adversarial Examples, in: International Conference on Learning Representations, 2015.
  * Goodman and Flaxman [2017] B. Goodman, S. Flaxman, European Union regulations on algorithmic decision-making and a “right to explanation”, AI magazine 38 (3) (2017) 50–57.
  * Goyal et al. [2019] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, S. Lee, Counterfactual visual explanations, in: ICML, PMLR, 2376–2384, 2019.
  * Grath et al. [2018] R. M. Grath, L. Costabello, C. L. Van, P. Sweeney, F. Kamiab, Z. Shen, F. Lecue, Interpretable credit application predictions with counterfactual explanations, arXiv preprint arXiv:1811.05245 .
  * Guidotti et al. [2019] R. Guidotti, A. Monreale, F. Giannotti, D. Pedreschi, S. Ruggieri, F. Turini, Factual and counterfactual explanations for black box decision making, IEEE Intelligent Systems 34 (6) (2019) 14–23.
  * Guidotti et al. [2018] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM computing surveys (CSUR) 51 (5) (2018) 93\.
  * Hanawa et al. [2021] K. Hanawa, S. Yokoi, S. Hara, K. Inui, Evaluation of Similarity-based Explanations, in: International Conference on Learning Representations, 2021\.
  * Hendricks et al. [2018] L. A. Hendricks, R. Hu, T. Darrell, Z. Akata, Generating counterfactual explanations with natural language, arXiv preprint arXiv:1806.09809 .
  * Hoffman et al. [2018] R. R. Hoffman, S. T. Mueller, G. Klein, J. Litman, Metrics for explainable AI: Challenges and prospects, arXiv preprint arXiv:1812.04608 .
  * Höltgen et al. [2021] B. Höltgen, L. Schut, J. M. Brauner, Y. Gal, DeDUCE: Generating Counterfactual Explanations Efficiently, arXiv preprint arXiv:2111.15639 .
  * Joshi et al. [2019] S. Joshi, O. Koyejo, W. Vijitbenjaronk, B. Kim, J. Ghosh, Towards realistic individual recourse and actionable explanations in black-box decision making systems, arXiv preprint arXiv:1907.09615 .
  * Kanamori et al. [2020] K. Kanamori, T. Takagi, K. Kobayashi, H. Arimura, DACE: Distribution-Aware Counterfactual Explanation by Mixed-Integer Linear Optimization, in: IJCAI-20, 2855–2862, 2020.
  * Karimi et al. [2020a] A.-H. Karimi, B. Schölkopf, I. Valera, Algorithmic Recourse: from Counterfactual Explanations to Interventions, arXiv preprint arXiv:2002.06278 .
  * Karimi et al. [2020b] A.-H. Karimi, J. von Kügelgen, B. Schölkopf, I. Valera, Algorithmic recourse under imperfect causal knowledge: a probabilistic approach, Advances in Neural Information Processing Systems 33.
  * Kaushik et al. [2020] D. Kaushik, E. Hovy, Z. Lipton, Learning The Difference That Makes A Difference With Counterfactually-Augmented Data, in: International Conference on Learning Representations, URL https://openreview.net/forum?id=Sklgs0NFvr, 2020\.
  * Keane and Kenny [2019] M. T. Keane, E. M. Kenny, How case-based reasoning explains neural networks: A theoretical analysis of XAI using post-hoc explanation-by-example from a survey of ANN-CBR twin-systems, in: Proceedings of the 27th International Conference on Case-Based Reasoning (ICCBR-19), Springer, 155–171, 2019\.
  * Keane et al. [2021] M. T. Keane, E. M. Kenny, E. Delaney, B. Smyth, If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques, arXiv preprint arXiv:2103.01035 .
  * Keane and Smyth [2020] M. T. Keane, B. Smyth, Good Counterfactuals and Where to Find Them: A Case-Based Technique for Generating Counterfactuals for Explainable AI (XAI), in: ICCBR, Springer, 163–178, 2020\.
  * Kendall and Gal [2017] A. Kendall, Y. Gal, What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, in: Advances in Neural Information Processing Systems, vol. 30, 2017.
  * Kenny et al. [2021] E. M. Kenny, C. Ford, M. Quinn, M. T. Keane, Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies, Artificial Intelligence 294 (2021) 1–25.
  * Kenny and Keane [2021] E. M. Kenny, M. T. Keane, On generating plausible counterfactual and semi-factual explanations for deep learning, in: Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI-21), 11575–11585, 2021.
  * Kim et al. [2016] B. Kim, R. Khanna, O. O. Koyejo, Examples are not enough, learn to criticize! criticism for interpretability, in: Advances in Neural Information Processing Systems, 2280–2288, 2016\.
  * Kim et al. [2014] B. Kim, C. Rudin, J. A. Shah, The bayesian case model: A generative approach for case-based reasoning and prototype classification, in: Advances in neural information processing systems, 1952–1960, 2014.
  * Kim et al. [2018] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al., Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav), in: International conference on machine learning, PMLR, 2668–2677, 2018\.
  * Kirfel and Liefgreen [2021] L. Kirfel, A. Liefgreen, What If (and How…)? - Actionability Shapes People’s Perceptions of Counterfactual Explanations in Automated Decision-Making, in: ICML Workshop on Algorithmic Recourse, 2021\.
  * Klaise et al. [2019] J. Klaise, A. Van Looveren, G. Vacanti, A. Coca, Alibi: Algorithms for monitoring and explaining machine learning models, URL https://github.com/SeldonIO/alibi, 2019\.
  * Klein et al. [2021] G. Klein, M. Jalaeian, R. Hoffman, S. Mueller, The Plausibility Gap: A model of sensemaking .
  * Koh and Liang [2017] P. W. Koh, P. Liang, Understanding black-box predictions via influence functions, in: International conference on machine learning, PMLR, 1885–1894, 2017\.
  * Kuhl et al. [2022] U. Kuhl, A. Artelt, B. Hammer, Keep Your Friends Close and Your Counterfactuals Closer: Improved Learning From Closest Rather Than Plausible Counterfactual Explanations in an Abstract Setting, arXiv preprint arXiv:2205.05515 .
  * Lage et al. [2019] I. Lage, E. Chen, J. He, M. Narayanan, B. Kim, S. Gershman, F. Doshi-Velez, An evaluation of the human-interpretability of explanation, arXiv preprint arXiv:1902.00006 .
  * Lagnado et al. [2013] D. A. Lagnado, T. Gerstenberg, R. Zultan, Causal responsibility and counterfactuals, Cognitive science 37 (6) (2013) 1036–1073.
  * Larasati et al. [2020] R. Larasati, A. De Liddo, E. Motta, The effect of explanation styles on user’s trust .
  * Laugel et al. [2019] T. Laugel, M.-J. Lesot, C. Marsala, X. Renard, M. Detyniecki, The dangers of post-hoc interpretability: Unjustified counterfactual explanations, in: IJCAI-19, 2801–2807, 2019\.
  * Leavitt and Morcos [2020] M. L. Leavitt, A. Morcos, Towards falsifiable interpretability research, arXiv preprint arXiv:2010.12016 .
  * LeCun [1998] Y. LeCun, The MNIST database of handwritten digits, http://yann. lecun. com/exdb/mnist/ .
  * Lewis [2013] D. Lewis, Counterfactuals, John Wiley & Sons, 2013.
  * Li et al. [2017] O. Li, H. Liu, C. Chen, C. Rudin, Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions, arXiv preprint arXiv:1710.04806 .
  * Lim et al. [2009] B. Y. Lim, A. K. Dey, D. Avrahami, Why and why not explanations improve the intelligibility of context-aware intelligent systems, in: Proceedings of the SIGCHI conference on human factors in computing systems, 2119–2128, 2009.
  * Lipton [2016] Z. C. Lipton, The mythos of model interpretability, arXiv preprint arXiv:1606.03490 .
  * Lucic et al. [2020] A. Lucic, H. Haned, M. de Rijke, Why does my model fail? contrastive local explanations for retail forecasting, in: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 90–98, 2020\.
  * Lundberg and Lee [2017] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Advances in Neural Information Processing Systems, 4765–4774, 2017.
  * Martens and Provost [2014] D. Martens, F. Provost, Explaining data-driven document classifications, MIS quarterly 38 (1) (2014) 73–100.
  * Miller [2019] T. Miller, Explanation in artificial intelligence: Insights from the social sciences, Artificial Intelligence 267 (2019) 1–38.
  * Molnar [2020] C. Molnar, Interpretable machine learning, Lulu.com, 2020.
  * Mothilal et al. [2020a] R. K. Mothilal, A. Sharma, C. Tan, Explaining machine learning classifiers through diverse counterfactual explanations, in: ACM FAccT, 607–617, 2020a.
  * Mothilal et al. [2020b] R. K. Mothilal, A. Sharma, C. Tan, Explaining machine learning classifiers through diverse counterfactual explanations, in: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 607–617, 2020b.
  * Mueller et al. [2019] S. T. Mueller, R. R. Hoffman, W. Clancey, A. Emrey, G. Klein, Explanation in human-AI systems: A literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI, arXiv preprint arXiv:1902.01876 .
  * Nugent and Cunningham [2005] C. Nugent, P. Cunningham, A case-based explanation system for black-box systems, Artificial Intelligence Review 24 (2) (2005) 163–178.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, C. Guestrin, Why should i trust you?: Explaining the predictions of any classifier, in: ACM SIGKDD, ACM, 1135–1144, 2016\.
  * Ross and Doshi-Velez [2018] A. Ross, F. Doshi-Velez, Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.
  * Rudin [2019] C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1 (5) (2019) 206–215.
  * Russell [2019] C. Russell, Efficient search for diverse coherent explanations, in: Conference on Fairness, Accountability, and Transparency, 20–28, 2019\.
  * Samangouei et al. [2018] P. Samangouei, A. Saeedi, L. Nakagawa, N. Silberman, Explaingan: Model explanation via decision boundary crossing transformations, in: European Conference on Computer Vision (ECCV), 666–681, 2018.
  * Schut et al. [2021] L. Schut, O. Key, R. Mc Grath, L. Costabello, B. Sacaleanu, Y. Gal, et al., Generating interpretable counterfactual explanations by implicit minimisation of epistemic and aleatoric uncertainties, in: International Conference on Artificial Intelligence and Statistics, PMLR, 1756–1764, 2021\.
  * Selvaraju et al. [2017] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, 618–626, 2017\.
  * Singla et al. [2020] S. Singla, B. Pollack, J. Chen, K. Batmanghelich, Explanation by Progressive Exaggeration, in: International Conference on Learning Representations, 2020\.
  * Smyth and Keane [2022] B. Smyth, M. T. Keane, A few good counterfactuals: generating interpretable, plausible and diverse counterfactual explanations, in: International Conference on Case-Based Reasoning, Springer, 18–32, 2022.
  * Sokol and Flach [2019] K. Sokol, P. A. Flach, Counterfactual explanations of machine learning predictions: opportunities and challenges for AI safety, SafeAI@ AAAI .
  * Sørmo et al. [2005] F. Sørmo, J. Cassens, A. Aamodt, Explanation in case-based reasoning–perspectives and goals, Artificial Intelligence Review 24 (2) (2005) 109–143.
  * van der Waa et al. [2021] J. van der Waa, E. Nieuwburg, A. Cremers, M. Neerincx, Evaluating XAI: A comparison of rule-based and example-based explanations, Artificial Intelligence 291 (2021) 103404.
  * Van Looveren and Klaise [2019] A. Van Looveren, J. Klaise, Interpretable counterfactual explanations guided by prototypes, arXiv preprint arXiv:1907.02584 .
  * Verma et al. [2020] S. Verma, J. Dickerson, K. Hines, Counterfactual Explanations for Machine Learning: A Review, arXiv preprint arXiv:2010.10596 .
  * Vermeire et al. [2022] T. Vermeire, D. Brughmans, S. Goethals, R. M. B. de Oliveira, D. Martens, Explainable image classification with evidence counterfactual, Pattern Analysis and Applications (2022) 1–21.
  * Wachter et al. [2017] S. Wachter, B. Mittelstadt, C. Russell, Counterfactual explanations without opening the black box: automated decisions and the GDPR, Harv.J.Law Tech. 31 (2017) 841.
  * Warren et al. [2022a] G. Warren, M. T. Keane, R. M. J. Byrne, Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI, in: IJCAI-ECAI’22 Workshop: Cognitive Aspects of Knowledge Representation, 2022a.
  * Warren et al. [2022b] G. Warren, B. Smyth, M. T. Keane, “Better” Counterfactuals, Ones People Can Understand: Psychologically-Plausible Case-Based Counterfactuals Using Categorical Features for Explainable AI (XAI), in: International Conference on Case-Based Reasoning, Springer, 63–78, 2022b.
  * Wexler et al. [2019] J. Wexler, M. Pushkarna, T. Bolukbasi, M. Wattenberg, F. Viégas, J. Wilson, The what-if tool: Interactive probing of machine learning models, IEEE transactions on visualization and computer graphics 26 (1) (2019) 56–65.
  * Woodward [2005] J. Woodward, Making things happen: A theory of causal explanation, Oxford university press, 2005\.
  * Zhang et al. [2021] R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, B. I. Rubinstein, Invertible concept-based explanations for cnn models with non-negative concept activation vectors, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 11682–11690, 2021\.
  * Zhao et al. [2021] W. Zhao, S. Oyama, M. Kurihara, Generating natural counterfactual visual explanations, in: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 5204–5205, 2021\.
  * Zhou et al. [2016] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: IEEE Conference on Computer Vision and Pattern Recognition, 2921–2929, 2016\.