# The effectiveness of feature attribution methods and its correlation with
automatic evaluation scores

Giang Nguyen
Auburn University
<EMAIL_ADDRESS>
&Daeyoung Kim
KAIST
<EMAIL_ADDRESS>
Anh Nguyen∗
Auburn University
<EMAIL_ADDRESS>
Corresponding authors. Work done when GN was at KAIST.

###### Abstract

Explaining the decisions of an Artificial Intelligence (AI) model is
increasingly critical in many real-world, high-stake applications. Hundreds of
papers have either proposed new feature attribution methods, discussed or
harnessed these tools in their work. However, despite humans being the target
end-users, most attribution methods were only evaluated on proxy automatic-
evaluation metrics [78, 80, 60]. In this paper, we conduct the first user
study to measure attribution map effectiveness in assisting humans in ImageNet
classification and Stanford Dogs fine-grained classification, and when an
image is natural or adversarial (i.e. contains adversarial perturbations).
Overall, feature attribution is surprisingly not more effective than showing
humans nearest training-set examples. On a harder task of fine-grained dog
categorization, presenting attribution maps to humans does not help, but
instead hurts the performance of human-AI teams compared to AI alone.
Importantly, we found automatic attribution-map evaluation measures to
correlate poorly with the actual human-AI team performance. Our findings
encourage the community to rigorously test their methods on the downstream
human-in-the-loop applications and to rethink the existing evaluation metrics.

Keywords: XAI, human evaluation, post-hoc explanations, interpretability,
prototypes

##  1 Introduction

Why did a computer vision system suspect that a person had breast cancer [77],
or was an US capitol rioter [2], or a shoplifter [3]? The explanations for
such high-stake predictions made by existing Artificial Intelligence (AI)
agents can impact human lives in various aspects, from social [22], to
scientific [54], and legal [3, 31, 23]. An image classifier’s decisions can be
explained via an _attribution map_ (AM) [13], i.e. a heatmap that highlights
the input pixels that are important for or against a predicted label. AMs can
be useful in localizing tumors in x-rays [61], reflecting biases [41] and
memory [56] of image classifiers, or teaching humans to distinguish
butterflies [48].

Via AMs, part of the thought process of AIs is now transparent to humans,
enabling human-AI collaboration opportunities. Can machines and humans work
together to improve the accuracy of humans in image classification? This
question remains largely unknown although, in other non-image domains, AMs
have been found useful to humans in the downstream tasks [66, 12, 47, 39].

Furthermore, it is also unknown whether any existing automatic evaluation
scores may predict such effectiveness of AMs to humans, the target user of
AMs. Most AMs were often only evaluated on proxy evaluation metrics such as
pointing game [78], object localization [80], or deletion [60], which may not
necessarily correlate with human-AI team accuracy on a downstream task.

In this paper, we conduct the first user-study to measure AM effectiveness in
assisting humans on the 120-class Stanford Dog [38] fine-grained
classification and 1000-class ImageNet [65] classification, which is the task
that most attribution methods were designed for. We asked 320 lay and 11
expert users to decide whether machine decisions are correct after observing
an input image, top-1 classification outputs, and explanations (Fig. 1(a)).
This task captures two related quantities: (1) human accuracy on image
categorization, here simplified to binary classification; and (2) model
debugging, i.e. understanding whether AI is correct or wrong. For a fuller
understanding of the effectiveness of AMs, we tested users on both real and
adversarial images [75]. Our main findings include:111Code and data are
available at http://github.com/anguyen8/effectiveness-attribution-maps

  1. 1.

AMs are, surprisingly, not more effective than nearest-neighbors (here, 3-NN)
in improving human-AI team performance on both ImageNet and fine-grained dog
classification (Sec. 3.2).

  2. 2.

On fine-grained dog classification, a harder task for a human-AI team than
1000-class ImageNet, presenting AMs to humans interestingly does not help but
instead hurts the performance of human-AI teams, compared to AI alone without
humans (Fig. 2b).

  3. 3.

On adversarial Stanford Dog images which are very hard for humans to label,
presenting confidence scores only to humans is the most effective method
compared to all other explanations tested including AMs and 3-NN examples
(Sec. 3.2).

  4. 4.

Despite being commonly used in the literature, some automatic evaluation
metrics—here, Pointing Game [78], localization errors [80], and IoU
[80]—correlate poorly with the actual human-AI team performance (Sec. 3.5).

  5. 5.

According to a study on 11 AI-expert users, 3-NN is significantly more useful
than GradCAM [67], a state-of-the-art attribution method for ImageNet
classification (Sec. 3.6).

(a) Given an input image, its top-1 predicted label (here, “lorikeet”) and
confidence score (A), we asked the user to decide Yes or No whether the
predicted label is accurate (here, the correct answer is No)—i.e. the accuracy
of human-AI teams _without_ visual explanations. We also compared this
baseline with the treatments where _one_ attribution map (B) or a set of three
nearest neighbors (C) is also provided to the user (in addition to the
confidence score).

Method | ImageNet | Stanford Dogs
---|---|---
$\mu$ | $\sigma$ | $\mu$ | $\sigma$
Confidence | 72.44 | 8.25 | 61.71 | 11.39
GradCAM | 72.58 | 8.11 | 60.56 | 9.27
EP | 73.85 | 6.88 | 56.67 | 10.57
SOD | 72.06 | 7.63 | 61.67 | 10.87
3-NN | 76.08 | 5.86 | 57.20 | 10.58
(b) Users of 3-NN outperform all other users including those using attribution
maps on ImageNet classification. However, on fine-grained dog classification
(Stanford Dogs), visual explanations (GradCAM, EP, 3-NN) surprisingly did not
help but hurt the human performance. Results from testing humans on both real
and adversarial images.

Figure 1: Overview of the methods compared in our user-study and its results.

##  2 Methods

###  2.1 Image classification tasks

To evaluate human-AI team performance on image classification, for each image,
we also presented to a user: (1) an AI’s top-1 predicted label, (2) its
confidence score, and, optionally (3) a visual explanation that is generated
in attempt to explain the predicted label. The user was then asked to decide
whether AI’s predicted labels are correct or not (see Fig. 1(a)). We performed
our experiments on two image datasets of ImageNet and Stanford Dogs, which are
of increasing difficulty.

##### ImageNet

We tested human-AI teams on ImageNet [65]—an image classification task that
most attribution methods were tested on or designed for [80, 67, 7, 60].
ImageNet is a $\sim$1.3M-image dataset spanning across 1000 diverse categories
from natural to man-made entities [65].

##### Stanford Dogs

To test whether our findings generalize to a fine-grained classification task
that is harder to human-AI teams than ImageNet, we repeated the user-study on
the 120-class Stanford Dogs [38] dataset, a subset of ImageNet. This dataset
is challenging due to the similarity between dog species and the large intra-
class variation [38]. Compared to ImageNet, Stanford Dogs is expected to be
harder to human-AI teams because lay users often have significantly less prior
knowledge about fine-grained dog breeds compared to a wide variety of everyday
objects in ImageNet.

###  2.2 Image classifiers

We took ResNet-34 [34] pretrained on ImageNet (73.31% top-1 accuracy) from
torchvision [50] as the target classifier (i.e., the AI) for both ImageNet and
Stanford Dogs classification tasks because the 1000-class ImageNet that the
model was pretrained on includes 120 Stanford Dogs classes. The visual
explanations in this paper were generated to explain this model’s predicted
labels. We chose ResNet-34 because ResNets were widely used in feature
attribution research [67, 28, 60, 7, 45].

###  2.3 Visual explanation methods

To understand the causal effect of adding humans in the loop, we compared the
performance of a standard AI-only system (i.e. no humans involved) and human-
AI teams. In each team, besides an input image and a corresponding top-1
label, humans are also presented with the corresponding confidence score from
the classifier, and a visual explanation (i.e. one heatmap or three 3-NN
images in; see examples from our user-study in Sec. A13).

##### AI-only

A common way to automating the process of accepting or rejecting a predicted
label is via confidence-score thresholding [14]. That is, a top-1 predicted
label is accepted if its associated confidence score is $\geq T$, a threshold.
We found the optimal confidence threshold $T^{*}$ that produces the highest
accepting accuracy on the entire validation set by sweeping across threshold
values $\in\\{0.05,0.10,...,0.95\\}$, i.e. at a 0.05 increment. The optimal
$T^{*}$ values are 0.55 for ImageNet and 0.50 for Stanford Dogs (details in
Sec. A6).

##### Confidence scores only

To understand the impact of visual explanations on human decisions, we also
used a baseline where users are asked to make decisions given no explanations
(i.e. given only the input image, its top-1 predicted label and confidence
score). To our best knowledge, this baseline has not been studied in computer
vision, but has been shown useful in some non-image domains for improving
human-AI team accuracy or user’s trust [79, 12].

##### GradCAM and Extremal Perturbation (EP)

We chose GradCAM [67] and Extremal Perturbation (EP) [28] as two
representatives for state-of-the-art attribution methods (Fig. 1(a)).
Representing for the class of white-box, _gradient_ -based methods [80, 16,
63], GradCAM relies on gradients and the activation map at the last conv layer
of ResNet-34 to compute a heatmap. GradCAM passed a weight-randomization
sanity check [6] and often obtained competitive scores in proxy evaluation
metrics (see Table 1 in [28] and Table 1&2 in [25]).

In contrast, EP is a representative for _perturbation_ -based methods [29, 7,
20]. EP searches for a set of $a$% of input pixels that maximizes the target-
class confidence score. We followed the authors’ best hyperparameters—summing
over four binary masks generated using $a\in\\{2.5,5,10,20\\}$ and a Gaussian
smoothing kernel with an std equal to 9% of the shorter side of the image (see
the code [1]). We used the TorchRay package [5] to generate GradCAM and EP
attribution maps.

We initially also tested vanilla Gradient [70], Integrated Gradient (IG) [74],
and SHAP [46] methods in a pilot study but discarded them from our study since
their AMs are too noisy to be useful to users in a pilot study.

##### Salient object detection (SOD)

To assess the need for heatmaps to explain a _specific classifer_ ’s
decisions, we also considered a classifier-agnostic heatmap baseline. That is,
we used a pre-trained state-of-the-art salient-object detection (SOD) method
called PoolNet [43], which uses a ResNet-50 backbone pre-trained on ImageNet.
PoolNet was trained to output a heatmap that highlights the _salient_ object
in an input image (Fig. 3)— a process that does _not_ take our image
classifier into account. Thus, SOD serves as a strong saliency baseline for
GradCAM and EP.

##### 3-NN

To further understand the pros and cons of AMs, we compared them to a
representative, prototype-based explanation method (e.g. [55, 18, 53]). That
is, for a given input image $\mathbf{x}$ and a predicted label $\mathbf{y}$
(e.g., “lorikeet”), we show the top-3 nearest images to $\mathbf{x}$ by
retrieving them from the same ImageNet training-set class $\mathbf{y}$ (Fig.
1(a)). To compute the distance between two images, we used the $L_{2}$
distance in the feature space of the last conv layer of ResNet-34 classifier
(i.e. $\mathsf{layer4}$ per PyTorch definition [4] or $\mathsf{conv5\\_x}$ in
He et al. [34]). We chose $\mathsf{layer4}$ as our pilot study found it to be
the most effective among the four main conv layers (i.e. $\mathsf{layer1}$ to
$\mathsf{layer4}$) of ResNet-34.

While $k$-NN has a wide spectrum of applications in machine learning and
computer vision [68], the effectiveness of human-AI collaboration using
prototype-based explanations has been rarely studied [64]. To the best of our
knowledge, we provided the first user study that evaluates the effectiveness
of post-hoc, nearest-neighbor explanations (here, 3-NN) on human-AI team
performance.

###  2.4 User-study design

####  2.4.1 Participants

Our user-study experiments were designed and hosted on Gorilla [11]. We
recruited lay participants via Prolific [57] (at $10.2/hr), which is known for
a high-quality user base [59]. Each Prolific participant self-identified that
English is their first language, which is the only demographic filter we used
to select users on Prolific. Participants come from diverse geographic
locations and backgrounds (see Sec. A9). Over the course of two small pilot
studies and one main, large-scale study, we collected in total over 466
complete submissions (each per user) after discarding incomplete ones.

In the main study, after filtering out submissions by validation scores
(described in Sec. 2.4.2), we had 161 and 159 _qualified_ submissions for our
ImageNet and Dogs experiments, respectively. Each of the 5 methods (i.e.
except for AI-only) was experimented by at least 30 users, and each (image,
explanation) pair was seen by at least two users (statistics of users and
trials in Sec. A8).

####  2.4.2 Tasks

Our Gorilla study contains three main sets of screens: (1) Introduction—where
each user is introduced to the study and relevant rules (details in Sec. A11);
(2) Training; and (3) Test. Each user is randomly assigned a set of test
images and _only one_ 222Showing multiple explanation types to the same user
complicates the user-training procedure and also makes the task harder to
users due to context switching. explanation method (e.g. GradCAM heatmaps) to
work with during the entire study.

##### Training

To familiarize with the image classification task (either ImageNet or Stanford
Dogs), users were given five practice questions. After answering each
question, they were shown feedback with groundtruth answers. In each training
screen, we also described each component in a screen via annotations (see
example screens in Sec. A11.2).

##### Validation and Test

After training, each user was asked to answer 40 Yes/No questions in total.
Before each question, a user was provided with a short WordNet [51] definition
of the predicted label and three random training-set images (see an example in
Fig. A5) correctly-classified into the predicted class with a confidence score
$\geq 0.9$. To control the quality of user responses, out of 40 trials, we
used 10 trials as validation cases where we carefully chose the input images
such that we expected participants who followed our instructions to answer
correctly (details in Sec. A11.3). We excluded those submissions that had
below 10/10 and 8/10 accuracy on the 10 validation trials from the ImageNet
and Dogs experiments, respectively. For the remaining (i.e., qualified)
submissions, we used the results of their 30 non-validation trials in our
study.

####  2.4.3 Images

We wish to understand the effectiveness of visual explanations when AIs are
correct vs. wrong, and when AIs face real vs. adversarial examples [75]. For
both ImageNet and Dogs experiments, we used the ResNet-34 classifier to sample
three types of images: (1) correctly-classified, real images; (2)
misclassified, real images; and (3) adversarial images (i.e. also
misclassified). In total, we used 3 types $\times$ 150 images = 450 images per
dataset. Each image was then used to generate model predictions and
explanations for comparing the 6 methods described in Sec. 2.3.

Filtering From the 50K-image ImageNet validation set, we sampled images for
both ImageNet and Dogs experiments. To minimize the impact of low-quality
images to users’ performance, we removed all 900 grayscale images and 897
images that have either width or height smaller than 224 px, leaving 48,203
and 5,881 images available for use in our ImageNet and Dogs experiments,
respectively. For Dogs, we further excluded all 71 dog images mislabeled by
ResNet-34 into non-dog categories (examples in Sec. A12) because they can
trivialize explanations, yielding 5,810 Dogs images available for selection
for the study.

Selecting natural images To understand human-AI team performance at varying
levels of difficulty for humans, we randomly divided images from the pool of
filtered, real images into three sets: Easy (E), Medium (M), and Hard (H).

Hard images are those correctly labeled by the classifier with a low
confidence score (i.e., $\in[0,0.3)$) and mislabeled with high confidence
(i.e., $\in[0.8,1.0]$). Vice versa, the Easy set contains those correctly
labeled with high confidence and mislabeled with low confidence. The Medium
set contains both correctly and incorrectly labeled images with a confidence
score $\in[0.4,0.6)$ i.e. when the AI is unsure (see confidence-score
distributions in Sec. A7). In each set (E/M/H), we sampled 50 images
correctly-labeled and 50 mislabeled by the model. In sum, per dataset, there
are 300 natural images divided evenly into 6 _controlled_ bins (see Fig. A1
for the ratios of these bins in the original datasets).

Generating adversarial images After the filtering above, we took the remaining
real images to generate adversarial examples via Foolbox [62] for the
ResNet-34 classifier using the Project Gradient Descent (PGD) framework [49]
with an $L_{\infty}$ bound $\epsilon=16/255$ for 40 steps, each of size of
$1/30$. We chose this setup because at weaker attack settings, most
adversarial images became _correctly_ classified after being saved as a JPEG
file [44], defeating the purpose of causing AIs to misbehave. Here, for each
dataset, we randomly sampled 150 adversarial examples (e.g., the input image
in Fig. 1(a)) that are _misclassified_ at the time presented to users in the
JPEG format and often contain so small artifacts that we assume to not bias
human decisions. Following the natural-image sampling, we also divided the 150
adversarial images into three sets (E/M/H), each containing 50 images.

###  2.5 Automatic evaluation metrics for attribution maps

Many attribution methods have been published; however, most AMs were not
tested on end-users but instead only assessed via proxy evaluation metrics. We
aim to measure the correlation between three common metrics—Pointing Game
[78], Intersection over Union (IoU) [80], and weakly-supervised localization
(WSL) [80]—with the actual human-AI team performance in our user study.

All three metrics are based on the assumption that an AM for explaining a
predicted label $\mathbf{y}$ should highlight an image region that overlaps
with a human-annotated bounding box (BB) for that category $\mathbf{y}$. For
each of 300 real, correctly-classified images from each dataset (described in
Sec. 2.4.3), we obtained its human-annotated BB from ILSVRC 2012 [65] for
using in the three metrics.

Pointing game [78] is a common metrics often reported in the literature (e.g.
[67, 28, 63, 24, 60, 76]). For an input image, a generated AM is considered a
“correct” explanation (i.e. a _hit_) if its highest-intensity point lies
inside the human-annotated BB. Otherwise, it is a _miss_. The accuracy for an
explanation method is computed by averaging over all images. We used the
TorchRay implementation of Pointing Game [5] and its default hyperparameters
(tolerance = 15).

Intersection over Union The idea is to compute the agreement in Intersection
over Union (IoU) [80] between a human-annotated BB and a binarized AM. A
heatmap was binarized at the method’s optimal threshold $\alpha$, which was
found by sweeping across values $\in\\{0.05x\mid
x\in{\mathbb{N}}\text{~{}and~{}}0<x<20\\}$.

Weakly-supervised localization (WSL) is based on IoU scores [80]. WSL counts a
heatmap correct if its binarized version has a BB that overlaps with the
human-labeled BB at an IoU $>0.5$. WSL is also commonly used e.g. [7, 67, 24].
For both GradCAM and EP, we found the binarization threshold $\alpha=0.05$
corresponds to their best WSL scores on 150 correctly-classified images (i.e.,
excluding mislabeled images because human-annotated BBs are for the
groundtruth labels).

##  3 Results

###  3.1 3-NN is more effective than AMs on ImageNet but both hurt user
accuracy on Dogs

We measure the per-user accuracy on ImageNet classification over our
controlled image sets (correctly-labeled, incorrectly-labeled, and adversarial
images; see Sec. 2.4.3), i.e. on both natural and adversarial images. We
repeated the same study for Stanford Dogs (see Table 1(b)).

ImageNet We found that 3-NN is significantly better than SOD and GradCAM (Mann
Whitney U-test; $p<0.035$). The differences among feature attribution methods
(GradCAM, SOD, EP) are _not_ statistically significant. In terms of mean and
standard deviation of per-user accuracy, users of 3-NN outperformed all other
users by a large margin of $\sim$3% (Table 1(b); 76.08% and 5.86%).

Stanford Dogs We found that users of 3-NN and EP performed statistically
significantly worse than the users with no explanations (Table 1(b);
Confidence) and SOD users (Mann Whitney U-test; $p<0.024$). There are no
significant differences among other pairs of methods. That is, interestingly,
on fine-grained Stanford Dog classification where an image may be real or
adversarial, visual explanations (i.e. both 3-NN and AMs) _hurt_ user
classification accuracy.

###  3.2 On natural images, how effective are attribution maps in human-AI
team image classification?

We wish to understand the effectiveness of AMs (GradCAM and EP) compared to
four baselines (AI-only, Confidence, SOD, and 3-NN) in image classification by
human-AI teams. Here, we compare 6 methods on 2 _natural_ -image sets:
ImageNet and Stanford Dogs (i.e. no adversarial images).

##### Experiment

Because a given image can be mapped to one of the 6 controlled bins as
described in Sec. 2.4.3, for each of the 6 methods, we computed its human-AI
team accuracy for each of the 6 bins where each bin has exactly 50 images
(Sec. A1 reports per-bin accuracy scores). To estimate the overall accuracy of
a method on the _original_ ImageNet and Dogs dataset, we computed the weighted
sum of its per-bin accuracy scores where the weights are the frequencies of
images appearing in each bin in practice. For example, 70.93% of the Dogs
images are correctly-classified with a high confidence score (Fig. A1; Easy
Correct).

(a) (b) (c) (d)

Figure 2:  3-NN is consistently among the most effective in improving human-AI
team accuracy (%) on natural ImageNet images (a), natural Dog images (b), and
adversarial ImageNet images (c). On the challenging adversarial Dogs images
(d), using confidence scores only helps humans the most in detecting AI’s
mistakes compared to using confidence scores and one visual explanation. Below
each sample dataset image is the top-1 predicted label (which was correct or
wrong) from the classifier.

ImageNet results On ImageNet, human-AI teams where humans use heatmaps
(GradCAM, EP, or SOD) and confidence scores outperformed AI-only by an
absolute gain of $\sim$6–8% in accuracy (Fig. 2a; 80.79% vs. 88.77%). That is,
when teaming up, humans and AI together achieve a better performance than AI
alone. Interestingly, only half of such improvement can be attributed to the
heatmap explanations (Fig. 2a; 84.79% vs. 88.77%). That is, users when
presented with the input image and the classifier’s top-1 label and confidence
score already obtained $+$4% boost over AI alone (84.79% vs. 80.79%).

Dogs results Interestingly, the trend did not carry over to fine-grained dog
classification. On average, humans when presented with (1) confidence scores
only or (2) confidence scores with heatmaps all _underperformed_ the accuracy
of AI-only (Fig. 2b; 81.14% vs. 76.45%). An explanation is that ImageNet
contains $\sim$50% of man-made entities, which contain many everyday objects
that users are familiar with. Therefore, the human-AI teaming led to a
substantial boost on ImageNet. In contrast, most lay users are not dog experts
and therefore do not have the prior knowledge necessary to help them in dog
identification, resulting in even worse performance than a trained AI (Fig.
2b; 81.14% vs. 76.45%). Interestingly, when providing users with nearest
neighbors, human-AI teams with 3-NN outperformed all other methods (Fig. 2b;
82.88%) including the AI-only.

Both datasets On both ImageNet and Dog distributions, 3-NN is among the most
effective. On average over 6 controlled bins, 3-NN also outperformed all other
methods by a clear margin of 2.71% (Sec. A2 & A4). Interestingly, SOD users
tend to reject more often than other users (Sec. A10), inadvertently causing a
high human-AI team accuracy on AI-misclassified images (Sec. A14.5).

###  3.3 On adversarial images, how effective are attribution maps in human-AI
team image classification?

As adversarial examples are posing a big threat to AI-only systems [8, 9, 58,
52], here, we are interested in understanding how effective explanations are
in improving human-AI team performance over AI-only, which is assumed to be 0%
accurate under adversarial attacks. That is, on ImageNet and Dogs, we compare
the accuracy of human-AI teams on 150 adversarial examples, which all caused
the classifier to misclassify. Note that a user was given a random mix of
natural and adversarial test images and was not informed whether an input
image is adversarial or not.

Adversarial ImageNet On both natural and adversarial ImageNet, AMs are on-par
or more effective than showing confidence scores alone (Fig. 2c). Furthermore,
the effect of 3-NN is a consistent +4% gain compared to Confidence (Fig. 2a &
c; Confidence vs. 3-NN). Aligned with the results on natural ImageNet and
Dogs, 3-NN remains the best method on adversarial ImageNet (Fig. 2c; 75.07%).
See Sec. A14.4 for adversarial images that were only correctly-rejected by
3-NN users but not others.

Adversarial Dogs Interestingly, on adversarial Dogs, adding an explanation
(either 3-NN or a heatmap) tend to cause users to agree with the model’s
incorrect decisions (Fig. 2d). That is, heatmaps often only highlight a coarse
body region of a dog without pinpointing an exact feature that might be
explanatory. Similarly, 3-NN often shows examples of an almost identical breed
to the groundtruth (e.g. mountain dog vs. Gordon setter), which is hard for
lay-users to tell apart (see qualitative examples in Sec. A14.6).

###  3.4 On ImageNet, why is 3-NN more effective than attribution maps?

Analyzing the breakdown accuracy of 3-NN in each controlled set of Easy,
Medium, and Hard, we found 3-NN to be the most effective method in the Easy
and Hard categories of ImageNet (Table A3).

Easy When AI mislabels with low confidence, 3-NN often presents contrast
evidence showing that the predicted label is incorrect, i.e. the nearest
examples from the predicted class is distinct from the input image (Fig. 1(a);
lorikeets have a distinct blue-orange pattern not found in bee eaters). The
same explanation is our leading hypothesis for 3-NN’s effectiveness on
adversarial Easy, ImageNet images. More examples in Sec. A14.3.

Hard Hard images often contain occlusions (Figs. A9 & A12), unusual instances
of common objects (Fig. A13), or could reasonably be in multiple categories
(Figs. A11 & A10). When the classifier is _correct_ but with low confidence,
we found 3-NN to be helpful in providing extra evidence for users to confirm
AI’s decisions (Fig. 3). In contrast, heatmaps often only highlight the main
object regardless of whether AI is correct or not (Fig. 1(a); GradCAM). More
examples are in Sec. A14.1.

Medium Interestingly, 3-NN was the best method on the Easy and Hard set, but
not on the Medium (Table A3). Upon a closer look, we surprisingly found
$\sim$63% of misclassified images by 3-NN human-AI teams to have debatable
groundtruth ImageNet labels (see Sec. A14.2 for failure cases of 3-NN).

(a) “african hunting dog”: dog-like mammals of Africa

(b) “caldron”: a very large pot used for boiling

Figure 3:  Hard images that were only correctly accepted by 3-NN users but not
other users of GradCAM, EP, or SOD. Despite the animals in the input image are
partially occluded, 3-NN provided closed-up examples of african hunting dogs,
enabling users to correctly decide the label (a). Choosing a single label for
a scene of multiple objects is challenging. However, 3-NN was able to retrieve
a nearest example showing a very similar scene enabling users to accept AI’s
correct decisions (b).
See full-res images in Figs. A9 & A10.

###  3.5 How do automatic evaluation scores correlate with human-AI team
performance?

IoU [80], WSL [80], and Pointing Game [78] are three attribution-map
evaluation metrics commonly used in the literature, e.g. in [60, 28, 29, 7,
67]. Here, we measure the correlation between these scores and the actual
human-AI team accuracy to assess how high-performance on such benchmarks
translate into the real performance on downstream tasks.

Experiment We took the EP and GradCAM heatmaps of the 150 real images that
were correctly-classified (see Sec. 2.4.3) from each dataset and computed
their IoU, WSL, and Pointing Game scores333Note that WSL and Pointing Game
scores are binary while IoU scores are real-valued. (see Sec. 2.5). We
computed the Pearson correlation between these scores and the human-AI team
accuracy obtained in the previous human-study (Sec. 2.4).

Results While EP and GradCAM are state-of-the-art methods under Pointing Game
[28] or WSL [29], we surprisingly found the accuracy of users when using these
AMs to correlate poorly with the IoU, WSL, and Pointing Game scores of
heatmaps. Only in the case of GradCAM heatmaps for ImageNet (Fig. 4(a)), the
evaluation metrics showed a small positive correlation with human-AI team
accuracy ($r=0.22$ for IoU; $r=0.15$ for WSL; and $r=0.21$ for Pointing Game).
In all other cases, i.e. GradCAM on Dogs; EP on ImageNet and Dogs (Fig.
4(b)–c), the correlation is negligible ($-0.12\leq r\leq 0.07$). That is, the
pointing or localization accuracy of feature attribution methods do not
necessarily reflect their effectiveness in helping users make correct
decisions in image classification.

(a) GradCAM’s IoU

(b) EP’s IoU

(c) EP’s Pointing Game

Figure 4:  The ImageNet localization performance (here, under IoU vs. human-
annotated bounding boxes) of GradCAM [67] and EP [28] attribution maps poorly
correlate with the human-AI team accuracy (y-axis) when users use these
heatmaps in image classification. Humans often can make correct decisions
despite that heatmaps poorly localize the object (see the range 0.0–0.2 on
x-axis).

###  3.6 Machine learning experts found Nearest-Neighbors more effective than
GradCAM

We have found in our previous study on lay-users that AMs can be useful to
human-AI teams, but not more than a simple 3-NN in most cases (Secs. 3.1–3.4).
Here, we aim to evaluate whether such conclusions carry over to the case of
machine learning (ML) expert-users who are familiar with feature attribution
methods, nearest-neighbors, and ImageNet. Because ML experts have a deeper
understanding into these algorithms than lay-users, our expert-study aims to
measure the utility of these two visual explanation techniques to ML
researchers and practitioners.

As GradCAM was the best attribution method in the lay-user ImageNet study
(Fig. 2), we chose GradCAM as the representative for attribution maps and
compare it with 3-NN on human-AI team image classification on ImageNet.

Experiment We repeated the same lay-user study but on a set of 11 expert users
who are Ph.D. students and postdoctoral researchers in ML and computer vision
and from a range of academic institutions. We recruited five GradCAM users and
six 3-NN users who are very familiar with feature attribution and nearest-
neighbor algorithms, respectively. Similar to the lay-user study, each expert
is presented with a set of randomly-chosen 30 images which include both
natural and adversarial images.

Results The experts working with 3-NN performed substantially better than the
GradCAM experts (Table 1; mean accuracy of 76.67% vs. 68.00%). 3-NN is
consistently more effective on both natural and adversarial image subsets.
Interestingly, the performance of GradCAM users also vary 3$\times$ more
(standard deviation of 8.69% vs. 2.98% in accuracy). Aligned with the lay-user
study, here, we found 3-NN to be more effective than AMs in improving human-AI
team performance where the users are domain experts familiar with the
mechanics of how explanations were generated.

Table 1:  3-NN is far more effective than GradCAM in human-AI team image classification of both natural and adversarial images. See the mean ($\mu$) and std ($\sigma$) in per-user accuracy over all images.  | Users | Avg. validation | Natural | Adversarial | $\mu$ | $\sigma$
---|---|---|---|---|---|---
accuracy | Accuracy | Trials | Accuracy | Trials
GradCAM | 5 | 9.80/10 | 67.31 | 70/104 | 69.57 | 32/46 | 68.00 | 8.69
3-NN | 6 | 9.83/10 | 78.45 | 91/116 | 73.44 | 47/64 | 76.67 | 2.98

##  4 Related Work

##### Attribution maps improved human accuracy in non-image domains

Our work is motivated by prior findings showing that AMs helped humans make
more accurate decisions on non-image classification tasks including predicting
book categories from text [66], movie-review sentiment analysis [66, 12],
predicting hypoxemia-risk from medical tabular data [47], or detecting fake
text reviews [40, 39]. In contrast, on house-price prediction, AMs showed
marginal utility to users and even hurt their performance in some cases [21].

##### Evaluating confidence scores on humans

AI confidence scores have been found to improve user’s trust on AI’s decisions
[79] and be effective in human-AI team prediction accuracy on several NLP
tasks [12]. In this work, we do not measure user trust but only the
complementary human-AI team performance. To the best of our knowledge, our
work is the first to perform such human-evaluation of AI confidence scores for
image classification.

Evaluating explanations on humans In NLP, using attribution methods as a word-
highlighter has been found to improve user performance in question answering
[26]. Bansal et al. [12] found that such human-AI team performance improvement
was because the explanations tend to make users more likely to accept AI’s
decisions regardless of whether the AI is correct or not. We found consistent
results that users with explanations tend to accept AI’s predicted labels more
(see Sec. A10) with the exception of SOD users who tend to reject more
(possibly due to the prediction-agnosticity of SOD heatmaps).

In image classification, Chu et al. [19] found that presenting IG heatmaps to
users did not significantly improve human-accuracy on predicting age from
facial photos. Different from [19], our study tested multiple attribution
methods (GradCAM, EP, SOD) and on the ImageNet classification task, which most
attribution methods were evaluated on under proxy metrics (e.g. localization).

Shen and Huang [69] showed users GradCAM [67], EP [28], and SmoothGrad [72]
heatmaps and measured user-performance in harnessing the heatmaps to identify
a label that an AI incorrectly predicts. While they measured the effect of
showing all three heatmaps to users, we compared each method separately
(GradCAM vs. EP vs. SOD) in a larger study. Similar to [69], Alqaraawi et al.
[10] and Folke et al. [27] tested the capability of attribution maps in
helping users understand the decision-making process of AI. Our work differs
from the above three papers [27, 69, 10] and similar studies in the text
domain [15, 33] in that we did not only measure how well users understand AIs,
but also the human-AI team accuracy on a standard _downstream_ task of
ImageNet.

Using AMs to debug models, Adebayo et al. [6] found that humans heavily rely
on predicted labels rather than AMs, which is strongly aligned with our
observations on the Natural ImageNet (in Sec. 3.2) where 3-NN outperforms AMs.

Adversarial attacks Concurrently, Folke et al. [27] also perform a related
study that nearest-neighbor measures the effectiveness of both AMs and
examples in helping users predict when AIs misclassify _natural adversarial_
images [37], which we do not study in this paper. Our work largely differs
from [27] in both the scale of the study, the type of image, and the
experiment setup. To our knowledge, we are the first to evaluate human users
on the well-known adversarial examples [49]. Previous work also found that it
is easy to minutely change the input image such that the gradient-based AMs
radically change [30]. However, none of such prior attempts studied the impact
of their attacks to the actual end-users of the heatmaps.

##  5 Discussion and Conclusion

Limitations An inherent limitation of our study is that it is not possible to
control the amount of prior knowledge that a participant has _before_ entering
the study. For example, a human with a strong dog expertise may perform better
at the fine-grained dog classification. In that case, the utility of
explanations is unknown in our study. We attempted to estimate the effect of
prior knowledge to human-AI team accuracy by asking each user whether they
know a class before each trial. We found prior knowledge to account for
$\sim$1-6% in accuracy (Sec. A5). Due to COVID and the large participant pool,
our study was done online; which, however, made it infeasible for us to
control various physical factors (e.g. user performing other activities during
the experiment) compared to a physical in-lab study. As most prior AM studies,
we do consider state-of-the-art methods for training humans on explanations
[71, 73] partly because each explanation method may need a different optimal
training procedure.

To our knowledge, our work is the first to (1) evaluate human-AI team
performance on the common ImageNet classification; (2) assess explanations on
adversarial examples; (3) reveal the weak correlation between automatic
evaluation metrics (Pointing Game, IoU [80], and WSL [80]) and the actual team
performance. Such poor correlation encourages future interpretability research
to take humans into their evaluation and to rethink the current automatic
metrics [42, 22]. We also showed the first evidence in the literature that a
simple 3-NN can outperform existing attribution maps, suggesting a combination
of two explanation types may be useful to explore in future work. The
superiority of 3-NN also suggests prototypical [17, 32] and visually-grounded
explanations [35, 36] may be more effective than heatmaps.

## Acknowledgement

This research was supported by the MSIT Grant No. IITP-2021-2020-0-01489, and
the Technology Innovation Program of MOTIE Grant No. 2000682. AN was supported
by the NSF Grant No. 1850117 and a donation from NaphCare Foundation.

## References

  * EP [2] Torchray/attribution_benchmark.py at master · facebookresearch/torchray.  https://github.com/facebookresearch/TorchRay/blob/master/examples/attribution_benchmark.py.  (Accessed on 04/28/2021).
  * [2] The fbi’s capitol riot investigation used surveillance technology that advocates say threatens civil liberties - the washington post.  https://www.washingtonpost.com/technology/2021/04/02/capitol-siege-arrests-technology-fbi-privacy/.  (Accessed on 04/11/2021).
  * [3] The new lawsuit that shows facial recognition is officially a civil rights issue | mit technology review.  https://www.technologyreview.com/2021/04/14/1022676/robert-williams-facial-recognition-lawsuit-aclu-detroit-police/.  (Accessed on 04/15/2021).
  * [4] vision/resnet.py at master · pytorch/vision.  https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L183.  (Accessed on 04/28/2021).
  * [5] facebookresearch/torchray: Understanding deep networks via extremal perturbations and smooth masks.  https://github.com/facebookresearch/TorchRay.  (Accessed on 04/28/2021).
  * Adebayo et al. [2018] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B.  Sanity checks for saliency maps.  _arXiv preprint arXiv:1810.03292_ , 2018.
  * Agarwal & Nguyen [2020] Agarwal, C. and Nguyen, A.  Explaining image classifiers by removing input features using generative models.  In _Proceedings of the Asian Conference on Computer Vision_ , 2020\.
  * Agarwal et al. [2019] Agarwal, C., Nguyen, A., and Schonfeld, D.  Improving robustness to adversarial examples by encouraging discriminative features.  In _2019 IEEE International Conference on Image Processing (ICIP)_ , pp. 3801–3505. IEEE, 2019.
  * Alcorn et al. [2019] Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W.-S., and Nguyen, A.  Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4845–4854, 2019.
  * Alqaraawi et al. [2020] Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., and Berthouze, N.  Evaluating saliency map explanations for convolutional neural networks: a user study.  In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , pp. 275–285, 2020.
  * Anwyl-Irvine et al. [2020] Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., and Evershed, J. K.  Gorilla in our midst: An online behavioral experiment builder.  _Behavior research methods_ , 52(1):388–407, 2020\.
  * Bansal et al. [2020a] Bansal, G., Wu, T., Zhu, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. S.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance.  _arXiv preprint arXiv:2006.14779_ , 2020a.
  * Bansal et al. [2020b] Bansal, N., Agarwal, C., and Nguyen, A.  Sam: The sensitivity of attribution methods to hyperparameters.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8673–8683, 2020b.
  * Bendale & Boult [2016] Bendale, A. and Boult, T. E.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1563–1572, 2016.
  * Chandrasekaran et al. [2018] Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., and Parikh, D.  Do explanations make vqa models more predictable to a human?  _arXiv preprint arXiv:1810.12366_ , 2018.
  * Chattopadhay et al. [2018] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 839–847. IEEE, 2018.
  * Chen et al. [2018] Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., and Rudin, C.  This looks like that: deep learning for interpretable image recognition.  _arXiv preprint arXiv:1806.10574_ , 2018.
  * Chen et al. [2019] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K.  This looks like that: Deep learning for interpretable image recognition.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/adf7ee2dcf142b0e11888e72b43fcb75-Paper.pdf.
  * Chu et al. [2020] Chu, E., Roy, D., and Andreas, J.  Are visual explanations useful? a case study in model-in-the-loop prediction.  _arXiv preprint arXiv:2007.12248_ , 2020.
  * Covert et al. [2020] Covert, I., Lundberg, S., and Lee, S.-I.  Feature removal is a unifying principle for model explanation methods.  _arXiv preprint arXiv:2011.03623_ , 2020.
  * Dinu et al. [2020] Dinu, J., Bigham, J., and Kolter, J. Z.  Challenging common interpretability assumptions in feature attribution explanations.  _arXiv preprint arXiv:2012.02748_ , 2020.
  * Doshi-Velez & Kim [2017] Doshi-Velez, F. and Kim, B.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ , 2017.
  * Doshi-Velez et al. [2017] Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S., O’Brien, D., Schieber, S., Waldo, J., Weinberger, D., and Wood, A.  Accountability of ai under the law: The role of explanation.  _arXiv preprint arXiv:1711.01134_ , 2017.
  * Du et al. [2018] Du, M., Liu, N., Song, Q., and Hu, X.  Towards explanation of dnn-based prediction with guided feature inversion.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1358–1367, 2018.
  * Elliott et al. [2019] Elliott, A., Law, S., and Russell, C.  Perturbations on the perceptual ball.  _arXiv preprint arXiv:1912.09405_ , 2019.
  * Feng & Boyd-Graber [2019] Feng, S. and Boyd-Graber, J.  What can ai do for me? evaluating machine learning interpretations in cooperative play.  In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ , pp. 229–239, 2019.
  * Folke et al. [2021] Folke, T., Li, Z., Sojitra, R. B., Yang, S. C.-H., and Shafto, P.  Explainable ai for natural adversarial images.  _arXiv preprint arXiv:2106.09106_ , 2021.
  * Fong et al. [2019] Fong, R., Patrick, M., and Vedaldi, A.  Understanding deep networks via extremal perturbations and smooth masks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2950–2958, 2019.
  * Fong & Vedaldi [2017] Fong, R. C. and Vedaldi, A.  Interpretable explanations of black boxes by meaningful perturbation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 3429–3437, 2017.
  * Ghorbani et al. [2019] Ghorbani, A., Abid, A., and Zou, J.  Interpretation of neural networks is fragile.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 3681–3688, 2019.
  * Goodman & Flaxman [2017] Goodman, B. and Flaxman, S.  European union regulations on algorithmic decision-making and a “right to explanation”.  _AI Magazine_ , 38(3):50–57, 2017.
  * Goyal et al. [2019] Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., and Lee, S.  Counterfactual visual explanations.  In _International Conference on Machine Learning_ , pp. 2376–2384. PMLR, 2019.
  * Hase & Bansal [2020] Hase, P. and Bansal, M.  Evaluating explainable ai: Which algorithmic explanations help users predict model behavior?  _arXiv preprint arXiv:2005.01831_ , 2020.
  * He et al. [2016] He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendricks et al. [2016] Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., and Darrell, T.  Generating visual explanations.  In _European Conference on Computer Vision_ , pp. 3–19. Springer, 2016.
  * Hendricks et al. [2018] Hendricks, L. A., Hu, R., Darrell, T., and Akata, Z.  Generating counterfactual explanations with natural language.  _arXiv preprint arXiv:1806.09809_ , 2018.
  * Hendrycks et al. [2021] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15262–15271, 2021.
  * Khosla et al. [2011] Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L.  Novel dataset for fine-grained image categorization: Stanford dogs.  In _First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition_ , Colorado Springs, CO, June 2011.
  * Lai & Tan [2019] Lai, V. and Tan, C.  On human predictions with explanations and predictions of machine learning models: A case study on deception detection.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pp. 29–38, 2019.
  * Lai et al. [2020] Lai, V., Liu, H., and Tan, C.  " why is’ chicago’deceptive?" towards building model-driven tutorials for humans.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pp. 1–13, 2020.
  * Lapuschkin et al. [2016] Lapuschkin, S., Binder, A., Montavon, G., Muller, K.-R., and Samek, W.  Analyzing classifiers: Fisher vectors and deep neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2912–2920, 2016.
  * Leavitt & Morcos [2020] Leavitt, M. L. and Morcos, A.  Towards falsifiable interpretability research.  _arXiv preprint arXiv:2010.12016_ , 2020.
  * Liu et al. [2019a] Liu, J.-J., Hou, Q., Cheng, M.-M., Feng, J., and Jiang, J.  A simple pooling-based design for real-time salient object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3917–3926, 2019a.
  * Liu et al. [2019b] Liu, Z., Liu, Q., Liu, T., Xu, N., Lin, X., Wang, Y., and Wen, W.  Feature distillation: Dnn-oriented jpeg compression against adversarial examples.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 860–868. IEEE, 2019b.
  * Lu et al. [2020] Lu, Y., Guo, W., Xing, X., and Stafford Noble, W.  Robust decoy-enhanced saliency maps.  _arXiv e-prints_ , pp. arXiv–2002, 2020.
  * Lundberg & Lee [2017] Lundberg, S. and Lee, S.-I.  A unified approach to interpreting model predictions.  _arXiv preprint arXiv:1705.07874_ , 2017.
  * Lundberg et al. [2018] Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  _Nature biomedical engineering_ , 2(10):749–760, 2018.
  * Mac Aodha et al. [2018] Mac Aodha, O., Su, S., Chen, Y., Perona, P., and Yue, Y.  Teaching categories to human learners with visual explanations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3820–3828, 2018.
  * Madry et al. [2017] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * Marcel & Rodriguez [2010] Marcel, S. and Rodriguez, Y.  Torchvision the machine-vision package of torch.  In _Proceedings of the 18th ACM international conference on Multimedia_ , pp. 1485–1488, 2010.
  * Miller [1995] Miller, G. A.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Munusamy Kabilan et al. [2018] Munusamy Kabilan, V., Morris, B., and Nguyen, A.  Vectordefense: Vectorization as a defense to adversarial examples.  _arXiv e-prints_ , pp. arXiv–1804, 2018.
  * Nauta et al. [2020] Nauta, M., Jutte, A., Provoost, J., and Seifert, C.  This looks like that, because… explaining prototypes for interpretable image recognition.  _arXiv preprint arXiv:2011.02863_ , 2020.
  * Nguyen et al. [2015] Nguyen, A., Yosinski, J., and Clune, J.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , CVPR ’15, pp. 427–436. IEEE, June 2015.  doi: 10.1109/CVPR.2015.7298640.
  * Nguyen et al. [2019] Nguyen, A., Yosinski, J., and Clune, J.  Understanding neural networks via feature visualization: A survey.  In _Explainable AI: interpreting, explaining and visualizing deep learning_ , pp. 55–76. Springer, 2019.
  * Nguyen et al. [2021] Nguyen, G., Chen, S., Jun, T. J., and Kim, D.  Explaining how deep neural networks forget by deep visualization.  In _International Conference on Pattern Recognition_ , pp. 162–173. Springer, 2021.
  * Palan & Schitter [2018] Palan, S. and Schitter, C.  Prolific. ac—a subject pool for online experiments.  _Journal of Behavioral and Experimental Finance_ , 17:22–27, 2018.
  * Papernot et al. [2016] Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A.  The limitations of deep learning in adversarial settings.  In _2016 IEEE European symposium on security and privacy (EuroS &P)_, pp. 372–387. IEEE, 2016.
  * Peer et al. [2017] Peer, E., Brandimarte, L., Samat, S., and Acquisti, A.  Beyond the turk: Alternative platforms for crowdsourcing behavioral research.  _Journal of Experimental Social Psychology_ , 70:153–163, 2017.
  * Petsiuk et al. [2018] Petsiuk, V., Das, A., and Saenko, K.  Rise: Randomized input sampling for explanation of black-box models.  _arXiv preprint arXiv:1806.07421_ , 2018.
  * Rajpurkar et al. [2017] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  _arXiv preprint arXiv:1711.05225_ , 2017.
  * Rauber et al. [2017] Rauber, J., Brendel, W., and Bethge, M.  Foolbox: A python toolbox to benchmark the robustness of machine learning models.  In _Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning_ , 2017.  URL http://arxiv.org/abs/1707.04131.
  * Rebuffi et al. [2020] Rebuffi, S.-A., Fong, R., Ji, X., and Vedaldi, A.  There and back again: Revisiting backpropagation saliency methods.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8839–8848, 2020.
  * Rudin et al. [2021] Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., and Zhong, C.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  _arXiv preprint arXiv:2103.11251_ , 2021.
  * Russakovsky et al. [2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Schmidt & Biessmann [2019] Schmidt, P. and Biessmann, F.  Quantifying interpretability and trust in machine learning systems.  _arXiv preprint arXiv:1901.08558_ , 2019.
  * Selvaraju et al. [2017] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Shakhnarovich et al. [2005] Shakhnarovich, G., Darrell, T., and Indyk, P.  Nearest-neighbor methods in learning and vision.  In _Neural Information Processing_ , 2005.
  * Shen & Huang [2020] Shen, H. and Huang, T.-H.  How useful are the machine-generated interpretations to general users? a human evaluation on guessing the incorrectly predicted labels.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 8, pp. 168–172, 2020.
  * Simonyan et al. [2013] Simonyan, K., Vedaldi, A., and Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Singla et al. [2014] Singla, A., Bogunovic, I., Bartók, G., Karbasi, A., and Krause, A.  Near-optimally teaching the crowd to classify.  In _International Conference on Machine Learning_ , pp. 154–162. PMLR, 2014.
  * Smilkov et al. [2017] Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ , 2017.
  * Su et al. [2017] Su, S., Chen, Y., Mac Aodha, O., Perona, P., and Yue, Y.  Interpretable machine teaching via feature feedback.  2017\.
  * Sundararajan et al. [2017] Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , pp. 3319–3328. PMLR, 2017.
  * Szegedy et al. [2013] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Wang et al. [2020] Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., and Hu, X.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 24–25, 2020.
  * Wu et al. [2019] Wu, N., Phang, J., Park, J., Shen, Y., Huang, Z., Zorin, M., Jastrzębski, S., Févry, T., Katsnelson, J., Kim, E., Wolfson, S., Parikh, U., Gaddam, S., Lin, L. L. Y., Ho, K., Weinstein, J. D., Reig, B., Gao, Y., Pysarenko, H. T. K., Lewin, A., Lee, J., Airola, K., Mema, E., Chung, S., Hwang, E., Samreen, N., Kim, S. G., Heacock, L., Moy, L., Cho, K., and Geras, K. J.  Deep neural networks improve radiologists’ performance in breast cancer screening.  _IEEE Transactions on Medical Imaging_ , pp. 1–1, 2019.  doi: 10.1109/TMI.2019.2945514.
  * Zhang et al. [2018] Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., and Sclaroff, S.  Top-down neural attention by excitation backprop.  _International Journal of Computer Vision_ , 126(10):1084–1102, 2018.
  * Zhang et al. [2020] Zhang, Y., Liao, Q. V., and Bellamy, R. K.  Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pp. 295–305, 2020.
  * Zhou et al. [2016] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2921–2929, 2016.

Appendix for:
The effectiveness of feature attribution methods and its correlation with
automatic evaluation scores

##  Appendix A1 Break-down human-AI team accuracy on image subsets

In Table A1, for each subset mentioned in Sec. 2.4.3, we let $Acc_{B,P}$
denote the accuracy on the subset, in which $B$ is the bin (E, M, H) and $P$
is the prediction result (Correct or Wrong) produced by ResNet-34.

Table A1: Human-AI team accuracy (%) on natural images in 6 controlled bins.
$Acc_{B,P}$.

|  $Acc_{B,P}$ ImageNet |  $Acc_{B,P}$ Stanford Dogs
---|---|---
E | M | H | E | M | H
Correct | Wrong | Correct | Wrong | Correct | Wrong | Correct | Wrong | Correct | Wrong | Correct | Wrong
Confidence | 92.23 | 78.64 | 88.78 | 59.22 | 75.51 | 43.16 | 86.67 | 66.96 | 74.07 | 32.77 | 64.41 | 25.86
GradCAM | 98.02 | 77.57 | 90.38 | 60.38 | 78.79 | 40.19 | 90.29 | 63.11 | 84.04 | 34.62 | 59.41 | 19.59
EP | 97.14 | 86.41 | 84.11 | 58.18 | 78.64 | 37.38 | 87.63 | 58.88 | 79.81 | 22.22 | 70.64 | 15.24
SOD | 95.69 | 82.57 | 88.60 | 55.65 | 66.96 | 43.75 | 83.96 | 70.19 | 70.18 | 40.78 | 59.43 | 28.04
3-NN | 96.64 | 88.60 | 86.84 | 54.95 | 83.93 | 44.76 | 97.46 | 56.00 | 76.92 | 24.55 | 60.42 | 18.75

##### Controlled-to-original accuracy conversion

While our experiments were not conducted on the real distributions of ImageNet
and Dogs, we can still estimate the human accuracy of the explanation methods
on the original datasets using the real ratios of subsets ($R_{B,P}$)
presented in Fig. A1.

From Table A1 and Fig. A1, we measured human accuracy of explanation methods
over the original distributions by the following formula:

$\sum_{B\in\\{E,M,H\\};P\in\\{C,W\\}}R_{B,P}\cdot Acc_{B,P}$

The estimated accuracy was reported in Fig. 2. It should be noted that the
above conversion was applied for natural images only because adversarial
images were synthetically generated, so no original distribution exists.

Figure A1: The ratios of image bins ($R_{B,P}$) in the controlled and real
distributions.

##  Appendix A2 How effective attribution maps in improving human-AI team
accuracy on the controlled distribution?

On average, over the controlled distribution (i.e. 6 bins, each of 50 images),
we found that 3-NN is the best method to help end-users categorize ImageNet
images. 3-NN outperforms other explanation methods by at least 2.71% (Table
A2; 76.59%). However, in fine-grained dog classification, explanations seem to
have low utility. All methods scored nearly the same and close to the random
baseline (i.e. 50% accuracy).

Table A2: Human-AI team accuracy with explanation methods on random images of the controlled distribution. NNs is the most effective in ImageNet experiment, while showing explanations could decrease human accuracy in fine-grained dog classification (Dogs).  | ImageNet | Dogs
---|---|---
Natural | Natural
Confidence | 73.17 | 58.33
GradCAM | 73.88 | 58.47
EP | 73.39 | 55.72
SOD | 72.25 | 58.91
3-NN | 76.59 | 56.73

##  Appendix A3 Are attribution maps more effective than 3-NN in improving
human accuracy on Easy, Medium, and Hard images of AI on the controlled
distribution?

In Table A3, we found 3-NN defeating other attribution methods and other
baselines by a wide margin on easy and hard images of ImageNet classification
task. It surpassed the second best methods (EP on E and Confidence on H) in
each image set by 3.13% on average.

Yet, 3-NN provided very little information for users working with the Dogs
experiment. None of attribution methods performed well on easy and hard images
of dogs. Because the differences among dog classes are minimal, only rightly
pointing out discriminative features could help users.

When AI classifies an image with confidence score around 0.5 (medium images),
GradCAM was the best explanation method on both ImageNet and Dogs data,
achieving 75.24% and 58.08%, respectively. However, the usefulness of
explanation methods for users with dog images was insignificant since the net
improvements over the random choice baseline were minimal.

Table A3: Human accuracy with explanation methods on easy (E), medium (H), and hard (H) images of the controlled distributions. | ImageNet | Dogs
---|---|---
| E | M | H | E | M | H
Confidence | 85.44 | 73.63 | 59.59 | 77.02 | 52.42 | 45.30
GradCAM | 87.50 | 75.24 | 58.74 | 76.70 | 58.08 | 39.90
EP | 91.83 | 70.97 | 57.62 | 72.55 | 51.72 | 43.46
SOD | 89.33 | 72.05 | 55.51 | 77.14 | 56.22 | 43.66
3-NN | 92.70 | 71.11 | 64.98 | 78.44 | 50.00 | 39.58
$\mu$ | 89.36 | 72.60 | 59.29 | 76.37 | 53.69 | 42.38

##  Appendix A4 Are attribution maps more effective than 3-NN in improving
human accuracy on correct/wrong images of the controlled distribution?

Explanations increase the chance humans agree with AI’s predictions in
sentiment classification and question answering [12]. While we examined the
human accuracy on correct and images of AI, it has been shown that
explanations also encouraged humans to accept AI’s predictions rather than
reject in image classification.

In Table A4, regarding ImageNet images, 3-NN was the most useful method for AI
error identification, at 63.33%. Besides, the explanations from 3-NN and
GradCAM seemed to benefit equally to humans when AI classify correctly (89.28%
and 89.14%, respectively).

EP and 3-NN did the best in explaining network’s correct predictions on dog
images (79.03% and 79.56%). We were surprised because all methods did not
outperform the random choice approach (50%) to identify misclassifications of
ResNet-34 on Dogs images.

Table A4: Human accuracy with explanation methods on correct and wrong images of the controlled distributions. | ImageNet | Dogs
---|---|---
Correct | Wrong | Correct | Wrong
Confidence | 85.62 | 60.80 | 75.14 | 41.71
GradCAM | 89.14 | 59.38 | 77.85 | 39.47
EP | 86.67 | 60.31 | 79.03 | 32.48
SOD | 83.77 | 60.42 | 71.17 | 46.18
3-NN | 89.28 | 63.33 | 79.56 | 33.01

##  Appendix A5 How does prior knowledge affect human accuracy?

Although we tried to mitigate the effect of prior knowledge by giving users
the definition and sample images of categories, we still would like to examine
how differently users score when never-seen-before vs. already-known objects
are presented.

We let $\Delta$ denote the gap between accuracy on known images vs. accuracy
on unknown images within a method. Looking into Table A5, SOD was affected the
most by prior knowledge compared to other explanation methods with $\Delta$ of
6.38% in ImageNet or 2.21% in Dogs. An explanation for the above observation
is that because SOD is model-agnostic, it provides the least classifier-
relevant information, and therefore humans would have to rely on the prior
knowledge the most to make decisions.

Table A5: Human accuracy on known and unknown images of both natural and
adversarial images.

| ImageNet | Dogs
---|---|---
Known | Unknown | $\Delta$ | Known | Unknown | $\Delta$
Accuracy | Trials | Accuracy | Trials | Accuracy | Trials | Accuracy | Trials
Confidence | 73.07 | 724 | 69.89 | 176 | 3.18 | 61.88 | 446 | 61.59 | 604 | 0.29
GradCAM | 72.69 | 714 | 72.22 | 216 | 0.47 | 60.78 | 459 | 60.32 | 441 | 0.46
EP | 74.11 | 734 | 73.01 | 226 | 1.10 | 55.39 | 334 | 57.38 | 596 | -1.84
SOD | 73.37 | 811 | 66.99 | 209 | 6.38 | 60.52 | 468 | 62.73 | 492 | -2.21
3-NN | 76.46 | 838 | 74.32 | 182 | 2.14 | 56.35 | 367 | 57.75 | 563 | -1.40

##  Appendix A6 AI-only thresholds

Using confidence score to automate the tasks, we found the two optimal
thresholds are 0.55 and 0.50 for ImageNet and Dogs, respectively (see Table
A6). These threshold values were tuned on around 50K ImageNet and 6K Stanford
Dogs images.

Table A6: Accuracy of AI-only on ImageNet and Dogs at different threshold
values.

$T$ | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.45 | 0.50 | 0.55 | 0.60 | 0.65 | 0.70 | 0.75 | 0.80 | 0.85 | 0.90 | 0.95
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ImageNet | 73.40 | 73.74 | 74.47 | 75.47 | 76.66 | 77.75 | 78.66 | 79.44 | 80.11 | 80.56 | 80.79 | 80.59 | 79.93 | 79.01 | 77.85 | 76.31 | 74.12 | 70.65 | 64.83
Dogs | 77.37 | 77.44 | 77.81 | 78.23 | 78.81 | 79.31 | 79.80 | 80.65 | 80.65 | 81.14 | 81.07 | 80.34 | 79.34 | 77.67 | 75.82 | 73.63 | 70.02 | 65.21 | 55.93

##  Appendix A7 Original distributions of ImageNet and Stanford Dogs

Fig. A2 shows the original distributions of ImageNet and Dogs by confidence
intervals. Although we skipped images having confidence in [0.3, 0.4) and
[0.6, 0.8), the human accuracy on the remaining intervals can still represent
that of the original distributions because the numbers of images in those
intervals are not major.

(a)

(b)

Figure A2: Image distribution of (a) ImageNet and (b) Dogs by confidence score
intervals.

Table A7 shows the distributions of easy, medium, and hard images of ImageNet
and Dogs dataset. It should be noted that we ignored a few confidence
intervals so the total numbers of images are not 50K and 6K.

Table A7: The real distributions of easy, medium, and hard images in ImageNet
and Dogs dataset.

| ImageNet | Dogs
---|---|---
Images | Percentage (%) | Images | Percentage (%)
E | 28967 | 75.19 | 3364 | 74.13
M | 6648 | 17.26 | 916 | 20.19
H | 2908 | 7.55 | 258 | 5.68
$\sum$ | 38523 | 100 | 4538 | 100

##  Appendix A8 Participants’ statistics

Table A8 shows the numbers of users, trials, and users per image in ImageNet
and Dogs experiment for each method.

Table A8: The number of users, trials, and users per image in ImageNet and Dogs experiment. A trial is one time we show an image to the user and asks for his decision.  | ImageNet | Dogs
---|---|---
Users | Trials | Users per image | Users | Trials | Users per image
Confidence | 30 | 900 | 2.00 | 35 | 1050 | 2.33
GradCAM | 31 | 930 | 2.07 | 30 | 900 | 2.00
EP | 32 | 960 | 2.13 | 31 | 930 | 2.07
SOD | 34 | 1020 | 2.27 | 32 | 960 | 2.13
3-NN | 34 | 1020 | 2.27 | 31 | 930 | 2.07
Total | 161 | 4830 |  | 159 | 4770 |

##  Appendix A9 Participants’ background and payment rate

User background We only use a single criterion for filtering users: Users must
be native English speakers. We think this is required for our study (and any
study that uses English) since the training, instructions, and ImageNet labels
are in English. We used this filter to avoid the cases where users make
arbitrary decisions without understanding some words. Prolific shows that our
users are diverse, aging from 18-77 (median=31) and coming from a diverse set
of countries (US, UK, Poland, India, Korea, Canada, Australia, South Africa,
etc.). Please see Prolific for more description of their online userbase,
which, according to a study, is more reliable than AMT Turkers [59].

Payment Our rate is higher than the Prolific recommended rate wage of
$9.60/hr. In fact, during the study, we had increased our rate to attract more
participants (up to $13.68/hr). As participants come from various countries in
the world, we did not consider minimum wage per region because this
recommended rate is suggested by Prolific and accepted by all participants.

##  Appendix A10 Participants’ acceptance-rejection rate

Table A9 shows the ratios of acceptance and rejection across methods in
ImageNet and Dogs (natural examples only). Consistent with [12], we found that
explanations increase the chance humans accept AI’s predictions except SOD. As
mentioned in Sec. A14.5, SOD users were most likely to reject AI’s labels
because this baseline gave users bad-quality heatmaps.

Table A9: The percentages of acceptance of rejection in ImageNet and Dogs experiment. | ImageNet | Dogs
---|---|---
Accept | Reject | Accept | Reject
Confidence | 62.33 | 37.67 | 66.67 | 33.33
GradCAM | 64.26 | 35.74 | 69.10 | 30.90
EP | 62.99 | 37.01 | 73.27 | 26.73
SOD | 61.97 | 38.03 | 62.66 | 37.34
3-NN | 63.56 | 36.44 | 73.40 | 26.60

##  Appendix A11 Sample experiment screens

Here we show our experiment user interface by screens.

###  A11.1 Instructions

We introduced Sam - the AI and explained the tasks to participants. Later, we
explicitly restricted the device used for the experiments to ensure the
display resolution will not affect human decisions.

Figure A3: Instruction screen of experiments.

###  A11.2 Training

We gave users 5 trials where each of them is followed by a feedback screen on
users’ decision. For GradCAM, EP, and SOD, as their heatmaps have the similar
format, the same interpretation was provided as shown in Fig. 4(a). 3-NN
displays to users three correct images of the predicted class in Fig. 4(b),
while Confidence only shows the input image, the predicted label, and the
confidence score.

(a)

(b)

Figure A4: The training screens for (a) heatmaps (GradCAM, EP, and SOD) and
(b) nearest neighbors with annotated components.

###  A11.3 Validation and Test

While we have 10 trials in evaluation and 30 trials in test, we did not tell
participants about the validation phase to avoid overfitting. These 40 trials
were showed continuously in which the definition and sample images of the
predicted class were given beforehand as in Fig. A5. No feedback was given in
validation and test.

(a)

(b)

Figure A5: Users are given the definition and three sample images of the
predicted classes (a). Users are asked to agree or disagree with the
prediction of AI using explanation(b).

As the purpose of validation was to filter out users not paying enough
attention to the experiments (e.g. random clicking), we carefully chose
clearly wrong and correct images by ResNet-34 (Fig. A6) to check users’
attention. It should be noted that the definition and sample images of the
predicted category were given in advance.

(a) ImageNet

(b) Stanford Dogs

Figure A6: Validation images for (a) ImageNet and (b) Dogs. There are 5
clearly correct and 5 clearly wrong predictions of AI in each experiment.
Above each image is the groundtruth label and the misclassified label (if the
image was misclassified by AI). In Stanford Dogs, the misclassified images
were synthetically generated using PGD-$L_{\text{inf}}$ adversarial attacks.

##  Appendix A12 Images mislabeled into non-dog categories (MIND samples)

In Dogs experiment, images misclassified into non-dog categories (MIND) were
discarded because users often can instantly reject those images without
explanation. As shown in Fig. A7, almost all MIND samples contain more than
one object.

(a)

(b)

(c)

(d)

Figure A7: Mislabeled Into Non-Dog samples (MINDs). There are 71 MINDs in Dogs
by ResNet-34. Above each image is the groundtruth label and the misclassified
label.

##  Appendix A13 Example explanations displayed to users

Below are example of explanations taken from the screens displayed to users
during our user-study. Here, we show the differences between GradCAM, EP, SOD,
and 3-NN explanations for the same input image predicted as “american coot”
(Fig. A8).

(a) GradCAM explanation

(b) EP explanation

(c) SOD explanation

(d) 3-NN explanation

Figure A8: The explanations from GradCAM, EP, SOD, and 3-NN for the input
image labeled “american coot” by the classifier. While the highlight from
GradCAM tends to be expansive, the focus of EP is narrow, and SOD attend to
the entire body of the bird. 3-NN presents similar scenes of a coot around the
pond.

##  Appendix A14 Qualitative examples supporting our findings

###  A14.1 Hard, real ImageNet images that were correctly labeled by 3-NN
users but not GradCAM, EP, or SOD users

Regarding hard images, we observed that images corrected by 3-NN but not
attribution maps and SOD often contain multiple concepts (Fig. A11), low
quality (Fig. A9), look-alike objects (Fig. A10 and A14), only a part of the
main object (Fig. A12), or objects with unusual appearances (Fig. A13). On
these images, while heatmaps did not highlight the discriminative features and
the confidence score was low, users tended to reject AI’s labels. In contrast,
3-NN helped users gain confidence that AI is correct when there are multiple
plausible labels.

Figure A9: Hard ImageNet image with low quality. 3-NN might help users
recognize the shape and color of “african hunting dog”, while attribution
methods gave users little information because users could not see what AI is
looking at clearly. Figure A10: Hard ImageNet image with probable look-alike
objects. 3-NN even found the same man, which strongly supports the AI
prediction, while EP and SOD highlighted incorrectly. Figure A11: Hard
ImageNet image with multiple plausible concepts present. The last image of
3-NN (right most) showed an image of a car labeled as “car wheel”, which
strongly helps users confirm the prediction of AI. However, users with other
explanation methods rejected the prediction because AI did not highlight
properly or showed low confidence. Figure A12: Hard ImageNet image with only a
part of the main object. 3-NN helped users recognize a “jean” leg, but other
heatmaps could not find the determinative features on the input image. Figure
A13: Hard ImageNet image with strange-looking objects. Although the “ladle” in
the input image looks strange, 3-NN showed other ladles also have unusual
appearance (the last neighbor image). Users might instantly rejected the
prediction because the three sample images are contrastive. Figure A14: Hard
ImageNet image with probable look-alike objects. 3-NN helped users gain
confidence while “overskirt”, “hoopskirt”, and “gown” look very similar.

###  A14.2 Medium, AI-misclassified, real ImageNet images that were
incorrectly accepted by 3-NN users

These images can be divided into two main categories: debatable ground-truth
(Figs. A15 & A16) and look-alike object (Figs. A17 & A18).

Figure A15: Medium ImageNet image with wrong labels. The input image and the
first NN are clearly the same but the annotated label is “missile”. Figure
A16: Medium ImageNet image with multiple objects present. The spider is
salient and 3-NN retrieved very similar images, but the ground truth is
“spider web”. Figure A17: Medium ImageNet image with fine-grained classes.
3-NN failed to show the difference between “cardigan” and “pembroke” to users.
Figure A18: Medium ImageNet image with confusing objects. 3-NN failed to show
the difference between “palace” and “monastery” to users.

###  A14.3 Easy, AI-misclassified, ImageNet images that were correctly
rejected by 3-NN users but not GradCAM and SOD users

3-NN helped users distinguish the two classes by showing contrastive examples
(e.g. “walking stick” vs. “african chameleon” in Fig. A19 or “horse cart” vs.
“grocery store” in Fig. A20).

Figure A19: Easy ImageNet image which was clearly misclassified by the AI.
3-NN easily pointed out the difference between “african chameleon” and
“walking stick” to users. Figure A20: Easy ImageNet image which was clearly
misclassified by the AI. 3-NN easily pointed out the difference between
“grocery store” and “horse cart” to users.

###  A14.4 Adversarial ImageNet images that were correctly rejected by 3-NN
users but not GradCAM, EP, or SOD users

##### In Fig. 2, what made 3-NN more effective than attribution maps on
Adversarial ImageNet?

As the adversarial attacks fooled AI by small perturbations, the misclassified
labels are not far from the ground truth (e.g. bee eater to lorikeet in Fig.
A22 or collie to shetland sheepdog in Fig. A26). The highlights of heatmaps
focused on parts of the main object and made the explanations compelling to
users. 3-NN helped users differentiate the two categories by looking at the
constrastive images of the predicted label and the ground truth (Fig. A21 and
Fig. A22).

Figure A21: Adversarial ImageNet image of “barracouta” which was misclassified
to “tench”. Users may use the difference in skin patterns of “tench” and
“barracouta” to make decision. Figure A22: Adversarial ImageNet image of “bee
eater” which was misclassified to “lorikeet”. 3-NN contrasted these two bird
breeds strongly.

###  A14.5 AI-misclassified Dogs images that are correctly rejected by SOD
users but not GradCAM, EP, or 3-NN users

##### In Table A1, what made SOD significantly more effective than other
methods on correcting AI-misclassified images of Dogs?

We found that GradCAM and EP often highlighted the entire face of the dogs,
which made the heatmaps persuasive to users although the predictions were
wrong. Regarding 3-NN, the misclassified category is visually similar to the
ground truth (i.e. eskimo dog vs. malamute), which was challenging for lay
users to distinguish. We assume that users expected explanations to be as
specific and relevant as possible because the differences among breeds are
minimal. SOD highlighted the entire body of the dogs (Fig. A23) or even
irrelevant areas (Fig. A24). This explains why users with SOD tended to ignore
rather than trust the AI. While 3-NN users leveraged the information of
nearest neighbors to identify AI’s errors, SOD users rejected predictions
because of the heatmaps’ low quality, which unintentionally improved the
accuracy on wrong Dogs images. The rejection rates of SOD users were highest
in both ImageNet (38.03%) and Dogs (37.34%) as shown in Table A9. Indeed, due
to the high rejection rate of SOD users, the accuracy on correct Dogs images
was lowest as shown in Table A1.

Figure A23: Wrong Dogs image of “eskimo dog” which was misclassified to
“malamute”. GradCAM and EP often highlighted the entire face of the dogs,
which makes the heatmaps persuasive to users although the predictions are
wrong. For 3-NN, the mislabeled category is visually similar to the ground
truth (e.g. eskimo dog vs. malamute), which is challenging for users to
distinguish, then they inclined to accept the predictions. SOD always
highlights the entire body of the dogs, explaining why users with SOD tended
to ignore rather than trust the AI. Figure A24: Wrong Dogs image of “flat-
coated retriever” which was misclassified to “newfoundland”. SOD highlighted
irrelevant areas, so users with SOD were more likely to reject.

###  A14.6 Adversarial Stanford Dogs images that were correctly rejected by
Confidence-only users but not GradCAM, EP, and 3-NN users

##### In Fig. 2, why did visual explanations hurt human-AI team performance
Adversarial Dogs (the hardest task)?

We found no explanation methods that benefit participants in this task. While
GradCAM and EP mostly concentrated on a body part of dogs, 3-NN showed images
of an almost identical breed (Fig. A25 and A26). Again, the improvement of SOD
came from the bad quality of its heatmap.

Figure A25: Adversarial Dogs image of “cocker spaniel” which was misclassified
to “bedlington terrier”. GradCAM and EP concentrated on the belly of dogs, and
3-NN showed images of an almost identical breed, making users trust the
prediction. Figure A26: Adversarial Dogs image of “collie” which was
misclassified to “shetland sheepdog”. GradCAM and EP concentrated on the face
of dogs, and 3-NN showed images of a very similar dog breed, making users
trust the prediction.