# Explainable Image Recognition
via Enhanced Slot-attention Based Classifier

Bowen Wang, Liangzhi Li, Jiahao Zhang, Yuta Nakashima, and Hajime Nagahara
This work was supported by World Premier International Research Center
Initiative (WPI), MEXT, Japan. This work is also supported by JSPS KAKENHI
24K20795 and JST FOREST Grant No. JPMJFR216O. Authors are with Institute of
Datability Science (IDS), Osaka University, 2-8 Yamadaoka, Suita, Osaka
565-0871. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>nagahara@ids.osaka-u.ac.jp)Bowen Wang, Yuta Nakashima, and Hajime Nagahara are
also with Premium Research Institute for Human Metaverse Medicine (WPI-PRIMe),
Osaka University, 2-2 Yamadaoka, Suita, Osaka 565-0871. (e-mail:
<EMAIL_ADDRESS>Corresponding author: Liangzhi LiOur code is
available at https://github.com/wbw520/scouter

###### Abstract

The imperative to comprehend the behaviors of deep learning models is of
utmost importance. In this realm, Explainable Artificial Intelligence (XAI)
has emerged as a promising avenue, garnering increasing interest in recent
years. Despite this, most existing methods primarily depend on gradients or
input perturbation, which often fails to embed explanations directly within
the model’s decision-making process. Addressing this gap, we introduce
E-SCOUTER, a visually explainable classifier based on the modified slot
attention mechanism. E-SCOUTER distinguishes itself by not only delivering
high classification accuracy but also offering more transparent insights into
the reasoning behind its decisions. It differs from prior approaches in two
significant aspects: (a) E-SCOUTER incorporates explanations into the final
confidence scores for each category, providing a more intuitive
interpretation, and (b) it offers positive or negative explanations for all
categories, elucidating “why an image belongs to a certain category” or “why
it does not.” A novel loss function specifically for E-SCOUTER is designed to
fine-tune the model’s behavior, enabling it to toggle between positive and
negative explanations. Moreover, an area loss is also designed to adjust the
size of the explanatory regions for a more precise explanation. Our method,
rigorously tested across various datasets and XAI metrics, outperformed
previous state-of-the-art methods, solidifying its effectiveness as an
explanatory tool.

###### Index Terms:

Explainable AI, Interpretability, Deep Learning, Image Classification,
Attention Mechanism.

##  I Introduction

Understanding the decision-making process of deep learning models has assumed
paramount importance, particularly in domains such as medical diagnosis [1, 2,
3], where the consequences of relying on black-box models can be profound.
Explainable artificial intelligence (XAI) has been proposed to unveil the
inner workings of models, uncovering the factors and features influencing
their predictions and enabling users to comprehend how and why specific
decisions are reached.

The most popular way for XAI to interpret a model’s decision involves
assessing the importance of pixels or regions within an input image. This has
been extensively explored to offer explanation for “why an image $x$ is
classified as category $l$” [4, 5, 6]. Meanwhile, the notion aiming to clarify
“why it is not classified as category $l$” [7, 8], presents an untapped
potential for enhancing understanding of the differences between categories.
Regardless of the explanation type, one important question that arises here
is: How do these regions contribute to the decision?

To explain this, let us consider a fully-connected (FC) classifier for
category $l$, represented by $y_{l}(x)=w^{\top}_{l}x+b_{l}$, where $x$ ,
$w_{l}$ , and $b_{l}$ represent the input image (or its feature vector),
learnable vector for $l$, and a learnable scalar, respectively. The training
involves identifying $w_{l}$, which may be decomposed into a set
$\mathcal{S}_{l}=\\{s_{li}\\}_{i}$ of discriminative supports, from the
training samples, with each support being associated with a specific weight
$\gamma_{li}$. The classifier can be rewritten as
$y_{l}(x)=(\sum_{i}\gamma_{li}s^{\top}_{li})x+b_{l}$, where one can always
choose $s_{li}$ such that $s_{li}^{\top}x\geq 0$ by appropriately choosing
$\gamma_{i}$. With this definition, $\gamma_{li}>0$ means the corresponding
$s_{il}$ is a positive support for category $l$ that increases $y_{l}$.
Conversely, a $s_{il}$ with $\gamma_{li}<0$ is a negative support.
$\mathcal{S}_{l}$ thus encompasses sets $\mathcal{S}^{+}_{l}$ and
$\mathcal{S}^{-}_{l}$ of positive and negative supports, respectively.

Figure 1: Explanations from E-SCOUTER. Using positive ($+$) and negative ($-$)
E-SCOUTER losses can emphasize the positive and negative supports
respectively, based on which one can understand why or why not the images are
classified into a certain category.

The bottom row of Fig. 1 displays an example from MNIST [9]. The acute angle
created by white line segments near the top-right corner can be a positive
support for category 7 as such a structure hardly appears in the other digits.
Conversely, the presence of a straight horizontal line segment around the top
and the absence of a horizontal line segment around the bottom are negative
supports for category 2. The former is also a negative support for 9.
Interpretive supports can also be found in natural image classification tasks,
e.g., over shark images (from ImageNet [10]) and bird images (from CUB200
[11]). A practical application in medical image analysis, as discussed in
[12], underscores the significance of identifying both positive and negative
supports. However, recent mainstream methodologies introduced in [5, 6, 13]
have not thoroughly explored this aspect.

In this paper, we introduce Enhanced Slot-based COnfigUrable and Transparent
classifiER (E-SCOUTER), which substitutes the FC classifier with the slot-
attention mechanism [14] to enhance transparency and provide both intuitive
and accurate explanations. Unlike existing XAI methods [15, 5, 16, 6, 17],
E-SCOUTER embeds explanations into the forward decision-making pipeline,
ensuring that the rationale for a classification decision is directly
accessible and interpretable. Our approach does not merely highlight decision-
relevant regions but also elucidates how they contribute, offering insights
into the final decision through both positive and negative supports, as
illustrated in Fig. 1.

A specialized loss function is designed to realize this capability. It allows
for fine-tuning the model’s behavior, switching between finding positive or
negative supports. Moreover, E-SCOUTER is distinguished by a bespoke attention
area loss, tailored to constrain the model’s focus and provide a more precise
explanation. Through extensive experimentation, we demonstrate that E-SCOUTER
is not just a competent classifier when compared to FC but also provides
superior explanatory insights than previous XAI methods. This makes it an
indispensable tool for many applications (e.g., medical diagnosis) where
understanding both the “why” and “why not” behind a model’s decision is
critical.

Our contributions are summarized as follows: 1) We proposed E-SCOUTER, a
transparent classifier, reasoning model decision with both positive and
negative supports. 2) By utilizing an area size loss functions, our method can
control the spatial size of the supports to be learned, facilitating a clearer
and more distinct explanation. 3) State-of-the-art (SOTA) interpretability
across several benchmark datasets. Our comprehensive evaluation demonstrates
E-SCOUTER’s robustness and versatility in various scenarios. 4) The case study
in the medical field further emphasizes the significance of both types of
supports, along with the necessary to control the area size of the explanatory
regions.

This paper is an extension of our conference paper published in ICCV 2021 [8].
In this version, 1) we overcome the category limitations, restricted to
handling fewer than 200 classification categories, by incorporating an
additional normalization step. This improvement also allows our method to have
a broader application across various scenarios. 2) E-SCOUTER shows a more
stable training process. Through several datasets and evaluation metrics, it
owns consistently higher classification accuracy and superior interpretability
than the former version, as well as, state-of-the-art performance than
previous XAI methods.

##  II Related Works

###  II-A Explainable AI

Explainable AI methods for image recognition are generally designed to
elucidate the relevance of individual pixels of an input image to a model’s
prediction, culminating in generating a heatmap that highlights relevant
regions. Such interpretability sheds light on what leads to a particular
decision, making it more transparent and accountable. To accomplish this
objective, there are primarily two paradigms [18], i.e., post-hoc and
intrinsic. Post-hoc methods generate a heatmap after the model makes the
decision [5, 6, 13]. Conversely, intrinsic methods embed heatmap generation in
the model’s forward path, often leveraging attention modules [19, 20].

The post-hoc paradigm has been extensively explored, and the most popular type
of method is based on back-propagation. Attribution techniques, e.g., saliency
[4] and its variants [4, 21, 22, 23, 24] were introduced to compute the
gradient of the input image. While saliency analysis can highlight important
regions, it tends to focus on low-level visual features, such as color,
contrast, and edges, without considering higher-level semantic information.
GradCAM [5], a neuron activation-based technique [25], combines the model’s
gradients with the activation map [15] of internal neurons. It captures the
importance of each spatial location in the feature maps relative to a specific
class. Multiple variants have appeared in the literature, e.g., GradCAM++
[16], Smooth-GradCAM++ [26], and Score-CAM [13]. All of them use a similar
idea to GradCAM. Perturbation provides another aspect of post-hoc, which does
not rely on a model’s structure and learned parameters (i.e., model-agnostic).
For instance, Extremal Perturbation [27] iteratively modifies the pixel values
of an input image while monitoring the model’s response to identify the most
salient input pixels. This strategy has been recently improved by designing
heuristics to control their solution space as in, e.g., RISE [6], IBA [17],
and IGOS [28]. Perturbation methods suffer from computation costs. Some recent
works [29] focus on efficiency.

The conceptual genesis for intrinsic methods lies in endowing the inherent
interpretability to a deep model [25]. Leveraging attention is a promising way
to realize this purpose [19, 20, 30, 31], while previous methods generally
take attention module as a part of the backbone model and are not intended to
interpret a classification task. There are also some concept-based frameworks
[32, 33, 34, 35] involving the concept-level explanation, which are different
from the per-pixel XAI methods (mentioned above).

SCOUTER [8] is an intrinsic method designed to reason classification models in
a single forward path. However, it suffers from training instability and
limitations regarding the number of classes it can handle, which restrict its
broader applicability. This paper introduces E-SCOUTER as an extension to
address the issues encountered in the original version for better stability
and scalability.

###  II-B Discriminant Counterfactual Explanation

Given an image of category $l$ and a counterfactual category $l^{\prime}$,
counterfactual explanation [36, 37, 38, 39, 40] try to find region
transformation between this image pair, which can change the model prediction
from $l$ to $l^{\prime}$. Among them, Discriminant Counterfactual Explanation
(DCE) [7] was introduced to elucidate “why an image $x$ is classified as
category $l$ instead of category $l^{\prime}$”, employing a more efficient
approach by harnessing the attributive explanations [4, 23, 24]. However, its
counterfactual explanation comes from the combination of two positive
explanations from an image pair, which can only be seen as a proxy of a
negative explanation. Notably, several post-hoc methods can be extended to
provide negative explanations by simply negating the gradient of the
prediction score. GradCAM [5] refers to its negative variant as a negative
explanation, indicating regions that have the potential to alter the model’s
decision. This strategy may not be straightforward because the linear
classifier may lead to the visualization failing to emphasize the support
regions for the negative explanation. E-SCOUTER directly generates a heatmap
to display important regions within the forward path, revealing the negative
part of a model’s decision.

###  II-C Slot-attention for Computer Vision

Self-attention [41] has been primarily explored in natural language processing
[42]. Self-attention layers process elements in an input sequence one by one,
aggregating information across the entire input sequence. This attention
mechanism has more recently transferred to computer vision as an essential
building block of various models, such as the Image Transformer [43],
DEtection TRansformer (DETR) [44], and Vision Transformer (ViT) [45]. Slot
attention [14] is motivated by this self-attention mechanism. It has been
employed to extract object-centric features from images. More recent research
has extended the utility of slot attention to various computer vision tasks,
including feature contrastive learning [46], diffusion for image generation
[47], and segmentation [48]. In this paper, we delve into the use of the slot
attention mechanism for explainable classification.

##  III Method

(a)

(b)

Figure 2: Classification pipeline. (a) E-SCOUTER as a classifier. (b) The
overview of E-SCOUTER for classification, where PE is position embedding, RE
is reshape operation, $\sigma$ is sigmoid activation, and ($\cdot$) denotes
dot multiplication.

The primary goal of an image classification model given an input image $x$ is
to determine the most possible category $l$ it belongs to, from a predefined
set $\mathcal{L}=\\{l_{1},l_{2},\ldots,l_{n}\\}$ of categories. This can be
done by using a backbone model $B$ to extract feature map
$F=B(x)\in\mathbb{R}^{h\times w\times c}$, where $h$, $w$, and $c$ are the
height, width, and the number of channels, respectively. After a global
average pooling, a multilayer perceptron (MLP) classifier with FC layers and
softmax converts $F$ into confidence scores $o=\\{o_{1},o_{2},\ldots,o_{n}\\}$
for respective categories in $\mathcal{L}$. Although FC provides substantial
learning capacity, it also makes it difficult to determine which features are
crucial for the prediction outcome and how these features interact with each
other.

E-SCOUTER substitutes for the MLP classifier to compute $o$ in order for
explainable image classification. As illustrated in Fig. 2(a), the entire
network, including the backbone, is trained with a customized loss function.
This loss function enables control over the desired size of relevant regions
and offers the switch between finding positive and negative supports.

###  III-A E-SCOUTER

The key component of E-SCOUTER is a slot-attention module [14], which
initially proposes slots to represent regions aggregated with the attention
mechanism [41]. Each slot is responsible for exclusively attending to a single
visual concept (e.g., a pattern and a part of an object) within the image
through the dedicated attention map, thereby producing distinct features of
the concept as output. This mechanism can be extended to identify relevant
regions to a specific decision within an input image. What follows details
E-SCOUTER.

As illustrated in Figure 2(b), the initial feature map $F$ undergoes a
transformation process. It starts with a $1\times 1$ convolution layer,
followed by the activation of a ReLU nonlinearity as
$F^{*}=\operatorname{ReLU}(\operatorname{Conv}(F))$. These transformations
reduce the dimensionality of $F$ from $c$ to $d$. Then, the spatial dimension
of $F^{*}$ is flattened to obtain a feature matrix
$V\in\mathbb{R_{+}}^{d\times s}$, where $s=hw$. To preserve spatial
information, a positional embedding PE [41, 14, 8] is added to the features,
denoted as $\tilde{F}=F^{*}+\text{PE}$, whose spatial dimension is also
flattened.

E-SCOUTER uses a single slot-attention module with $n$ slots as a classifier,
which is applied to feature map $F$ (Fig. 2(a)). Each slot is associated with
a specific category $l\in\mathcal{L}$ and is responsible for identifying a
support of $l$ that increases (or decreases) the confidence score for $l$. A
slot for $l$ is a learnable vector $w_{l}$ that describes a visual concept
associated with $l$, which can be collectively denoted by
$W\in\mathbb{R}^{n\times d}$ for all $l\in\mathcal{L}$.

The original slot-attention module updates the slots with a gated recurrent
unit (GRU) for more defined responses around the edges between the objects of
interest and the rest. We also adopt this design choice in E-SCOUTER. Let
$W^{(t)}$ be the slots after the $t$-th iteration, where $t=1,2,\ldots,T$ and
$W^{(1)}=W$. We employ two multilayer perceptrons (MLPs), denoted as $Q$ and
$K$, each comprising three FC layers with ReLU nonlinearities in between.
These MLPs transform $W^{(t)}$ and $\tilde{F}$ to be ‘query’ and ‘key’ as

$Q(W^{(t)})\in\mathbb{R}^{n\times d},\quad K(\tilde{F})\in\mathbb{R}^{d\times
s}.$ (1)

E-SCOUTER uses the dot-product similarity [41] between $W$ and $\tilde{F}$ to
compute attention matrix $A^{(t)}$

$A^{(t)}=\sigma(Q(W^{(t)})K(\tilde{F}))\quad\in(0,1)^{n\times s},$ (2)

where $\sigma$ is the element-wise sigmoid. This attention matrix encapsulates
the dependency between the slots and spatial elements within the feature map,
showing the spatial positions in the feature map at which the concept
associated with each slot appears.

We further normalize the attention weights to compactify the region. Let
$a^{(t)}_{l}$ be the row vector for category $l$. We normalize the activation
over the spatial positions by

$\displaystyle\bar{a}^{(t)}_{l}=\frac{a^{(t)}_{l}}{a^{(t)}_{l}\mathbf{1}_{s}+1},$
(3)

where $\mathbf{1}_{s}$ is a column vector in $\mathbb{R}^{s}$ with all
elements being 1. The normalized attention matrix is denoted by
$\bar{A}^{(t)}$. The attention weights are then applied to $V$ to obtain the
features associated with each slot:

$\displaystyle U^{(t)}=\bar{A}^{(t)}{V}^{\top}\quad\in\mathbb{R}_{+}^{n\times
d}.$ (4)

The normalization by Eq. (3) is a critical step forward from the original
SCOUTER [8], effectively mitigating excessive attention values that could
impede training progress, while preserving the magnitude of smaller
activation.

The slots $W^{(t)}$ are then updated through a GRU as:

$W^{(t+1)}=\operatorname{GRU}(U^{(t)},W^{(t)}).$ (5)

This operation takes $U^{(t)}$ and $W^{(t)}$ as input, giving $W^{(t+1)}$ as
the hidden state. Following the original slot-attention, $T$ is defaulted to
3.

The output of E-SCOUTER is the summation of all feature values corresponding
to category $l$ in $U^{(T)}$, i.e.,

$o=U^{(T)}\mathbf{1}_{d}\quad\in\mathbb{R}_{+}^{n}.$ (6)

Eq. (4) and Eq. (6) give a vivid interpretation of E-SCOUTER:

$o=\bar{A}^{(T)}V^{\top}\mathbf{1}_{d}\quad\in\mathbb{R}_{+}^{n}.$ (7)

It is worth noting that, in contrast to the original slot attention module,
E-SCOUTER omits a linear transformation applied to the features $V$, as it
already possesses a sufficient number of learnable parameters in the
convolution layer Conv applied to $F$, $Q$, $K$, GRU, and others, ensuring
flexibility.

Interpretation. The elements of $V$ may be interpreted as the indicators of
the presence of a support (or concept) for one of the categories111It should
be noted that each element of the column vectors of $V$ is not necessarily an
indicator of a support; some elements in a column vector can combinatorially
represent one., and $V^{\top}\mathbf{1}_{d}\in\mathbb{R}^{d}$ can be then seen
as a map that shows the presence of supports for any category. Meanwhile,
$\bar{A}$ gives the spatial positions corresponding to each slot (or
category). Therefore, $o$, which is a multiplication of $\bar{A}$ and
$V^{\top}\mathbf{1}_{d}\in\mathbb{R}^{d}$, gives the weighted sum of support
indicators for each category.

Typical classifiers use FC layers to compute the confidence for each category,
while E-SCOUTER uses Eq. (6), which is the weighted sum of the support
indicators. An obvious advantage of this design is the interpretability of the
decision-making process, as the regions with a larger weighted value have a
greater influence on the final prediction. Therefore, by visualizing the
attention map, we can directly know which regions are supporting the model’s
final decision.

###  III-B E-SCOUTER Loss

The overall pipeline is implemented in an end-to-end manner and can be trained
by minimizing the softmax cross-entropy loss $\ell_{\text{CE}}$ of confidences
$o$ and ground-truth label $y\in\mathcal{L}$.

With the design of E-SCOUTER, we presume that the supports for a category, or
equivalently the regions specified by $\bar{A}$, only occupy smaller areas in
the image because of the locality of visual elements, which is encoded in Eq.
(3). The expected areas, though, can vary for different domains (or datasets).
More explicit control over the areas can be advantageous to encode our beliefs
about them. Hence, we introduce the area loss $\ell_{\operatorname{Area}}$ to
regulate the size of support regions, given by:

$\ell_{\operatorname{Area}}=\sum_{l\in\mathcal{L}}\mathbf{1}_{s}^{\top}\bar{a}_{l}^{(T)}$
(8)

summing all the elements in $\bar{A}^{(T)}$.

The overall loss is:

$\ell_{\operatorname{E-SCOUTER}}=\ell_{\operatorname{CE}}+\lambda\ell_{\operatorname{Area}},$
(9)

where $\lambda$ serves as a factor to adjust the weight of the area loss. By
increasing the value of $\lambda$, E-SCOUTER tends to focus on smaller
regions. Conversely, a smaller value of $\lambda$ encourages the model to
prioritize larger areas. The ablation study in Section IV-E provides
additional insights and details.

###  III-C Positive and Negative Explanation

The E-SCOUTER loss in Eq. (9) only offers a positive explanation and we
introduce a hyper-parameter $e\in\\{+1,-1\\}$ to Eq. (6) as:

$o=e\cdot U^{(T)}\mathbf{1}_{d}\quad\in\mathbb{R}_{+}^{n}.$ (10)

This hyper-parameter is pivotal in configuring E-SCOUTER to learn either
positive or negative supports. Since all elements of $U^{(T)}$ are constrained
to be non-negative, the model exhibits distinct behaviors for different values
of $e$.

Utilizing the softmax activation and cross-entropy loss, the model is trained
to assign the highest confidence value, $o_{l}$, to the ground-truth category
$l=y$, while giving smaller values, $o_{l}$, to the others (i.e., $l\neq y$).
As both $\bar{A}^{(T)}$ and $V$ are non-negative, all elements in $o$ are non-
negative as well.

For $e=+1$, larger values of $o_{l}$ can only be produced when some elements
in $\bar{a}^{(T)}_{l}$, which is the row vector in $\bar{A}^{(T)}$
corresponding to category $l$, are close to 1. Conversely, smaller values of
$o_{l}$ are obtained when all elements in $\bar{a}^{(T)}_{l}$ are close to 0.
Therefore, when $e$ is set to $+1$, the model learns to identify positive
supports $\mathcal{S}^{+}_{l}$ as a larger value of an element in $V$
positively amends $o_{l}$. The visualization of $\bar{a}^{(T)}_{l}$ thus
serves as a positive explanation, as illustrated in Fig. 1 (left).

In contrast, when $e=-1$, all elements in $o$ are negative, resulting in
$o_{l}$ being close to 0 when $l=y$, while $o_{l}$ gives a smaller value when
$l\neq y$. Due to the non-negativity of $V$, all elements in
$\bar{a}^{(T)}_{l}$ must be close to 0 for $l=y$, whereas smaller values of
$o_{l}$ are generated when $\bar{a}^{(T)}_{l^{\prime}}$ includes elements
close to 1. The model thus learns to discover negative supports
$\mathcal{S}^{-}_{l}$, which do not manifest in images of the ground-truth
category. Consequently, $\bar{a}^{(T)}_{l^{\prime}}$ can be used as a negative
explanation, as illustrated in Fig. 1 (right).

##  IV Experiments

###  IV-A Experimental Settings

We evaluated E-SCOUTER using two widely-recognized datasets: ImageNet [49] and
CUB200 [11]. These datasets benchmark the performance of classification
models. To cover small-scale tasks, which may be more realistic, we also
conducted evaluations on the MNIST [9] and Caltech [50] datasets.
Additionally, we undertook two case studies in medical image analysis. The
first case study focuses on diagnosing glaucoma using the ACRIMA dataset [51],
and the second involves multi-label classification for chest x-ray diagnosis
using the X-ray14 dataset [52].

All images used in our evaluation are the size of $260\times 260$. The number
of channels $c$ in $F$ is due to the backbone, and in $F^{\prime}$ is $d=64$.
Following the original slot-attention [14], the iterations of GRU defaulted to
$T=3$. All models underwent training on the training set of respective
datasets with a batch size of 256 for 40 epochs, and performance metrics were
computed on the validation set using the trained models at the last epoch. We
employed the AdamW optimizer [53] with an initial learning rate of $10^{-4}$,
which was reduced by a factor of ten after 30 epochs. Only horizontal flipping
is used as data augmentation. The implementations of the underlying CNN models
were sourced from the PyTorch timm library [54]. All our experiments were done
with a GPU server with 4 NVIDIA A100.

Figure 3: Classification performance of different models with FC classifier,
E-SCOUTER+ ($\lambda=10$), and E-SCOUTER- ($\lambda=10$). The horizontal axis
is the number of categories, where the first $n$ categories of the ImageNet
dataset were used; the vertical axis is the accuracy.

###  IV-B Evaluation Metrics

We utilize six metrics to assess the performance of E-SCOUTER in comparison to
other XAI methods. Metrics are computed based on the visualization results. To
evaluate positive explanations, the ground-truth object region, provided for
each image, covering the region corresponding to the ground-truth label
$y$,222Some datasets provide ground-truth object regions. is used as their
ground-truth region. On the other hand, each image has $n-1$ negative
explanations, one for each non-ground-truth category. For an image of category
$y$, we identify the least similar category (LSC) $l^{\prime}$ to $y$ to
evaluate the negative explanation. This choice is based on the observation
that LSC images consistently exhibit unequivocal negative explanations.

To find LSC, we use the semantic similarity between categories $y$ and
$l\in\mathcal{L}$ based on [55] as

$l^{\prime}=\operatorname*{arg\,min}_{l\in\mathcal{L}}\frac{2\cdot\operatorname{Depth}(\operatorname{LCM}(y,l))}{\operatorname{Depth}(y)+\operatorname{Depth}(l)},$
(11)

where $\operatorname{Depth}(\cdot)$ provides the depth of the category in
WordNet [56], and $\operatorname{LCM}(y,l)$ identifies the lowest common
ancestor of categories $y$ and $l^{\prime}$. For the CUB200 dataset, WordNet
is not applicable because there is no accurate relation for bird species, so
we computed similarity based on category-wise feature embedding using the
following equation:

$l^{\prime}=\operatorname*{arg\,min}_{l\in\mathcal{L}}\cos(E(y),E(l)),$ (12)

where $\cos(\cdot)$ denotes cosine similarity, and $E(\cdot)$ represents the
average of feature embeddings of a category’s training samples, extracted from
a pre-trained CLIP [57].

It should be noted that, regardless of positive and negative explanations, the
object regions used as ground truth should be viewed as just a rough proxy of
the desired explanation of respective XAI methods. For E-SCOUTER, a positive
explanation should pinpoint a visual concept (associated with the slot), which
should be at least in the object region but does not necessarily cover the
whole region.

We use the following six metrics for evaluation. In the definitions, we use
$x$ and $\bar{x}$ to represent the input image and the object region for the
target category (i.e., $y$ for a positive explanation and $l^{\prime}$ for a
negative explanation), as well as confidence $c(x)\in\mathcal{R}$. The object
region can be derived from the bounding box sourced from ImageNet [49] or the
segment mask in CUB200 [11]. We also denote the relevance map for the target
category as $r(x)$, which is resized to match $x$’s size.333For positive
E-SCOUTER, $c(x)=o_{y}$ and $r=a^{(T)}_{y}$.

(i) Precision represents a broadened version of the pointing game [58]. As
higher relevance scores should be concentrated in $\bar{x}$ for a successful
explanation, Precision measures the degree of the concentration by

$\displaystyle\text{Precision}=\frac{\sum_{p\in\bar{x}}r_{p}(x)}{\sum_{p\in
x}r_{p}(x)},$ (13)

where $r_{p}(x)$ give the relevance score of pixel $p$.

(ii) Insertion [6] is determined by incrementally adding pixels, prioritized
by their relevance, to an initially blank image and observing the evolution of
confidence. A successful explanation should give a higher relevance score to a
pixel relevant to the category, resulting in a swift increase in confidence
along with adding more pixels. Insertion is defined as the area under the
curve of the number of added pixels versus the confidence.

(iii) Deletion[6] is computed by gradually removing pixels from the original
image based on their relevance, in contrast to the insertion. Confidence
should rapidly decline if the explanation is meaningful. Deletion is defined
in the same way as the insertion.

(iv) Infidelity [59] measures how much perturbation
$\phi\in\mathbb{R}^{\gamma}$ to the input image $x\in\mathbb{R}^{\gamma\times
3}$ impacts the confidence and relevance map, given by

$\text{Infidelity}=\mathbb{E}_{\phi}[(\phi^{\top}r-(c(x)-c(x-\phi)))^{2}],$
(14)

where $\gamma$ is the number of pixels in the input image, and the expectation
is computed for $\phi\sim\mathcal{N}(0,\sigma^{2})$ with $\sigma=0.2$.444In
Eq. (14), we slightly abuse the notation $x-\phi$ to represent subtraction
with broadcasting $\phi$ to all channels of $x$.

(v) Stability [32] gauges the stability of an XAI method when the input
undergoes minor perturbations, such as Gaussian noise, given by

$\operatorname{Stability}=\frac{\|r(x)-r(x^{\prime})\|_{2}}{\|x-x^{\prime}\|_{2}},$
(15)

where $x^{\prime}$ is $x$ with minor white noise. Adding minor noise hardly
change the confidence, but the relevance map may change a lot if the method is
unstable. For this metric, the smaller the better.

(vi) Time required to obtain the explanation, including feature extraction for
classification, is also measured as a metric for computational burden.

For metrics (ii)-(v), we use the official implementation released in GitHub.
For metric (vi), we measure the average time over 10,000 samples from
ImageNet’s validation set.

###  IV-C E-SCOUTER as a Classifier

E-SCOUTER serves as an alternative to FC classifiers, offering intrinsic
interpretability. It is essential to quantify its classification efficacy
across a variety of backbone models and datasets. Fig. 3 illustrates the
performance of both positive and negative variants of E-SCOUTER (denoted as
E-SCOUTER+ and E-SCOUTER-, respectively) on ImageNet. The horizontal and
vertical axes are the number of first $n$ categories of the original ImageNet
categories used during training and the accuracy. An FC classifier,
E-SCOUTER+, and E-SCOUTER- are represented by yellow, red, and grey lines,
respectively. As the category number increases, the performance declines for
the FC classifier and E-SCOUTERs.

TABLE I: Classification accuracy across various datasets using $\lambda=10$
and ResNet [60] variants as the backbone.

Models | Classifier | ImageNet [49] | Caltech [50] | CUB200 [11]
---|---|---|---|---
ResNet-18 | FC | 0.6606 | 0.7918 | 0.7325
E-SCOUTER + | 0.6675 | 0.8058 | 0.7240
E-SCOUTER - | 0.6589 | 0.7950 | 0.7056
ResNet-50 | FC | 0.7550 | 0.8470 | 0.7850
E-SCOUTER + | 0.7604 | 0.8615 | 0.7730
E-SCOUTER - | 0.7522 | 0.8452 | 0.7544
ResNet-101 | FC | 0.7616 | 0.8755 | 0.7984
E-SCOUTER + | 0.7638 | 0.8794 | 0.7901
E-SCOUTER - | 0.7572 | 0.8687 | 0.7763

TABLE II: Comparison of E-SCOUTER and an FC classifier on ImageNet [49] in
terms of required computational resource ($n$ is 1,000 and the size of input
images are $260\times 260$).

| Params (M) | Flops (G)
---|---|---
Model | FC | E-SCOUTER | FC | E-SCOUTER
ResNet-18 [60] | 11.6895 | 11.3107 | 2.6527 | 2.7321
ResNet-50 [60] | 25.5570 | 23.7406 | 6.0158 | 6.1016
ResNeSt-26 [61] | 17.0694 | 15.2530 | 5.2087 | 5.2945
ResNeSt-50 [61] | 27.4832 | 25.6668 | 7.7841 | 7.8699
DenseNet-121 [62] | 7.9788 | 7.1208 | 3.7535 | 3.8337
DenseNet-169 [62] | 14.1494 | 12.6924 | 4.4399 | 4.5221
MobileNet-75 [63] | 2.5106 | 1.9372 | 0.2447 | 0.3249
MobileNet-100 [63] | 3.9329 | 3.1348 | 0.3409 | 0.4221
EfficeintNet-B2 [64] | 9.1099 | 7.8926 | 1.0541 | 1.1372
EfficeintNet-B5 [64] | 30.3897 | 28.5733 | 3.7067 | 3.7925
Inception-V3 [65] | 23.8345 | 22.0181 | 3.9594 | 4.0388
Inception-V4 [66] | 42.6798 | 41.3426 | 8.5120 | 8.5906

TABLE III: Evaluation of the explanations based on the ResNet-50 backbone over
ImageNet [49] and CUB200 [11] in terms of Precision (Pre.), Insertion (Ins.),
Deletion (Del.), Infidelity (Inf.) and Stability (Sta.). Time (in second) is
measured with 10,000 samples from the ImageNet validation set on a single
NVIDIA A100 GPU.

|  | CUB200 [11] | ImageNet[10]
---|---|---|---
| Methods | Pre. $\uparrow$ | Ins. $\uparrow$ | Del. $\downarrow$ | Inf. $\downarrow$ | Sta. $\downarrow$ | Pre. $\uparrow$ | Ins. $\uparrow$ | Del. $\downarrow$ | Inf. $\downarrow$ | Sta. $\downarrow$ | Time (s)
Positive | Saliency [4] | .4012 | .5562 | .1299 | .6008 | .1078 | .5644 | .5701 | .1417 | .8129 | .0968 | .0470
SmoothGrad [23] | .6408 | .6819 | .1553 | .6107 | .1006 | .6137 | .6537 | .1495 | .6834 | .0743 | .8950
InteGrad [24] | .6217 | .6728 | .1599 | .6178 | .1106 | .5987 | .6502 | .1545 | .6937 | .0817 | .8420
Occlusion [21] | .4145 | .5850 | .1566 | .6623 | .2483 | .5472 | .5723 | .2352 | .7751 | .1761 | .2210
GradCAM [5] | .6301 | .7474 | .0615 | .6428 | .1462 | .6850 | .6840 | .1308 | .8661 | .0714 | .0140
GradCAM++ [16] | .6355 | .7535 | .0628 | .6470 | .1323 | .7097 | .6921 | .1270 | .7908 | .0613 | .0240
ScoreCAM [13] | .6517 | .7601 | .0594 | .6271 | .1242 | .7128 | .7003 | .1241 | .7728 | .0627 | .1558
GroupCAM [67] | .6449 | .7580 | .0688 | .6140 | .1208 | .7089 | .7115 | .1327 | .7583 | .0644 | .1380
SESS [68] | .6625 | .7517 | .0651 | .6207 | .1524 | .7020 | .7036 | .1356 | .7152 | .1024 | .0512
RISE [6] | .5370 | .7312 | .0603 | .6039 | .2377 | .5564 | .6875 | .1315 | .5590 | .2541 | 5.179
Extremal Perturbation [27] | .5012 | .7250 | .1108 | .6361 | .2542 | .5297 | .6383 | .1555 | .5821 | .2957 | 4.826
IBA [17] | .5715 | .7084 | .1024 | .6159 | .1771 | .6440 | .7027 | .1363 | .6411 | .1554 | .8960
IGOS++ [69] | .5543 | .6018 | .1564 | .7155 | .3472 | .6969 | .6081 | .2265 | .6137 | .2773 | 6.027
Greedy-AS [70] | .5960 | .6817 | .1359 | .6380 | .1854 | .6101 | .6152 | .1715 | .6440 | .1513 | 7915.
E-SCOUTER +λ=1 | .6838 | .7329 | .0537 | .4682 | .2497 | .7200 | .7030 | .1367 | .3982 | .1853 | .2200
|  E-SCOUTER +λ=3 | .6910 | .7301 | .0495 | .5087 | .2202 | .7391 | .6980 | .1225 | .4207 | .1138 | .2160
|  E-SCOUTER +λ=10 | .7029 | .7183 | .0514 | .5133 | .2053 | .7459 | .6802 | .1184 | .4412 | .0987 | .2090
Negative | Saliency [4] | .5173 | .6673 | .0842 | .6535 | .1212 | .5437 | .5298 | .1644 | .6075 | .1396 | .0420
SmoothGrad [23] | .6350 | .6742 | .1018 | .6498 | .0798 | .6081 | .6125 | .1710 | .6351 | .1051 | .8890
Occlusion [21] | .4749 | .5398 | .2541 | .7278 | .2005 | .5240 | .5571 | .2114 | .7508 | .1737 | .2230
GradCAM [5] | .6881 | .6965 | .0721 | .6597 | .1078 | .6469 | .6391 | .1485 | .8209 | .0621 | .0151
GradCAM++ [16] | .6920 | .6990 | .0702 | .6517 | .1086 | .6563 | .6475 | .1438 | .8071 | .0619 | .0263
E-SCOUTER -λ=1 | .7136 | .6496 | .0608 | .5988 | .1130 | .7103 | .6028 | .1381 | .3755 | .1046 | .2236
E-SCOUTER -λ=3 | .7180 | .6717 | .0575 | .5839 | .1235 | .7241 | .6220 | .1306 | .3468 | .0876 | .2184
E-SCOUTER -λ=10 | .7285 | .6609 | .0580 | .6117 | .1199 | .7326 | .6107 | .1450 | .4514 | .0752 | .2170

Notably, E-SCOUTER+ consistently outperforms FC classifiers in terms of
classification accuracy across different backbone models. E-SCOUTER-, while
still comparable to the FC classifiers for a smaller number of categories,
exhibits a slight decrease by approximately less than 1% in some backbones,
like DenseNet [62], when $n$ comes closer to 1000. This observation suggests
that E-SCOUTER is competent even for large datasets, addressing a major
limitation of the original SCOUTER [8]. The improvement over the original is
elaborated in Section IV-F.

We further extended our experiments to the CUB200 [11] and Caltech [50]
datasets using the ResNet variants [60] as a backbone (Table I). The results
demonstrate that E-SCOUTER generalizes well across different domains and
achieves performance comparable to the FC classifiers. However, a notable
limitation is observed in its application to CUB200, particularly with
E-SCOUTER-. Unlike ImageNet, CUB200 is designed to differentiate among similar
bird images, and each category contains a limited number of images (50 per
class). These characteristics make it challenging to identify consistent and
effective supports that appear across all images within a category.

Table II presents a computational cost of E-SCOUTERs and the FC classifiers.
E-SCOUTER incurs floating-point operations per second (FLOPS) that are
slightly higher than that of the FC classifiers, while it maintains a slightly
lower number of parameters. The increase in FLOPS is attributed to the
inclusion of shallow FC layers (such as the $K$ and $Q$ layers) and a GRU
module within the slot attention module. When this additional computational
expense is juxtaposed against the variations in computational costs and
parameter numbers of different backbones, we would say the extra overhead
introduced by E-SCOUTER is almost negligible. This observation underscores the
efficiency of E-SCOUTER, and the slight increase in FLOPS can be seen as a
small trade-off for the significant benefits of model interpretability.

###  IV-D Explainability

####  IV-D1 Quantitative Analysis

This section undertakes a detailed analysis to understand how the
explainability of E-SCOUTER stands in comparison to other XAI methods. Table
III gives the scores of the metrics in Section IV-B over CUB200 and ImageNet.
Higher values ($\uparrow$) signify enhanced Precision and Insertion metrics.
Conversely, lower values ($\downarrow$) indicate better performance for
Deletion, Infidelity, Stability, and Time. To ensure robust analysis, we used
10,000 random samples in the ImageNet validation set and all validation
samples in CUB200. We also evaluated SOTA positive explanation methods. These
methods do not necessarily offer a negative variant. Hence, the comparison for
the negative variant is over a small subset of the methods, including GradCAM
[5] and Saliency [4] that use the negative direction of gradients555In
GradCAM, the negative direction of gradients is used to interpret the
counterfactual samples (shown in Fig. 8). It aims at explaining why an image
does not belong to category $l$, which is consistent with our negative
explanation..

E-SCOUTER demonstrates remarkable strengths for positive explanations. On
ImageNet, E-SCOUTER achieves the highest Precision (0.7459 for $\lambda$=10),
showing its ability to localize foreground objects. The Deletion score (0.1184
for $\lambda$=10) is also notably low. However, in terms of Insertion,
E-SCOUTER is worse than GroupCAM, though still among the top. It also achieves
the lowest Infidelity (0.3982 for $\lambda$=1), which shows E-SCOUTER’s
ability to explain the model decision even upon input perturbation faithfully.
Meanwhile, E-SCOUTER gave lower Stability scores with a certain margin than
gradient-based methods, showing that E-SCOUTER is more sensitive to small
changes in input images. This is possibly caused by the overall attention of
the slot computation, which counts noise more easily in the final decision-
making. Similarly, on CUB200, E-SCOUTER exhibits the best Precision (0.7029
for $\lambda$=10), lowest Deletion (0.0495 for $\lambda$=3), and lowest
Fidelity (0.4682 for $\lambda$=1), underscoring its effectiveness in
discerning fine-grained features for bird species classification. From the
Time metric, E-SCOUTER is more computationally expensive than the gradient-
based methods because of the iterative update of the slot. Yet, it is faster
than the perturbation-based methods.

(a) Explanation Confusion Matrix: why model predicts the images of [GT
Category] are [Predicted Category].

(b) Explanation Consistency: why model predicts the images of a same category
(“8”) are “8”.

Figure 4: Positive supports for MNIST [9]. Using ResNet-18 [60] as backbone
and $\lambda$=10.

(a) Explanation Confusion Matrix: why model not predicts the images of [GT
Category] are [Predicted Category].

(b) Explanation Consistency: why model predicts the images of the same
category (“8”) are not “7”.

Figure 5: Negative supports for MNIST [9]. Using ResNet-18 [60] as backbone
and $\lambda$=10.

We also observe that $\lambda$ influences the scores. Specifically, a larger
value of $\lambda$ tends to reduce the size of the area $A$ as designed, which
makes regions with higher attention weights more likely to fall into the
object region $\bar{x}$, resulting in a higher Precision.

There seems to be a trade-off between Insertion and Deletion. With a larger
$\lambda$, E-SCOUTER tries to find supports within smaller regions.
Consequently, deleting this area can effectively lower confidence. This
behavior aligns with our design objective to identify supporting evidence for
decisions. On the other hand, when computing Insertion, E-SCOUTER needs more
pixels to regain confidence. This requirement stems from the slot’s attention
mechanism, which necessitates a view of enough pixels to retain the
confidence. Stability is also decreased with an increase of $\lambda$. This
implies that concentrating on a smaller region can reduce the impact of minor
perturbations in breaking consistency between the model’s output and
explanation. However, limiting the size of the support region can result in a
sluggish response to significant perturbations, often resulting in worse
Infidelity scores.

Figure 6: Visualized positive explanations using E-SCOUTER+ and existing XAI
methods. We applied $\lambda$ equals to 1, 3, and 10 during the training
process.

For negative explanations, E-SCOUTER achieves the highest Precision scores
(0.7326 at $\lambda$=10). Notably, it records the lowest Deletion (0.1306 at
$\lambda$=3) and Infidelity (0.3468 at $\lambda$=3). Compared to its positive
variants, E-SCOUTER- demonstrates more significant improvements in
performance. This is particularly evident in Precision, where it surpasses
GradCAM++ by approximately 0.08. However, its weaknesses are still in
Insertion and Stability. One notable distinction from E-SCOUTER+ is the
influence of $\lambda$. A larger $\lambda$ generally aids E-SCOUTER in
deriving more effective explanations, but the best performance is achieved at
$\lambda=3$. This means that excessively prioritizing the area loss can lead
to sub-optimal explanations. These results indicate E-SCOUTER’s sensitivity to
$\lambda$.

In summary, across both datasets and variants, E-SCOUTER consistently
outperforms the SOTA methods in Precision, Deletion, and Infidelity, while
maintaining competitive scores in Insertion and Stability. This underscores
its efficacy in providing interpretable and reliable explanations for model
decisions. The time efficiency of E-SCOUTER is also notable, with it being
relatively low compared to perturbation-based methods.

####  IV-D2 Specificity and Consistency of Supports

To better understand E-SCOUTER’s interpretability, we visualized its attention
weights $\bar{a}_{l}$, which serves as an explanation for category $l$. Our
focus is three-fold: (i) identifying supports that E-SCOUTER relies on for
decision-making, (ii) discerning how these supports vary across different
categories, and (iii) examining their consistency within the same category.
MNIST [9] is particularly suitable for this analysis due to the clear
similarities and differences among its categories (i.e., digits), offering a
simpler context than ImageNet and other natural image datasets.

Firstly, we visualize $\bar{a}_{l}$ based on E-SCOUTER+ for
$l=\\{\texttt{0},\dots,\texttt{9}\\}$ from left to right for a random input
image of each category from top to down. We can see that the supports that
E-SCOUTER found in the input image are specific to the category. Secondly, we
give visualizations of $\bar{a}_{l}$ of some random input images of category
$l$ (i.e., when $l$ is the ground-truth (GT) category $y$). They can provide
some insights into the consistency of the supports. For this consistency
analysis, we specifically examine the digit 8 with ten randomly selected
samples from MNIST’s test set.

From the visualizations in Fig. 4(a), it is evident that E-SCOUTER+ adeptly
identifies supports for images belonging to their ground-truth categories.
Notably, strong attention is primarily observed only in the explanations for
the ground-truth category, which is placed diagonally. This highlights
E-SCOUTER+’s effectiveness in focusing on category-specific features. Turning
to the consistency analysis, Fig. 4(b) identifies the middle crossing part in
digit 8 for all samples, which is a distinctive feature of this digit. This
observation underscores E-SCOUTER+’s ability to find consistent supports for
the category.

Similarly, Fig. 5(a) gives visualization for E-SCOUTER-. Unlike the positive
variants, it does not identify supports for the ground-truth categories.
Instead, E-SCOUTER- consistently gives strong attention for non-ground-truth
categories. Given the simplicity of digit recognition, E-SCOUTER- employs
simple yet effective supports to refute most non-ground-truth categories. For
instance, to distinguish an image of 1 from others, it consistently focuses on
the central part of the vertical line because it can be a positive support for
digit 1 and a negative support for the others.666The vertical line is also a
negative support for digit 7, which also has a slanted line. This support may
respond to a line at a certain angle (i.e., closer to vertical) or may also
cover the (absence of) a horizontal line of digit 7. When differentiating an
image of 8 from digit 7, the horizontal line is highlighted as the negative
support. As shown in Fig. 5(b), this pattern of support is also consistently
observed across other image samples of different digit 8.

####  IV-D3 Qualitative Analysis for Natural Images

This section compares over natural images for E-SCOUTER and competing methods,
including saliency-based methods (Saliency [4] and SmoothGrad [23]), gradient-
based methods (GradCAM++ [16] and ScoreCAM [13]), and perturbation-based
methods (RISE [6] and IBA [17]).

(a) Negative support for Cock and Hen.

(b) Negative support for White Shark and Hammerhead.

(c) Negative support for Husky and Timer Wolf.

(d) Negative support for Red Fox and Kit Fox.

Figure 7: Visualized negative explanations using E-SCOUTER- and existing
methods. We applied $\lambda$ equals to -1, -3, and -10 during the training
process. Every image is marked on the left with its GT category label. The
accompanying explanations are specifically designed to elucidate why an image
of a certain GT category should not be classified into another category in the
pair.

Fig. 6 shows some examples of positive explanation by our method. They clearly
illustrate that E-SCOUTER+ effectively identifies supports that precisely
cover the crucial features distinctive to each category. For instance, the
American Eagle image, E-SCOUTER+ highlights the black and white region at the
neck. GradCAM++ and ScoreCAM also cover this area, encompassing a larger
region. This trend is shared among other examples. For the Horizontal Bar
image, our method mostly focuses on the part where the hand grips the pole. In
the Cup image, it highlights the junction between the cup body and handle.
Similarly, it identifies the part around the wing root for the Air Plane
image. For an image with multiple target objects, as in the images in the
second row from the top, E-SCOUTER+ covers all penguins or even differentiates
among them when $\lambda$ is higher (e.g., $\lambda$=10). It is also
noteworthy that increasing $\lambda$ tends to reduce the size of the
explanation area slightly. This constraint compels the model to seek out more
fine-grained visual supports that are specific to each category.

E-SCOUTER also provides negative explanations, as demonstrated in Fig. 7. We
show pairs of images of closely related categories to highlight its capability
to pinpoint negative supports. The ground-truth category for each image is
shown on the left, while the negative supports for the other category are
highlighted (e.g., Fig. 7(a)’s top image is of Cock, and the negative supports
for Hen are highlighted in the images). In Fig. 7(a), E-SCOUTER-’s negative
support for the image of Cock not being Hen consistently focuses on the cock’s
face, particularly noting the red face as a key distinguishing characteristic.
Conversely, when distinguishing the hen image from Cock, the model’s attention
is drawn to the neck feathers, likely influenced by the typically more lush
feathers found around a cock’s neck. This nuanced approach of E-SCOUTER-
starkly contrasts with other methods like GradCAM, which tend to cover the
entire foreground target, thus failing to isolate specific differentiating
features. SmoothGrad, while has similar explanation region as E-SCOUTER- in
its negative support for Cock, diverges in the case of negative support for
Hen by focusing on less distinctive areas like the feet and tail.

Figure 8: Visualization of positive and negative supports for counterfactual
samples. Each image is marked on the left with its assumed target category
label. TABLE IV: Performance comparison of E-SCOUTER and its variants on
ImageNet dataset. $\lambda$ is set to $10$ during training and ResNet-50 is
adopted as the backbone. The explanation performance is measured on the GT
category for the positive explanation and on the least similar class (LSC) for
the negative explanation.

Exp. Type | Variants | Computational Costs | Classification | Explainability
---|---|---|---|---
Params (M) | Flops (G) | Accuracy | Pre. $\uparrow$ | Ins. $\uparrow$ | Del. $\downarrow$ | Inf. $\downarrow$ | Sta. $\downarrow$
Positive |  E-SCOUTER + | 23.7406 | 6.1016 | 0.7595 | 0.7459 | 0.6802 | 0.1184 | 0.4412 | 0.0987
w.o. GRU | 23.3527 | 6.0755 | 0.7548 | 0.7319 | 0.6743 | 0.1232 | 0.4650 | 0.1068
w.o. PE | 23.7406 | 6.1016 | 0.7514 | 0.7283 | 0.6704 | 0.1292 | 0.4742 | 0.1132
Negative |  E-SCOUTER - | 23.7406 | 6.1016 | 0.7507 | 0.7326 | 0.6107 | 0.1450 | 0.4514 | 0.0752
w.o. GRU | 23.3527 | 6.0755 | 0.7460 | 0.7304 | 0.6059 | 0.1515 | 0.4827 | 0.1003
w.o. PE | 23.7406 | 6.1016 | 0.7413 | 0.7317 | 0.6001 | 0.1567 | 0.4902 | 0.0936

TABLE V: Area sizes of the explanations based on category similarity.
Calculation is implemented over 5,000 samples randomly selected from the
validation set of ImageNet.

Methods | Target Categories
---|---
GT | Highly-similar | Similar | Dissimilar
E-SCOUTER +λ=10 | .2863 | .1291 | .0714 | .0373
E-SCOUTER -λ=10 | .0116 | .0225 | .0572 | .1704

Fig. 7(b) also illustrates how E-SCOUTER- distinguishes different breeds of
sharks. The teeth serve as the primary feature for differentiating White Shark
from Hammer Headed, where the converse situation is identified mainly by the
head part. Similarly, Fig. 7(c) demonstrates that E-SCOUTER- consistently
focuses on the face to distinguish between Husky and Timber Wolf. Moreover, in
Fig. 7(d), the distinction is made evident as highlighting the face for Red
Fox and the ears for Kit Fox. These examples showcase the precision of
E-SCOUTER- in identifying specific features crucial for differentiating
similar categories. In addition, it is evident that increasing the value of
$\lambda$ effectively narrows the focus of the attention area, thereby
improving the ability to pinpoint negative supports (refer to Table III).

To further showcase E-SCOUTER’s effectiveness in simultaneously identifying
both positive and negative supports in a single image, we present examples
involving two objects. The first image (the left image at the top) contains
both Tiger Cat (a cat) and Bull Mastiff (a dog). The three images from the
left in the row are positive explanations for the category on the left, and
the other three images are negative explanations for the same category.
E-SCOUTER excels in locating the foreground object in its positive and
negative explanation. For instance, in the positive explanation, E-SCOUTER+
accurately identifies the heads of both the dog and cat when explaining “why
it is category $l$”. Similarly, in the negative explanation, it also
highlights the head region to illustrate “why it is not category $l$”.
GradCAM++ and SmoothGrad provide both positive and negative explanations, but
they are not designed to pinpoint discerning features. For example, GradCAM++
seems to spot the foreground objects, encompassing both the cat and dog for
Tiger Cat. This tendency is also seen in the second example, showing both
Hammer Headed and Stingray, where E-SCOUTER successfully distinguishes them.
These results demonstrate E-SCOUTER’s ability to learn fine-grained
explanations in complex scenarios where multiple categories appear
simultaneously.

####  IV-D4 Semantic Similarity of Categories and Explanations

E-SCOUTER is designed to give a larger relevance score (as an explanation) to
distinguishing features of the ground-truth category (or of non-ground-truth
in the negative variant), while showing a smaller value to other regions. It
is interesting to explore how E-SCOUTER reacts to semantically similar
categories. We make pairs of categories based on their similarity scores
defined in Eq. (11). Specifically, if the similarity between a pair of
categories is equal to or higher than $0.9$, the pair is marked as ‘highly
similar.’ If it is equal to or higher than 0.7 but less than 0.9, it is marked
as ‘similar,’ and any other pairs are considered ‘dissimilar.’ To quantify the
strength of explanation (SE), we use the sum of relevance scores, formulated
as

$\text{SE}_{l}(x)=\sum_{p\in x}r_{p}^{l}(x)/|x|,$ (16)

where we extend the notation of relevance score $r_{p}(x)$ to $r_{p}^{l}(x)$
to clarify the category, and $|x|$ gives the number of pixels in $x$.

Table V summarizes SE evaluated over the image of a category for the other
category of highly similar, similar, and dissimilar category pairs. Note that
SE is averaged over 5,000 samples randomly selected from the validation set of
ImageNet. A distinct trend emerges from these results: SE for E-SCOUTER+
decreases as the semantic similarity decreases. Conversely, E-SCOUTER- tends
to yield larger SE for categories that are less similar. This adaptive power
to explanatory regions based on category similarity is meaningful to
application scenarios requiring globally interpretable measures.

###  IV-E Ablation Study

This section quantifies the impact of GRU and PE, while retaining the others
unchanged.

To this end, we evaluate two variants: One is E-SCOUTER without GRU (i.e.,
$T=0$). The other variant is without PE, feeding flattened input features to
E-SCOUTER without positional information. These variants are trained and
evaluated over ImageNet with ResNet-50 backbone and $\lambda=10$. Table IV
shows the metrics. We observed that incorporating GRU incurs a minor increase
in the number of parameters and FLOPs. PE does not change computational
metrics. Notably, the exclusion of GRU leads to a decline in classification
accuracy. Removal of PE also worsens the accuracy. Moreover, the table clearly
demonstrates that GRU and PE contribute to the XAI metrics. We can conclude
that the absence of GRU and PE not only diminishes classification performance
but also adversely affects all aspects of explainability, underscoring their
vital roles in E-SCOUTER.

Figure 9: Classification and area size variance with different value of
$\lambda$. Experiments are implemented on ImagegNet and use ResNet-50 as
backbone. TABLE VI: Comparison between E-SCOUTER and SCOUTER in different XAI
metrics. All experiments are implemented on ImageNet (200 categories) with
ResNet-50 as a backbone and $\lambda$=10.

Method | Explainability
---|---
Pre. $\uparrow$ | Ins. $\uparrow$ | Del. $\downarrow$ | Inf. $\downarrow$ | Sta. $\downarrow$
E-SCOUTER + | .7392 | .7240 | .1218 | .4286 | .1045
SCOUTER+ | .7520 | .7085 | .1132 | .4325 | .1426
E-SCOUTER - | .7388 | .6573 | .1309 | .4032 | .0765
SCOUTER- | .7404 | .6456 | .1286 | .4217 | .0947

We also investigated the influence of the parameter $\lambda$ on the size of
explanatory regions and classification accuracy. We train the model five times
for each value of $\lambda$. Fig. 9 (left) demonstrates the influence of
$\lambda$ on the size of explanatory regions, in line with our area loss
design. The size is quantified similarly to the area loss in Eq. (8) but only
for the ground-truth categories (LSC for E-SCOUTER-). Formally, using the
notation $r_{p}(x)$, the area size is defined as:

$\text{Area size}=\sum_{p\in x}r_{p}(x)/|x|,$ (17)

where $|x|$ gives the number of pixels in $x$. Setting $\lambda$ to 0 results
in a larger area size. As $\lambda$ increases, the area size decreases.

Fig. 9 (right) illustrates the variation in accuracy. Our findings are two-
fold: (i) Higher values of $\lambda$ tend to improve accuracy for both the
positive and negative variants. (ii) A smaller attention region aids the model
in identifying significant features, particularly in E-SCOUTER-. Reasons here.
It is also noteworthy that the accuracy of E-SCOUTER- is initially more
diverse, and it becomes more stable as $\lambda$ increases. It is important to
note the possible trade-off involved: as depicted in Figure 9 (right), larger
$\lambda$ may adversely affect classification performance as we can see a
slight decrease in accuracy for E-SCOUTER- with $\lambda$=10. Furthermore, as
discussed in Section IV-D1, explainability also declines with higher
$\lambda$. In conclusion, $\lambda$ emerges as a crucial hyper-parameter for
E-SCOUTER. Tuning $\lambda$ for a target dataset can be the key to achieving a
balance between higher accuracy and better explainability.

Figure 10: Comparison of classification accuracy between E-SCOUTER and SCOUTER
across different category number for training ImageNet. We adopt 5 times
training for each category number setting (using ResNet-50 and $\lambda$=10).

###  IV-F Comparison to SCOUTER

We compare E-SCOUTER to the original SCOUTER. In this updated version, we have
made two key improvements.

Stability in training with more categories : Incorporating a normalization
step (refer to Eq. 3) has significantly stabilized training, enabling
E-SCOUTER to handle more than 200 categories, allowing for its utility in
various scenarios. To experimentally show this, we trained the model five
times for each category number, presenting violin plots with line plots of the
mean values. The improvement is evident in Fig. 10, where the classification
accuracies of E-SCOUTER+ (red line) and SCOUTER+ (yellow line) are depicted on
the left and of E-SCOUTER- (grey line) and SCOUTER- (blue line) on the right.
The plots clearly show a gradual decrease in E-SCOUTER’s accuracy. As
discussed in Section IV-C, E-SCOUTER’s performance is on par with the FC
classifier. SCOUTER achieves high accuracy for 200 categories in some training
attempts, but its performance significantly deteriorates with more categories.
Notably, SCOUTER experiences training failures (the accuracy is stuck to
almost the chance rate) even with fewer categories, while E-SCOUTER maintains
stability up to 1,000 categories. We found that our new normalization can
prevent slots from generating excessive attention values, which may
approximate hard attention and lead to training failures, stabilizing the
training.

Higher Interpretability: E-SCOUTER not only excels in terms of accuracy but
also exhibits superior interpretability in both negative and positive
versions. This enhanced interpretability is crucial, as it provides deeper
insights into the decision-making process of the model, making it more
transparent and trustworthy. As illustrated in Table VI (only using 200
categories for training), when evaluated against the five key XAI metrics,
E-SCOUTER outperforms SCOUTER (slightly lower in Precision and Delation),
maintaining a noticeable margin. These findings underscore the substantial
improvements from E-SCOUTER.

###  IV-G Case Study on Medical Image Diagnosis

E-SCOUTER incorporates an area loss feature that limits the size of the
support region. This is particularly advantageous in applications like medical
image classification, where smaller, more focused support regions can
accurately highlight symptoms, providing greater informative value. Notably,
existing methods lacked the capability to furnish negative explanations — a
critical need in medical diagnostics, where understanding the rationale for
ruling out certain conditions is as important as diagnosing them. E-SCOUTER is
able to address this for applications such as glaucoma detection (see Fig. 11)
and chest X-ray analysis (refer to Fig. 12). In Table VII, we present a
comparative analysis of both E-SCOUTER variants against the FC classifier,
utilizing two datasets. The comparison metrics include Area Under the Curve
(AUC), Accuracy (Acc.), F1-score, and an XAI metric Precision (Pre.). For the
ACRIMA dataset [51], the Precision metric is based on the foreground region of
the Optic Cup (OC), as done by the REFUGE [71]. In the case of the X-ray
dataset, Precision is determined using lesion annotations provided by the
Xray-14 dataset [52].

TABLE VII: Classification Performance on ACRIMA [51] and X-ray14 [52]
datasets. ResNet-50 is adopted as the backbone and $\lambda$ is defaulted as
10. For FC, we use GrandCAM++ [16] to compute the Precision (Pre.).

Dataset | Classifier | AUC | Acc. | F1 | Pre.
---|---|---|---|---|---
ACRIMA [51] | FC | .9992 | .9921 | .9866 | .3407
E-SCOUTER + | 1.000 | 1.000 | 1.000 | .4852
E-SCOUTER - | .9997 | .9929 | .9932 | .5008
Xray-14 [52] | FC | .7787 | .9453 | .8525 | .1858
E-SCOUTER + | .7801 | .9460 | .8560 | .2549
E-SCOUTER - | .7746 | .9405 | .8427 | .2316

Figure 11: Explanations for a positive sample in the glaucoma diagnosis
dataset. Top row shows the input image and previous XAI methods explanations.
Bottom row presents E-SCOUTER+ (first and second columns) and E-SCOUTER-
(third and fourth columns) explanations for Normal (N.) and Glaucoma (G.)
cases, using $\lambda=10$. Figure 12: Explanations for samples from the
Xray-14 [52] dataset. The results are compared among E-SCOUTER and other
methods. The disease name for each input sample is listed on the left and the
lesion area is annotated with a red bounding box.

We deployed E-SCOUTER with a $\lambda=10$ on the ACRIMA dataset [51], which
categorizes images into Normal and Glaucoma categories. Utilizing a ResNet-50
architecture as the backbone, we observed better classification performance
from both E-SCOUTER+ and E-SCOUTER- models compared to the FC classifier, as
detailed in Table VII. Our method also has much better Precision to locate the
OC area. Of particular note is the preference for E-SCOUTER in clinical
settings, where doctors value precise identification of regions in the optic
disc. As illustrated in Fig. 11, E-SCOUTER ’s visualizations pinpoint
clinically meaningful features, such as changes in vessel shape associated
with optic cup enlargement. While the IBA model also identifies small regions,
it often includes irrelevant or non-informative areas. In this context,
E-SCOUTER ’s capability to provide fine-grained negative explanations proves
particularly beneficial, especially when machine diagnoses diverge from
doctors’ initial assessments.

E-SCOUTER exhibits outstanding performance on the Xray-14 dataset, with the
positive variant surpassing the FC classifier across all classification
metrics. Although there is a slight decrease in performance in the negative
variant, it still remains comparable. A key strength of our method lies in its
ability to align the explanation area closely with lesion annotations. As
indicated in Table VII, both variants achieve higher Precision compared to
GradCAM++. Fig. 12 showcases a variety of visualization samples for each
disease category, where it is evident that E-SCOUTER typically generates
smaller explanation areas that more accurately coincide with the lesion
annotations (indicated by red bounding boxes). This is particularly notable in
complex diagnoses, such as nodules, where other methods struggle to identify
compact regions as decisive factors. This underscores E-SCOUTER’s
effectiveness in discriminative localization [15]. In conclusion, our approach
not only captures critical lesion features but also aligns more consistently
with medical professionals’ analyses.

##  V Discussion

We present E-SCOUTER, a visual explainable classifier. This method
distinguishes itself through its robust performance across diverse
explainability metrics and datasets.

As a classifier, E-SCOUTER showcases competitive performance in comparison to
traditional FC classifiers. Our extensive experiments demonstrate that
E-SCOUTER, in both its positive and negative variants, consistently maintains
high accuracy levels across a range of datasets and backbone models. These
results affirm E-SCOUTER’s effectiveness as a classifier. Moreover, its
contribution lies in the explainability. In recent years, XAI has emerged as a
critical focus, with the understanding of the rationale behind AI decisions
becoming increasingly important, particularly in high-stakes domains.
E-SCOUTER propels this field forward by offering explanations that are not
only more precise but also more reflective of the model’s actual decision-
making process. Additionally, by employing our unique attention area loss,
E-SCOUTER gains the capability to regulate the size of explanatory regions.
This feature enables E-SCOUTER to adaptively align with specific task
requirements, providing accurate explanations for model decisions.

E-SCOUTER’s capability for negative explanation is another key innovation. It
adeptly differentiates between similar categories by identifying negative
supports in an image. This ability to elucidate the “why not” behind a
classification decision is especially valuable in scenarios requiring fine-
grained classification. Furthermore, negative explanations can be invaluable
for machine teaching. Understanding the reasons behind a model’s exclusion of
certain categories can be as crucial as comprehending its final decision.

In the domain of medical image analysis, E-SCOUTER’s capacity to provide
explanations is exceptionally beneficial. As discussed in Section IV-G, it
pinpoints lesion areas in medical images, a critical aspect for diagnostic
accuracy. The precision with which E-SCOUTER identifies and elucidates
specific regions within an image equips healthcare professionals with a deeper
understanding of AI-based diagnostic decisions, thus bolstering trust in AI-
assisted medical diagnosis. In summary, E-SCOUTER signifies a leap forward in
XAI for image recognition. Its dual capability of providing both positive and
negative explanations, alongside its competitive performance relative to
traditional classifiers, and its potential applicability in crucial areas,
like medical imaging, render it an important step forward in the XAI field.

##  VI Conclusion

In this paper, we introduced a visually explainable classifier, E-SCOUTER.
Built upon slot attention, it enhances the model’s ability to focus on
relevant image regions and ensures that the explanations provided align with
the model’s internal reasoning processes. Our extensive experimental
evaluations have demonstrated that our method not only excels in providing
accurate and meaningful explanations of its classification decisions but also
maintains a high level of classification performance. These results attest to
the effectiveness of E-SCOUTER as a SOTA tool in the field of XAI. We believe
it contributes to the ongoing discourse in the AI community regarding
explainability and sets a new benchmark for future developments.

## References

  * [1] J. Amann, A. Blasimme, E. Vayena, D. Frey, and V. I. Madai, “Explainability for artificial intelligence in healthcare: a multidisciplinary perspective,” _BMC medical informatics and decision making_ , vol. 20, no. 1, pp. 1–9, 2020.
  * [2] B. H. Van der Velden, H. J. Kuijf, K. G. Gilhuijs, and M. A. Viergever, “Explainable artificial intelligence (xai) in deep learning-based medical image analysis,” _Medical Image Analysis_ , vol. 79, p. 102470, 2022.
  * [3] S. Reddy, “Explainability and artificial intelligence in medicine,” _The Lancet Digital Health_ , vol. 4, no. 4, pp. e214–e215, 2022.
  * [4] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” in _Workshop, ICLR_ , 2013.
  * [5] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in _CVPR_ , 2017, pp. 618–626.
  * [6] V. Petsiuk, A. Das, and K. Saenko, “RISE: Randomized input sampling for explanation of black-box models,” in _BMVC_ , 2018.
  * [7] P. Wang and N. Vasconcelos, “Scout: Self-aware discriminant counterfactual explanations,” in _CVPR_ , 2020, pp. 8981–8990.
  * [8] L. Li, B. Wang, M. Verma, Y. Nakashima, R. Kawasaki, and H. Nagahara, “Scouter: Slot attention-based classifier for explainable image recognition,” in _ICCV_ , 2021, pp. 1046–1055.
  * [9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [10] J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _CVPR_ , 2009, pp. 248–255.
  * [11] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-UCSD Birds 200,” California Institute of Technology, Tech. Rep. CNS-TR-2010-001, 2010.
  * [12] J. Chang, J. Lee, A. Ha, Y. S. Han, E. Bak, S. Choi, J. M. Yun, U. Kang, I. H. Shin, J. Y. Shin _et al._ , “Explaining the rationale of deep learning glaucoma decisions with adversarial examples,” _Ophthalmology_ , vol. 128, no. 1, pp. 78–88, 2021.
  * [13] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu, “Score-cam: Score-weighted visual explanations for convolutional neural networks,” in _CVPRW_ , 2020, pp. 24–25.
  * [14] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf, “Object-centric learning with slot attention,” _NeurIPS_ , vol. 33, pp. 11 525–11 538, 2020.
  * [15] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _CVPR_ , 2016, pp. 2921–2929.
  * [16] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in _WACV_ , 2018, pp. 839–847.
  * [17] S. Karl, S. Leon, T. Federico, and L. Tim, “Restricting the flow: Information bottlenecks for attribution,” in _ICLR_ , 2020.
  * [18] G. Ras, N. Xie, M. Van Gerven, and D. Doran, “Explainable deep learning: A field guide for the uninitiated,” _Journal of Artificial Intelligence Research_ , vol. 73, pp. 329–396, 2022.
  * [19] J. Kim and J. Canny, “Interpretable learning for self-driving cars by visualizing causal attention,” in _ICCV_ , 2017, pp. 2942–2950.
  * [20] D. Mascharka, P. Tran, R. Soklaski, and A. Majumdar, “Transparency by design: Closing the gap between performance and interpretability in visual reasoning,” in _CVPR_ , 2018, pp. 4942–4950.
  * [21] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _ECCV_. Springer, 2014, pp. 818–833.
  * [22] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” in _ICLR_ , 2014.
  * [23] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” in _Workshop, ICML_ , 2017.
  * [24] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _ICML_ , 2017, pp. 3319–3328.
  * [25] N. Akhtar, “A survey of explainable ai in deep visual modeling: Methods and metrics,” _arXiv preprint arXiv:2301.13445_ , 2023.
  * [26] D. Omeiza, S. Speakman, C. Cintas, and K. Weldermariam, “Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models,” _arXiv preprint arXiv:1908.01224_ , 2019.
  * [27] R. Fong, M. Patrick, and A. Vedaldi, “Understanding deep networks via extremal perturbations and smooth masks,” in _ICCV_ , 2019, pp. 2950–2958.
  * [28] Q. Zhongang, K. Saeed, and F. Li, “Visualizing deep networks by optimizing with integrated gradients,” in _AAAI_ , 2020.
  * [29] T. Fel, M. Ducoffe, D. Vigouroux, R. Cadène, M. Capelle, C. Nicodème, and T. Serre, “Don’t lie to me! robust and efficient explainability with verified perturbation analysis,” in _CVPR_ , 2023, pp. 16 153–16 163.
  * [30] N. Xie, F. Lai, D. Doran, and A. Kadav, “Visual entailment: A novel task for fine-grained image understanding,” _arXiv preprint arXiv:1901.06706_ , 2019.
  * [31] S. Stalder, N. Perraudin, R. Achanta, F. Perez-Cruz, and M. Volpi, “What you see is what you classify: Black box attributions,” in _NeurIPS_ , vol. 35, 2022, pp. 84–94.
  * [32] D. Alvarez-Melis and T. S. Jaakkola, “Towards robust interpretability with self-explaining neural networks,” _NeurIPS_ , 2018.
  * [33] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su, “This looks like that: Deep learning for interpretable image recognition,” in _NeurIPS_ , vol. 32, 2019.
  * [34] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang, “Concept bottleneck models,” in _ICML_ , 2020, pp. 5338–5348.
  * [35] B. Wang, L. Li, Y. Nakashima, and H. Nagahara, “Learning bottleneck concepts in image classification,” in _CVPR_ , 2023.
  * [36] S. Verma, V. Boonsanong, M. Hoang, K. E. Hines, J. P. Dickerson, and C. Shah, “Counterfactual explanations and algorithmic recourses for machine learning: A review,” _arXiv preprint arXiv:2010.10596_ , 2020.
  * [37] S. Vandenhende, D. Mahajan, F. Radenovic, and D. Ghadiyaram, “Making heads or tails: Towards semantically consistent visual counterfactuals,” in _ECCV_ , 2022, pp. 261–279.
  * [38] S. Khorram and L. Fuxin, “Cycle-consistent counterfactuals by latent transformations,” in _CVPR_ , 2022, pp. 10 203–10 212.
  * [39] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, “Counterfactual visual explanations,” in _ICML_ , 2019, pp. 2376–2384.
  * [40] R. Kommiya Mothilal, D. Mahajan, C. Tan, and A. Sharma, “Towards unifying feature attribution and counterfactual explanations: Different means to the same end,” in _AAAI_ , 2021, pp. 652–663.
  * [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , vol. 30, 2017.
  * [42] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [43] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, “Image transformer,” in _ICML_ , 2018, pp. 4055–4064.
  * [44] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_. Springer, 2020, pp. 213–229.
  * [45] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021.
  * [46] X. Wen, B. Zhao, A. Zheng, X. Zhang, and X. Qi, “Self-supervised visual representation learning with semantic grouping,” _NeurIPS_ , vol. 35, pp. 16 423–16 438, 2022.
  * [47] J. Jiang, F. Deng, G. Singh, and S. Ahn, “Object-centric slot diffusion,” _arXiv preprint arXiv:2303.10834_ , 2023.
  * [48] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, “Groupvit: Semantic segmentation emerges from text supervision,” in _CVPR_ , 2022, pp. 18 134–18 144.
  * [49] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009, pp. 248–255.
  * [50] G. Griffin, A. Holub, and P. Perona, “Caltech256 object category dataset,” California Institute of Technology, Tech. Rep., 2007.
  * [51] A. Diaz-Pinto, S. Morales, V. Naranjo, T. Köhler, J. M. Mossi, and A. Navea, “CNNs for automatic glaucoma assessment using fundus images: an extensive validation,” _Biomedical Engineering Online_ , vol. 18, no. 1, p. 29, 2019.
  * [52] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _CVPR_ , 2017, pp. 2097–2106.
  * [53] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [54] R. Wightman, “Pytorch image models,” 2019.
  * [55] Z. Wu and M. Palmer, “Verbs semantics and lexical selection,” in _ACL_ , 1994, p. 133–138.
  * [56] G. A. Miller, “Wordnet: a lexical database for english,” _Communications of the ACM_ , vol. 38, no. 11, pp. 39–41, 1995.
  * [57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021, pp. 8748–8763.
  * [58] J. Zhang, S. A. Bargal, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff, “Top-down neural attention by excitation backprop,” _IJCV_ , vol. 126, no. 10, pp. 1084–1102, 2018.
  * [59] C.-K. Yeh, C.-Y. Hsieh, A. Suggala, D. I. Inouye, and P. K. Ravikumar, “On the (in) fidelity and sensitivity of explanations,” _NeurIPS_ , vol. 32, 2019.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [61] H. Zhang, C. Wu, Z. Zhang, Y. Zhu, H. Lin, Z. Zhang, Y. Sun, T. He, J. Mueller, R. Manmatha _et al._ , “Resnest: Split-attention networks,” in _CVPR_ , 2022, pp. 2736–2746.
  * [62] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _CVPR_ , 2017, pp. 4700–4708.
  * [63] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [64] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _ICML_ , 2019, pp. 6105–6114.
  * [65] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _CVPR_ , 2016, pp. 2818–2826.
  * [66] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _AAAI_ , vol. 31, no. 1, 2017.
  * [67] Q. Zhang, L. Rao, and Y. Yang, “Group-cam: Group score-weighted visual explanations for deep convolutional networks,” _arXiv preprint arXiv:2103.13859_ , 2021.
  * [68] O. Tursun, S. Denman, S. Sridharan, and C. Fookes, “Sess: Saliency enhancing with scaling and sliding,” in _ECCV_. Springer, 2022, pp. 318–333.
  * [69] S. Khorram, T. Lawson, and L. Fuxin, “iGOS++ integrated gradient optimized saliency by bilateral perturbations,” in _Conference on Health, Inference, and Learning_ , 2021, pp. 174–182.
  * [70] Cheng-Yu, C.-K. Hsieh, X. Yeh, P. Liu, S. Ravikumar, S. Kim, Kuma, and C.-J. Hsieh, “Evaluations and methods for explanation through robustness analysis,” in _ICLR_ , 2021.
  * [71] J. I. Orlando, H. Fu, J. B. Breda, K. Van Keer, D. R. Bathula, A. Diaz-Pinto, R. Fang, P.-A. Heng, J. Kim, J. Lee _et al._ , “Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs,” _Medical image analysis_ , vol. 59, p. 101570, 2020.

|  Bowen Wang received a B.S. and M.S. degree in Computer Science from Anhui
University, China, in 2016 and in Medical Information from Osaka University,
Japan, in 2020, respectively, and a Ph.D. degree in Computer Science from
Osaka University, in 2023. He has received the best paper award in APAMI 2020
and is a member of IEEE. He is currently a specially-appointed researcher with
the Institute for Datability Science (IDS), Osaka University, Japan. His
research interests in computer vision, explainable AI, city perception, and
medical AI.
---|---
|  Liangzhi Li currently serves as the Chief Scientist at Xiamen Meet You Co.,
Ltd, and as the Head of Meetyou AI Lab (MAIL) since 2023. He also holds a
professorship at the Computer Science Department of Qufu Normal University
from the same year. He received the B.S and M.S degrees in Computer Science
from South China University of Technology (SCUT), China, in 2012 and 2016,
respectively, and the Ph.D. degree in Engineering from Muroran Institute of
Technology, Japan, in 2019. After his graduation, he worked as a researcher
(2019-2021) at Osaka University and later served as an Assistant Professor
(2021-2023). His main fields of research interest include computer vision,
explainable AI, and medical imaging. In his current roles, he is focusing on
advancing technology-driven solutions for business enhancements, contributing
to the field of computer science through educational excellence and carrying
out pioneering research in the realm of artificial intelligence.
---|---
|  Jiahao Zhang earned his Bachelor’s degree in Computer Science from
Zhengzhou University, China, and his Master’s degree in Medical Information
Research from Osaka University. He is currently pursuing his Ph.D. at the
Institute for Datability Science, Osaka University. His research focuses on
computer vision and medical AI.
---|---
|  Yuta Nakashima (M’09) received the B.E. and M.E. degrees in communication
engineering and the Ph.D. degree in engineering from Osaka University, Osaka,
Japan, in 2006, 2008, and 2012, respectively. From 2012 to 2016, he was an
Assistant Professor at the Nara Institute of Science and Technology. He is
currently a Professor at the Institute for Datability Science, Osaka
University. He was a Visiting Scholar at the University of North Carolina at
Charlotte in 2012 and at Carnegie Mellon University from 2015 to 2016. His
research interests include computer vision and machine learning and their
applications. His main research includes video content analysis using machine
learning approaches. Prof. Nakashima is a member of ACM, IEICE, and IPSJ.
---|---
|  Hajime Nagahara received the Ph.D. degree in system engineering from Osaka
University, Suita, Japan, in 2001. Since 2017, he is a Professor with the
Institute for Datability Science, Osaka University. He was a Research
Associate with the Japan Society for the Promotion of Science from 2001 to
2003. He was an Assistant Professor with the Graduate School of Engineering
Science, Osaka University, from 2003 to 2010. He was an Associate Professor
with the Faculty of Information Science and Electrical Engineering, Kyushu
University, from 2010 to 2017. He was a Visiting Associate Professor with CREA
University of Picardie Jules Verns, in 2005. He was a Visiting Researcher with
Columbia University in 2007–2008 and 2016–2017. His research interests include
computational photography and computer vision. He was the recipient of an ACM
VRST2003 Honorable Mention Award in 2003, IPSJ Nagao Special Researcher Award
in 2012, ICCP2016 Best Paper Runners-up, and SSII Takagi Award in 2016.
---|---