(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  University of Bonn 22institutetext: Goethe University
Frankfurt 33institutetext: MIT CSAIL 44institutetext: MIT-IBM Watson AI
Lab55institutetext: IBM Research

# LeGrad: An Explainability Method for Vision Transformers via Feature
Formation Sensitivity

Walid Bousselham  1 122 Angie Boggust  33 Sofian Chaybouti  1 122
Hendrik Strobelt  44 5 5 Hilde Kuehne  1 12244

###### Abstract

Vision Transformers (ViTs), with their ability to model long-range
dependencies through self-attention mechanisms, have become a standard
architecture in computer vision. However, the interpretability of these models
remains a challenge. To address this, we propose LeGrad, an explainability
method specifically designed for ViTs. LeGrad computes the gradient with
respect to the attention maps of ViT layers, considering the gradient itself
as the explainability signal.We aggregate the signal over all layers,
combining the activations of the last as well as intermediate tokens to
produce the merged explainability map. This makes LeGrad a conceptually simple
and an easy-to-implement tool for enhancing the transparency of ViTs. We
evaluate LeGrad in challenging segmentation, perturbation, and open-vocabulary
settings, showcasing its versatility compared to other SotA explainability
methods demonstrating its superior spatial fidelity and robustness to
perturbations.111A demo and the code is available at
https://github.com/WalBouss/LeGrad

Figure 1: LeGrad explainability maps: For a given vision-language model and a
textual prompt, LeGrad generates a heatmap indicating the part of the image
that is most sensitive to that prompt. LeGrad performs equally well across a
large range of model sizes, ranging from ViT-B/16(150M params.) to ViT-
bigG/14(2B params.).

##  1 Introduction

In the realm of computer vision, Vision Transformers (ViTs)[16] have
revolutionized the field with their ability to model long-range dependencies
through self-attention mechanisms [43, 27, 45, 4, 30]. Explanability methods
designed for convolutional or feed-forward neural networks are not directly
applicable to ViTs due to their architectural requirements, like GradCAM’s
[37] reliance on convolutional layers and Layer-wise Relevance Propagation’s
(LRP) [5] specific layer-wise propagation rules. While ViT-specific
interpretability techniques exist, including adaptations of traditional
methods [12, 37, 13], attention-based techniques [1, 44, 12, 13], and text-
based explanations [22, 20, 1], the interpretability especially of large-scale
architectures remains a challenge.

To address this, we propose a Layerwise Explainability method that considers
the Gradient with respect to the attention maps, LeGrad. LeGrad is
specifically designed for ViTs as leverages the self-attention mechanism
intrinsic to ViTs to generate relevancy maps highlighting the most influential
parts of an image for the model’s prediction. Compared to other methods,
LeGrad computes gradient with respect to the attention maps, as e.g. opposed
to CheferCam [12, 13], which uses the gradient to weight the attention maps.
This is done independently for each layer. The final explainability signal is
then pooled over all layers of the ViT. Note that using a layerweise gradient,
compared to other signals, allows to sum up over different layers without
further need for normalization. To further improve the signal, the gradient is
clipped by a ReLU function preventing negative gradients to impact positive
activations. The proposed approach is conceptually simple and versatile, as it
only requires the gradient w.r.t. to the ViT’s attention maps. This
facilitates its adoption across various applications and architecture,
including larger ViTs such as ViT-BigG as well as attention pooling
architectures [26, 49], enhancing the transparency of ViTs.

We evaluate the proposed method for various ViT backbones on three challenging
tasks: segmentation, open-vocabulary detection, and perturbation on ImageNet
[34, 19] as well as OpenImagesV7 [6]. It shows that while current methods
struggle especially with the diverse object categories in OpenImagesV7, LeGrad
can achieve performance gains of $2\times$–$5\times$, reaching a score of
$48.4$ p-mIoU on OpenImagesV7 using OpenCLIP-ViT-B/16. Furthermore, we
demonstrate the applicability of LeGrad to very large models, such as the ViT-
BigG/14 [14] with 2.5 billion parameters. Finally, it also adapts well to
different feature aggregation strategies employed by ViTs [32, 49], making it
a versatile tool for explainability.

We summarize the contributions as follows: (1) We propose LeGrad as a
layerwise explainability method based on the gradient with respect to ViTs
attention maps. (2) As the layerwise explainability allows to easily pool over
many layers, LeGrad scales to large architectures such as ViT-BigG/14 and is
applicable to various feature aggregation methods. (3) We evaluate LeGrad on
various tasks and benchmarks, showing its improvement compared to other state-
of-the-art explainability methods especially for large-scale open vocabulary
settings.

##  2 Related Work

Gradient-Based Explanation Methods Feature-attribution methods (also known as
saliency methods) are a commonly used explanation technique that explain model
decisions by assigning a score to each image pixel representing its importance
to the model’s output. Largely, these methods can be categorized into two
groups [29] — gradient-based methods that compute explanations based on the
gradient of the model’s prediction with respect to each input pixel [38, 17,
41, 40, 39, 24, 37] and perturbation-based methods that measure pixel-
importance by successively perturbing the input images and measuring the
impact on the model’s output [28, 33, 31, 11, 48]. While both types of methods
have been used successfully to identify reasoning, spurious correlation, and
trustworthiness in traditional computer vision models [7, 10], gradient-based
methods are often more computationally efficient since they only require a
single backwards pass and simple to understand since they are a direct
function of the model’s parameters and do not rely on additional models or
image modifications. However, many existing gradient-based methods were
designed for convolutional and feed-forward model architectures, so it is non-
trivial to directly apply them to ViTs since ViTs do not contain spatial
feature maps and include complex interactions between patches induced by the
self-attention mechanism. Instead, we build on the simplicity and
computational efficiency of gradient-based explanation methods to develop
LeGrad, a gradient-based feature-attribution method specifically designed for
ViT architectures.

While most gradient-based methods were designed prior to the widespread use of
ViTs, researchers have recently made efforts to adapt existing methods or
develop new ones specifically for transformers. Chefer et al. [12] extend LRP
[5] to transformers by integrating gradients within the self-attention layers.
However, this approach is computationally heavy and is inflexible to
architecture changes as it requires specific implementation for each module of
the network. To circumvent that complexity, CheferCAM [13] weights the
attention by their gradient and aggregates it through the layer via matrix
multiplication. However, the use of gradients to weigh the attention heads’
importance makes this method class-specific. LeGrad on the other hand is
class-agnostic and computationally efficient, making it simple to use and more
effective than these other methods (see Sec. 4 for a quantitative evaluation).

Attention-Based Explanation Methods A separate line of research has proposed
using attention maps, as opposed to gradients, as a way to explain for
transformers’ decisions [1, 44]. One attention-based method, rollout [1],
traces the flow of importance through the transformer’s layers by linearly
combining the attention maps via matrix multiplication. Attention flow [1]
contextualizes the attention mechanism as a max-flow problem; however, it is
computationally demanding and has not been extensively evaluated for vision
tasks. While these methods offer insights into the attention mechanism, they
often neglect the non-linear interactions between attention heads and the
subsequent layers. Moreover, they may not adequately distinguish between
positive and negative contributions to the final decision, leading to
potentially misleading interpretations, as found in Chefer et al. [12]. While
LeGrad also focuses on the attention maps to generate its explanation, it uses
the gradient w.r.t. to the attention maps, thereby assessing the sensitivity
of the attention maps to a change in the patch tokens. LeGrad does not attempt
to track the flow of information throughout the transformer layers but,
rather, focuses on assessing the sensitivity of the model prediction at each
layer for every location of the image.

Text-based Explanations of Intermediate Representations Research has also
explored text-based methods for interpreting intermediate representations in
vision models. For instance, leveraging CLIP’s language-image space,
researchers have provided text descriptions for active neuron regions [22, 20]
and projected model features into text-based concept banks [18]. In
particular, TextSpan [18] focuses on the explainability of CLIP-like models.
It refrains from using gradient computation by aggregating the intermediate
features’ similarities along a given text direction, creating a relevancy map
for a text query. LeGrad advances this line of work by focusing on the
sensitivity of feature representations within ViTs, offering a clear and
concise method to generate relevancy maps that can be adapted to various
feature aggregation strategies.

Evaluating Explanation Methods As the popularity of feature-attribution based
explanation methods has grown, so has research evaluating their ability to
faithfully reflect the model’s decision making process [41, 46, 25, 35, 23, 3,
2, 42, 21, 50]. Faithfulness is an important criteria because explanation
methods that highlight human-salient pixels may look correct even if the model
relies on spurious correlations and give human interpreters unwarranted
confidence in the model. These evaluations can be categorized based on the
aspect of faithfulness that they test [8], such as the method’s sensitivity to
meaningful changes to the input, model, or data [41, 46, 25, 35, 23, 3, 2, 42]
and its ability to highlight a succinct and meaningful pixel region [21, 50].
We evaluate LeGrad on both of these types of tests, demonstrating that it is
faithful to the model’s decision making process by testing that removing
LeGrad’s important pixels impacts the model’s output and checking that
LeGrad’s output corresponds to a concise object region.

Figure 2: Overview of LeGrad: Given a text prompt or a classifier
$\mathcal{C}$, for each layer $l$, an activation $s^{l}$ is computed and used
to compute the explainability of that layer. The layerwise explainability maps
are then merged to produce LeGrad’s output.

##  3 Method

In this section, we first introduce ViT’s mechanics and the different feature
aggregation mechanisms used for this architecture. We then explain the details
of LeGrad, starting by a single layer and then extending it to multiple
layers.

###  3.1 Feature formation in ViTs

The ViT architecture is a sequence-based model that processes images by
dividing them into a grid of $n$ patches. These patches are linearly embedded
and concatenated with a class token
$z_{0}^{0}=z^{0}_{[CLS]}\in\mathbb{R}^{d}$, which is designed to capture the
global image representation for classification tasks. The input image $I$ is
thus represented as a sequence of $n+1$ tokens
$Z^{0}=\\{z_{0}^{0},z_{1}^{0},\dots,z_{n}^{0}\\}$, each of dimension $d$, with
positional encodings added to retain spatial information.

The transformation of the initial sequence $Z^{0}\in\mathbb{R}^{(n+1)\times
d}$ through the ViT involves $L$ layers, each performing a series of
operations. Specifically, each layer $l$ applies multi-head self-attention
(MSA) followed by a multilayer perceptron (MLP) block, both with residual
connections:

$\displaystyle\hat{Z}^{l}$ $\displaystyle=\text{MSA}^{l}(Z^{l-1})+Z^{l-1},$
(1) $\displaystyle Z^{l}$
$\displaystyle=\text{MLP}^{l}(\hat{Z}^{l})+\hat{Z}^{l}.$

After $L$ layers, the image representation can be obtained via various
strategies:

[CLS] token: The class token approach, as introduced in the original ViT paper
[16], uses the processed class token as the image embedding
$\bar{z}_{[CLS]}=z^{L}_{0}$. This method relies on the transformer’s ability
to aggregate information from the patch tokens into the class token during
training.

Attentional Pooler: as used in [26, 47, 49], employs an additional multi-head
attention layer with a learnable query token $q_{pool}\in\mathbb{R}^{d}$. This
token interacts with the final layer patch tokens to produce the pooled
representation $\bar{z}_{AttnPool}$:

$\bar{z}_{AttnPool}=\text{softmax}\left(\frac{q_{pool}\cdot(W_{K}Z^{L})^{T}}{\sqrt{d}}\right)(W_{V}Z^{L}),$
(2)

where $W_{K},W_{V}\in\mathbb{R}^{d\times d}$ are learnable projection
matrices.

Independent of the feature aggregation strategy, it is important for an
explainability method to account for the iterative nature of feature formation
in ViTs. The goal should be to capture the contributions of all layers for the
final representation, providing insights into the model’s decision-making
process.

###  3.2 Explainability Method: LeGrad

We denote the output tokens of each block $l$ of the ViT as
$Z^{l}=\\{z_{0}^{l},z_{1}^{l},\dots,z_{n}^{l}\\}\in\mathbb{R}^{(n+1)\times
d}$, where $d$ is the dimensionality of each token.

Consider a classifier $\mathcal{C}\in\mathbb{R}^{d\times C}$, where $C$ is the
number of classes. This classifier can be learned during training for a
supervised classification task (e.g., ImageNet) or can be formed from text
embeddings of class descriptions for a zero-shot classifier, as in the case of
vision-language models like CLIP, generating a prediction $\bar{y}$. This
prediction $\bar{y}$ is obtained by passing the aggregated feature
representation of the ViT, noted $\bar{z}$, through the classifier
$\mathcal{C}$ and selecting the class with the highest activation:

$\bar{y}=\bar{z}\cdot\mathcal{C}\in\mathbb{R}^{C},$ (3)

The following is the description of how to obtain the explainability map for a
single layer, using the last layer as an example and then how to generalize it
to multiple layers. The overall method is visualized Figure 2.

Figure 3: LeGrad for a single layer.

Process for a Single Layer: To compute a 2D map that highlights the image
regions most influential for the model’s prediction of a particular class, we
focus on the activation with respect to the target class $\hat{c}$, denoted by
$s=\bar{y}_{[\hat{c}]}$. The attention operation within a ViT is key to
information sharing, and thus our method concentrates on this process.

We compute the gradient of the activation $s$ with respect to the attention
map of the last layer, as shown in Figure 3, denoted as
$\mathbf{A}^{L}\in\mathbb{R}^{h\times n\times n}$:

$\nabla\mathbf{A}^{L}=\frac{\partial
s}{\partial\mathbf{A}^{L}}\in\mathbb{R}^{h\times(n+1)\times(n+1)},$ (4)

where $h$ is the number of heads in the self-attention operation. Negative
contributions are eliminated by applying a ReLU function, and the gradient is
averaged across the patch and head dimensions:

$\hat{E^{L}}(s)=\frac{1}{h\cdot(n+1)}\sum_{h}\sum_{i}\left(\nabla\mathbf{A}^{L}_{h,i,.}\right)^{+}\in\mathbb{R}^{n+1}.$
(5)

The final explainability map is obtained by removing the column corresponding
to the [CLS] token to isolate the influence of the patch tokens, reshaping it
into a 2D map, and applying min-max normalization:

$E^{L}(s)=\text{norm}(\text{reshape}(\hat{E}^{L}(s)_{1:}))\in\mathbb{R}^{W\times
H}.$ (6)

Process for Multiple Layers: Recognizing that information aggregation occurs
over several layers, we extend the process to all layers. For each layer $l$,
we calculate the activation score $s^{l}$ using the intermediate tokens
$Z^{l}$ and derive the explainability map accordingly:

$\hat{E}^{l}(s^{l})=\frac{1}{h\cdot(n+1)}\sum_{h}\sum_{i}\left(\nabla\mathbf{A}^{l}_{h,i,.}\right)^{+}\in\mathbb{R}^{n+1}.$
(7)

Prior to normalization, we average the explainability maps from each layer:

$\displaystyle\bar{\mathbf{E}}$
$\displaystyle=\frac{1}{L}\sum_{l}\hat{E}^{l}(s^{l})_{1:}$ (8)
$\displaystyle\mathbf{E}$
$\displaystyle=\text{norm}(\text{reshape}(\bar{\mathbf{E}}))\in R^{W\times
H}.$

This approach queries each patch token at a given layer about its influence on
the prediction at that stage.

Adaptation to Attentional Pooler: For ViTs using an attentional pooler (e.g.
SigLIP [49]), a slight modification is made to compute the activation $s^{l}$
at each layer. We apply the attentional pooler module $Attn_{pool}$ to each
intermediate representation $Z^{l}$ to obtain a pooled query
$q^{l}\in\mathbb{R}^{d}$. The activation $s^{l}$ with respect to the desired
class $c$ is then computed as
$s^{l}=q^{l}\cdot\mathcal{C}_{:,c}\in\mathbb{R}$. Instead of considering the
self-attention map, we use the attention map of the attentional pooler,
denoted $\mathbf{A}_{pool}\in\mathbb{R}^{h\times 1\times n}$. Thus, for every
layer $l$, $\nabla A^{l}=\frac{\partial s^{l}}{\partial A^{l}_{pool}}$.

##  4 Experiments

###  4.1 Evaluation Settings

Implementation details: For the empirical evaluation of LeGrad, we utilized a
suite of Vision Transformer models sourced from the OpenCLIP library [14],
trained on the LAION dataset [36]. The models employed in our experiments
include ViT-B/16, ViT-L/14, ViT-H/14, and ViT-bigG/14.For SigLIP [49], we use
the ViT-B/16 with the "webli" weights. Each model processes input images at a
resolution of $224\times 224$ pixels. A key advantage of LeGrad is its
hyperparameter-free nature, which obviates the need for hyperparameter tuning.
Furthermore, we do not retraining or fine-tune the vision-language models,
thereby demonstrating their capabilities to localize queries by solely being
trained on image-level data without any localization annotations during
training or fine-tuning phases.

Object Segmentation: Following standard benchmarks [12, 13, 18] we evaluate
the ability of explainability methods to accurately localize an object in the
image. $\diamondsuit$Task: To do so, we generate image heatmaps based on the
activation of the groundtruth class for models trained with a classifier or
based on the the class descriptions "A photo of a [class]" for vision-language
models. Subsequently, we apply a threshold to binarize these heatmaps (using a
threshold of 0.5), thereby obtaining a foreground/background
segmentation.$\diamondsuit$Metric: We assess the quality of this segmentation
by computing the mIoU (mean Intersection over Union), pixel accuracy and the
mAP (mean Average Precision) zero-shot segmentations produced by different
explainability methods. This benchmark serves as a testbed for evaluating the
spatial fidelity of the explainability method. $\diamondsuit$Dataset: In our
evaluation of heatmap-based explainability methods, we adhere to a
standardized protocol and use the ImageNet-Segmentation dataset [19], with
$4,276$ images that provide segmentation annotations.

Open-vocabulary: For vision-language models, we extend our evaluation to
encompass open-vocabulary scenarios by generating explainability maps for
arbitrary text descriptions. This allows us to assess the quality of
explainability methods beyond the common classes found in ImageNet.
$\diamondsuit$Task: We generate a heatmap for each class object present in the
image, binarize them (using a threshold of 0.5) and assess the localization
accuracy. $\diamondsuit$Dataset/Metric: We employ the OpenImageV7 dataset [6],
which offers annotations for a diverse array of images depicting a broad
spectrum of objects and scenarios. Following [9], our evaluation utilizes the
point-wise annotations of the validation set, which contains 36,702 images
labeled with 5,827 unique class labels. Each image is associated with both
positive and negative point annotations for the objects present. In our
analysis, we focus exclusively on the classes that are actually depicted in
each image.

Perturbation-Based: To measure LeGrad’s ability to faithfully identify
features important to the model, we employ a perturbation-based methodology.
$\diamondsuit$Task: Given a classification dataset, we begin by generating
explainability maps for every image using the different explainability
methods. The analysis then consists of two complementary perturbation tests:
positive and negative. In the positive perturbation test, image regions are
occluded in descending order of their attributed relevance, as indicated by
the explainability maps. Conversely, the negative perturbation test occludes
regions in ascending order of relevance (see the Annex for more details and
visualizations of positive/negative perturbations). $\diamondsuit$Metric: For
both perturbation scenarios, we quantify the impact on the model’s accuracy by
computing the area under the curve (AUC) for pixel erasure, which ranges from
0% to 90%. This metric provides insight into the relationship between the
relevance of image regions and the model’s performance. The tests are
applicable to both the predicted and ground-truth classes, with the
expectation that class-specific methods will show improved performance in the
latter. This dual perturbation approach enables a comprehensive evaluation of
the network’s interpretability by highlighting the importance of specific
image regions in the context of the model’s classification accuracy.
$\diamondsuit$Dataset: Following common practices, we use the ImageNet
validation set, which contains $50K$ images and covers $1,000$ classes.

###  4.2 Comparison to State-of-the-Art

Segmentation Analysis: We first evaluate LeGrad as well as other methods in
the context of image segmentation using the ImageNet-segmentation dataset. As
shown in Table 2, LeGrad achieved a mIoU of 58.7%, surpassing other SOTA
explainability methods. Notably, it outperformed CheferCAM as a gradient-based
method for ViTs, and TextSpan as a non-gradient-based method, indicating its
robustness in capturing relevant image features for classification tasks.

Open-Vocabulary Performance: We further evaluate the performance on the
OpenImagesV7 dataset testing the capabilities of all methods in handling
diverse object categories. Note that for GradCAM we searched over the layers
and took the one that was performing the best. As shown in Table 2, LeGrad
outperforms all other SOTA methods, with performance gains ranging from
$2\times$ to $5\times$ compared to the second-best performing method. This can
be seen as an indicator for LeGrad’s capacity for fine-grained recognition.

Perturbation-Based Evaluation: We further performed the perturbation-based
evaluation on the ImageNet-val datasets. As detailed in Table 3, LeGrad’s
performance is here comparable to TextSpan for positive perturbations and
slightly superior for negative perturbations. For all other methods, LeGrad
outperformes both attention-based (e.g., "rollout" and "raw attention") and
gradient-based methods (e.g., GradCAM, AttentionCAM, and CheferCAM) across
various model sizes and for both predicted and ground truth classes,
emphasizing its ability to identify and preserve critical image regions for
accurate classification.

Performance on SigLIP: We finally evaluate the adaptability b regarding the
performance on SigLIP-B/16, a Vision-Language model employing an attentional
pooler as shown in Table 4. The results underscore the methods performance
across both negative and positive perturbation-based benchmarks. Notably, in
the open-vocabulary benchmark on OpenImagesV7, LeGrad achieved a p-mIoU of
25.4, significantly surpassing GradCAM’s 7.0 p-mIoU, the next best method.
These findings affirm the versatility of LeGrad, demonstrating its robust
applicability to various pooling mechanisms within Vision Transformers.
Further details on the methodological adaptations of LeGrad and other
evaluated methods for compatibility with SigLIP are provided in the annex.

Method | Pixel Acc.$\uparrow$ | mIoU$\uparrow$ | mAP$\uparrow$
---|---|---|---
LRP | 52.81 | 33.57 | 54.37
Partial-LRP | 61.49 | 40.71 | 72.29
rollout | 60.63 | 40.64 | 74.47
Raw attention | 65.67 | 43.83 | 76.05
GradCAM | 70.27 | 44.50 | 70.30
CheferCAM | 69.21 | 47.47 | 78.29
TextSpan | 73.01 | 40.26 | 81.4
LeGrad | 77.52 | 58.66 | 82.49
Table 1: Object Segmentation: method comparison on ImageNet-S using a
CLIP(ViT-B/16) model.

| p-mIoU $\uparrow$
---|---
Method | B/16 | L/14 | H/14 | bigG/14
rollout | 8.75 | 6.85 | 5.82 | 4.74
Raw attention | 0.94 | 1.60 | 0.85 | 0.07
GradCAM | 8.72 | 2.80 | 2.46 | 0.14
AttentionCAM | 5.87 | 4.74 | 1.20 | 0.72
CheferCAM | 5.87 | 2.51 | 9.49 | 9.71
TextSpan | 9.44 | 21.73 | 23.74 | 20.18
LeGrad | 48.38 | 47.69 | 46.51 | 36.47
Table 2: Open-Vocabulary Segmentation Performance: methods comparison on
OpenImagesV7 using different openCLIP model size.

|  | Negative |  | Positive
---|---|---|---|---
| Method | Predicted $\uparrow$ | Target $\uparrow$ |  | Predicted $\downarrow$ | Target $\downarrow$
ViT-B/16 | rollout | 44.36 | 44.36 |  | 23.03 | 23.03
Raw attention | 46.97 | 46.97 |  | 20.23 | 20.23
GradCAM | 32.01 | 45.26 |  | 36.52 | 22.86
AttentionCAM | 39.56 | 39.68 |  | 34.28 | 34.12
CheferCAM | 47.91 | 49.28 |  | 18.66 | 17.70
TextSpan | 50.92 | 52.81 |  | 15.10 | 14.26
LeGrad | 50.24 | 52.27 |  | 15.06 | 13.97
ViT-L/14 | rollout | 40.46 | 40.46 |  | 29.46 | 29.46
Raw attention | 47.14 | 47.14 |  | 23.49 | 23.49
GradCAM | 45.24 | 47.08 |  | 23.81 | 22.68
AttentionCAM | 45.81 | 45.84 |  | 31.18 | 31.03
CheferCAM | 49.69 | 50.37 |  | 20.67 | 20.14
TextSpan | 53.17 | 54.42 |  | 16.77 | 16.12
LeGrad | 53.11 | 54.48 |  | 15.98 | 15.23

|  | Negative |  | Positive
---|---|---|---|---
| Method | Predicted $\uparrow$ | Target $\uparrow$ |  | Predicted $\downarrow$ | Target $\downarrow$
ViT-H/14 | rollout | 49.37 | 49.37 |  | 32.91 | 32.91
Raw attention | 54.42 | 54.42 |  | 29.25 | 29.25
GradCAM | 45.26 | 45.48 |  | 40.98 | 40.91
AttentionCAM | 51.62 | 51.65 |  | 36.72 | 36.56
CheferCAM | 56.55 | 56.56 |  | 26.17 | 26.16
TextSpan | 60.14 | 61.91 |  | 20.07 | 19.14
LeGrad | 60.02 | 61.72 |  | 19.30 | 18.26
ViT-BigG/14 | rollout | 38.44 | 38.44 |  | 48.72 | 48.72
Raw attention | 56.56 | 56.56 |  | 31.25 | 31.25
GradCAM | 40.06 | 40.98 |  | 57.53 | 56.31
AttentionCAM | 54.68 | 54.74 |  | 39.28 | 39.27
CheferCAM | 57.95 | 58.27 |  | 29.21 | 28.45
TextSpan | 63.32 | 64.95 |  | 21.96 | 21.06
LeGrad | 62.62 | 64.67 |  | 21.33 | 20.28

Table 3: SOTA Perturbation Performance: Comparison of interpretability methods on the ImageNet-val using a different model size. | ImageNet | OpenImagesV7
---|---|---
| Negative | Positive |
Method | Predicted $\uparrow$ | Target $\uparrow$ | Predicted $\downarrow$ | Target $\downarrow$ | p-mIoU $\uparrow$
rollout | 47.81 | 47.81 | 25.74 | 25.74 | 0.07
Raw attention | 44.42 | 44.42 | 25.85 | 25.85 | 0.09
GradCAM | 41.25 | 44.42 | 35.10 | 33.50 | 6.97
AttentionCAM | 45.62 | 45.71 | 45.01 | 44.92 | 0.19
CheferCAM | 47.12 | 49.13 | 22.35 | 21.15 | 1.94
LeGrad | 50.08 | 51.67 | 18.48 | 17.55 | 25.40
Table 4: SOTA comparison on SigLIP-B/16: Comparison of interpretability
methods on perturbation-based tasks on ImageNet-val and open-vocabulary
localization on OpenImagesV7.

###  4.3 Ablation

####  4.3.1 Ablation Studies on Layer Contributions

Ablation studies are conducted to understand the impact of the number of
layers considered in the computation of LeGrad’s explainability maps. We
investigate this aspect across two distinct benchmarks: a perturbation-based
evaluation using the ImageNet validation set and an open-vocabulary
segmentation task on the OpenImagesV7 dataset. The experiments are performed
on various model sizes from the openCLIP library, including ViT-B/16,
ViT-L/14, and ViT-H/14.

Perturbation-Based Evaluation: In the perturbation-based evaluation, see
Figure 4(a), we employ a negative perturbation test as described in Section
4.1, using the ground-truth class for reference. The results indicate that the
ViT-B/16 model’s performance is optimal when fewer layers are included in the
explainability map computation. Conversely, larger models such as ViT-L/14 and
ViT-H/14 show improved performance with the inclusion of more layers. This
suggests that in larger models, the aggregation of information into the [CLS]
token is distributed across a greater number of layers, necessitating a more
comprehensive layer-wise analysis for accurate explainability.

Open-Vocabulary Segmentation: For the open-vocabulary segmentation task,
Figure 4(b), all models demonstrate enhanced performance with the inclusion of
additional layers in the explainability map computation. The optimal number of
layers varies with the model size, with larger models requiring a greater
number of layers. This finding aligns with existing literature [18], which
suggests that the information aggregation process in ViTs is more distributed
in larger architectures. However, it is also observed that beyond a certain
number of layers, the performance gains plateau, indicating that the inclusion
of additional layers does not further enhance the explainability map’s
quality.

These ablation studies underscore the importance of layer integration in the
construction of explainability maps for Vision Transformers. They further show
that while the inclusion of more layers can be beneficial, especially for
larger models, there exists an optimal range beyond which no additional
interpretability is gained. Indeed, we note in Figure 4 that including the
last $40\%$ of the ViT layers in LeGrad’s explainability computation is a good
trade-off. This can be helpful for the practical application of LeGrad, as it
allows for a more targeted and computationally efficient approach to
generating explainability maps without compromising on their quality.

(a) AUC for Negative perturbation on ImageNet-val for different layers used
for LeGrad

(b) point-mIoU on OpenImagesV7 for different layers used for LeGrad.

Figure 4: Ablation on the number of layers used in LeGrad for different
architecture sizes.

####  4.3.2 Ablations of ReLU and Layer Aggregation

We finally scrutinize the design choices underpinning LeGrad in Table 5.
Specifically, we investigate the effect of discarding negative gradients
before aggregating layer-specific explainability maps vai ReLU, as well as the
implications of leveraging intermediate feature tokens $Z^{l}$ to compute
gradients for each respective layer. We use the framework of the perturbation
benchmark delineated in Section 4.1 and both ViT-L/14 and ViT-H/14 models. The
results indicate that the omission of either component induces decline in
performance, thereby affirming the role these elements play in the
architecture of the method.

| L/14 | H/14
---|---|---
ReLU | All Layers | Negative$\uparrow$ | Positive$\downarrow$ | Negative$\uparrow$ | Positive$\downarrow$
✗ | ✗ | 47.81 | 20.80 | 57.57 | 21.73
✓ | ✗ | 49.32 | 19.95 | 59.55 | 19.50
✗ | ✓ | 52.01 | 16.80 | 60.28 | 18.26
✓ | ✓ | 54.48 | 15.23 | 61.72 | 18.26
Table 5: Ablation study: "ReLU" corresponds to whether or not negative
gradients are set to 0. "All layers" corresponds to whether or not the
intermediate tokens are used to compute the gradient for every layer or if
only the features from the last layer are used. Numbers are the AUC score for
the perturbation base benchmark using the target class to compute the
explainability map.

###  4.4 Qualitative analysis

####  4.4.1 Ablation Studies on Layer Contributions

To elucidate the interpretability of Vision Transformers (ViTs) through our
proposed LeGrad method, we extend our examination beyond quantitative ablation
studies to a qualitative analysis. This analysis scrutinizes the localization
maps generated by LeGrad for the last ten layers of various ViT models, as
well as the final map that merge these individual layer contributions. The
experiments are performed on various model sizes from the openCLIP library,
including ViT-B/16, ViT-L/14, and ViT-H/14.

The qualitative results, illustrated in Figure 5, reveal that the localization
of the prompt is not confined to a single layer but is distributed across
multiple layers. This observation is consistent with the findings from the
ablation studies in Section 4.3, which underscore the necessity of
incorporating multiple layers into the explainability framework to capture the
full scope of the model’s decision-making process.

A notable distinction is visible in the behavior of smaller models, such as
ViT-B/16, with that of larger counterparts like ViT-L/14 and ViT-H/14. In the
case of ViT-B/16, the explainability signal predominantly emanates from the
final layers, particularly the last two. Conversely, in larger models, earlier
layers also substantially influence the merged explainability map. For
instance, in Figure 5, layers 8 and 9 of ViT-H/14 appear to distinctly focus
on the heads of the cats, while other layers’ contributions are more diffused.
This observation not only corroborates the utility of incorporating multiple
layers into the explainability analysis but also suggests a more distributed
information aggregation process into the [CLS] token in larger models, as
posited in the literature [18].

The qualitative analysis thus reinforces the hypothesis that the mechanism by
which ViTs integrate information across layers is size-dependent, with larger
models engaging a broader range of layers in the synthesis of the global
feature representation. This is important for the practical application of
LeGrad, as it informs the selection of layers to be included in the
explainability map computation, ensuring both computational efficiency and the
retention of fidelity.

Figure 5: Qualitative analysis of the impact of each layer for different model
sizes using "a photo of a cat" as prompt. In smaller models the explainability
signal predominantly emanates from the final layers while in larger models
lower layers also contribute to the explainability map.

####  4.4.2 Qualitative Comparison to SOTA

Here, we present a qualitative analysis of the explainability maps generated
by LeGrad in comparison to other state-of-the-art (SOTA) methods. The visual
results are depicted in Figure 6, which includes a diverse set of
explainability approaches such as gradient-based methods (e.g., CheferCAM,
GradCAM), attention-based methods (e.g., Raw Attention weights visualization,
Rollout), and methods that integrate intermediate visual representations with
text prompts (e.g., TextSpan).

Our observations indicate that raw attention visualizations tend to highlight
a few specific pixels with high intensity, often associated with the
background rather than the object of interest. This pattern, consistent with
findings in the literature [9, 15], suggests that certain tokens
disproportionately capture attention weights. Consequently, methods that rely
on raw attention weights to construct explainability maps, such as CheferCAM,
exhibit similar artifacts. For instance, in the localization of "Wheel"
(Figure 6, row 2), the wheel is marginally accentuated amidst a predominance
of noisy, irrelevant locations.

In contrast, LeGrad is resilient to such distractions by leveraging gradients
with respect to attention maps. The presence of uniform noisy activations
across different prompts results in minimal gradients for these regions,
effectively filtering them out from the final heatmaps. This characteristic
enables LeGrad to produce more focused and relevant visual explanations.

The presence of noisy activations is also observed in methods like TextSpan
and GradCAM, which utilize intermediate feature representations. These
artifacts tend to surface in the same spatial regions as those highlighted by
raw attention weights, suggesting a potential link to tokens with inflated
attention. While the underlying cause of this behavior demands further
investigation, it is beyond the scope of the current work. Eventually, Figure
6 showcases LeGrad’s capability to accurately localize a wide range of objects
and concepts, from commonplace items like "baskets" to abstract notions such
as "numbers" and activities like "swimming." The method’s robustness and
versatility underscore its effectiveness in providing interpretable insights
into the decision-making processes of vision transformers.

Figure 6: SOTA Qualitative Comparison: visual comparison of different
explainability methods on images from OpenImagesV7.

##  5 Conclusion

In this work, we proposed LeGrad as an explainability method that highlights
the decision-making process of Vision Transformers (ViTs) across all layers
various feature aggregation strategies. We validated the method’s
effectiveness in generating interpretable visual explanations that align with
the model’s reasoning. Our approach offers a granular view of feature
formation sensitivity, providing insights into the contribution of individual
layers and attention mechanisms. The evaluation across object segmentation,
perturbation-based metrics, and open-vocabulary scenarios underscores LeGrad’s
versatility and fidelity in different contexts. By facilitating a deeper
understanding of ViTs LeGrad paves the way for more transparent and
interpretable foundation models.

## References

  * [1] Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928 (2020)
  * [2] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I.J., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS). pp. 9525–9536 (2018)
  * [3] Alvarez-Melis, D., Jaakkola, T.S.: On the robustness of interpretability methods. CoRR abs/1806.08049 (2018)
  * [4] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6836–6846 (2021)
  * [5] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015)
  * [6] Benenson, R., Ferrari, V.: From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142 (2022)
  * [7] Boggust, A., Hoover, B., Satyanarayan, A., Strobelt, H.: Shared interest: Measuring human-ai alignment to identify recurring patterns in model behavior. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. pp. 1–17 (2022)
  * [8] Boggust, A., Suresh, H., Strobelt, H., Guttag, J., Satyanarayan, A.: Saliency cards: A framework to characterize and compare saliency methods. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. pp. 285–296 (2023)
  * [9] Bousselham, W., Petersen, F., Ferrari, V., Kuehne, H.: Grounding everything: Emerging localization properties in vision-language transformers. arXiv preprint arXiv:2312.00878 (2023)
  * [10] Carter, B., Jain, S., Mueller, J., Gifford, D.: Overinterpretation reveals image classification model pathologies. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 15395–15407 (2021)
  * [11] Carter, B., Mueller, J., Jain, S., Gifford, D.K.: What made you do this? Understanding black-box decisions with sufficient input subsets. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 567–576. PMLR (2019)
  * [12] Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visualization. arXiv preprint arXiv:2012.09838 (2020)
  * [13] Chefer, H., Gur, S., Wolf, L.: Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 397–406 (2021)
  * [14] Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2818–2829 (2023)
  * [15] Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv preprint arXiv:2309.16588 (2023)
  * [16] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [17] Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Technical Report, Univeristé de Montréal (2009)
  * [18] Gandelsman, Y., Efros, A.A., Steinhardt, J.: Interpreting clip’s image representation via text-based decomposition. In: The Twelfth International Conference on Learning Representations (2023)
  * [19] Gao, S., Li, Z.Y., Yang, M.H., Cheng, M.M., Han, J., Torr, P.: Large-scale unsupervised semantic segmentation. IEEE TPAMI (2022)
  * [20] Goh, G., †, N.C., †, C.V., Carter, S., Petrov, M., Schubert, L., Radford, A., Olah, C.: Multimodal neurons in artificial neural networks. Distill (2021)
  * [21] Gomez, T., Fréour, T., Mouchère, H.: Metrics for saliency map evaluation of deep learning explanation methods. In: Pattern Recognition and Artificial Intelligence ICPRAI. Lecture Notes in Computer Science, vol. 13363, pp. 84–95. Springer (2022)
  * [22] Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., Andreas, J.: Natural language descriptions of deep visual features. In: International Conference on Learning Representations (2021)
  * [23] Hooker, S., Erhan, D., Kindermans, P., Kim, B.: A benchmark for interpretability methods in deep neural networks. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 9734–9745 (2019)
  * [24] Kapishnikov, A., Bolukbasi, T., Viégas, F.B., Terry, M.: XRAI: Better attributions through regions. In: Proceedings of the International Conference on Computer Vision (ICCV). pp. 4947–4956. IEEE (2019)
  * [25] Kindermans, P., Hooker, S., Adebayo, J., Alber, M., Schütt, K.T., Dähne, S., Erhan, D., Kim, B.: The (un)reliability of saliency methods. In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, vol. 11700, pp. 267–280. Springer (2019)
  * [26] Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., Teh, Y.W.: Set transformer: A framework for attention-based permutation-invariant neural networks. In: International conference on machine learning. pp. 3744–3753. PMLR (2019)
  * [27] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
  * [28] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)
  * [29] Molnar, C.: Interpretable Machine Learning (2019)
  * [30] Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4195–4205 (2023)
  * [31] Petsiuk, V., Das, A., Saenko, K.: RISE: Randomized input sampling for explanation of black-box models. In: Proceedings of the British Machine Vision Conference (BMVC). p. 151. BMVA (2018)
  * [32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [33] Ribeiro, M.T., Singh, S., Guestrin, C.: "Why should i trust you?": Explaining the predictions of any classifier. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD). pp. 1135–1144. ACM (2016)
  * [34] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252 (2015)
  * [35] Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.: Evaluating the visualization of what a deep neural network has learned. IEEE Transactions Neural Networks and Learning Systems 28(11), 2660–2673 (2017)
  * [36] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S.R., Crowson, K., Schmidt, L., Kaczmarczyk, R., Jitsev, J.: LAION-5b: An open large-scale dataset for training next generation image-text models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022)
  * [37] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: ICCV. pp. 618–626 (2017)
  * [38] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. In: Proceedings of the International Conference on Learning Representations (ICLR), Workshop Track (2014)
  * [39] Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M.: Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017)
  * [40] Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.A.: Striving for simplicity: The all convolutional net. In: Proceedings of the International Conference on Learning Representations (ICLR), Workshop Track (2015)
  * [41] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 3319–3328. JMLR. org (2017)
  * [42] Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., Preece, A.D.: Sanity checks for saliency metrics. In: Proceedings of the Conference on Artificial Intelligence. pp. 6021–6029. AAAI (2020)
  * [43] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International conference on machine learning. pp. 10347–10357. PMLR (2021)
  * [44] Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 5797–5808 (2019)
  * [45] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34, 12077–12090 (2021)
  * [46] Yeh, C., Hsieh, C., Suggala, A.S., Inouye, D.I., Ravikumar, P.: On the (in)fidelity and sensitivity of explanations. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 10965–10976 (2019)
  * [47] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)
  * [48] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. pp. 818–833. Springer (2014)
  * [49] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 (2023)
  * [50] Zhang, J., Lin, Z.L., Brandt, J., Shen, X., Sclaroff, S.: Top-down neural attention by excitation backprop. In: European Conference on Computer Vision (ECCV). vol. 9908, pp. 543–559. Springer (2016)

Supplementary material
LeGrad: An Explainability Method for Vision Transformers via Feature Formation
Sensitivity

##  7 Overview

In this supplementary material, we first provide in section 8 a link to a demo
of our novel method, LeGrad, showcasing its practical application. Detailed
information regarding the implementation, including pre-trained weights and
baseline methodologies for comparative evaluation, is presented in section 9.
An expanded evaluation of LeGrad’s performance, specifically on Vision
Transformers (ViT) trained with the ImageNet dataset, is documented in section
10. Section 10.2 offers a visual representation of the perturbation benchmarks
utilized to assess the efficacy of various explainability approaches.
Additional qualitative examples are provided in section 10.3 to further
illustrate the capabilities of our method. Lastly, section 11 includes a
disclaimer addressing the use of personal and human subjects data within our
research.

##  8 Demo

An image is worth a $1000$ words, we, therefore, provide a demo at:
https://huggingface.co/spaces/WalidBouss/LeGrad-Demo.

##  9 Implementation details

###  9.1 Pretrained weights

The experiments conducted in our study leverage a suite of models with varying
capacities, including ViT-B/16, ViT-L/14, ViT-H/14, and ViT-bigG/14. These
models are initialized with pretrained weights from the OpenCLIP library
respectively identified by: "laion2b_s34b_b88k", "laion2b_s32b_b82k",
"laion2b_s32b_b79k", and "laion2b_s39b_b160k". For the SigLIP method, we
utilize the ViT-B/16 model equipped with the "webl" weights.

###  9.2 Detailed Description of Baselines

In this section, we provide a concise overview of the baseline methods against
which our proposed approach is compared.

GradCAM: While originally designed for convolutional neural networks (CNNs),
GradCAM can be adapted for Vision Transformers (ViTs) by treating the tokens
as activations. To compute the GradCAM explainability map for a given
activation $s$, we calculate the gradient of $s$ with respect to the token
dimensions. The gradients are aggregated across all tokens and serve as
weights to quantify the contribution of each token dimension. Formally, for
intermediate tokens
$Z^{l}=\\{z_{0}^{l},z_{1}^{l},\dots,z_{n}^{l}\\}\in\mathbb{R}^{(n+1)\times
d}$, the GradCAM map $E_{GradCAM}$ is defined as:

$\displaystyle w$ $\displaystyle=\frac{1}{n}\sum_{i=0}^{n}\frac{\partial
s}{\partial z_{i}^{l}}\quad\in\mathbb{R}^{d\times 1\times 1}$ (9)
$\displaystyle\hat{E}_{GradCAM}$
$\displaystyle=\left(\frac{1}{d}\sum_{k=1}^{d}w_{d}*Z^{l}_{1:,d}\right)^{+}\quad\in\mathbb{R}^{n}$
$\displaystyle E_{GradCAM}$
$\displaystyle=\text{norm}(\text{resize}(\hat{E}_{GradCAM}))\quad\in\mathbb{R}^{W\times
H},$

with $d$ representing the token dimension, $*$ denoting element-wise
multiplication, and the superscript $+$ indicating the ReLU operation. We
empirically determined that applying GradCAM to layer 8 of ViT-B/16 yields
optimal results.

AttentionCAM: This method extends the principles of GradCAM to ViTs by
utilizing the attention mechanism within the transformer’s architecture.
AttentionCAM leverages the gradient signal to weight the attention maps in the
self-attention layers. Specifically, for the last block’s self-attention maps
$\mathbf{A}^{L}$, the AttentionCAM map $E_{AttnCAM}$ is computed as:

$\begin{aligned} \nabla\mathbf{A}^{L}&=\frac{\partial
s}{\partial\mathbf{A}^{L}}\in\mathbb{R}^{h\times(n+1)\times(n+1)}\\\
w&=\frac{1}{n^{2}}\sum_{i,j}\nabla\mathbf{A}^{L}_{:,i,j}\in\mathbb{R}^{h}\\\
\hat{E}_{AttnCAM}&=\sum_{p}^{h}\left(w_{p}*A^{L}_{p,:,:}\right)\in\mathbb{R}^{(n+1)\times(n+1)}\\\
E_{AttnCAM}&=\text{norm}(\text{resize}(\hat{E}_{AttnCAM})_{0,1:})\end{aligned},$
(10)

where $h$ denotes the number of heads in the self-attention mechanism.

Raw Attention: This baseline considers the attention maps from the last layer,
focusing on the weights associated with the [CLS] token. The attention heads
are averaged and the resulting explainability map is normalized. The Raw
Attention map $E_{Attn}$ is formalized as:

$\displaystyle\hat{E}_{Attn}$
$\displaystyle=\mathbf{A}^{L}_{:,0,1:}\in\mathbb{R}^{h\times 1\times n}$ (11)
$\displaystyle E_{Attn}$
$\displaystyle=\text{norm}(\text{resize}(\frac{1}{h}\sum_{k=1}^{h}(\hat{E}_{Attn})_{k}))\quad\in\mathbb{R}^{W\times
H}$

These baselines provide a comprehensive set of comparative measures to
evaluate the efficacy of our proposed method in the context of explainability
for ViTs.

###  9.3 Adaptation of Baseline Methods to Attentional Pooler

In the main manuscript, we introduced our novel method, LeGrad, and its
application to Vision Transformers (ViTs) with attentional poolers. Here, we
provide supplementary details on how LeGrad and other baseline methods were
adapted ViTs employing attentional poolers:

CheferCAM: Following the original paper [13] that introduces CheferCAM, we
considered the attentional pooler as an instance of a "decoder transformer"
and applied the relevancy update rules described in equation (10) of that
paper [13], (following the example of DETR). Since the attentional pooler has
no skip connection we adapted the relevancy update rule not to consider the
skip connection.

AttentionCAM: For AttentionCAM, instead of using the attention maps of the
last layer, we use the attention maps of the attentional pooler. We found this
variant to work better.

Raw Attention: Similarly, the Raw Attention baseline was adjusted by
substituting the attention maps from the last self-attention layer with those
from the attentional pooler.

Other Baselines: For the remaining baseline methods, no alterations were
necessary. These methods were inherently compatible with the attentional
pooler, and thus could be applied directly without any further adaptation.

The adaptations described above ensure that each baseline method is
appropriately tailored to the ViTs with attentional poolers, allowing for a
fair comparison with our proposed LeGrad method.

###  9.4 Mitigation of sensitivity to irrelevant regions:

We observe that for all evaluated explainability methods, SigLIP displays high
activations in image regions corresponding to the background. These
activations appeared to be invariant to the input, regardless of the gradient
computation’s basis, these regions were consistently highlighted. To address
this issue, we computed the explainability map for a non-informative prompt,
specifically "a photo of". We then leveraged this map to suppress the
irrelevant activations.

Namely, for an activation $s$ under examination, we nullify any location where
the activation exceeds a predefined threshold (set at 0.8) in the map
generated for the dummy prompt. Formally, let $E^{s}$ denote the
explainability map for activation $s$, and $E^{empty}$ represent the map for
the prompt "a photo of". The correction procedure is defined as follows:

$E^{s}_{E^{empty}>th}=0,$ (12)

where $th=0.8$. This method effectively addresses the issue without resorting
to external data regarding the image content.

##  10 Additional results

###  10.1 Image Classification

In this section, we extend our evaluation of the proposed LeGrad method to
Vision Transformers (ViTs) that have been trained on the ImageNet dataset for
the task of image classification. The results of this evaluation are presented
in Table 7, also providing a comparison with other state-of-the-art
interpretability methods.

It shows that LeGrad achieves superior performance on the perturbation-based
benchmark, particularly in scenarios involving positive perturbations.

Another observation is that even elementary interpretability approaches, such
as examining the raw attention maps from the final attention layer of the ViT,
demonstrate competitive results. In fact, these basic methods surpass more
complex ones like GradCAM (achieving an AUC of 53.1 versus 43.0 for negative
perturbations).

| Negative | Positive
---|---|---
Method | Predicted $\uparrow$ | Target $\uparrow$ | Predicted $\downarrow$ | Target $\downarrow$
rollout [1] | 53.10 | 53.10 | 20.06 | 20.06
Raw attention | 45.55 | 45.55 | 24.01 | 24.01
GradCAM [37] | 43.17 | 42.97 | 26.89 | 26.99
AttentionCAM [13] | 41.53 | 42.03 | 33.54 | 34.05
Trans. Attrib. [12] | 54.19 | 55.09 | 17.01 | 16.36
Partial-LRP [44] | 50.28 | 50.29 | 19.82 | 19.80
CheferCAM [13] | 54.68 | 55.70 | 17.30 | 16.75
LeGrad | 54.72 | 56.43 | 15.20 | 14.13
Table 7: SOTA comparison on ViT-B/16: Comparison of interpretability methods
on perturbation-based tasks on ImageNet-val for a ViT trained on ImageNet.

###  10.2 Pertubation example

Figure 8 illustrates the perturbation-based benchmark of Section 4.1 in the
main paper. Given the explainability map generated by the interpretatability
method, for the negative (respectively positive) we progressively remove the
most important (respectively the least important) part of the image. We then
look at the decline in the model accuracy.

Figure 8: Example of positive/negative perturbations: illustration of positive
and negative perturbations used in the perturbation-based benchmark. (Top
row): positive perturbation. Bottom row: negative perturbations

###  10.3 Qualitative examples

Finally, Figure 9 provides additional qualitative comparisons with state-of-
the-art explainability methods, illustrating the efficacy of the proposed
approach.

Figure 9: Qualitative Comparison to SOTA: visual comparison of different
explainability methods on images from OpenImagesV7

##  11 Personal and human subjects data

We acknowledge the use of datasets such as ImageNet and OpenImagesV7, which
contain images sourced from the internet, potentially without the consent of
the individuals depicted. We also recognize that the VL models used in this
study were trained on the LAION-2B dataset, which may include sensitive
content. We emphasize the importance of ethical considerations in the use of
such data.