(csquotes) Package csquotes Warning: Load ’inputenc’ before ’csquotes’

# PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-
Aware Prototype Alignment and Discrimination

Anant Khandelwal
Glance AI
<EMAIL_ADDRESS>
###### Abstract

The potential for zero-shot generalization in vision-language (V-L) models
such as CLIP has spurred their widespread adoption in addressing numerous
downstream tasks. Previous methods have employed test-time prompt tuning to
adapt the model to unseen domains, but they overlooked the issue of imbalanced
class distributions. In this study, we explicitly address this problem by
employing class-aware prototype alignment weighted by mean class probabilities
obtained for the test sample and filtered augmented views. Additionally, we
ensure that the class probabilities are as accurate as possible by performing
prototype discrimination using contrastive learning. The combination of
alignment and discriminative loss serves as a geometric regularizer,
preventing the prompt representation from collapsing onto a single class and
effectively bridging the distribution gap between the source and test domains.
Our method, named PromptSync, synchronizes the prompts for each test sample on
both the text and vision branches of the V-L model. In empirical evaluations
on the domain generalization benchmark, our method outperforms previous best
methods by 2.33% in overall performance, by 1% in base-to-novel
generalization, and by 2.84% in cross-dataset transfer tasks.

##  1 Introduction

Training Vision-Language Models (VLMs) with large-scale image-text pairs is
known for imparting robust generalization capabilities across diverse
downstream tasks [31, 20, 44, 41, 42, 1]. However, training these models from
scratch for each downstream task is very time-consuming. Moreover, the essence
of pre-training with a large-scale dataset is lost when the pre-trained model
is not generalizable across downstream tasks. This is due to unexpected
changes in data distribution, and the sensitivity to these shifts leads to a
decline in performance [16, 32, 30]. To tackle this, there exist three most
commonly used techniques: fine-tuning [28], prompt tuning [47], adapter [13],
and LoRA [19]. Among these, prompt tuning is the simple, recent, and most
widely used technique for foundation models [47, 46, 21, 22, 43]. However,
prompt learning/tuning approaches are used during the training phase to learn
representative prompts based on the training data for the downstream task.
This approach does not specifically address the distribution shift present in
the dataset. Recent methods, TPT [35] and PromptAlign [33], adjusts the
learnable prompt tokens dynamically during testing to enable test-time
adaptation and align the context of the test sample as per the seen
distribution by the model. Specifically, TPT [35] updates the learnable prompt
tokens (keeping the model parameters frozen) by minimizing the entropy of
top-$N_{k}$ confidently predicted samples, acquired through diverse augmented
views of the incoming test sample. Additionally, PromptAlign [33] aligns token
distribution of the test sample in the visual branch with the pre-computed
statistics of the complete proxy source dataset irrespective of the fact that
one class distribution may have different mean and variance than the other
classes.

In this work, we demonstrate the multi-modal test-time adaptation of prompts.
In contrast to PromptAlign, which aligns the distribution for the complete
source dataset with test sample, we propose class-aware prototype alignment to
address the distributional shift on a class-wise basis. For instance, in an
open world there are 360 different breeds of dogs compared to only 71 for
cats, leading to one class having higher variance than the others. For each
test sample, we obtain randomly augmented views (for both text and image) that
are fed to the model for prompt tuning on both the textual and visual
branches. We adapt the learnable prompt tokens by aligning the prototype for
test sample and confident augmented views with the pre-computed class
prototypes (obtained from the proxy source dataset) weighted by the mean
probability of each class obtained from confident augmented views. Before
alignment, we update the prompt tokens on both the text and visual branches
using prototype discrimination and then use updated prompts to align the test
sample and augmented views with class prototypes using mean class
probabilities. This is based on the idea that prototype vector can capture the
complete information of mean and variance for each class distribution and
hence it mitigates the class collapse (during test time adaptation) due to
high variance of particular classes. Empirical evaluation of our methods shows
state-of-the-art Top-1 accuracy for three tasks: domain generalization, base-
to-novel generalization, and cross-dataset transfer. This validates the
effectiveness of our method in enhancing zero-shot generalization. Our
contributions can be summarized as follows:

  * •

We propose a class-aware prototype alignment technique for individual test
samples to align the context of each test sample with the source distribution
on a class-wise basis, thereby mitigating the effects of distributional shift
between classes.

  * •

We propose class-aware prototype discrimination to discover the class
distribution for efficient alignment. Additionally, we propose the offline
computation of class prototypes from a proxy source dataset for foundation V-L
models.

  * •

We propose multi-modal test-time prompt tuning for both text and visual
branches. Empirical evaluation on base-to-novel generalization, domain
generalization, and cross-dataset transfer shows the efficiency of our method
over existing methods.

##  2 Related Work

Vision-Language (V-L) foundation models like CLIP [31] and ALIGN [20] have
emerged as robust zero-shot generalizable models. They integrate image and
text modalities through pre-training on extensive image-text pairs. However,
adapting these models to specific downstream tasks with limited data remains
challenging. Recent methods explore prompt tuning in CLIP-like models,
treating prompts as continuous learnable vectors and fine-tuning them while
keeping the model parameters frozen. CoOp [47] proposed fine-tuning CLIP by
learning a set of prompts in the text encoder. CoCoOp [46], an improvement
over CoOp, dynamically conditions the text prompts by the image embeddings.
MaPLe [21] is a deep prompting baseline that tunes prompts on both text and
image branches, further conditioning image prompts on text prompts using a V-L
coupling function. However, these approaches necessitate training data for
prompt learning, limiting adaptation to novel datasets during test time.
Recent approaches like TPT [35] aim to learn prompts exclusively at test time
but encounter challenges in handling distribution misalignment between CLIP’s
pre-training data and downstream test data. PromptAlign [33] addresses this by
introducing token distribution alignment in the image branch. However, it does
not account for the potential variance in class distributions. In contrast,
our method, inspired by a multi-modal prompting variant [21], actively aligns
class prototypes by leveraging a proxy dataset as a substitute for unavailable
CLIP pre-training data. To our knowledge, our approach is the first to
explicitly address class-aware distribution misalignment in V-L foundational
models during test time.

##  3 Methodology

Revisiting CLIP: Our approach is based on the pre-trained V-L model:
Contrastive Language-Image Pre-Training (CLIP). It consists of a text and
visual encoder (denoted by $\mathcal{F}_{t}$ and $\mathcal{F}_{v}$,
respectively, and their pre-trained parameters are represented by
$\theta_{\textsc{\tiny{CLIP}}}=\\{\theta_{t},\theta_{v}\\}$, respectively),
used for mapping the text and image to the vector representation,
respectively. The input image is $\mathbf{X}$, which is divided into $M$
patches, and the [CLS] token is prepended to these $M$ patch tokens that are
projected to produce
$\tilde{\mathbf{X}}_{v}=\\{\mathbf{e}_{\tiny{\textsc{[CLS]}}},\mathbf{e}_{1},\mathbf{e}_{2},......\mathbf{e}_{M}\\}$,
where $e_{i}$ is the embedding for the corresponding patch token in
$\mathbf{X}$. The image encoder produces latent visual feature representation
$\tilde{\boldsymbol{f}}_{v}=\mathcal{F}_{v}(\tilde{\mathbf{X}}_{v},\theta_{v})$
with transformer blocks from $\tilde{\mathbf{X}}_{v}$. The class label $y$ is
embedded within a text template, such as “a photo of a <CLS>” resulting in
$\tilde{\mathbf{X}}_{t}=\\{\tiny{SOS},\mathbf{t}_{1},\mathbf{t}_{2},...,\mathbf{t}_{L},\mathbf{c}_{k},\tiny{EOS}\\}$,
where SOS and EOS are the start and end token embeddings and
$\mathbf{t}_{l}|_{l=1}^{L}$ and $\mathbf{c}_{k}$ are the token embeddings
corresponding to the text template and the class label, respectively.
Similarly, the text encoder $\mathcal{F}_{t}$ encodes $\tilde{\mathbf{X}}_{t}$
with transformer blocks to produce latent text feature representation
$\tilde{\boldsymbol{f}}_{t}=\mathcal{F}_{t}(\tilde{\mathbf{X}}_{t},\theta_{t})$.
For zero-shot inference, each text feature for class labels
$y=\\{1,2,.....C\\}$ is paired with an image feature to compute the similarity
score
$s_{i}=\textrm{sim}(\tilde{\boldsymbol{f}}_{t_{i}}\cdot\tilde{\boldsymbol{f}}_{v})$
where $\textrm{sim}(\cdot)$ denotes cosine similarity. The predicted
probability on X for each $y_{i}$ is given as
$p(y_{i}|\textbf{X})=\frac{e^{\textrm{sim}(\tilde{\boldsymbol{f}}_{t_{i}}\cdot\tilde{\boldsymbol{f}}_{v})/\tau}}{\sum_{j=1}^{C}e^{\textrm{sim}(\tilde{\boldsymbol{f}}_{t_{j}}\cdot\tilde{\boldsymbol{f}}_{v})/\tau}}$,
where $\tau$ is the temperature of softmax.
Prompt Tuning: CLIP integrates a considerable pool of knowledge derived from
its training on millions of image-text pairs characterized by varying degrees
of noise. Prompt tuning methods aim to extract the rich features learned by
the CLIP model. Recent approaches [46, 47, 21, 3, 43] append extra learnable
prompts to the input of image and text encoders while keeping them frozen.
Modified input prompts with frozen encoders generate undistorted and rich CLIP
features, where prompt tuning tries to map the context to the source
distribution, i.e., the CLIP pre-training dataset. In our work, we use a
recent multi-modal prompting baseline [21] where prompt tuning is performed on
both the text and image encoders. Specifically, the image and text encoders
process the input
$\tilde{\mathbf{X}}_{v}=\\{\mathbf{e}_{\tiny{\textsc{[CLS]}}},\mathbf{p}_{v},\mathbf{e}_{1},\mathbf{e}_{2},......\mathbf{e}_{M}\\}$
and
$\tilde{\mathbf{X}}_{t}=\\{\tiny{SOS},\mathbf{p}_{t},\mathbf{t}_{1},\mathbf{t}_{2},...,\mathbf{t}_{L},\mathbf{c}_{k},\tiny{EOS}\\}$
respectively. The learnable prompts $\textbf{p}_{v}$ and $\textbf{p}_{t}$
represent the $V$ visual and $T$ textual tokens, respectively. We will call
prompts $\textbf{p}_{t}$ and $\textbf{p}_{v}$ as p only. Our approach is based
on deep prompting, as in[21], along with text and image prompts at subsequent
transformer blocks. We suggest referring to [21] for more details on baseline
architecture.
Test Time Adaptation: Test-time adaptation aims to boost generalisation in a
zero-shot manner. Existing methods, Test time prompt tuning (TPT)[35] and
PromptAlign[33], both are introduced to provide the model context that is
customized for each individual test sample in order to extract rich knowledge
from CLIP. For both methods, several augmented views
$\mathcal{H}(\textbf{X}_{test})$ are generated from the given test sample
$\textbf{X}_{test}$. The average entropy for the filtered views (selected
using a confidence threshold) is then used to update the prompts p using the
following unsupervised objective:

$\mathcal{L}_{ent}=\textrm{arg}\underset{\textbf{p}}{\textrm{min}}-\sum_{i=1}^{C}\tilde{p}_{\textbf{p}}(y_{i}|\textbf{X}_{test})\log\tilde{p}_{\textbf{p}}(y_{i}|\textbf{X}_{test})$
(1)

where $\tilde{p}_{\textbf{p}}(y_{i}|\textbf{X}_{test})$ is the average of
vector class probabilities (over the filtered augmented views) produced by the
model. Additionally, PromptAlign uses distribution alignment loss, which
aligns the mean and variance of filtered augmented views of the test sample
with source statistics across layers of the model.

Figure 1: Architecture of proposed PromptSync method for zero-shot
generalization in CLIP. During test time, we updates the learnable prompts
using discriminative and alignment of class prototypes. For a single test
example, we obtain multiple augmented views and obtain the mean class
probabilities after parameter updates with discriminating loss. Mean class
probabilities act as weights in the class-prototype alignment with filtered
augmented views. Gradient are accumulated over multiple iterations before
final update to the learnable prompts.

###  3.1 Proposed Method: PromptSync

The multi-modal test-time prompt tuning method, PromptAlign [33], updates text
and visual prompts using entropy loss and distribution alignment loss with
highly confident augmented views (obtained from a test sample
$\textbf{X}_{\text{test}}$). PromptAlign, despite considering the
distribution, does not take into account the fact that the distribution of
each class/domain can be entirely different from other classes/domains, and
hence using the source statistics of mean and variance for distribution
alignment can still be suboptimal. Inspired by prototype learning [38] and
Extreme-Multi-PatchSSL (EMP-SSL) [39], which establish a prototype/benchmark
for each class/sample, we propose class-wise prototype alignment between
original and augmented views for both source and test samples. The
architecture of PromptSync is shown in Figure 1. We use the parameter update
from prototype discrimination to generate the class probabilities for the test
sample and its augmented views. We accumulate the average of gradients from
prototype alignment loss weighted by class probabilities for confident
augmented views. The accumulated gradient over multiple iterations is then
applied for prompt tuning during test-time adaptation.

###  3.2 Class-aware Prototype Generation

We generated prototypes for each class for both text and visual branches. The
prototype for each class is computed using proxy source dataset. For a test
sample, $\textbf{X}_{test}$ and its $N_{k}$ random views (generated using a
set of augmentations $\mathcal{H}$ on $\textbf{X}_{test}$) the prototype
vector is generated. Let’s denote the token $e_{i}$ features of a sample
$x\in\\{\textbf{X}_{test}+\mathcal{H}(\textbf{X}_{test})\\}$, at the output of
the text encoder and visual encoder as $E_{T}(x,e_{i})$ and $E_{V}(x,e_{i})$,
respectively. The prototype for a sample from text and visual branches is
given as:

$\displaystyle h_{x}^{\\{t,v\\}}$
$\displaystyle=\frac{1}{|P|}\sum_{i=1}^{|P|}E_{\\{T,V\\}}(x,e_{i})$ (2)
$\displaystyle h_{\textrm{\tiny{CLS}},x}^{v}$
$\displaystyle=E_{V}(x,e_{\textrm{\tiny{CLS}}})$

where $|P|$ represents the total number of tokens (learnable and non-learnable
for both text and visual), excluding EOS, SOS, and CLS. $t,v$ represents
textual branch and visual branch respectively. For the proxy source dataset,
the class-aware prototype is obtained as:

$\displaystyle
h_{c_{k}}^{t}=\frac{1}{|\mathcal{D}(c_{k})|}\sum_{x\in\mathcal{D}(c_{k})}h_{x}^{t},$
(3) $\displaystyle
h_{c_{k}}^{v}=\frac{1}{|\mathcal{D}(c_{k})|}\sum_{x\in\mathcal{D}(c_{k})}h_{x}^{v},$
(4) $\displaystyle
h_{\textrm{\tiny{CLS}},c_{k}}^{v}=\frac{1}{|\mathcal{D}(c_{k})|}\sum_{x\in\mathcal{D}(c_{k})}h_{\textrm{\tiny{CLS}},x}^{v}$
(5)

where $\mathcal{D}(c_{k})$ contains all samples for class $c_{k}$. The
prototypes for augmented views are calculated using the augmented samples for
each class $c_{k}$ denoted as $\mathcal{D}^{aug}(c_{k})$ and the corresponding
prototypes are denoted as $h_{c_{k}}^{aug,t}$, $h_{c_{k}}^{aug,v}$ and
$h_{\textrm{\tiny{CLS}},c_{k}}^{aug,v}$ respectively.

###  3.3 Prototype Discriminating Loss

The discriminating loss is responsible for training learnable prompts to
distinguish the context of samples from one class compared to other classes.
This goal is achieved by pushing the class prototype $h_{c_{k}}^{m},\textrm{
}m\in\\{t,v\\}$ for both text and visual branches away from the prototype of
class $h_{c_{j}}^{m},\textrm{ }m\in\\{t,v\\}$ where $c_{k}\neq c_{j}$.
Likewise, we pull prototypes $h_{c_{k}}^{m}$ and $h_{c_{k}}^{aug,m}$ for same
class and push away augmented ones for $c_{j}\neq c_{k}$. In this regard,
contrastive learning [7, 14, 8, 23] offers a solution to pull prototypes of
positive pairs and push away negative pairs. We refer to [34] to propose our
discriminating loss $\mathcal{L}_{D}$, formally expressed as:

$\mathcal{L}_{pos}(c_{k})=\frac{1}{|\mathcal{H}|}\sum_{aug\in\mathcal{H}}e^{\textrm{sim}(h_{c_{k}}^{m},h_{c_{k}}^{aug,m})/\tau}$
(6)
$\mathcal{L}_{neg}(c_{k})=\frac{1}{|\mathcal{H}|}\sum_{aug\in\mathcal{H}}\sum_{c=1,c\neq
c_{k}}^{C}e^{\textrm{sim}(h_{c_{k}}^{m},h_{c}^{m})/\tau}\\\
+e^{\textrm{sim}(h_{c_{k}}^{aug,m},h_{c}^{m})/\tau}+e^{\textrm{sim}(h_{c_{k}}^{m},h_{c}^{aug,m})/\tau}$
(7) $\mathcal{L}_{D}=-\frac{1}{|m|*C}\sum_{\forall
m}\sum_{c=1}^{C}\log\frac{\mathcal{L}_{pos}}{\mathcal{L}_{neg}}$ (8)

where $c_{k}\in[1,C]$ and $m\in\\{t,v\\},i.e.|m|=2$. The prototypes
$h_{c_{k}}^{m}$ and $h_{c_{k}}^{aug,m}$ additionally contains
$h_{\textrm{\tiny{CLS}},c_{k}}^{v},h_{\textrm{\tiny{CLS}},c_{k}}^{aug,v}$ when
$m=v$. Resulting prompt update $\mathbf{p}\rightarrow\hat{\mathbf{p}}$
(learnable prompt tokens) is obtained after applying gradients for the
discriminating loss. Since the proxy dataset will remain same for all the test
instances, the updated prompt can be saved and restored each time for an
incoming test sample. We presented the study on performance and latency with
and without saving these updated prompts in Appendix 10. For the rest of the
paper we generalize our method without requiring to save these updated
prompts.

###  3.4 Prototype Alignment Loss

Loss $\mathcal{L}_{D}$ can effectively separate different classes, it is not
able to tune the prompt for the test sample, which comes from a different
distribution than the source distribution. Hence, we propose the prototype
alignment of the test sample (and its augmented views) with the class
prototype obtained from the source distribution. We propose to weigh the
prototype alignment by the probability of the test sample lying in the
particular class. Lets denote the probability
$\tilde{p}_{\mathbf{\hat{p}}}[c]$ (as mentioned in Eq.1) as the mean of
probabilities (for class c) produced with the updated prompt
$\hat{\mathbf{p}}$ across filtered augmented views (preserved after the
confidence selection filter) including test sample ($F$). The amplitude and
angle alignment of sample $x_{i}$ with the class prototypes for both text and
visual branches is calculated as follows:

$\displaystyle\mathcal{L}_{amp}^{{}^{\prime}}(x_{i})=\sum_{c=1}^{C}\tilde{p}_{\mathbf{\hat{p}}}[c]||p_{x_{i}}^{m}-p_{c}^{m}||^{2}$
(9)
$\displaystyle\mathcal{L}_{ang}^{{}^{\prime}}(x_{i})=\sum_{c=1}^{C}\tilde{p}_{\mathbf{\hat{p}}}[c]\textrm{sim}(p_{x_{i}}^{m},p_{c}^{m})$
(10)

where $m\in\\{t,v\\}$. However, there is an issue with MSE loss since it gives
an equal penalty (e.g. $\mathcal{L}_{amp}^{{}^{\prime}}$ = 0.1) for an
increase from 1.2 to 1.3 and 1.7 to 1.8. But we wanted to penalise more for
1.2 to 1.3 since increase in MSE in the smaller range should be penalised more
to preserve the base class performance. Hence we penalise with logarithm i.e.
we use $\mathcal{L}_{amp}=\log\mathcal{L}_{amp}^{{}^{\prime}}$. Similarly, the
penalty should be applied to $1/\mathcal{L}_{ang}^{{}^{\prime}}$ for angle
alignment. The updated amplitude and angle alignment loss is
$\mathcal{L}_{amp}(x_{i})=\log\mathcal{L}_{amp}^{{}^{\prime}}$ and
$\mathcal{L}_{ang}(x_{i})=\log(1/\mathcal{L}_{ang}^{{}^{\prime}})=-\log\mathcal{L}_{ang}^{{}^{\prime}}$
respectively. We combined the amplitude and angle loss with equal importance
and hence the prototype alignment loss $\mathcal{L}_{A}$ is given as:

$\displaystyle\mathcal{L}_{A}=\frac{1}{|F|}\sum_{x_{i}\in
F}(\mathcal{L}_{amp}(x_{i})+\mathcal{L}_{ang}(x_{i}))$ (11)
$\displaystyle=-\frac{1}{|F|}\sum_{x_{i}\in
F}\log\frac{\mathcal{L}_{ang}^{\prime}(x_{i})}{\mathcal{L}_{amp}^{\prime}(x_{i})}$

###  3.5 Algorithm Details

In order to compute the prototype discriminating loss on the source dataset,
we require the pre-training dataset of the CLIP model. However, it was trained
on over 400 million image-text pairs, which are not publicly available.
Nevertheless, in previous works[2, 33], CLIP has been heavily tuned on the
ImageNet[11] dataset to achieve excellent zero-shot performance. Hence, we use
ImageNet as the proxy for the source dataset to compute prototypes for each
class. These prototypes are computed offline for both the sample and its
augmented views, and they are used directly during test-time adaptation.
During each iteration of test-time adaptation, the meta-train stage is entered
first. The model starts training using the prototype discriminating objective
$\underset{\boldsymbol{p}}{\textrm{{argmin} }}\mathcal{L}_{D}$, and gradients
are calculated, resulting in the prompt update
$\mathbf{p}\rightarrow\hat{\mathbf{p}}$ ($i^{th}$ iteration). Subsequently,
the meta-test stage is executed. Here, the augmented views are first filtered
using a confidence threshold over predicted probabilities using the updated
prompts $\hat{\mathbf{p}}$. The mean probabilities
$\tilde{p}_{\mathbf{\hat{p}}}$ are computed over $F$ and used as weights in
$\mathcal{L}_{A}$. The model is trained on $F$, and the gradient of prototype
alignment loss $\nabla_{{\mathbf{p}}}\mathcal{L}_{A}$ is calculated. We
average out the gradients over all samples in $F$. Finally, the prompts
$\boldsymbol{p}$ is updated using combined objective:
$\mathcal{L}_{ent}+\underset{\boldsymbol{p}}{\textrm{{argmin}
}}\mathcal{L}_{A}$. For $n>1$ we accumulate the averaged gradients before
final prompt update.

| Imagenet V2 | Imagenet Sketch | Imagenet A | Imagenet R | OOD Avg
---|---|---|---|---|---
CLIP | 60.86 | 46.09 | 47.87 | 73.98 | 57.20
CLIP+TPT | 64.35 | 47.94 | 54.77 | 77.06 | 60.81
CoOp | 64.20 | 47.99 | 49.71 | 75.21 | 59.28
CoOp+TPT | 66.83 | 49.29 | 57.95 | 77.27 | 62.84
Co-CoOp | 64.07 | 48.75 | 50.63 | 76.18 | 59.91
Co-CoOp+TPT | 64.85 | 48.27 | 58.47 | 78.65 | 62.61
MaPLe | 64.07 | 49.15 | 50.90 | 76.98 | 60.28
MaPLe+TPT | 64.87 | 48.16 | 58.08 | 78.12 | 62.31
PromptAlign | 65.29 | 50.23 | 59.37 | 79.33 | 63.55
PromptSync | 67.54 | 53.42 | 61.92 | 80.64 | 65.88
Table 1: Comparison on the domain generalization setting. Prompt tuning methods are trained on ImageNet and evaluated on datasets with domain shifts |  | Camera
---
(Yaw/ Pitch/ Roll)
| Pose
---
(Yaw/ Pitch/ Roll)
Scale | Texture | Lighting | Worlds
MaPLe | 48.73/ 39.93/ 32.13 | 48.10/ 28.40/ 27.80 | 46.90 | 37.90 | 15.50 | 32.13
MaPLe+TPT | 57.04/ 45.99/ 39.23 | 56.26/ 35.64/ 33.26 | 54.87 | 43.73 | 22.52 | 42.00
PromptAlign | 58.14/ 46.93/ 40.45 | 57.43/ 36.31/ 34.32 | 56.18 | 44.97 | 23.06 | 43.24
PromptSync | 59.84/ 48.54/ 41.92 | 59.72/ 38.84/ 36.64 | 58.12 | 45.98 | 25.02 | 44.84
Table 2: Comparison on the domain generalization setting for distribution
alignment. MaPLe is trained on ImageNet and evaluated on OOD dataset i.e., PUG

##  4 Experiments

We have evaluated PromptSync on different benchmark settings (Appendix 9) with
different datasets described below:
Datasets: For domain generalisation setting, we follow PromptAlign [33] and
evaluated our method on four out-of-distribution (OOD) variants of ImageNet
[11]: ImageNetV2 [32], ImageNet-Sketch [40], ImageNet-A [18] and ImageNet-R
[17]. We also consider the evaluation on a recent and challenging benchmark,
namely, Photorealistic Unreal Graphics (PUG) dataset [4], comprised of
different textures, sizes, orientations, and backgrounds. For cross-dataset
transfer setting, we follow TPT [35] and evaluate the performance on 10
diverse image classification datasets with varying complexities for visual
recognition tasks. This includes Caltech 101 [12] for generic objects. Five
fine-grained datasets (spanning images of animals, flowers and transportation)
are StanfordCars [24], Food101 [5], Flowers102 [27], FGVC-Aircraft [25],
OxfordPets [29]. Moreover, four datasets, namely, SUN397 [37], DTD [10],
UCF101 [36], and EUROSAT [15], comprise scenes, textures, human actions, and
satellite imagery, respectively. For base-to-novel generalisation, we follow
[21] and evaluate our method on ImageNet and the 10 image classification
datasets.
Baselines: We compared PromptSync with existing few-shot prompt learning
methods for CLIP adaptation; these are CoOp [47], CoCoOp [46], TPT [35], and
PromptAlign [33]. MaPLe [21] is a multi-modal prompt learning baseline that
adapts CLIP by learning prompts on both text and visual branches. TPT [35] and
PromptAlign [33] are the test-time prompt tuning methods that tune the prompt
for each incoming test sample, achieving state-of-the-art performance in
prompt learning.
Implementation Details: We ran all experiments on a single NVIDIA A100 40GB
GPU. Following [21], we trained on ImageNet with 16-shot training data
selected at random for each class using 2 prompt tokens for a depth of 3
layers (on CLIP ViT-B/16 backbone architecture). We optimized the prompts on
both the text and visual branches using a single test image. We augmented each
test image with 127 different views using random resized crops, background
substitution, horizontal flip augmentations, and visual corruption. For text
augmentation, we used hyponyms, synonyms, and meronyms from WordNet[26].
Moreover, we generated various text prompts from pre-trained LLMs [6].
Additionally, we randomly masked one of the learnable tokens for 15% of
augmented views. We computed the gradients of alignment loss for a batch size
of 128 images, including the original image. During the meta-train stage, we
updated the original parameters (using a single iteration) and then optimized
the prompts in the meta-test stage by calculating the gradients of alignment
loss w.r.t. the updated parameters accumulated for a single ($n=1$) iteration
to facilitate the one-to-one comparison with baselines. We obtained the top
10% confident predictions of augmented views based on the lowest entropy. We
used the AdamW optimizer and a learning rate $\beta$ of $5e^{-4}$ for the
fine-grained datasets and $0.04$ for the rest of the datasets.

###  4.1 Domain Generalization

We demonstrate that all test-time adaptation methods exhibit better
performance (Table 1) compared to the pre-trained CLIP model, highlighting the
advantage of tuning V-L models at test time. PromptSync achieves the highest
Top-1 accuracy averaged across all the domains of ImageNet variants.
Furthermore, we evaluated the ImageNet-trained model on various out-of-
distribution (OOD) datasets and observed consistent improvement in performance
compared to existing state-of-the-art (SOTA) approaches. The detailed results
for each domain dataset are presented in Tables 1 and 2. This confirms that
alignment and discriminative training with augmented views on both the text
and visual branches enhance the generalization performance of V-L models like
CLIP.

Datasets | Sets |  | CoOp
---
(IJCV22)
| CoCoOp
---
(CVPR22)
| ProDA
---
(CVPR22)
| MaPLe
---
(CVPR23)
| MaPLe + TPT
---
(CVPR23)
| PromptAlign
---
(NIPS23)
| PromptSync
---
Average |  | Base
---
Novel
| 82.38
---
67.96
| 80.47
---
71.69
| 81.56
---
72.30
| 82.24
---
75.09
| 82.16
---
74.95
| 83.19
---
75.88
| 84.17
---
77.17
ImageNet |  | Base
---
Novel
| 76.46
---
66.31
| 75.98
---
70.43
| 75.40
---
70.23
| 76.67
---
70.54
| 77.73
---
72.24
| 78.26
---
72.59
| 79.23
---
73.84
Caltech101 |  | Base
---
Novel
| 97.80
---
93.27
| 97.96
---
93.81
| 98.27
---
93.23
| 98.00
---
94.27
| 98.54
---
94.29
| 98.60
---
94.50
| 98.62
---
94.67
OxfordPets |  | Base
---
Novel
| 94.47
---
96.00
| 95.20
---
97.69
| 95.43
---
97.83
| 95.43
---
97.80
| 95.23
---
97.37
| 95.38
---
97.56
| 95.44
---
97.83
Stanford Cars |  | Base
---
Novel
| 75.67
---
67.53
| 70.49
---
73.59
| 74.70
---
71.20
| 72.90
---
73.97
| 74.00
---
75.20
| 75.02
---
75.71
| 76.42
---
77.21
Flowers102 |  | Base
---
Novel
| 97.27
---
67.13
| 94.87
---
71.75
| 97.70
---
68.68
| 95.93
---
72.40
| 96.24
---
72.10
| 96.61
---
72.34
| 97.73
---
73.78
Food101 |  | Base
---
Novel
| 89.37
---
88.77
| 90.70
---
91.29
| 90.30
---
88.57
| 90.70
---
92.07
| 91.13
---
92.03
| 91.63
---
92.68
| 92.39
---
92.95
FGVC Aircraft |  | Base
---
Novel
| 39.67
---
31.23
| 33.41
---
23.71
| 36.90
---
34.13
| 37.27
---
35.53
| 34.31
---
35.81
| 37.21
---
37.27
| 40.91
---
39.31
SUN397 |  | Base
---
Novel
| 80.85
---
68.34
| 79.74
---
76.86
| 78.67
---
76.93
| 80.80
---
78.70
| 81.15
---
79.18
| 81.57
---
79.48
| 84.28
---
83.01
DTD |  | Base
---
Novel
| 79.97
---
48.60
| 77.01
---
56.00
| 80.67
---
56.48
| 80.30
---
59.23
| 82.20
---
59.91
| 82.60
---
60.55
| 83.49
---
62.03
Eurosat |  | Base
---
Novel
| 90.10
---
53.00
| 87.49
---
60.04
| 83.90
---
66.00
| 93.63
---
72.87
| 91.02
---
68.96
| 94.10
---
72.71
| 94.63
---
73.19
UCF101 |  | Base
---
Novel
| 84.53
---
67.37
| 82.33
---
73.45
| 85.23
---
71.97
| 82.97
---
78.57
| 82.23
---
77.34
| 84.11
---
79.30
| 85.75
---
81.29

Table 3: Comparison on Base-to-novel generalization setting. PromptSync shows
consistent improvement on both base and novel classes over previous methods

###  4.2 Base to Novel Generalization

Table 3 presents the detailed performance report of PromptSync on base and
novel classes across 11 recognition datasets. On average, our strategy
outperforms the model performance by 1.29% on base classes and nearly 1% on
novel classes. We observe that PromptAlign, based on a distribution alignment
strategy, outperforms for novel classes in most cases, with an average
improvement of 0.79% compared to the best-performing model. However, the
margin of improvement is very low. In contrast, with TPT, the performance
drops in some instances, such as for OxfordPets, Eurosat, and UCF101. This
demonstrates that: 1) test-source alignment is crucial for prompt tuning. 2)
Prompt tuning alone in the text branch is not sufficient for zero-shot
generalization. Since distribution alignment does not promote discriminative
learning and the entropy loss on the test dataset is noisy, PromptSync
outperforms with class-aware prototype discrimination and alignment across
different augmented views. Averaging the gradients further motivates domain-
agnostic prompt tuning on both the text and visual branches. This enhances the
zero-shot generalization of the V-L model compared to other state-of-the-art
approaches. Moreover, our strategy for prompt tuning does not lose information
for base classes.

| Caltech | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average
---|---|---|---|---|---|---|---|---|---|---|---
CLIP | 93.35 | 88.25 | 65.48 | 67.44 | 83.65 | 23.67 | 62.59 | 44.27 | 42.01 | 65.13 | 63.58
CLIP+TPT | 94.16 | 87.79 | 66.87 | 68.98 | 84.67 | 24.78 | 65.50 | 47.75 | 42.44 | 68.04 | 65.10
CoOp | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88
CoOp + TPT | 93.15 | 89.48 | 66.77 | 68.48 | 86.48 | 20.51 | 66.06 | 43.32 | 37.73 | 68.91 | 64.08
CoCoOp | 93.79 | 90.46 | 64.90 | 70.85 | 83.97 | 22.29 | 66.89 | 45.45 | 39.23 | 68.44 | 64.63
CoCoOp + TPT | 88.57 | 85.33 | 59.68 | 55.31 | 80.64 | 16.89 | 60.24 | 38.93 | 48.55 | 63.35 | 59.75
ProDA | 86.70 | 88.20 | 60.10 | 77.50 | 80.80 | 22.20 | - | 50.90 | 58.50 | - | 65.62
MaPLe | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30
MaPLe+TPT | 93.59 | 90.72 | 66.50 | 72.37 | 86.64 | 24.70 | 67.54 | 45.87 | 47.80 | 69.19 | 66.50
PromptAlign | 94.01 | 90.76 | 68.50 | 72.39 | 86.65 | 24.80 | 67.54 | 47.24 | 47.86 | 69.47 | 66.92
PromptSync | 95.78 | 91.89 | 69.24 | 77.68 | 87.72 | 25.91 | 67.98 | 50.99 | 59.36 | 71.04 | 69.76

Table 4: Comparison on cross-dataset transfer setting. Prompt tuning methods
are trained on ImageNet and evaluated on cross-datasets

Method |  | Entropy
---
Loss
| Alignment
---
Loss
| Discriminative
---
Loss
| Top-1
---
Acc.
MaPLe | ✗ | ✗ | ✗ | 50.90
MaPLe+TPT | $\checkmark$ | ✗ | ✗ | 58.08
PromptAlign† | ✗ | $\checkmark$ | ✗ | 50.85
PromptAlign | $\checkmark$ | $\checkmark$ | ✗ | 59.37
PromptSync† | ✗ | ✗ | $\checkmark$ | 56.67
PromptSync | $\checkmark$ | $\checkmark$ | $\checkmark$ | 61.92

Table 5: Ablation Study. Analysis of Alignment, Discriminative and Entropy
minimization loss. The average of Top-1 accuracy(%) across three seeds is
reported

###  4.3 Cross-Dataset Transfer

In Table 4, we compared the transfer performance of PromptSync with existing
state-of-the-art methods using prompt learning. We evaluated methods for
transfer performance across diverse cross-datasets. PromptSync consistently
outperforms the previous best method, i.e., PromptAlign [33], across all
cross-datasets, providing an average improvement of $2.84\%$. Compared to
PromptAlign, which outperforms the previous method MaPLe + TPT by a very small
margin, i.e., $0.42\%$, our method shows a significant average improvement of
$3.26\%$ over MaPLe + TPT. Other methods, CoOp and CoCoOp, on average, perform
worse than zero-shot CLIP + TPT (except ProDA [45]). This affirms that both
text-visual alignment and domain-agnostic parameter updates result in better
transfer generalization across cross-datasets in V-L models. As opposed to our
method, the previous approaches were not consistent in performance across all
datasets, which further affirms the advantage of a domain-agnostic training
strategy.

| $\mathcal{L}_{amp}$ | $\mathcal{L}_{ang}$ | sub | sum_exp | $\mathcal{L}_{A}$
---|---|---|---|---|---
PromptSync | 59.84 | 58.81 | 57.86 | 59.83 | 61.92

Table 6: Ablation analysis to alignment loss variants. All results are on
ImageNet-A dataset

##  5 Ablation

Class-Aware Prototype Alignment: Table 5 summarizes the comparison between two
alignment strategies: distribution alignment of the test sample with the
class-agnostic source distribution. All results are on the ImageNet-A dataset.
PromptAlign adopted distribution alignment along with averaged cross-entropy
for prompt tuning. However, we perform domain-agnostic parameter updates with
class-aware prototype alignment for the test sample. As shown in Table 5,
PromptAlign† without entropy loss is as good as vanilla MaPLe. This is due to
the fact that distributional alignment does not promote any discriminative
learning in the absence of entropy loss. However, because entropy loss is
noisy due to the poor performance of the vanilla zero-shot V-L model, we
propose the stronger discriminative loss of class prototype alignment for
prompt tuning with source and test samples with augmented views. PromptSync†
without entropy loss outperforms the corresponding counterpart PromptAlign†.
This is because the class-aware prototype alignment has both alignment and
discriminative properties, thus improving test-time adaptation on its own.
With additional signals from predicted probabilities for each class, the
class-aware prototype alignment acts as a geometric regularizer, mitigating
class collapse in prompt representation.
Loss variants: We conducted an ablation study on amplitude and angle loss for
the class-aware prototype alignment objective. Table 6 compares three loss
choices: 1) amplitude loss, 2) angle loss, and 3) amplitude + angle loss.
Clearly, the combination of amplitude and angle performs better than other
choices. The formulation for the combination of amplitude and angle loss is
the same as in equation 11. We further investigated other variants, i.e.,
combining two of them without taking the log: 1) subtraction between amplitude
and angle
$\mathcal{L}_{amp}^{{}^{\prime}}-\mathcal{L}_{ang}^{{}^{\prime}}$(sub) 2) the
summation of exponential of both losses
$\textrm{exp}(\mathcal{L}_{amp}^{{}^{\prime}})+\textrm{exp}(1/\mathcal{L}_{ang}^{{}^{\prime}})$(sum_exp).
Clearly, the formulation in equation 11 ($\mathcal{L}_{A}$) performs best
among other variants. Ablation on the proxy dataset is given in Appendix 12,
and ablation on performance and latency with and without saving updated
prompts is provided in Appendix 10. We also compared the number of augmented
views and prompt updates in Appendix 11.

##  6 Performance and Latency

The experiments presented in the Table 7 (Appendix) involve a comparison of
different methods, namely MaPLe + TPT, PromptAlign, PromptSync*, and
PromptSync. In these experiments, we evaluated the top-1 average accuracy (%)
and latency (in hours for a single prompt update) of each method.
Specifically, we investigated PromptSync with and without saving the updated
prompt obtained after prototype discrimination, with the variant denoted as
PromptSync* indicating the adaptation of prompt tokens for test samples after
restoring saved prompt tokens.

The results, as shown in Table 7, include latency measurements represented in
hours for a single prompt update, and all evaluations are conducted on the
ImageNet-A dataset. Notably, the PromptSync* variant demonstrates a faster
processing time compared to the full PromptSync method, with only a marginal
drop in performance. This outcome underscores the achieved generalization
through prototype alignment. Furthermore, in comparison to previous methods
such as MaPLe + TPT and PromptAlign, the PromptSync* variant exhibits only a
slight increase in latency (0.03 hours) while still improving overall
performance.

##  7 Sensitivity Comparison

We further performed the sensitivity comparison of our method as compared to
other state-of-the-art baselines. In Appendix, Figure 2(a) shows the
comparison of performance during test time adaptation as the number of views
increases. All the results are on ImageNet-A dataset. In comparison to
PromptAlign and MaPLe + TPT, their performance almost plateaus around 64 views
with insignificant improvement further, while PromptSync shows a consistent
improvement with the increase in views and insignificant improvement beyond
128. This proves the generalizability achieved by our method since it
optimises base CLIP over a larger number of possible shifts in the dataset,
resulting in better performance. Figure 2(b) shows the performance comparison
as the number of prompt update steps increases. All the methods increase their
performance with an increase in the number of steps; however, our method shows
better adaptation to the test sample with more steps in comparison to
PromptAlign and MaPLe + TPT. For apples-to-apples comparison we perform a
single-step update (128 views) following TPT [35].

##  8 LAION400M Proxy Dataset Analysis

Given CLIP’s impressive zero-shot performance on ImageNet, we opted for
ImageNet as a viable proxy source dataset, aligning with prior research [33].
We worked with a subset of LAION400M, comprising 2.5 million images (2 times
the size of ImageNet). Furthermore, we carried out an ablation study on the
alignment strategy using LAION400M as the source dataset, a dataset known to
mirror CLIP’s training dataset [9]. The results for this ablation study is
shown in Table 8 (Appendix). Notably, the performance impact remains
consistent when utilizing this subset of LAION400M alongside ImageNet. Source
class prototypes are computed on the proxy source data to derive the
distribution for alignment during test time. As this proxy dataset aligns with
the model’s training set, this offline computation remains unchanged despite
environmental shifts and only necessitates computation once.

## Conclusion

In summary, PromptSync significantly improves zero-shot generalization in
vision-language models. Our approach, addressing class dominance and variance,
outperforms existing methods by 2.33% overall, with a 1% boost in base-to-
novel generalization and 2.84% in cross-dataset transfer on a domain
generalization benchmark. This underscores PromptSync’s effectiveness in
enhancing the robustness of vision-language models.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bahng et al. [2022a] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Exploring visual prompts for adapting large-scale models.  _arXiv preprint arXiv:2203.17274_ , 2022a.
  * Bahng et al. [2022b] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Visual prompting: Modifying pixel space to adapt pre-trained models.  _arXiv preprint arXiv:2203.17274_ , 3:11–12, 2022b.
  * Bordes et al. [2023] Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S Morcos.  Pug: Photorealistic and semantically controllable synthetic data for representation learning.  _arXiv preprint arXiv:2308.03977_ , 2023.
  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 446–461. Springer, 2014.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2020a] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020a.
  * Chen et al. [2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020b.
  * Cherti et al. [2023] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2818–2829, 2023.
  * Cimpoi et al. [2014] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3606–3613, 2014.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _2004 conference on computer vision and pattern recognition workshop_ , pages 178–178. IEEE, 2004.
  * He et al. [2021] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_ , 2021.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7):2217–2226, 2019.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _arXiv preprint arXiv:1903.12261_ , 2019.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349, 2021a.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271, 2021b.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Khattak et al. [2023a] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19113–19122, 2023a.
  * Khattak et al. [2023b] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.  Self-regulating prompts: Foundational model adaptation without forgetting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15190–15200, 2023b.
  * Khosla et al. [2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  _Advances in neural information processing systems_ , 33:18661–18673, 2020.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pages 554–561, 2013.
  * Maji et al. [2013] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Miller [1995] George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian conference on computer vision, graphics & image processing_, pages 722–729. IEEE, 2008.
  * Oquab et al. [2014] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1717–1724, 2014.
  * Parkhi et al. [2012] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 3498–3505. IEEE, 2012.
  * Quinonero-Candela et al. [2008] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  _Dataset shift in machine learning_.  Mit Press, 2008.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Samadh et al. [2023] Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan.  Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Sha et al. [2023] Zeyang Sha, Xinlei He, Ning Yu, Michael Backes, and Yang Zhang.  Can’t steal? cont-steal! contrastive stealing attacks against image encoders.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16373–16383, 2023.
  * Shu et al. [2022] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao.  Test-time prompt tuning for zero-shot generalization in vision-language models.  _Advances in Neural Information Processing Systems_ , 35:14274–14289, 2022.
  * Soomro et al. [2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  A dataset of 101 human action classes from videos in the wild.  _Center for Research in Computer Vision_ , 2(11), 2012.
  * Sun et al. [2020] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _International conference on machine learning_ , pages 9229–9248. PMLR, 2020.
  * Tan et al. [2022] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang.  Fedproto: Federated prototype learning across heterogeneous clients.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8432–8440, 2022.
  * Tong et al. [2023] Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun.  Emp-ssl: Towards self-supervised learning in one training epoch.  _arXiv preprint arXiv:2304.03977_ , 2023.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Yao et al. [2021] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  _arXiv preprint arXiv:2111.07783_ , 2021.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zang et al. [2022] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Unified vision and language prompt learning.  _arXiv preprint arXiv:2210.07225_ , 2022.
  * Zhai et al. [2022] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18123–18133, 2022.
  * Zhang et al. [2021] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen.  Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12414–12424, 2021.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16816–16825, 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022b.

Supplementary Material

##  9 Benchmark Settings

Method | Top-1 Average Accuracy(%) | Latency
---|---|---
MaPLe + TPT | 58.08 | 0.41
PromptAlign | 59.37 | 0.46
PromptSync* | 61.88 | 0.49
PromptSync | 61.92 | 0.65

Table 7: Performance and Latency: Performance and Latency comparison of
PromptSync with state-of-the-art baselines and its variant which reuse the
learned prompt tokens after prototype discrimination without learning them for
each incoming test sample.

Figure 2: Sensitivity Comparison. (a) Top-1 accuracy improves with number of
augmented views (b) Top-1 accuracy improves consistently with number of prompt
update steps.

Method | Flowers | DTD | Pets | Cars | UCF | Caltech | Food | SUN | Aircraft | Eurosat | Avg
---|---|---|---|---|---|---|---|---|---|---|---
ImageNet | 77.68 | 50.99 | 91.89 | 69.24 | 71.04 | 95.78 | 87.72 | 67.98 | 25.91 | 59.36 | 69.74
LAION | 77.68 | 51.00 | 91.88 | 69.25 | 71.03 | 95.79 | 87.75 | 68.00 | 25.90 | 59.35 | 69.76

Table 8: Performance impact analysis using both ImageNet and LAION400M subset

Base-to-Novel Generalisation: Following MaPLe [21], we evaluate PromptSync on
a zero-shot setting. We split the dataset into base and novel classes. The
model is trained only on the base classes in a few-shot setting and evaluated
on the base and novel classes.
Cross-dataset Transfer: We evaluate PromptSync on the ImageNet[11] pre-trained
model on other datasets to determine the transfer performance. Following
CoCoOp[46], our model is trained on all 1000 ImageNet classes in a few-shot
manner.
Domain Generalisation: We evaluate PromptSync on out-of-distribution (OOD)
datasets for domain generalizability. Similar to cross-dataset, we evaluate
our ImageNet-trained model directly on OOD datasets, which are described in
Section 4.

##  10 Performance and Latency

The experiments presented in the table 7 above involve a comparison of
different methods, namely MaPLe + TPT, PromptAlign, PromptSync*, and
PromptSync. In these experiments, we evaluated the top-1 average accuracy (%)
and latency (in hours for a single prompt update) of each method.
Specifically, we investigated PromptSync with and without saving the updated
prompt obtained after prototype discrimination, with the variant denoted as
PromptSync* indicating the adaptation of prompt tokens for test samples after
restoring saved prompt tokens.

The results, as shown in Table 7, include latency measurements represented in
hours for a single prompt update, and all evaluations are conducted on the
ImageNet-A dataset. Notably, the PromptSync* variant demonstrates a faster
processing time compared to the full PromptSync method, with only a marginal
drop in performance. This outcome underscores the achieved generalization
through prototype alignment. Furthermore, in comparison to previous methods
such as MaPLe + TPT and PromptAlign, the PromptSync* variant exhibits only a
slight increase in latency (0.03 hours) while still improving overall
performance.

##  11 Sensitivity Comparison

We further performed the sensitivity comparison of our method as compared to
other state-of-the-art baselines. Figure 2(a) shows the comparison of
performance during test time adaptation as the number of views increases. All
the results are on ImageNet-A dataset. In comparison to PromptAlign and MaPLe
+ TPT, their performance almost plateaus around 64 views with insignificant
improvement further, while PromptSync shows a consistent improvement with the
increase in views and insignificant improvement beyond 128. This proves the
generalizability achieved by our method since it optimises base CLIP over a
larger number of possible shifts in the dataset, resulting in better
performance. Figure 2(b) shows the performance comparison as the number of
prompt update steps increases. All the methods increase their performance with
an increase in the number of steps; however, our method shows better
adaptation to the test sample with more steps in comparison to PromptAlign and
MaPLe + TPT. For apples-to-apples comparison we perform a single-step update
(with 128 views) following TPT [35].

##  12 LAION400M Proxy Dataset Analysis

Given CLIP’s impressive zero-shot performance on ImageNet, we opted for
ImageNet as a viable proxy source dataset, aligning with prior research [33].
We worked with a subset of LAION400M, comprising 2.5 million images (2 times
the size of ImageNet). Furthermore, we carried out an ablation study on the
alignment strategy using LAION400M as the source dataset, a dataset known to
mirror CLIP’s training dataset [9]. The results for this ablation study is
shown in Table 8. Notably, the performance impact remains consistent when
utilizing this subset of LAION400M alongside ImageNet. Source class prototypes
are computed on the proxy source data to derive the distribution for alignment
during test time. As this proxy dataset aligns with the model’s training set,
this offline computation remains unchanged despite environmental shifts and
only necessitates computation once.


j\leq L-1.$

Using Theorem G.1, we know that the objective of prefix-tuning can be
formulated as a discrete-time control problem. We then use the MSA described
in Theorem G.2 to determine the co-process that solves the control problem and
realize that the MSA is equivalent to back-propagation with Theorem G.3, which
recovers the training phase of prefix-tuning. As a result, we can conclude
that prefix-tuning seeks the optimal control of the pretrained models for
specific downstream tasks.

Our robust prefix-tuning framework can also be formalized as the close-loop
control of the obtained prefix for robust downstream tasks. According to Chen
et al. (2021), the close-loop control framework for robust neural networks is
defined as

$\displaystyle\min_{\\{\pi_{0},\dots,\pi_{L-1}\\}}\sum_{t=0}^{L-1}\ell(x_{t},\pi_{t}(x_{t})),$
(38) $\displaystyle x_{t+1}=f_{t}(x_{t},\pi_{t}(x_{t})),~{}x_{0}=x,~{}0\leq
t\leq L-1.$

where $\ell(x_{t},\pi_{t}(x_{t}))$ is the running loss. According to Eq. (26),
the goal of our robust prefix-tuning framework can be formulated into the form

$\displaystyle\min_{\\{\psi^{(0)},\dots,\psi^{(L-1)}\\}}\mathbb{E}_{(x,y)\sim\mathcal{D}_{te}}\sum_{j=0}^{L-1}\mathcal{R}_{\Phi,\theta^{(j)}}^{(j)}\left(h_{o}^{(j)},\psi^{(j)}\left(h_{o}^{(j)}\right)\right),$
(39) $\displaystyle
h_{o}^{(j+1)}=h_{o}^{(j)}+\widetilde{\mathcal{G}}_{\Phi,\theta^{(j)}}^{(j)}\left(h_{o}^{(j)},\psi^{(j)}\left(h_{o}^{(j)}\right)\right),~{}h_{o}^{(0)}=z_{o}={\rm[ANS]},~{}0\leq
j\leq L-1.$

Since Theorems G.1, G.2 and G.3 can also be applied to Eq. (39), the
optimization of our robust prefix is equivalent to seeking the close-loop
control for robust downstream tasks.

In conclusion, recent years have witnessed the paradigm shift in NLP from
adapting LMs to downstream tasks to adapting downstream tasks to LMs (Liu et
al., 2021). In this work, we provide the interpretation of prefix-tuning as
well as our robust prefix-tuning framework from the optimal control
perspective. We hope that our work can bring insight to the future design of
robust and efficient finetuning approaches for large-scale pretrained models.


# Sequential Integrated Gradients: a simple but effective method for
explaining language models

Joseph Enguehard
Babylon Health
Skippr
<EMAIL_ADDRESS>
###### Abstract

Several explanation methods such as Integrated Gradients (IG) can be
characterised as path-based methods, as they rely on a straight line between
the data and an uninformative baseline. However, when applied to language
models, these methods produce a path for each word of a sentence
simultaneously, which could lead to creating sentences from interpolated words
either having no clear meaning, or having a significantly different meaning
compared to the original sentence. In order to keep the meaning of these
sentences as close as possible to the original one, we propose Sequential
Integrated Gradients (SIG), which computes the importance of each word in a
sentence by keeping fixed every other words, only creating interpolations
between the baseline and the word of interest. Moreover, inspired by the
training procedure of several language models, we also propose to replace the
baseline token "pad" with the trained token "mask". While being a simple
improvement over the original IG method, we show on various models and
datasets that SIG proves to be a very effective method for explaining language
models.111An implementation of this work can be found at
https://github.com/josephenguehard/time_interpret

##  1 Introduction

Figure 1: Comparison between IG, DIG, and our method: SIG. While DIG improves
on IG by creating discretized paths between the data and the baseline, it can
produce sentences with a different meaning compared to the original one. Our
method tackles this issue by fixing every word to their true value except one,
and moving the remaining word along a straight path (SIG)

Language models such as BERT Devlin et al. (2018) have demonstrated to be
effective on various tasks, for instance on sentiment analysis Hoang et al.
(2019), machine translation Zhu et al. (2020), text summarization Liu (2019)
or intent classification Chen et al. (2019). However, with the increased
performance and usage of such models, there has been a parallel drive to
develop methods to explain predictions made by these models. Indeed, BERT and
its variations are complex models which do not allow a user to easily
understand why a certain prediction has been produced. On the other hand, it
is important to be able to explain a model’s predictions, especially when this
model is used to make high-stake decisions, or when there is a risk of a
discriminating bias, for instance when detecting hate speech on social media
Sap et al. (2019).

As a result, developing effective methods to explain not only language models,
but also machine learning models in general, has recently gained significant
attention. Many different methods have therefore been proposed such as: LIME
Ribeiro et al. (2016), Grad*Inp Shrikumar et al. (2016), Integrated Gradients
(IG) Sundararajan et al. (2017), DeepLift Shrikumar et al. (2017) or
GradientShap Lundberg and Lee (2017). Among these methods, some can be
characterised as path-based, which means that they rely on a straight line
between the data and an uninformative baseline. For instance, IG computes
gradients on interpolated points along such a path, while DeepLift and
GradientShap can be seen as approximations of IG Ancona et al. (2017);
Lundberg and Lee (2017).

While these methods aim to be used on any type of models and data, some have
been tailored to the specificity of language models. For instance, Sanyal and
Ren (2021) challenge the use of continuous paths on a word embedding space
which is inherently discrete. They propose as a result Discretized Integrated
Gradient (DIG), which replaces the continuous straight path with a discretized
one, where interpolated points are words.

In our work, we suggest another potential issue when applying path-based
explanation methods on language models. These models are usually designed to
be used on individual or multiple sentences, in order to perform for instance
sentiment analysis or question answering. However, a path-based method applied
on such models creates straight lines between each word and a baseline
simultaneously. When interpolated points are grouped together to form a
sentence, this sentence could have a very different meaning compared with the
original one.

As a result, we propose a simple method to alleviate this potential issue:
computing the importance of each word in a sentence or a text by keeping fixed
every other word and only creating interpolations between the baseline and the
word of interest. After computing the importance of each word in this way, we
normalise these attributions across the sentence or text we aim to explain. We
call this method Sequential Integrated Gradients (SIG), as, although we focus
in this work on language models, such a method could be used on any sequential
modelling. We also propose to use the token "mask" as a baseline, when
possible, as its embedding has been trained to replace part of sentences when
training language models. As a result, our method follows closely the training
procedure of these models.

##  2 Method

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.412 | 0.112 | 0.375 | -0.199 | 0.0760 | 0.426 | -0.263 | 0.0923 | 0.439
DeepLift | -0.624 | 0.170 | 0.271 | -0.261 | 0.0932 | 0.408 | -0.244 | 0.0898 | 0.438
GradientShap | -1.32 | 0.303 | 0.258 | -0.896 | 0.261 | 0.314 | -0.622 | 0.219 | 0.388
IG | -1.96 | 0.445 | 0.151 | -1.44 | 0.405 | 0.226 | -0.981 | 0.345 | 0.352
DIG | -1.69 | 0.384 | 0.167 | -0.824 | 0.263 | 0.278 | -0.777 | 0.287 | 0.345
SIG | -2.02 | 0.473 | 0.0992 | -1.62 | 0.440 | 0.216 | -1.19 | 0.392 | 0.312

Table 1: Comparison of SIG with several feature attribution methods on three
language models fine-tuned on the SST2 dataset. For $\uparrow$ metrics, the
higher the better, while for $\downarrow$ ones, the lower the better.

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.153 | 0.0766 | 0.209 | -0.0892 | 0.0432 | 0.300 | -0.291 | 0.0887 | 0.298
DeepLift | -0.269 | 0.117 | 0.159 | -0.124 | 0.0557 | 0.269 | -0.285 | 0.0701 | 0.366
GradientShap | -0.832 | 0.289 | 0.137 | -0.606 | 0.204 | 0.144 | -0.874 | 0.172 | 0.308
IG | -1.50 | 0.534 | 0.0428 | -1.35 | 0.441 | 0.0327 | -1.58 | 0.302 | 0.224
DIG | -0.779 | 0.304 | 0.133 | -0.663 | 0.186 | 0.108 | -1.06 | 0.207 | 0.232
SIG | -1.95 | 0.564 | 0.00409 | -1.37 | 0.404 | -3.31E-05 | -2.12 | 0.364 | 0.124

Table 2: Comparison of SIG with several feature attribution methods on three
language models fine-tuned on the IMDB dataset.

#### SIG formulation

Let’s define a language model as a function
$\textrm{F}(\textbf{x}):\mathbb{R}^{m\times n}\rightarrow\mathbb{R}$. The
input x is here modelled as a sequence of $m$ words, each having $n$ features.
These features are usually constructed by an embedding layer. We denote
$\textbf{x}_{i}$ the $\textrm{i}^{\textrm{th}}$ word of a sentence (or of a
text, depending on the input of the model), and $x_{ij}$ the
$\textrm{j}^{\textrm{th}}$ feature of the $\textrm{i}^{\textrm{th}}$ word. The
output of F is a value in $\mathbb{R}$, which is, in our experiments, a
measure of the sentiment for a given sentence. We now define the baseline for
each word $\textbf{x}_{i}$ as
$\overline{\textbf{x}}^{i}=(\textbf{x}_{1},...,\texttt{<mask>},...,\textbf{x}_{m})$.
The baseline is therefore identical to x except at the
$\textrm{i}^{\textrm{th}}$ position, where the word $\textbf{x}_{i}$ is
replaced by the embedding of the word "mask"222Certain language models, such
as GPT-2 (Radford et al., 2019), do not have a ”mask” token. A ”pad” token
should be therefore used for such models., a token used in many language model
to replace part of the sentence during training. Moreover, we use the notation
$\overline{\textbf{x}}^{i}$ instead of $\overline{\textbf{x}}_{i}$ as
$\overline{\textbf{x}}^{i}$ corresponds to an entire sentence, not to be
mistaken with a single word like $\textbf{x}_{i}$.

In this setting, we keep the baseline as similar to the original sentence as
possible, only changing the word of interest. This method of explaining a word
is also kept similar to the way these language models are usually pre-trained,
by randomly masking part of sentences.

Let’s now define our Sequential Integrated Gradients (SIG) method. For a word
$\textbf{x}_{i}$ and a feature $j$, SIG is defined as:

$\textrm{SIG}_{ij}(\textbf{x}):=(x_{ij}-\overline{x}_{ij})\times\\\
\int_{0}^{1}\frac{\partial\textrm{F}(\overline{\textbf{x}}^{i}+\alpha\times(\textbf{x}-\overline{\textbf{x}}^{i}))}{\partial
x_{ij}}\,d\alpha$

Similar to the original IG Sundararajan et al. (2017), we compute the gradient
of F along a straight line between $\overline{\textbf{x}}^{i}$ and x for each
word $\textbf{x}_{i}$, the main difference being that the baseline differs for
each word. Also similar to the original IG, we approximate in practice the
integral with Riemann summation.

Finally, we compute the overall attribution of a word by computing the sum
over the feature dimension j, and normalising the result:

$\textrm{SIG}_{i}(\textbf{x}):=\frac{\sum_{j}\textrm{SIG}_{ij}}{||\textrm{SIG}||}$

#### Axioms satisfied by SIG

The original Integrated Gradients method satisfies a few axioms that are
considered desirable for any explanation methods to have. Among these axioms,
SIG follows implementation invariance, which states that attributions should
be identical if two models are functionally equivalent. Moreover, SIG follows
completeness in a specific way: for each word $\textbf{x}_{i}$, we have the
following result:

$\sum_{j}SIG_{ij}(\textbf{x})=\textrm{F}(\textbf{x})-\textrm{F}(\overline{\textbf{x}}^{i})$

This means that for each word, the sum of its attribution across all features
j is equal to the difference between the output of the model as x and at its
corresponding baseline $\overline{\textbf{x}}^{i}$. However, it does not
entail that
$\sum_{ij}SIG_{ij}(\textbf{x})=\textrm{F}(\textbf{x})-\textrm{F}(\overline{\textbf{x}})$,
where $\overline{\textbf{x}}$ would be an overall baseline filled with <mask>.

Moreover, this last axiom entail another one called sensitivity, which here
means that if, for a certain word, the input x has the same influence on the
output of F as its corresponding baseline $\overline{\textbf{x}}^{i}$, then
$\sum_{j}SIG_{ij}(\textbf{x})=0$.

Finally, we show in Appendix A that SIG preserves symmetry for each word on
the embedding dimension, but that this axiom is not true in general.

#### Using mask instead of pad as a baseline

We propose in this study to replace, as the baseline, the commonly used "pad"
token with the "mask" token, on language models having such token. This seems
to go against the intuition that the baseline should be uninformative, as
"mask" is a trained token. To support the usage of "mask", we argue that,
because <PAD> (denoting the embedding of "pad") is untrained, it could be
arbitrarily close to some words, and far from others. Oh the other hand,
<MASK> has been trained to replace random words, making it ideally as close to
one word as to any other.

Another way to see it is to compare it with images. It is natural for images
to choose the baseline as a black image, as this baseline has no information.
However, there is no such guarantee in NLP. For instance, the embedding of
"pad": <0, 0, 0, …, 0> could perfectly be very close to an embedding of a word
with a specific meaning, which would harm the explanations. On the other hand,
<MASK> has been trained to replace any word, and therefore seems more suited
to be the baseline.

##  3 Experiments

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.257 | 0.0681 | 0.315 | -0.121 | 0.0617 | 0.363 | -0.438 | 0.143 | 0.438
DeepLift | -0.332 | 0.101 | 0.260 | -0.163 | 0.0804 | 0.348 | -0.452 | 0.123 | 0.450
GradientShap | -0.452 | 0.237 | 0.212 | -0.389 | 0.194 | 0.299 | -0.715 | 0.204 | 0.438
IG | -0.540 | 0.341 | 0.163 | -0.787 | 0.354 | 0.242 | -1.19 | 0.307 | 0.410
DIG | -0.487 | 0.273 | 0.181 | -0.426 | 0.223 | 0.286 | -1.05 | 0.293 | 0.414
SIG | -0.533 | 0.331 | 0.134 | -0.869 | 0.361 | 0.251 | -1.52 | 0.390 | 0.349

Table 3: Comparison of SIG with several feature attribution methods on three
language models fine-tuned on the Rotten Tomatoes dataset.

| steps | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | Delta | Time
---|---|---|---|---|---|---
IG | 50 | -0.981 | 0.345 | 0.352 | 0.304 | $t$
SIG | 50 | -1.19 | 0.392 | 0.312 | 4.82 | N $\times$ $t$
IG | 250 | -0.999 | 0.352 | 0.355 | 0.055 | $t^{\prime}$
IG | 10 $\times$ N | -0.998 | 0.351 | 0.352 | 0.066 | N $\times$ $t^{\prime}$ / 25
SIG | 10 | -1.14 | 0.373 | 0.322 | 4.93 | N $\times$ $t^{\prime}$ / 25

Table 4: Comparison of IG and SIG with different numbers of interpolations on
BERT fine-tuned on the SST2 dataset.
$t$ and $t^{\prime}$ represent the amount of time to calculate IG with 50 and
250 steps respectively, and N represents the number of words on the input data
(for instance in one sentence). On the SST2 dataset, we have an average of: N
$\approx 25$ words per sentence.
On top of the table, we compare IG and SIG using a fixed number of steps. On
the bottom of the table, we compare IG with 250 steps against SIG with 10
steps. Since N $\approx 25$, we have N $\times$ $t^{\prime}$ / 25 $\approx$
$t^{\prime}$. For a fairer comparison, we also compare IG with a variable
number of steps: 10 $\times$ N for each sentence, against SIG with 10 steps.
These two methods have the same time complexity.
Delta is defined as
$\sum_{ij}Attr_{ij}(\textbf{x})-(\textrm{F}(\textbf{x})-\textrm{F}(\overline{\textbf{x}}))$.
Contrary to IG, SIG has a high delta value, as in general
$\sum_{ij}SIG_{ij}(\textbf{x})\neq\textrm{F}(\textbf{x})-\textrm{F}(\overline{\textbf{x}})$.

###  3.1 Experiments design

We evaluate SIG against various explanation methods by closely following the
experimental setup of Sanyal and Ren (2021). As such, we use the following
language models: BERT Devlin et al. (2018), DistilBERT Sanh et al. (2019) and
RoBERTa Liu et al. (2019). We also use the following datasets: SST2 Socher et
al. (2013), IMDB Maas et al. (2011) and Rotten Tomatoes Pang and Lee (2005),
which classify sentences into positive or negative sentiments or reviews.
Moreover, we use the HuggingFace library to recover processed data and
pretrained models Wolf et al. (2019).

Following Sanyal and Ren (2021), we use the following evaluation metrics: Log-
Odds Shrikumar et al. (2017), Comprehensiveness DeYoung et al. (2019) and
Sufficiency DeYoung et al. (2019). These metrics mask the top or bottom 20 %
important features, according to an attribution method, and measure by how
much the prediction of the language model changes using this masked data,
compared with the original one. For more details on these metrics, please see
Sanyal and Ren (2021).

Finally, we use the following feature attribution methods to compare our
methods against: Grad*Inp Shrikumar et al. (2016), Integrated Gradients
Sundararajan et al. (2017), DeepLift Shrikumar et al. (2017), GradientShap
Lundberg and Lee (2017) and Discretized IG (DIG) Sanyal and Ren (2021) using
the GREEDY heuristics. Moreover, as in Sanyal and Ren (2021), we use 50
interpolation steps for all methods expect from DIG, for which we use 30
steps.

Method | Example
---|---
IG | “a well-made and often lovely depiction of the mysteries of friendship.
SIG | “a well-made and often lovely depiction of the mysteries of friendship.
IG | “a hideous , confusing spectacle , one that may well put the nail in the coffin of any future rice adaptations.”
SIG | “a hideous , confusing spectacle , one that may well put the nail in the coffin of any future rice adaptations.”
IG | "this is junk food cinema at its greasiest."
SIG | "this is junk food cinema at its greasiest."
IG | "a remarkable 179-minute meditation on the nature of revolution."
SIG | "a remarkable 179-minute meditation on the nature of revolution."

Table 5: Examples of attributions on several sentences of the SST2 dataset.
The underlined bold tokens represent the most important token in the sentence,
while bold tokens represent the top 20 % tokens in the sentence, according to
each attribution method.

###  3.2 Results

#### Comparison with other feature attribution methods

We present of Tables 1, 2 and 3 a comparison of the performance of SIG with
the attribution methods listed in 3.1. We observe that SIG significantly
outperforms all other methods across most datasets and language models we
used. This tends to confirm that the change of overall meaning of a sentence
by combining interpolations simultaneously is an important issue which needs
to be tackled.

#### Comparison between IG and DIG

Although results in Sanyal and Ren (2021) show that DIG outperforms other
methods, including IG, this is not the case when using "mask" as a token. This
result seems to undermine the intuition of Sanyal and Ren (2021) that the
discrete nature of the embedding space is an important factor when explaining
a language model. We also show in Appendix C that the requirement of having a
monotonic path, stressed by Sanyal and Ren (2021), is not necessary.

#### Choice of the baseline token

We also provide in Appendix B results using "pad" as a baseline. Comparison
between Tables 1, 2 and 3 on one hand, and Tables 6, 7, 8 on the other hand
show that IG greatly improves using the "mask" token as a baseline. This seems
to confirm our intuition of using this token instead of "pad". Moreover, SIG
performs similarly using either token, which demonstrates the robustness of
this method across these two baseline tokens.

#### Time complexity of SIG

One important drawback of SIG is its time complexity, which is dependent on
the number of words in the input data. In Table 4, we compare the original IG
with SIG, using different numbers of steps. We define $t$ and $t^{\prime}$ as
the time complexity of computing IG with respectively 50 and 250 steps, and N
the number of words in the input data. This table shows that, although
reducing the number of steps results in a decrease of performance, SIG with 10
steps still performs better than both IG with 250 steps and IG with 10
$\times$ N steps, while having the same time complexity.

Moreover, as noted in Sanyal and Ren (2021), using IG with a large number of
steps decreases Delta =
$\sum_{ij}IG_{ij}(\textbf{x})-(\textrm{F}(\textbf{x})-\textrm{F}(\overline{\textbf{x}}))$,
while not improving performance. As a result, when computing attributions on
long sentences or large texts, we recommend using SIG with a reduced number of
steps instead of IG.

#### Comparison of IG and SIG on several examples

We provide on Table 5 several examples of explained sentences, using IG and
SIG. Both methods tend to agree on short sentences, while more disagreements
appear on larger ones. For each example, we display in underlined bold the
most important token, and in bold the top 20 % most important tokens,
according to each method.

##  4 Conclusion

In this work, we have defined an attribution method specific to text data:
Sequential Integrated Gradients (SIG). We have shown that SIG yields
significantly better results than the original Integrated Gradients (IG), as
well as other methods specific to language models, such as Discretized
Integrated Gradients (DIG). This suggests that keeping the meaning of
interpolated sentences close to the original one is key to producing good
explanations. We have also shown that, although SIG can be computationally
intensive, reducing the number of interpolations still yields better results
than IG with a greater number of interpolations.

We have also highlighted in this work the benefit of using the token "mask" as
a baseline, instead of "pad". Although SIG seems to be robust across both
tokens, this is especially important when using IG, as it significantly
improves the quality of explanations. Using the trainable token "mask" is
indeed closer to the training procedure of language models, and should yield
better interpolations as a result. We recommend therefore using this token as
a baseline, when possible, when explaining predictions made by a language
model.

Moreover, while this study was conducted on bidirectional language models such
as BERT, SIG could also be used on auto-regressive models such as GPT-2
(Radford et al., 2019), by iteratively computing the attribution of a token,
while keeping previous tokens fixed, and masking future tokens if any has been
already computed.

## Limitations

We see two main limitations of this work. The first one concerns the diversity
of the language models and datasets used. BERT, DistilBERT and RoBERTa have
similar architecture, and SST2, IMDB and Rotten Tomatoes are datasets designed
to evaluate the sentiment of English text. It would therefore be interesting
to validate the robustness of our results on more diverse languages, tasks and
language models. In this short paper, we decided for brevity to follow the
experiment design of Sanyal and Ren (2021), while being aware of its inherent
limitations.

The second limitation of this work concerns the time complexity of SIG. As it
needs to compute explanations for each word individually, this method can
become very computationally expensive when applied on large text data. To
alleviate this issue, we first made it possible to compute gradients in
parallel, using an internal batch size similar to how Captum Kokhlikyan et al.
(2020) implemented the Integrated Gradients method. Secondly, as discussed in
3.2, it is possible to reduce the number of interpolated points, which makes
the computation faster while retaining better performance than the original
IG.

In this work, we ran our experiments on a machine with 16 CPUs, and one Nvidia
Tesla T4 GPU. With this setting, computing SIG on SST2 and Rotten Tomatoes
takes around one hour for each model. On the larger IMDB, computing SIG, on
2000 randomly sampled inputs, takes around 5 days for BERT and RoBERTa, and 2
days for DistilBERT.

## Ethics Statement

The methods presented in this work aim to explain language models, and can as
such present ethical issues related to this task. Discriminating biases can
indeed be present in text data on which a language model is trained, and such
a model can acquire and propagate these biases Sap et al. (2019). As the
presented methods aim to explain a language model without additional
knowledge, these methods could also display discriminating biases learnt by a
language model.

Moreover, common explanation methods such as Integrated Gradients has proved
to be prone to adversarial attacks Dombrowski et al. (2019), and can be
misleading when used on out of sample data Slack et al. (2021). There is no
reason to believe our methods would be more robust compared to existing
methods such as IG.

The proposed methods can also be characterised as gradient-based, as they rely
on computing gradients on the input data, an uninformative baseline, or on
interpolated points between them. As noted by Mittelstadt et al. (2019), such
methods are only local and may not give a clear explanation of the model
globally.

## Acknowledgement

The author would like to thank Vitalii Zhelezniak for his thoughtful comments
and suggestions, including using the "mask" token as a baseline. We also thank
Anthony Hu for his detailed initial review of this paper.

## References

  * Ancona et al. (2017) Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2017.  Towards better understanding of gradient-based attribution methods for deep neural networks.  _arXiv preprint arXiv:1711.06104_.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * DeYoung et al. (2019) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2019.  Eraser: A benchmark to evaluate rationalized nlp models.  _arXiv preprint arXiv:1911.03429_.
  * Dombrowski et al. (2019) Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. 2019.  Explanations can be manipulated and geometry is to blame.  _Advances in Neural Information Processing Systems_ , 32.
  * Hoang et al. (2019) Mickel Hoang, Oskar Alija Bihorac, and Jacobo Rouces. 2019.  Aspect-based sentiment analysis using BERT.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics_ , pages 187–196, Turku, Finland. Linköping University Electronic Press.
  * Kokhlikyan et al. (2020) Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020.  Captum: A unified and generic model interpretability library for pytorch.
  * Liu (2019) Yang Liu. 2019.  Fine-tune bert for extractive summarization.  _arXiv preprint arXiv:1903.10318_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30.
  * Lundstrom et al. (2022) Daniel D Lundstrom, Tianjian Huang, and Meisam Razaviyayn. 2022.  A rigorous study of integrated gradients method and extensions to internal neuron attributions.  In _International Conference on Machine Learning_ , pages 14485–14508. PMLR.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 142–150.
  * Mittelstadt et al. (2019) Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019.  Explaining explanations in ai.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 279–288.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  _arXiv preprint cs/0506075_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  " why should i trust you?" explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sanyal and Ren (2021) Soumya Sanyal and Xiang Ren. 2021.  Discretized integrated gradients for explaining language models.  _arXiv preprint arXiv:2108.13654_.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th annual meeting of the association for computational linguistics_ , pages 1668–1678.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pages 3145–3153. PMLR.
  * Shrikumar et al. (2016) Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. 2016.  Not just a black box: Learning important features through propagating activation differences.  _arXiv preprint arXiv:1605.01713_.
  * Slack et al. (2021) Dylan Slack, Anna Hilgard, Himabindu Lakkaraju, and Sameer Singh. 2021.  Counterfactual explanations can be manipulated.  _Advances in Neural Information Processing Systems_ , 34:62–75.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020.  Incorporating bert into neural machine translation.  _arXiv preprint arXiv:2002.06823_.

##  Appendix A On the symmetry-preserving axiom of Sequential Integrated
Gradients

This section is divided into two parts. First, we show that SIG preserves
symmetry for each word along the embedding dimension. Second, we provide a
counterexample to show that symmetry does not hold in general.

#### Symmetry of $\textrm{SIG}_{i}$

Let us use the same notations as in Section 2. We want to compute the
attribution of a word $\textbf{x}_{i}$ on a model F, using the baseline
$\overline{\textbf{x}}^{i}$. Let’s define the function:

$\textrm{F}_{i}(\textbf{x}):=\textrm{F}(\textbf{x}_{1},...,\textbf{x},...,\textbf{x}_{m})$

$\textrm{F}_{i}$ corresponds to F where only the $\textrm{i}^{\textrm{th}}$
word is not fixed. Here, x corresponds to a word, not a sentence.

For such a function, SIG corresponds to the regular IG method: the baseline is
$<\textrm{mask}>$ and SIG constructs a straight line between this baseline and
$\textbf{x}_{i}$. As a result, if $\textrm{F}_{i}$ is symmetric on two
embedding features $j_{1}$ and $j_{2}$, SIG preserves this symmetry:
$\textrm{SIG}_{ij_{1}}(\textbf{x})=\textrm{SIG}_{ij_{2}}(\textbf{x})$.

#### Non symmetry of SIG

The fact that SIG does not preserve symmetry in general is due to the choice
of the baseline. As a counterexample, let’s define a language F which takes as
an input two words $\textbf{x}_{1}$ and $\textbf{x}_{2}$. This language model
is moreover symmetric:
$\textrm{F}(\textbf{x}_{1},\textbf{x}_{2})=\textrm{F}(\textbf{x}_{2},\textbf{x}_{1})$.

Here, the original IG method would preserve the symmetry: as the baseline is
$(\texttt{<mask>},\texttt{<mask>})$, when $\textbf{x}_{1}=\textbf{x}_{2}$, we
have $IG(\textbf{x})_{1}=IG(\textbf{x})_{2}$. However, SIG doesn’t preserve
the symmetry due to its baseline: we would have:
$\overline{\textbf{x}}^{1}=(\texttt{<mask>},\textbf{x}_{2})$ and
$\overline{\textbf{x}}^{2}=(\textbf{x}_{1},\texttt{<mask>})$. As a result,
$\textrm{SIG}(\textbf{x})_{1}=\textrm{SIG}(\textbf{x})_{2}$ only if
$\textbf{x}_{1}=\textbf{x}_{2}=\texttt{<mask>}$.

##  Appendix B Additional results using the "pad" token

We present in this section results using the "pad" token instead of the "mask"
one. These results for the three datasets: SST2, IMDB and Rotten Tomatoes can
be found respectively on Tables 6, 7 and 8.

When using the "pad" token as a baseline, SIG seems to perform similarly
compared with using the "mask" one, while other methods perform significantly
worse. This demonstrates both the need to use "mask" as a token, and the
robustness of the SIG method across different baselines.

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.402 | 0.112 | 0.375 | -0.318 | 0.085 | 0.398 | -0.454 | 0.092 | 0.439
DeepLift | -0.196 | 0.053 | 0.489 | -0.270 | 0.0784 | 0.439 | -0.283 | 0.061 | 0.463
GradientShap | -0.753 | 0.191 | 0.328 | -0.514 | 0.146 | 0.386 | -0.471 | 0.146 | 0.425
IG | -0.954 | 0.251 | 0.273 | -0.726 | 0.227 | 0.315 | -0.658 | 0.235 | 0.398
DIG | -1.222 | 0.310 | 0.237 | -0.812 | 0.249 | 0.287 | -0.879 | 0.292 | 0.374
SIG | -1.993 | 0.466 | 0.108 | -1.346 | 0.398 | 0.244 | -1.30 | 0.393 | 0.331

Table 6: Comparison of SIG with several baselines on three language models
fine-tuned on the SST2 dataset. For $\uparrow$ metrics, the higher the better,
while for $\downarrow$ ones, the lower the better.

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.189 | 0.082 | 0.209 | -0.216 | 0.047 | 0.315 | -0.654 | 0.087 | 0.299
DeepLift | -0.032 | -0.005 | 0.515 | -0.149 | 0.031 | 0.374 | -0.519 | 0.027 | 0.465
GradientShap | -0.315 | 0.117 | 0.302 | -0.351 | 0.110 | 0.213 | -0.622 | 0.088 | 0.358
IG | -0.474 | 0.186 | 0.201 | -0.499 | 0.169 | 0.114 | -0.577 | 0.117 | 0.288
DIG | -0.812 | 0.297 | 0.153 | -0.626 | 0.187 | 0.099 | -0.971 | 0.192 | 0.229
SIG | -2.157 | 0.585 | 0.0062 | -0.856 | 0.291 | 0.0207 | -1.96 | 0.352 | 0.152

Table 7: Comparison of SIG with several baselines on three language models
fine-tuned on the IMDB dataset.

Method | DistilBERT | RoBERTa | BERT
---|---|---|---
LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$ | LO $\downarrow$ | Comp $\uparrow$ | Suff $\downarrow$
Grad*Inp | -0.152 | 0.068 | 0.315 | -0.211 | 0.062 | 0.363 | -0.806 | 0.143 | 0.438
DeepLift | -0.077 | 0.017 | 0.372 | -0.198 | 0.056 | 0.370 | -0.457 | 0.076 | 0.474
GradientShap | -0.326 | 0.147 | 0.250 | -0.264 | 0.103 | 0.348 | -0.697 | 0.161 | 0.429
IG | -0.424 | 0.208 | 0.190 | -0.360 | 0.151 | 0.312 | -0.795 | 0.201 | 0.414
DIG | -0.501 | 0.257 | 0.184 | -0.346 | 0.153 | 0.310 | -1.06 | 0.267 | 0.416
SIG | -0.753 | 0.378 | 0.109 | -0.771 | 0.318 | 0.266 | -1.55 | 0.360 | 0.393

Table 8: Comparison of SIG with several baselines on three language models
fine-tuned on the Rotten Tomatoes dataset.

##  Appendix C Challenge of the monotonic assumption of the path

Sanyal and Ren (2021) stipulate that the path between a baseline and an input
needs to be monotonic to allow approximating the integral in IG using Riemann
summation. However, while this is true for a Riemann integral, it is also
possible to approximate the Riemann–Stieltjes integral, which is a
generalisation of Riemann integral, and does not need a monotonic path. We
define the Riemann–Stieltjes integral of
$f:[\textbf{a},\textbf{b}]\to\mathbb{R}$ as:

$\int_{\textbf{x}=\textbf{a}}^{\textbf{b}}f(\textbf{x})\,dg(\textbf{x})$

where $g:[0,1]\to[\textbf{a},\textbf{b}]$ designates a path. Let us define a
partition over $[0,1]$ as $t_{k}$ such as $0\leq t_{1}\leq...\leq t_{n}\leq
1$. We can then approximate the integral with the sum:

$\sum_{i=0}^{n-1}f(g(c_{i}))\times[g(t_{i+1})-g(t_{i})]$

where $c_{i}\in[t_{i},t_{i+1}]$. As such, while the partition
$t_{i},\,i\in\\{1,...,n\\}$ needs to be monotonic, the function g does not
need to have this constraint. As a result, we could define a path-based IG
method as:

$IG_{\gamma}(\textbf{x})_{i}:=\int_{\gamma}\frac{\partial\textrm{F}(\textbf{x})}{\partial
x_{i}}\,dx_{i}$

where $\gamma$ is not necessarily monotonic.

Lundstrom et al. (2022) provide more insights on this topic, and in particular
show that the implementation invariance, completeness and sensitivity axioms
hold for non-monotonic paths.

For this reason, we decided not to include a combination of DIG and SIG in
this study. However, an implementation of this method and the corresponding
results can be found in the repository published with this paper.