Controllable Prompt Tuning For
Balancing Group Distributional Robustness

Hoang Phan | Andrew Gordon Wilson | Qi Lei
---|---|---

New York University

{hvp2011<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Models trained on data composed of different groups or domains can suffer from
severe performance degradation under distribution shifts. While recent methods
have largely focused on optimizing the worst-group objective, this often comes
at the expense of good performance on other groups. To address this problem,
we introduce an optimization scheme to achieve good performance across groups
and find a good solution for all without severely sacrificing performance on
any of them. However, directly applying such optimization involves updating
the parameters of the entire network, making it both computationally expensive
and challenging. Thus, we introduce Controllable Prompt Tuning (CPT), which
couples our approach with prompt-tuning techniques. On spurious correlation
benchmarks, our procedures achieve state-of-the-art results across both
transformer and non-transformer architectures, as well as unimodal and
multimodal data, while requiring only $0.4\%$ tunable parameters.

##  1 Introduction

Spurious correlation or shortcut learning arises when a classifier relies on
non-predictive features that coincidentally correlate with class labels among
training samples. For example, a _majority_ of waterbirds are often found near
water, while landbirds mostly appear near land. Even when being provided with
a small number of _minority group examples_ , such as waterbirds on land, or
landbirds on water, model predictions are often still dominated by spurious
features, in this case, the background, ignoring the salient feature, the
foreground [68].

A variety of approaches aim to improve worst group performance, by
oversampling high-loss samples [53, 90], undersampling the majority group [69]
or re-training a last layer on group-balanced validation data [42]. A
particularly popular and effective baseline, _Group Distributionally Robust
Optimization_ (GroupDRO) [68, 69], directly minimizes an estimated upper bound
on worst group loss. However, all these procedures generally neglect knowledge
transfer between groups. Moreover, high-error samples can potentially include
noisy-label data that could hurt the model’s predictive ability if we try to
learn them exhaustively [61]. Besides, DRO-based methods, in particular, are
susceptible to overfitting, wherein their test performances can decline to the
same as ERM with sufficiently long training [31, 64, 88].

Figure 1:  Accuracy curves on Waterbirds for four groups during training.
Vertical lines indicate early stopping epochs when models perform best on the
validation set.

To visually illustrate these limitations, we plot in Figure 1 the performance
of a ResNet50 on the Waterbirds benchmark, trained with ERM, GroupDRO, and our
approach (to be explained later). The plots demonstrate that while GroupDRO
can improve the performance of the minority group (green line) early in
training compared to ERM, it rapidly overfits and fails to maintain this
performance over time. After training for ten epochs, the test performance gap
between the minority and majority (blue line) groups grows sharply. To address
these critical limitations, we introduce a systematic approach to train a
model that exhibits consistently high performance across all groups.
Specifically, at each iteration, we identify a descending direction that
benefits the groups simultaneously rather than focusing on an individual
group. Figure 1 shows that our proposed method performs almost equally well in
every group, thanks to our balancing mechanism.

Moreover, we aim to control the group-wise learning process by ensuring that
the magnitude of loss for different groups is inversely proportional to a
predefined vector $\mathbf{c}$. Varying this hyperparameter $\mathbf{c}$
adjusts the trade-off between the worst group and the average performance,
yielding a more flexible method than prior work that only focuses on
maximizing the worst group accuracy. However, computation can grow linearly
with the number of controlling vectors $\mathbf{c}$. To overcome this
challenge, we propose decoupling our optimization framework with parameter-
efficient fine-tuning (PEFT) techniques that considerably reduce the number of
trainable parameters. With the model compression ability powered by prompt-
tuning, our proposed model can scale well for a large number of controlling
vectors with a small parameter increase, e.g., $0.01\%$ and $0.003\%$ per
value of $\mathbf{c}$ for Vision Transformer [23] and CLIP [66], respectively.

Contribution. In this work, we introduce CPT, a Controllable Prompt Tuning
method to prevent models from learning spurious features. In summary, our key
contributions are depicted as follows:

  * •

Drawing inspiration from the principles of multi-objective optimization
theory, we introduce a novel balancing mechanism to address the challenges of
group distributional robustness. Our method not only considers the worst group
but also leverages the gradient information from all groups to determine an
ultimate updating direction that benefits all of them.

  * •

While the above rigid balancing procedure works well in different scenarios,
we extend our method by introducing a controlling vector that allows us to
dynamically adjust the priority across groups. Additionally, we integrate
prompt tuning techniques to enhance the scalability of our method as the
complexity and number of controlling vectors increase and to make it
applicable for de-biasing large transformers-based models.

  * •

We conduct intensive experiments on many different benchmarks, where CPT
consistently exhibits superior performance. Our method surpasses the current
state-of-the-art baselines on Waterbirds and CelebA datasets while updating
$0.4\%$ parameters. Moreover, it also outperforms recently proposed methods
that aim to de-bias Vision Transformer and CLIP models with minimal training
cost.

##  2 Related work

We first review relevant methods to enhance distributional robustness and then
discuss prior work related to the two essential parts of our proposed method:
transformer and parameter-efficient fine-tuning.

#### Distributional robustness

can be compromised for reasons like selection bias or spurious correlation
across tasks. For selection bias on individual samples, prior works seek to
reduce bias via practices like importance reweighting [36, 72, 16, 46], hard
sample reweighting [53, 60] or distribution matching and discrepancy
minimization [17, 7, 8, 63], and domain-adversarial algorithms [27, 58] across
tasks in their feature representation space. For selection bias on groups or
subpopulation, namely subpopulation shift or dataset imbalance, label
propagation [11, 8] or other consistency regularization [59, 84] are used to
generalize the prediction to broader domains. In particular, GroupDRO [68, 69]
or other relevant methods [44, 39] select descending direction more in favor
of the worse performing/minority group(s) during training.

Improving distributional robustness also requires mitigating the effect of
environmental features. Prior work uses model ensemble [45] or model soups
[81] to learn rich features and reduce the effect of spurious features. More
practices include invariant feature representation learning [6, 14], domain
generalization [44, 73, 71] or through data augmentation [83, 86].

Transformer In the past few years, transformer [77] has emerged as a dominant
architecture for a wide range of applications, exhibiting their profound
capacity in modeling different modalities: languages [21, 55, 76], vision [23,
12], speech [67, 79], and time series [49, 30]. Due to its ability to process
multiple modalities, transformer-based architectures were widely adopted for
image understanding using different types of supervision. For example, Vision
Transformer (ViT) [23] utilizes masked patches for pre-training and image-
label supervision for fine-tuning. Notably, Contrastive Language-Image Pre-
training (CLIP) [66], which was pre-trained on billion-scale image-text pairs,
has demonstrated excellent zero-shot performance in image classification
tasks. Despite obtaining impressive results on various downstream tasks, over-
parameterization is still a crucial problem for transformer [26, 62] which
potentially causes overfitting [48] and unintended biases [1, 80, 24, 89] more
severely than small-scale architectures [69].

Parameter-Efficient Fine-Tuning While conventional training updates the entire
network, prior work has shown that it is possible to achieve performance
comparable with full fine-tuning by instead updating a small set of parameters
[51, 33, 34]. Early work in this line of research includes adapter [37, 13,
74], which adds a lightweight fully connected network between the layers of a
frozen pre-trained model. Since then, there have been many other methods that
update subsets of the network [32, 28, 87] or reparametrize the weights of the
network using low-rank decompositions [2, 38, 25]. Another appealing PEFT
strategy is prompt tuning, which prepends a few trainable tokens before the
input sequence or hidden state to encode task-specific knowledge to the pre-
trained model [47, 50, 94, 40].

##  3 Background

In this section, we first formulate the problem of group robustness and recap
the technical details of GroupDRO, then revisit some concepts of multi-
objective optimization.

###  3.1 Problem formulation

Formally, we consider the conventional setting of classifying an input
$x\in\mathcal{X}$ as a target $y\in\mathcal{Y}$, where
$\mathcal{X},\mathcal{Y}$ are input and label spaces, respectively. We are
given a training dataset composed of $K$ groups from the set $\mathcal{G}$,
where each group $g\in\mathcal{G}$ consists of $n_{g}$ instances sampled from
the probability distribution $P_{g}(\mathcal{X},\mathcal{Y})$. Since the
numbers of examples are different among groups in the training data, we
consider groups with relatively large $n_{g}$ as majority groups and those
with small $n_{g}$ as minority groups. Our main goal is to develop a model
that performs effectively across all groups within $\mathcal{G}$ to avoid
learning spurious correlations.

Following previous methods [69], we adopt two metrics: worst group accuracy,
indicating the minimum test accuracy across all groups $g\in\mathcal{G}$, and
average accuracy, which represents the weighted average test accuracy with the
weights corresponding to the relative proportions of each group in the
training data. It is worth noting that commonly used datasets for group
robustness often exhibit skewed training set distributions, the weighted
average score is thus dominated by the performance of the model on groups that
include large numbers of training data. Therefore, we propose to evaluate the
model performance by additionally reporting the mean (unweighted average)
accuracy score across different groups.

###  3.2 Group distributionally robust optimization

GroupDRO learns the model parameters $\theta$ by directly optimizing for the
worst-group training loss as follows:

$\max_{g\in\mathcal{G}}\mathbb{E}_{(x,y)\sim
P_{g}(\mathcal{X},\mathcal{Y})}\left[\ell\left(f_{\theta}(x),y\right)\right]$

for some loss function
$\ell:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}$. Then, their
updating formula at each step is given by:

$\quad g^{*}=\arg\max\ell_{g}(\theta)\quad$ and
$\quad\theta^{t+1}=\theta^{t}-\eta\nabla_{\theta}\ell_{g^{*}}$, where $\eta$
is the learning rate.

###  3.3 Multi-objective Optimization

Assume that we are given $m$ objectives functions $f_{i}(x)$, $i\in[m]$ where
$x\in\mathbb{R}^{d}$ and each $f_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is a
scalar-valued function (we use the $[m]$ notation to denote the set
$\\{1,2,\dots,m\\}$) . In the multi-objective optimization (MOO) problem, we
are interested in minimizing a vector-valued objective function whose $i$-th
component is $f_{i}(x)$:

$\vec{F}(x)=[f_{1}(x),f_{2}(x),\dots,f_{m}(x)]$ (1)

In general, an optimal solution for this objective vector function does not
exist since each of the individual function $f_{i}(x)$ does not necessarily
guarantee to have the same minimum solution. Alternatively, we expect to
obtain a solution, from which we cannot improve any specific objective without
hurting another. According to the above formulation, the solution [95] of the
multi-objective minimization problem is formally defined as follows:

######  Definition 1.

(Pareto dominance) Let $x_{1},x_{2}$ be two solutions for the multi-objective
optimization problem in Equation 1, $x_{1}$ dominates $x_{2}$ $(x_{1}\prec
x_{2})$ if and only if $f_{i}(x_{1})\leq f_{i}(x_{2})\forall i\in[m]$ and
$\exists j\in[m]$ s.t. $f_{j}(x_{1})<f_{j}(x_{2})$.

######  Definition 2.

(Pareto optimality) A solution $x^{*}$ in problem 1 is said to be Pareto
optimal if it is not dominated by any other solution. Therefore, the solutions
set of the multi-objective minimization problem is given by
$\mathcal{P}:=\left\\{x\mid\nexists x^{\prime}:x^{\prime}\prec x\right\\}$.
This set of all Pareto optimal solutions is called the Pareto set while the
collection of their images in the objective space is called the Pareto front.

Gradient-based MOO Désidéri [20] shows that the descent direction can be found
in the convex hull composed of gradients corresponding to different objectives
$\\{\boldsymbol{h}^{i}:=\nabla f_{i}\\}_{i=1}^{m}$:

$\mathcal{CH}=\left\\{\boldsymbol{H}^{\top}\boldsymbol{\alpha}=\sum_{i=1}^{m}\alpha_{i}\boldsymbol{h}^{i}\mid\alpha\in\Delta_{m}\right\\}$

where $\Delta_{m}$ denotes the $m-1$ dimension simplex.

Moreover, they introduce the Multiple Gradient Descent Algorithm (MGDA), which
calculates the minimum-norm gradient vector $\boldsymbol{h}$ that lies in the
convex hull: $\boldsymbol{h}=\text{argmin}_{h\in\mathcal{CH}}||h||_{2}$. This
approach can guarantee that the obtained solutions lie on the Pareto front,
from which we cannot find an updating direction that decreases all objectives
simultaneously.

##  4 Proposed method

In GroupDRO, the updating formula solely focuses on the worst-performing
group, which is susceptible to challenges when groups exhibit varying loss
magnitude, noise, and transfer characteristics. For example, among the high-
loss group, there might exist a few outliers that have large input shifts or
even wrong labels, which can raise the training loss of the whole group and
prevent the model from fitting other groups.

###  4.1 Balancing group distributional robustness

Figure 2: Overview of our method on the Waterbirds dataset. Our main objective
is not only to improve model performance across groups by optimizing their
loss functions $\ell_{1},\ell_{2},\ell_{3},\ell_{4}$, but also to maximize the
entropy over this loss distribution.

In this paper, we go beyond the worst group approach of GroupDRO to examine
all groups at a time. To encourage the model to learn from different groups at
each step to avoid struggling on just the hardest one only, we propose to
minimize the following $K$-dimension loss vector:

$\vec{\mathcal{L}}(\theta)=\big{\\{}\mathbb{E}_{(x,y)\sim
P_{g^{k}}(\mathcal{X},\mathcal{Y})}\left[\ell\left(f_{\theta}(x),y\right)\right]\big{\\}}_{k=1}^{K}$
(2)

where $K$ is the number of groups and $g^{k}$ is the $k$-th group.

At $t$-th iteration, we search for an update vector that minimizes the
weighted sum of group losses:

$\displaystyle
d_{t}(w)=w^{\top}\nabla\vec{\mathcal{L}}(\theta_{t})=\sum_{k=1}^{K}w_{k}\nabla\ell_{k}(\theta_{t})$
(3)

Then, the updating formula of our algorithm is then given by
$\theta_{t+1}=\theta_{t}-\eta d_{t}(w)$ for some step size
$\eta\in\mathbb{R}^{+}$. The coefficient $w$ is chosen such that our update
rule can benefit all groups $d_{t}(w)^{\top}\nabla\ell_{k}(\theta_{t})\geq 0$.
To this end, a straightforward approach that is readily applicable to optimize
the objective (2) is applying MGDA to find the solution of (3) that yields the
minimum Euclidean norm of the composite gradient:

$\displaystyle
d_{t}^{*}=\operatorname*{arg\,min}_{w\in\Delta_{{}_{K}}}\left\|\sum_{k=1}^{K}w_{k}\nabla\ell_{k}(\theta_{t})\right\|_{2}$
(4)

However, MGDA is often biased toward objectives with smaller gradient
magnitudes [54], and lacks controllable property (i.e. could not adjust the
group that we want to focus more or less at each update), which can be
utilized to balance the loss between groups. Hence, we introduce a new
objective that steers the updating direction to obtain our desired balanced
magnitude among groups. Assuming that the entropy $H(p)$ measures the
diversity of a continuous distribution $p$, this behavior could be achieved
via maximization of the entropy with respect to the distribution of loss
functions:

$\displaystyle\mathcal{L}_{ent}(\theta)$
$\displaystyle=H\big{(}\text{softmax}\big{(}\vec{\mathcal{L}}(\theta))\big{)}=H\Big{(}\text{softmax}\big{(}\big{\\{}\mathbb{E}_{P_{g^{k}}}\left[\ell\left(f_{\theta}(x),y\right)\right]\big{\\}}_{k=1}^{K}\big{)}\Big{)}$
(5)

######  Theorem 3.

Assume that the loss function $\ell$ is differentiable up to the first order
with respect to $\theta$, then following

$d_{ent}:=\sum_{i=1}^{K}\nabla\ell_{i}(\theta)\Big{[}p_{i}\log(p_{i})-p_{i}\sum_{j=1}^{K}\log(p_{j})p_{j}]\Big{]}$

where $p_{i}=\frac{e^{\ell_{i}(\theta)}}{\sum_{j=1}^{K}e^{\ell_{j}(\theta)}}$,
maximizes the objective $\mathcal{L}_{ent}(\theta)$.

The proof of Theorem 3 is deterred to Appendix A. Theorem 3 provides us with
the updating rule that increases the entropy term in Equation (5) the most. To
balance the learning across groups, we seek the coefficient $w$ that maximizes
the cosine similarity with $d_{ent}$ and does not conflict against the
gradient vector of each group. Denotes
$w^{\prime}_{i}=p_{i}\log(p_{i})-p_{i}\sum_{j=1}^{K}\log(p_{j})p_{j}$, this is
equivalent to optimizing:

$\displaystyle w^{*}$
$\displaystyle=\operatorname*{arg\,max}_{w\in\Delta_{K}}d_{t}(w)^{\top}d_{ent}$
(6)
$\displaystyle\text{s.t.}\;\;\;d_{t}(w)^{\top}\nabla\ell_{k}(\theta_{t})\geq
0\;\forall\,k\in[K].$

While the balance between loss magnitude is not achieved, we strictly follow
the updating direction that decreases training errors of high-loss groups but
slightly relaxes constraints in the optimization problem (6) to allow bounded
increases among groups that already have small loss magnitudes.

$\displaystyle
w^{*}=\operatorname*{arg\,max}_{w\in\Delta_{K}}d_{t}(w)^{\top}d_{ent}$
$\displaystyle\;\;\;\;=\operatorname*{arg\,max}_{w\in\Delta_{K}}w^{\top}\nabla\vec{\mathcal{L}}(\theta)^{\top}\nabla\vec{\mathcal{L}}(\theta)w^{\prime}$
(7) $\displaystyle\text{s.t.}\;\;\;\forall\,k\in[K]:$
$\displaystyle\;\;d_{t}(w)^{\top}\nabla\ell_{k}(\theta_{t})\geq
0\;\text{if}\;\ell_{k}(\theta_{t})=\max_{k\in[K]}\ell_{k}(\theta_{t}).$
$\displaystyle\;\;d_{t}(w)^{\top}\nabla\ell_{k}(\theta_{t})\geq
d_{ent}^{\top}\nabla\ell_{k}(\theta_{t})\;\text{if}\;\ell_{k}(\theta_{t})<\max_{k\in[K]}\ell_{k}(\theta_{t}).$

The second constraint guarantees that the obtained solution is not worse than
$d_{ent}$ on those groups in which the model is performing well or when
$d_{ent}$ itself intrinsically supports learning them. Once the loss magnitude
is almost balanced ($||d_{ent}||\leq\epsilon$), we simply update the model
parameter by the average group gradients, i.e.
$w=(\frac{1}{K},\frac{1}{K},\dots,\frac{1}{K})$.

In summary, the key difference between our proposed method and GroupDRO is
that we solve a small $K$-dimensional optimization problem to find the
updating direction that not only focuses on the worst group but also improves
the performance in other groups and balances the loss magnitude between them
at the same time. In practical settings, the group number $K$ is often a small
number (e.g. $K=4$), thus could be efficiently solved using standard linear
programming libraries [22, 5] without introducing significant computational
overhead (Appendix B).

###  4.2 Controllable prompt tuning for balancing group distributional
robustness

To achieve the controllable property, we apply the optimization procedure in
section 4.1 on
$\ell_{k}(\theta)={c}_{k}\mathbb{E}_{P_{g^{k}}}\left[\ell\left(f_{\theta}(x),y\right)\right]$
where $\mathbf{c}\in\mathbb{R}^{K}$ is a pre-defined vector that is used to
adjust the magnitude of group losses. Intuitively, the principle behind our
proposed objective is to enforce the loss vector $\vec{\mathcal{L}}(\theta)$
to be inversely proportional to $\mathbf{c}$, from which we can adjust the
group loss magnitude, and thus control the trade-off between them by varying
$\mathbf{c}$. For example, we can put more weight on penalizing loss terms of
some specific groups to accelerate the learning progress on those groups, or
conversely slow down the learning to avoid overfitting on oversampled groups
[19].

In practice, tuning the value of $\mathbf{c}$ to obtain our desired model
behavior requires learning multiple models individually and independently.
Therefore, the computational complexity of our proposed method can grow
linearly with the number of vectors $\mathbf{c}$, thus being very expensive in
the context of deep learning. To reduce overhead, we adopt the usual practice
of parameter-efficient fine-tuning methods and optimize only a small portion
of the model. In particular, we freeze the main backbone while introducing
lightweight prompt sets, one for each task. For example, we employ the same
ViT backbone for the two datasets Waterbirds, CelebA, but using different sets
of prompts (Section 5).

Overview of our proposed method is depicted in Figure 2, given an input image
belonging to one of a group $\\{g^{1},g^{2},\dots,g^{K}\\}$ and a set of
classes, we solve the optimization problem (4.1) to find the reweighting
coefficient $w$ that maximizes the entropy $\mathcal{L}_{ent}(\theta)$ while
reasonably decreases group losses. While Figure 2 illustrates our proposed
method on the CLIP model on the image classification task, it applies to other
transformer-based models and task types. In our experiment, we exploit both
image-end prompts and language-end prompts tuning on frozen ViT and CLIP
backbones, respectively.

Prompt tuning for Vision Transformer: Regarding the prompt design for a ViT of
$N$ layers, we use the set of continuous learnable tokens
$\\{\mathbf{P}_{n}\\}_{n=0}^{N-1}$. Each $\mathbf{P}_{n}\in\mathbb{R}^{L\times
D}$, where $D$, $L\in\mathbb{N}$ indicate the prompt length and latent space
dimension, respectively. Subsequently, the input at the $(n+1)$-th layer
follows the format of $\left[x_{n},\mathbf{P}_{n},\mathbf{E}_{n}\right]$,
where $\mathbf{x}_{n},\mathbf{E}_{n}\in\mathbb{R}^{D}$ represent the [CLASS]
token and image patch embeddings after $n$-th layer while $[\cdot,\cdot]$
denotes the concatenation operator. Thus, the number of trainable parameters
for this image-end prompt set is $L\times N\times D$.

Prompt tuning for CLIP: Being trained on an enormous amount of image-text
pairs, CLIP offers an expressive representation for both language and image
inputs by using two separate encoders, one for each modality, as shown in
Figure 2. Since the image encoder can be either ResNet [35], ConvNeXt [57] or
ViT [23], we conduct prompt tuning on the transformer text encoder for
consistency. In short, we employ a learnable context prompt of length $L$:
$\mathbf{P}\in\mathbb{R}^{L\times D}$, to replace the hand-crafted prompt used
by the original CLIP (i.e., “this is a photo of a "). Based on this design,
the input for the text encoder of each class has the format of
$\left[x,\mathbf{P}\right]$, where $x$ is the [CLASS] embedding of the
corresponding class name and has the same dimension of $D$. These learnable
tokens $\mathbf{P}$ are better at capturing domains-specific knowledge than
those artificial prompt [92, 93] and can be learned to de-bias the original
CLIP model while introducing a few trainable parameters ($N\times D$).

##  5 Experiments

In this section, we evaluate the effectiveness of the proposed CPT method on
benchmark image datasets in the presence of spurious features: Waterbirds
[68], CelebA [56], MetaShift [52] and ISIC [15]. Due to the space limit
constraint, we briefly provide an overview of our experimental setup below,
see Appendix B for more details and additional results.

###  5.1 Datasets

Waterbirds is created by placing bird photos from the Caltech-UCSD Birds
dataset [78] with background images taken from Places [91]. The target
attributes are foreground objects (i.e. landbird or waterbird) while spurious
correlations are background (i.e. land or water landscape).

CelebA is a factual dataset containing $200$K face images of celebrities. The
goal is to classify blond/non-blond hair color. Statistically, more than
94$\%$ of blond hair examples in the training set are women.

MetaShift is another real-world dataset [52] for cat-dog classification, where
the cat and dog images are spuriously correlated with indoor (e.g. bed) and
outdoor objects (e.g. bike). For evaluation, models are given dog and cat
images associated with shelf background, which does not appear in the training
set. Thus, this dataset represents a more coherent distributional shift,
compared to other benchmarks.

ISIC is an even more challenging dataset with the appearance of multiple
spurious features [15], which includes dermoscopic images of skin lesions with
target attributes benign or melanoma. Visual examples of those datasets, along
with the size for each group in train, validation, and test sets are given in
Appendix B.

###  5.2 Baselines

Following the previous studies, we compare CPT against other state-of-the-art
methods for mitigating spurious correlations, including GroupDRO [68], Deep
Feature Reweighting (DFR) [42], Progressive Data Expansion (PDE) [19],
Subsample [19], Common Gradient Descent (CGD) [64] with the access to the
group information during training. Besides, we consider those methods require
group labels for validation: Learning from failure (LfF) [60], Just Train
Twice (JTT) [53], Predictive Group Invariance (PGI) [3], Contrastive Input
Morphing (CIM) [75], Correct-n-Contrast (CNC) [90], or those do not use group
annotations at all like Empirical Risk Minimization (ERM) and Automatic
Feature Reweighting (AFR) [65]. Moreover, we also report the results from two
recent studies of spurious correlation on ViT [29] and CLIP [85].

Regarding the baselines for those experiments further taking the distribution
shift into account, we follow exactly the protocol of [82] and compare our
method to Empirical Risk Minimization (ERM), Upweighting (UW), Invariant risk
minimization (IRM) [6], Information Bottleneck based Invariant Risk
Minimization (IB-IRM) [4], Variance Risk Extrapolation (V-REx) [44],
CORrelation ALignment (CORAL) [73], and Fish [71]; instance reweighting
methods: GroupDRO [68], JTT [53], Domain mixup (DM-ADA) [83], Selective
Augmentation (LISA) [86] and the current state-of-the-art method on those
benchmarks Discover and Cure (DISC) [82].

###  5.3 Model training and evaluation

We examine our proposed method on ResNet50 [35], ViT-B/16 [23] and different
variants of OpenAI’s CLIP [66] to align with previous work [42, 29, 85]. For
prompt tuning on ViT, we employ prompts of lengths $L=5$ in the MetaShift and
ISIC, or $L=10$ in the Waterbirds and CelebA datasets. Similarly, $L$ is set
to $16$ for all CLIP backbones throughout our experiment. Unless otherwise
explicitly mentioned in the experiment, the value of the controlling vector
$c$ is set to the default value of $(1,\dots,1)$. Other detailed
configurations are relegated to Appendix B.

Due to the large number of groups in ISIC, we calculate the AUROC score [10],
following [82, 9] while computing average, worst-group and mean (if possible)
accuracy scores for other datasets. For a fair and precise comparison, results
of baselines are taken from recent studies [19, 82, 64, 90, 41] and reported
directly from original papers.

###  5.4 Results

Waterbirds and CelebA We start by validating CPT on two well-established
benchmarks for investigating spurious correlation, namely Waterbirds and
CelebA in Table 5.4. From the results, we see that CPT consistently achieves
the best worst-group accuracy compared with the recent state-of-the-art
methods such as DFR and PDE on both datasets. Remarkably, our method only
updates a tiny portion ($0.4\%$) of parameters. As such, CPT is much more
computationally efficient than methods that need to update the entire network.
Furthermore, we see that CPT has strong gains in average accuracy, thus not
only outperforming other spurious correlation methods on both metrics but also
closing the gap to the average accuracy of ERM. Another interesting
observation is that those baselines aim to balance the learning between groups
via downsampling the minority groups (Subsample), reweighting group losses
inversely proportional to the group populations (UW), or computing scaling
weights using group training performance to upweight the worst-group loss
(GroupDRO), are effective in alleviating spurious correlation. Even though,
CPT still exhibits impressive performance compared to the aforementioned
methods by using an adaptive balancing mechanism at each iteration instead of
relying on fixed reweighting coefficients.

Table 1: Overall results on Waterbirds and CelebA datasets with best methods
are highlighted in bold. Performance is evaluated on the test set with models
early stopped at the highest worst-group accuracy on the validation set.

|  Train  group info |  Validation  group info |  Train once | Waterbirds | CelebA
---|---|---|---|---|---
Method | Worst | Average | # Params | Worst | Average | # Params
ERM | $\times$ | $\times$ | $\surd$ | $70.0_{\pm 2.3}$ | $97.1_{\pm 0.1}$ |  $23$M | $45.0_{\pm 1.5}$ | $94.8_{\pm 0.2}$ |  $23$M
AFR | $\times$ | $\times$ | $\times$ | $90.4_{\pm 1.1}$ | $94.2_{\pm 1.2}$ |  $23$M | $82.0_{\pm 0.5}$ | $91.3_{\pm 0.3}$ |  $23$M
LfF | $\times$ | $\surd$ | $\times$ | $78.0_{N/A}$ | $91.2_{N/A}$ |  $23$M | $77.2_{N/A}$ | $85.1_{N/A}$ |  $23$M
SSA | $\times$ | $\surd$ | $\times$ | $89.0_{\pm 0.6}$ | $92.2_{\pm 0.9}$ |  $23$M | $89.8_{\pm 1.3}$ | $92.8_{\pm 1.3}$ |  $23$M
JTT | $\times$ | $\surd$ | $\times$ | $86.7_{N/A}$ | $93.3_{N/A}$ |  $23$M | $81.1_{N/A}$ | $88.0_{N/A}$ |  $23$M
PGI | $\times$ | $\surd$ | $\times$ | ${{79.5}}_{\pm 1.9}$ | $95.5_{\pm 0.8}$ |  $23$M | $85.3_{\pm 0.3}$ | $87.3_{\pm 0.1}$ |  $23$M
CIM | $\times$ | $\surd$ | $\times$ | ${{77.2}}_{N/A}$ | $95.6_{N/A}$ |  $23$M | $83.6_{N/A}$ | $90.6_{N/A}$ |  $23$M
CnC | $\times$ | $\surd$ | $\times$ | $88.5_{\pm 0.3}$ | $90.9_{\pm 0.1}$ |  $23$M | $88.9_{\pm 1.3}$ | $88.9_{\pm 0.5}$ |  $23$M
DFR | $\times$ | $\surd$ | $\times$ | $92.9_{\pm 0.2}$ | $94.2_{\pm 0.4}$ |  $23$M | $88.3_{\pm 1.1}$ | $91.3_{\pm 0.3}$ |  $23$M
UW | $\surd$ | $\surd$ | $\surd$ | $88.0_{\pm 1.3}$ | $95.1_{\pm 0.3}$ |  $23$M | $83.3_{\pm 2.8}$ | $92.9_{\pm 0.2}$ |  $23$M
Subsample | $\surd$ | $\surd$ | $\surd$ | $86.9_{\pm 2.3}$ | $89.2_{\pm 1.2}$ |  $23$M | $86.1_{\pm 1.9}$ | $91.3_{\pm 0.2}$ |  $23$M
LISA | $\surd$ | $\surd$ | $\times$ | $89.2_{\pm 0.6}$ | $91.8_{\pm 0.3}$ |  $23$M | $89.3_{\pm 1.1}$ | $92.4_{\pm 0.4}$ |  $23$M
GroupDRO | $\surd$ | $\surd$ | $\surd$ | $86.7_{\pm 0.6}$ | $93.2_{\pm 0.5}$ |  $23$M | $86.3_{\pm 1.1}$ | ${92.9}_{\pm 0.3}$ |  $23$M
CGD | $\surd$ | $\surd$ | $\surd$ | $88.9_{\pm 0.8}$ | $91.3_{\pm 0.6}$ |  $23$M | $90.0_{\pm 0.8}$ | ${92.5}_{\pm 0.2}$ |  $23$M
DISC | $\surd$ | $\surd$ | $\times$ | ${{88.7}}_{\pm 0.4}$ | $93.8_{\pm 0.7}$ |  $23$M | N/A | N/A | N/A
PDE | $\surd$ | $\surd$ | $\surd$ | ${{90.3}}_{\pm 0.3}$ | $92.4_{\pm 0.8}$ |  $23$M | ${{91.0}}_{\pm 0.4}$ | $92.0_{\pm 0.6}$ |  $23$M
CPT | $\surd$ | $\surd$ | $\surd$ | ${\mathbf{93.5}}_{\pm 0.4}$ | ${{96.3}}_{\pm 0.2}$ |  $\mathbf{94}$k | $\mathbf{92.0}_{\pm 0.3}$ | ${93.2}_{\pm 0.3}$ |  $\mathbf{94}$k

De-bias ViT and CLIP Motivated by recent studies focusing on investigating
spurious correlations of pre-trained transformer-based models [85, 18], we
apply our proposed algorithm on ViT and CLIP and then compare them against
those work. It is worth noting that, the authors did not experiment on CelebA
with the blond hair target. Hence, we only show their original results on
Waterbirds in Figure 3 and Table 2. As can be seen in Figure 3, while CPT
enjoys much less computation and memory overhead (only $0.1\%$ and $0.04\%$
compared to ViT-B/16 itself and BiT-M-R50x3 [29]) it still outperforms the
second-best model by a large margin. This significant improvement comes from
our prompting design and optimization contribution, respectively. Details of
backbones are given in Appendix B.

Figure 3: Results of fine-tuning ViT backbones on Waterbirds. Error bars
represent the standard deviation over independent runs.

As shown in Table 2, CPT${}_{\text{balance}}$ (i.e. CPT with $c=(1,\dots,1)$)
clearly outperforms all other alternatives using the same number of trainable
parameters, which proves the effectiveness of our optimization framework.
Furthermore, when varying the coefficient $c$, one can control the trade-off
between group performance, yielding a much more flexible learning algorithm
for group distributional robustness. Indeed, increasing coefficients of the
majority groups (e.g. $c=(2,1,1,1.5)$ in this setting) helps improve the
average accuracy of CPT to perform on par or better than Yang et al. [85] on
both metrics while updating only $0.05\%$ and $1\%$ of the model parameters.
Thus, this again suggests the clear efficiency advantage of our prompt tuning,
which offers scalability for controlling adjustment. Since CPT has a larger
capacity to control the learning between groups, we can sacrifice worst-group
accuracy for the gain in performance on other groups (line 11) and vice-versa
(line 10). Intriguingly, while obtaining impressive performance on CLIP-RN50,
Contrastive Adapter [89] falls far behind other baselines on CLIP
ViT-L/14@336px.

Table 2: Performance of CLIP models on Waterbirds. $\dagger$ indicates our
implemented results. We use Bold font to indicate the best methods for
different numbers of trainable parameters. N/A denotes those methods do not
report the number of trainable parameters.

Model | ResNet50 | ViT-L/14@336px
---|---|---
| Average | Worst | # Params | Average | Worst | # Params
Pre-trained CLIP | ${90.8}$ | $44.9$ | $0$ | $88.5$ | $34.0$ | $0$
Fine-tuned CLIP | $81.3$ | ${77.1}$ | $14789632$ | ${97.2}$ | $89.7$ | $786432$
ERM | ${93.5}$ | $54.4$ | $14789632$ | $96.8$ | $58.1$ | $786432$
ERM Adapter | $96.0$ | ${63.0}$ | $524288$ | $97.8$ | $76.1$ | $524288$
Contrastive Adapter | $88.2$ | ${82.5}$ | $263424$ | $94.5$ | $85.3$ | $197632$
DFR | $91.8$ | $63.9$ | N/A | $96.1$ | $65.9$ | N/A
FairerCLIP | $84.3$ | $75.4$ | N/A | $92.2$ | $86.0$ | N/A
GroupDRO | $83.3$ | $73.7$ | $14789632$ | $94.1$ | ${90.8}$ | $786432$
Yang et al. [85] | $83.2$ | ${77.5}$ | $14789632$ | $96.9$ | ${90.5}$ | $786432$
CPT${}_{\text{balance}}$ | $85.8$ | $\mathbf{82.8}$ | $14789632$ | $94.0$ | $\mathbf{92.9}$ | $786432$
ERM† | ${85.7}$ | $64.6$ | $8192$ | $97.4$ | $84.3$ | $12288$
GroupDRO† | $81.0$ | $76.7$ | $8192$ | $95.8$ | $90.5$ | $12288$
CPT | $83.3$ | $78.4$ | $8192$ | $97.0$ | $90.2$ | $12288$
CPT${}_{\text{balance}}$ | $81.8$ | $\mathbf{79.8}$ | $8192$ | $95.6$ | $\mathbf{91.9}$ | $12288$

Table 3: Experimental results on MetaShift and ISIC dataset.

| MetaShift | ISIC |
---|---|---|---
Method | Average | Worst | AUROC | # Params
ERM | $72.9_{\pm 1.4}$ | $62.1_{\pm 4.8}$ | $36.4_{\pm 0.7}$ |  $23$M
UW | $72.1_{\pm 0.9}$ | $60.5_{\pm 3.8}$ | $39.2_{\pm 0.6}$ |  $23$M
IRM | $73.9_{\pm 0.8}$ | $64.7_{\pm 2.1}$ | $45.5_{\pm 3.6}$ |  $23$M
IB-IRM | $74.8_{\pm 0.2}$ | $65.6_{\pm 1.1}$ | $38.6_{\pm 1.7}$ |  $23$M
V-REx | $72.7_{\pm 1.7}$ | $60.8_{\pm 5.5}$ | $24.5_{\pm 6.4}$ |  $23$M
CORAL | $73.6_{\pm 0.4}$ | $62.8_{\pm 2.7}$ | $37.9_{\pm 0.7}$ |  $23$M
Fish | $64.4_{\pm 2.0}$ | $53.2_{\pm 4.5}$ | $42.0_{\pm 0.8}$ |  $23$M
GroupDRO | $73.6_{\pm 2.1}$ | $66.0_{\pm 3.8}$ | $36.4_{\pm 0.9}$ |  $23$M
JTT | $74.4_{\pm 0.6}$ | $64.6_{\pm 2.3}$ | $33.8_{\pm 0.0}$ |  $23$M
DM-ADA | $74.0_{\pm 0.8}$ | $65.7_{\pm 1.4}$ | $35.8_{\pm 1.0}$ |  $23$M
LISA | $70.0_{\pm 0.7}$ | $59.8_{\pm 2.3}$ | $38.0_{\pm 1.3}$ |  $23$M
DISC | $75.5_{\pm 1.1}$ | $73.5_{\pm 1.4}$ | $55.1_{\pm 2.3}$ |  $23$M
CPT | $\mathbf{79.8}_{\pm 2.6}$ | $\mathbf{77.0}_{\pm{0.8}}$ | $\mathbf{64.9}_{\pm 1.2}$ |  $\mathbf{47}$k

MetaShift and ISIC Results on those datasets are depicted in Table 3, along
with the number of parameters required for each baseline. We mostly pick those
that utilize group information during training for better benchmarking the
efficiency of our proposed framework. In summary, the performance of CPT
exceeds other methods by large margins, especially $\approx 10$ AUROC score on
ISIC. On MetaShift, it is also the best method in terms of average and worst-
group accuracy, improving the performance by approximately $4\%$.

###  5.5 Ablation studies

In this section, we provide different ablation studies to see how each
component in our proposed method individually contributes to the overall
improvement of CPT.

Optimization and Backbones Table 4 (rows 5 and 7) eliminates the entropy
objective and uses the average gradient $1/K\sum\nabla\ell_{i}(\theta)$ to
update the model. While this strategy can improve mean accuracy among groups,
compared to ERM, the performance has a significant drop compared to our
optimization procedure. This conclusion suggests that conducting simple
gradient averaging when there are still large differences among group losses
could cause severe bias in training. On the other hand, there is no
significant difference between the performance of fully fine-tuning ResNet50
(row 6) and prompt tuning on ViT-B/16 (row 8), which indicates that our method
does not benefit much from more powerful backbones. Instead, we remind readers
that prompting in CPT is introduced to scale up our proposed algorithm. Thus,
even applying our proposed balancing method alone can boost the performance of
ResNet to exceed GroupDRO by large margins on WaterBirds $(5.2\%)$ and CelebA
$(5.1\%)$.

Table 4: Ablation results for different architectures and optimization
procedures.

Dataset | Method | Average | Worst | # Params
---|---|---|---|---
MetaShift | ERM | $72.9_{\pm 1.4}$ | $62.1_{\pm 4.8}$ |  $23$M
GroupDRO | $73.6_{\pm 2.1}$ | $66.0_{\pm 3.8}$ |  $23$M
DISC | $75.5_{\pm 1.1}$ | $73.5_{\pm 1.4}$ |  $23$M
CPT${}_{\text{w/o entropy}}$ (ResNet50) | ${76.1}_{\pm 0.7}$ | ${62.8}_{\pm 1.8}$ |  $23$M
CPT (ResNet50) | ${78.2}_{\pm 2.5}$ | ${76.9}_{\pm 2.0}$ |  $23$M
CPT${}_{\text{w/o entropy}}$ | ${77.3}_{\pm 0.8}$ | ${74.7}_{\pm 2.5}$ |  $\mathbf{47}$k
| CPT | $\mathbf{79.8}_{\pm 2.6}$ | $\mathbf{77.0}_{\pm{0.8}}$ |  $\mathbf{47}$k
Waterbirds | ERM | $97.1_{\pm 0.1}$ | $70.0_{\pm 2.3}$ |  $23$M
GroupDRO | $93.2_{\pm 0.5}$ | $86.7_{\pm 0.6}$ |  $23$M
CPT (ResNet50) | ${90.9}_{\pm 0.2}$ | ${91.9}_{\pm 0.3}$ |  $23$M
CPT | ${{96.3}}_{\pm 0.2}$ | ${\mathbf{93.5}}_{\pm 0.4}$ |  $\mathbf{94}$k
CelebA | ERM | $94.8_{\pm 0.2}$ | $45.0_{\pm 1.5}$ |  $23$M
GroupDRO | ${92.9}_{\pm 0.3}$ | $86.3_{\pm 1.1}$ |  $23$M
CPT (ResNet50) | ${92.7}_{\pm 0.3}$ | ${91.4}_{\pm 0.8}$ |  $23$M
CPT | ${93.2}_{\pm 0.3}$ | $\mathbf{92.0}_{\pm 0.3}$ |  $\mathbf{94}$k

Similarly, Table 5 presents the scores of ERM, GroupDRO, and CPT on a wide
range of CLIP backbones, including CLIP with ResNet101, ViT-B/16, ViT-B/32,
ViT-L/14. Here we simply use the balanced version of CPT instead of
hyperparameter tuning on $c$. Therefore, note that we do not try to beat
GroupDRO on all metrics, but focus on investigating how our optimization
procedure helps close the gap among groups. In summary, balancing the learning
among groups results in large improvements in worst-group and mean accuracies
for all backbones.

Table 5: Performance on other CLIP variants, where CPT has the lowest gap
between worst and average accuracy.

Model | Last layers ResNet101 | Prompting ResNet101 | Last layer ViT-B/16 | Prompting ViT-B/16
---|---|---|---|---
# Params | 13740544 | 8192 | 393216 | 8192
Accuracy | Average | Worst | Mean | Average | Worst | Mean | Average | Worst | Mean | Average | Worst | Mean
Pre-trained | $91.7$ | $46.8$ | $71.1$ | $91.7$ | $46.8$ | $71.1$ | $90.6$ | $50.1$ | $75.5$ | $90.6$ | $50.1$ | $75.5$
ERM | $89.3$ | $69.4$ | $80.2$ | $88.6$ | $67.2$ | $80.0$ | $95.7$ | $77.4$ | $87.8$ | $96.1$ | $82.2$ | $89.7$
GroupDRO | $88.5$ | $74.9$ | $82.1$ | $86.3$ | $74.0$ | $80.7$ | $91.4$ | $88.5$ | $90.2$ | $93.3$ | $85.5$ | $90.4$
CPT | $87.7$ | $\mathbf{75.2}$ | $\mathbf{83.9}$ | $83.1$ | $\mathbf{78.9}$ | $\mathbf{81.2}$ | $91.4$ | $\mathbf{89.4}$ | $\mathbf{90.5}$ | $92.2$ | $\mathbf{89.2}$ | $\mathbf{91.5}$
Model | Last layer ViT-B/32 | Prompting ViT-B/32 | Last layer ViT-L/14 | Prompting ViT-L/14
# Params | 393216 | 8192 | 786432 | 12288
Accuracy | Average | Worst | Mean | Average | Worst | Mean | Average | Worst | Mean | Average | Worst | Mean
Pretrained | $90.1$ | $49.0$ | $72.2$ | $90.1$ | $49.0$ | $72.2$ | $89.4$ | $36.1$ | $83.9$ | $89.4$ | $36.1$ | $83.9$
ERM | $94.7$ | $73.5$ | $85.2$ | $94.0$ | $76.0$ | $85.7$ | $96.6$ | $85.8$ | $91.8$ | $96.8$ | $83.1$ | $91.6$
GroupDRO | $90.7$ | $81.6$ | $86.8$ | $88.3$ | $82.3$ | $85.9$ | $95.8$ | $88.9$ | $92.8$ | $95.5$ | $86.3$ | $91.5$
CPT | $87.4$ | $\mathbf{86.3}$ | $\mathbf{87.4}$ | $86.8$ | $\mathbf{85.2}$ | $\mathbf{87.5}$ | $94.8$ | $\mathbf{90.7}$ | $\mathbf{92.9}$ | $94.9$ | $\mathbf{90.0}$ | $\mathbf{92.6}$

Figure 4: Performance of ResNet50 at: the last checkpoint used for evaluation
(highest worst group accuracy on validation set), denoted by $\bigtriangleup$,
and the checkpoint where the performance on the minority group is highest,
denoted by $\square$. Results are obtained on three random seeds.

Gap among group performance Apart from Figure 1, we visualize the performance
of each comparative method at its early stopping epoch and the point when it
actually obtains the highest score on the minority group in Figure 4. Those
points collapse in the case of ERM since it does not have any balancing
mechanism. Unsurprisingly, the gap between minority and majority performance
of ERM is largest ($>10\%$) while this figure decreases for GroupDRO and CPT.
It is true that GroupDRO can obtain a relatively high score on the minority
group early in training, even on par with CPT. However, its majority group
performance at this point has not converged yet ($\approx 90.5\%$), and when
converged, its performance on the minority group declines considerably
($91\%\rightarrow 88\%$). By contrast, the is only a small difference in terms
of minority performance for CPT at those points, allowing to have equally good
performance across groups through training.

Table 6: Performance of the model after training for one epoch. Here, we
simply adjust the optimization in favor of two minority groups to observe
their improvement.

Group $g$ (size) |  0 $(44\%)$ |  1 $(41\%)$ |  2 $(14\%)$ |  3 $(1\%)$ | Worst | Avg | Mean
---|---|---|---|---|---|---|---
GroupDRO | $92.0$ | $92.5$ | $91.5$ | $86.1$ | $86.1$ | $92.08$ | $89.4$
CPT $c=(1,1,1,1)$ | $\mathbf{92.9}$ | $\mathbf{93.4}$ | $92.8$ | $91.7$ | $\mathbf{91.7}$ | $\mathbf{93.1}$ | $92.7$
CPT $c=(1,1,1,2)$ | $92.0$ | $91.1$ | $93.1$ | $\mathbf{95.6}$ | $91.1$ | $91.9$ | $93.0$
CPT $c=(1,1,2,1)$ | $91.2$ | $91.8$ | $\mathbf{95.2}$ | $95.0$ | $91.2$ | $92.0$ | $\mathbf{93.3}$

Effect of controlling vector Table 6 presents the distribution of the CelebA
dataset and the performance of GroupDRO and our proposed method with different
values of the controlling vectors. We can see that setting $c$ equal to the
default value $(1,1,1,1)$ can perfectly balance the accuracy score across
groups while GroupDRO encounters difficulty in fitting the minority one.
Furthermore, by varying the value of $c$, we could observe different behavior
of the model in each group. For example, the accuracy scores on two smaller
groups ($2\&3$) increase significantly, as a result of the loss magnitudes
decrease when we put more weight on penalizing those two. With a tiny
sacrifice on the performance of two majority groups, the model could gain
considerable improvement in minority groups and the overall performance.

Figure 5: GradCAM of ResNet50 on in-domain samples. Figure 6: GradCAM of
ResNet50 on OOD samples.

Saliency maps  We plot the saliency maps [70] produced when predicting the
target attribute of in-domain images in Figure 5 and out-of-domain images in
Figure 6. Pixels highlighted by warmer colors indicate higher contributions to
the model predictions. Our proposed method can learn causal features rather
than focusing on spurious background features when making predictions. Hence,
balancing the learning among groups not only de-bias the classifier in-domain
but also robustifies it in tackling distributional shifts.

##  6 Conclusion

In this paper, we propose Controllable Prompt Tuning, a novel training method
that not only improves the performance across groups but also allows us to
control the tradeoff between them. Our proposed method can effectively de-bias
the training of different types of backbones and achieves state-of-the-art
performance on benchmark datasets.

## References

  * Agarwal et al. [2021] S. Agarwal, G. Krueger, J. Clark, A. Radford, J. W. Kim, and M. Brundage.  Evaluating clip: towards characterization of broader capabilities and downstream implications.  _arXiv preprint arXiv:2108.02818_ , 2021.
  * Aghajanyan et al. [2020] A. Aghajanyan, L. Zettlemoyer, and S. Gupta.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  _arXiv preprint arXiv:2012.13255_ , 2020.
  * Ahmed et al. [2020] F. Ahmed, Y. Bengio, H. Van Seijen, and A. Courville.  Systematic generalisation with group invariant predictions.  In _International Conference on Learning Representations_ , 2020.
  * Ahuja et al. [2021] K. Ahuja, E. Caballero, D. Zhang, J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish.  Invariance principle meets information bottleneck for out-of-distribution generalization.  _Advances in Neural Information Processing Systems_ , 34:3438–3450, 2021.
  * Andersen et al. [2020] M. Andersen, J. Dahl, and L. Vandenberghe.  Cvxopt: Convex optimization.  _Astrophysics Source Code Library_ , pages ascl–2008, 2020.
  * Arjovsky et al. [2019] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Ben-David et al. [2010] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan.  A theory of learning from different domains.  _Machine learning_ , 79:151–175, 2010.
  * Berthelot et al. [2021] D. Berthelot, R. Roelofs, K. Sohn, N. Carlini, and A. Kurakin.  Adamatch: A unified approach to semi-supervised learning and domain adaptation.  In _International Conference on Learning Representations_ , 2021.
  * Bissoto et al. [2020] A. Bissoto, E. Valle, and S. Avila.  Debiasing skin lesion datasets and models? not so fast.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 740–741, 2020.
  * Bradley [1997] A. P. Bradley.  The use of the area under the roc curve in the evaluation of machine learning algorithms.  _Pattern recognition_ , 30(7):1145–1159, 1997.
  * Cai et al. [2021] T. Cai, R. Gao, J. Lee, and Q. Lei.  A theory of label propagation for subpopulation shift.  In _International Conference on Machine Learning_ , pages 1170–1182. PMLR, 2021.
  * Chen et al. [2021] C.-F. R. Chen, Q. Fan, and R. Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 357–366, 2021.
  * Chen et al. [2022a] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  _Advances in Neural Information Processing Systems_ , 35:16664–16678, 2022a.
  * Chen et al. [2022b] Y. Chen, E. Rosenfeld, M. Sellke, T. Ma, and A. Risteski.  Iterative feature matching: Toward provable domain generalization with logarithmic environments.  _Advances in Neural Information Processing Systems_ , 35:1725–1736, 2022b.
  * Codella et al. [2019] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  _arXiv preprint arXiv:1902.03368_ , 2019.
  * Cortes et al. [2010] C. Cortes, Y. Mansour, and M. Mohri.  Learning bounds for importance weighting.  _Advances in neural information processing systems_ , 23, 2010.
  * Cortes et al. [2015] C. Cortes, M. Mohri, and A. Muñoz Medina.  Adaptation algorithm and theory based on generalized discrepancy.  In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 169–178, 2015.
  * Dehdashtian and Boddeti [2024] L. Dehdashtian, Sepehr Wang and V. Boddeti.  Fairerclip: Debiasing zero-shot predictions of clip in rkhss.  In _International Conference on Learning Representations_ , 2024.
  * Deng et al. [2023] Y. Deng, Y. Yang, B. Mirzasoleiman, and Q. Gu.  Robust learning with progressive data expansion against spurious correlation.  _Advances in neural information processing systems_ , 2023.
  * Désidéri [2012] J.-A. Désidéri.  Multiple-gradient descent algorithm (mgda) for multiobjective optimization.  _Comptes Rendus Mathematique_ , 350(5-6):313–318, 2012.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Diamond and Boyd [2016] S. Diamond and S. Boyd.  Cvxpy: A python-embedded modeling language for convex optimization.  _The Journal of Machine Learning Research_ , 17(1):2909–2913, 2016.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Du et al. [2022] Y. Du, F. Wei, Z. Zhang, M. Shi, Y. Gao, and G. Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14084–14093, 2022.
  * Edalati et al. [2022] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh.  Krona: Parameter efficient tuning with kronecker adapter.  _arXiv preprint arXiv:2212.10650_ , 2022.
  * Fan et al. [2019] A. Fan, E. Grave, and A. Joulin.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_ , 2019.
  * Ganin et al. [2016] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gheini et al. [2021] M. Gheini, X. Ren, and J. May.  Cross-attention is all you need: Adapting pretrained transformers for machine translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1754–1765, 2021.
  * Ghosal and Li [2023] S. S. Ghosal and Y. Li.  Are vision transformers robust to spurious correlations?  _International Journal of Computer Vision_ , pages 1–21, 2023.
  * Gruver et al. [2023] N. Gruver, M. A. Finzi, S. Qiu, and A. G. Wilson.  Large language models are zero-shot time series forecasters.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Gulrajani and Lopez-Paz [2020] I. Gulrajani and D. Lopez-Paz.  In search of lost domain generalization.  In _International Conference on Learning Representations_ , 2020.
  * Guo et al. [2021] D. Guo, A. Rush, and Y. Kim.  Parameter-efficient transfer learning with diff pruning.  In _Annual Meeting of the Association for Computational Linguistics_ , 2021.
  * He et al. [2023] H. He, J. Cai, J. Zhang, D. Tao, and B. Zhuang.  Sensitivity-aware visual parameter-efficient fine-tuning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11825–11835, 2023.
  * He et al. [2021] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_ , 2021.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Heckman [1979] J. J. Heckman.  Sample selection bias as a specification error.  _Econometrica: Journal of the econometric society_ , pages 153–161, 1979.
  * Houlsby et al. [2019] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR, 2019.
  * Hu et al. [2021] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Hu et al. [2018] W. Hu, G. Niu, I. Sato, and M. Sugiyama.  Does distributionally robust supervised learning give robust classifiers?  In _International Conference on Machine Learning_ , pages 2029–2037. PMLR, 2018.
  * Huang et al. [2023] Q. Huang, X. Dong, D. Chen, W. Zhang, F. Wang, G. Hua, and N. Yu.  Diversity-aware meta visual prompting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10878–10887, 2023.
  * Kim et al. [2023] N. Kim, J. Kang, S. Ahn, J. Ok, and S. Kwak.  Removing multiple biases through the lens of multi-task learning.  _International Conference on Machine Learning_ , 2023.
  * Kirichenko et al. [2022] P. Kirichenko, P. Izmailov, and A. G. Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Kolesnikov et al. [2020] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby.  Big transfer (bit): General visual representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 491–507. Springer, 2020.
  * Krueger et al. [2021] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pages 5815–5826. PMLR, 2021.
  * Kumar et al. [2022] A. Kumar, T. Ma, P. Liang, and A. Raghunathan.  Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift.  In _Uncertainty in Artificial Intelligence_ , pages 1041–1051. PMLR, 2022.
  * Lei et al. [2021] Q. Lei, W. Hu, and J. Lee.  Near-optimal linear regression under distribution shift.  In _International Conference on Machine Learning_ , pages 6164–6174. PMLR, 2021.
  * Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, 2021.
  * Li et al. [2023] B. Li, Y. Hu, X. Nie, C. Han, X. Jiang, T. Guo, and L. Liu.  Dropkey for vision transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22700–22709, 2023.
  * Li et al. [2019] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  _Advances in neural information processing systems_ , 32, 2019.
  * Li and Liang [2021] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics)_ , pages 4582–4597, 2021.
  * Lialin et al. [2023] V. Lialin, V. Deshpande, and A. Rumshisky.  Scaling down to scale up: A guide to parameter-efficient fine-tuning.  _arXiv preprint arXiv:2303.15647_ , 2023.
  * Liang and Zou [2021] W. Liang and J. Zou.  Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts.  In _International Conference on Learning Representations_ , 2021.
  * Liu et al. [2021] E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn.  Just train twice: Improving group robustness without training group information.  In _International Conference on Machine Learning_ , pages 6781–6792. PMLR, 2021.
  * Liu et al. [2020] L. Liu, Y. Li, Z. Kuang, J.-H. Xue, Y. Chen, W. Yang, Q. Liao, and W. Zhang.  Towards impartial multi-task learning.  In _International Conference on Learning Representations_ , 2020.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. [2015] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Liu et al. [2022] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11976–11986, 2022.
  * Long et al. [2018] M. Long, Z. Cao, J. Wang, and M. I. Jordan.  Conditional adversarial domain adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Miyato et al. [2018] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1979–1993, 2018.
  * Nam et al. [2020] J. Nam, H. Cha, S. Ahn, J. Lee, and J. Shin.  Learning from failure: De-biasing classifier from biased classifier.  _Advances in Neural Information Processing Systems_ , 33:20673–20684, 2020.
  * Oh et al. [2022] D. Oh, D. Lee, J. Byun, and B. Shin.  Improving group robustness under noisy labels using predictive uncertainty.  _arXiv preprint arXiv:2212.07026_ , 2022.
  * Panahi et al. [2021] A. Panahi, S. Saeedi, and T. Arodz.  Shapeshifter: a parameter-efficient transformer using factorized reshaped matrices.  _Advances in Neural Information Processing Systems_ , 34:1337–1350, 2021.
  * Phan et al. [2023] H. Phan, T. Le, T. Phung, A. T. Bui, N. Ho, and D. Phung.  Global-local regularization via distributional robustness.  In _International Conference on Artificial Intelligence and Statistics_ , pages 7644–7664. PMLR, 2023.
  * Piratla et al. [2021] V. Piratla, P. Netrapalli, and S. Sarawagi.  Focus on the common good: Group distributional robustness follows.  In _International Conference on Learning Representations_ , 2021.
  * Qiu et al. [2023] S. Qiu, A. Potapczynski, P. Izmailov, and A. G. Wilson.  Simple and fast group robustness by automatic feature reweighting.  In _International Conference on Machine Learning_. PMLR, 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Radford et al. [2023] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR, 2023.
  * Sagawa et al. [2019] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang.  Distributionally robust neural networks.  In _International Conference on Learning Representations_ , 2019.
  * Sagawa et al. [2020] S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang.  An investigation of why overparameterization exacerbates spurious correlations.  In _International Conference on Machine Learning_ , pages 8346–8356. PMLR, 2020.
  * Selvaraju et al. [2017] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Shi et al. [2021] Y. Shi, J. Seely, P. Torr, N. Siddharth, A. Hannun, N. Usunier, and G. Synnaeve.  Gradient matching for domain generalization.  In _International Conference on Learning Representations_ , 2021.
  * Shimodaira [2000] H. Shimodaira.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of statistical planning and inference_ , 90(2):227–244, 2000.
  * Sun et al. [2017] B. Sun, J. Feng, and K. Saenko.  Correlation alignment for unsupervised domain adaptation.  _Domain adaptation in computer vision applications_ , pages 153–171, 2017.
  * Sung et al. [2022] Y.-L. Sung, J. Cho, and M. Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5227–5237, 2022.
  * Taghanaki et al. [2021] S. A. Taghanaki, K. Choi, A. H. Khasahmadi, and A. Goyal.  Robust representation learning via perceptual similarity metrics.  In _International Conference on Machine Learning_ , pages 10043–10053. PMLR, 2021.
  * Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. [2023] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_ , 2023.
  * Wang et al. [2021] J. Wang, Y. Liu, and X. Wang.  Are gender-neutral queries really gender-neutral? mitigating gender bias in image search.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1995–2008, 2021.
  * Wortsman et al. [2022] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pages 23965–23998. PMLR, 2022.
  * Wu et al. [2023] S. Wu, M. Yuksekgonul, L. Zhang, and J. Zou.  Discover and cure: Concept-aware mitigation of spurious correlation.  _International Conference on Machine Learning_ , 2023.
  * Xu et al. [2020] M. Xu, J. Zhang, B. Ni, T. Li, C. Wang, Q. Tian, and W. Zhang.  Adversarial domain adaptation with domain mixup.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 6502–6509, 2020.
  * Yang et al. [2023a] S. Yang, Y. Dong, R. Ward, I. S. Dhillon, S. Sanghavi, and Q. Lei.  Sample efficiency of data augmentation consistency regularization.  In _International Conference on Artificial Intelligence and Statistics_ , pages 3825–3853. PMLR, 2023a.
  * Yang et al. [2023b] Y. Yang, B. Nushi, H. Palangi, and B. Mirzasoleiman.  Mitigating spurious correlations in multi-modal models during fine-tuning.  _International Conference on Machine Learning_ , 2023b.
  * Yao et al. [2022] H. Yao, Y. Wang, S. Li, L. Zhang, W. Liang, J. Zou, and C. Finn.  Improving out-of-distribution robustness via selective augmentation.  In _International Conference on Machine Learning_ , pages 25407–25437. PMLR, 2022.
  * Zaken et al. [2022] E. B. Zaken, Y. Goldberg, and S. Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, 2022.
  * Zhai et al. [2022] R. Zhai, C. Dan, J. Z. Kolter, and P. K. Ravikumar.  Understanding why generalized reweighting does not improve over erm.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhang and Ré [2022] M. Zhang and C. Ré.  Contrastive adapters for foundation model group robustness.  _Advances in Neural Information Processing Systems_ , 35:21682–21697, 2022.
  * Zhang et al. [2022] M. Zhang, N. S. Sohoni, H. R. Zhang, C. Finn, and C. Re.  Correct-n-contrast: a contrastive approach for improving robustness to spurious correlations.  In _International Conference on Machine Learning_ , pages 26484–26516. PMLR, 2022.
  * Zhou et al. [2017] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba.  Places: A 10 million image database for scene recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(6):1452–1464, 2017.
  * Zhou et al. [2022] K. Zhou, J. Yang, C. C. Loy, and Z. Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022.
  * Zhu et al. [2023a] B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang.  Prompt-aligned gradient for prompt tuning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15659–15669, 2023a.
  * Zhu et al. [2023b] J. Zhu, S. Lai, X. Chen, D. Wang, and H. Lu.  Visual prompt multi-modal tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9516–9526, 2023b.
  * Zitzler and Thiele [1999] E. Zitzler and L. Thiele.  Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach.  _IEEE transactions on Evolutionary Computation_ , 3(4):257–271, 1999.

Supplement to “Controllable Prompt Tuning For

Balancing Group Distributional Robustness"

Due to space constraints, some details were omitted from the main paper. We
therefore include proof for Theorem 4.1 in the main paper (Appendix A) and
more detailed experimental results (Appendix B) in this supplementary
material.

##  Appendix A Proof of Theorem 4.1

Theorem 4.1.  Assume that the loss function $\ell$ is differentiable up to the
first order with respect to $\theta$, then following d_ent :=∑_i=1^K
∇ℓ_i(θ)[p_i log(p_i)-p_i ∑_j=1^K log(p_j) p_j] where
$p_{i}=\frac{e^{\ell_{i}(\theta)}}{\sum_{j=1}^{K}e^{\ell_{j}(\theta)}}$,
maximizes the objective $\mathcal{L}_{\text{ent }}(\theta)$:

$\displaystyle\mathcal{L}_{ent}(\theta)$
$\displaystyle=H\big{(}\text{softmax}\big{(}\vec{\mathcal{L}}(\theta))\big{)}=H\Big{(}\text{softmax}\big{(}\big{\\{}\mathbb{E}_{P_{g^{k}}}\left[\ell\left(f_{\theta}(x),y\right)\right]\big{\\}}_{k=1}^{K}\big{)}\Big{)}$

Proof: For brevity of notations, we denote
$\mathbb{E}_{P_{g^{i}}}\left[\ell\left(f_{\theta}(x),y\right)\right]$ as
$\ell_{i}(\theta)$ and
$p_{i}=\frac{e^{\ell_{i}(\theta)}}{\sum_{j=1}^{K}e^{\ell_{j}(\theta)}}$. The
derivative of $\mathcal{L}_{ent}$ with respect to $\theta$ is computed as:

$\displaystyle\nabla\mathcal{L}_{ent}(\theta)$
$\displaystyle=-\sum_{i=1}^{K}\big{\\{}\nabla
p_{i}[\log(p_{i})+1]\big{\\}}=-\sum_{i=1}^{K}\Bigg{\\{}[\log(p_{i})+1]\nabla\frac{e^{\ell_{i}(\theta)}}{\sum_{j=1}^{K}e^{\ell_{j}(\theta)}}\Bigg{\\}}$
$\displaystyle=-\sum_{i=1}^{K}\Bigg{\\{}[\log(p_{i})+1]\frac{\sum_{j=1}^{K}e^{\ell_{j}(\theta)}e^{\ell_{i}(\theta)}\nabla\ell_{i}(\theta)-e^{\ell_{i}(\theta)}\sum_{j=1}^{K}e^{\ell_{j}(\theta)}\nabla{\ell_{j}(\theta)}}{(\sum_{j=1}^{K}e^{\ell_{j}(\theta)})^{2}}\Bigg{\\}}$
$\displaystyle=-\frac{1}{(\sum_{j=1}^{K}e^{\ell_{j}(\theta)})^{2}}\sum_{i=1}^{K}\Big{\\{}[\log(p_{i})+1]e^{\ell_{i}(\theta)}\sum_{j=1}^{K}e^{\ell_{j}(\theta)}(\nabla\ell_{i}(\theta)-\nabla\ell_{j}(\theta))\Big{\\}}$
$\displaystyle=-\frac{1}{(\sum_{j=1}^{K}e^{\ell_{j}(\theta)})^{2}}\sum_{i=1}^{K}\Big{[}\log(p_{i})e^{\ell_{i}(\theta)}\sum_{j=1}^{K}e^{\ell_{j}(\theta)}(\nabla\ell_{i}(\theta)-\nabla\ell_{j}(\theta))\Big{]}$
$\displaystyle=-\frac{1}{(\sum_{j=1}^{K}e^{\ell_{j}(\theta)})^{2}}\sum_{i=1}^{K}\nabla\ell_{i}(\theta)\Big{[}\log(p_{i})e^{\ell_{i}(\theta)}\sum_{j=1}^{K}e^{\ell_{j}(\theta)}-e^{\ell_{i}(\theta)}\sum_{j=1}^{K}\log(p_{j})e^{\ell_{j}(\theta)}\Big{]}$
$\displaystyle=\sum_{i=1}^{K}\nabla\ell_{i}(\theta)[p_{i}\sum_{j=1}^{K}\log(p_{j})p_{j}-p_{i}\log(p_{i})]$
(8)

We conclude the proof. It is noteworthy that when the loss function $\ell$ is
positive, one can also simplify the computation by minimizing the following
balancing function:

$\displaystyle\mathcal{L}^{\prime}_{ent}(\theta)$
$\displaystyle:=H\Big{(}\Big{\\{}\frac{\mathbb{E}_{P_{g^{i}}}\left[\ell\left(f_{\theta}(x),y\right)\right]}{\sum_{j=1}^{K}\mathbb{E}_{P_{g^{j}}}\left[\ell\left(f_{\theta}(x),y\right)\right]}\Big{\\}}_{i=1}^{K}\Big{)}$

Eliminating the exponential term and denoting
$q_{i}=\frac{{\ell_{i}(\theta)}}{\sum_{j=1}^{K}{\ell_{j}(\theta)}}$, the
gradient direction of the entropy term is given by:

$\displaystyle\nabla\mathcal{L}^{\prime}_{ent}(\theta)$
$\displaystyle=\sum_{i=1}^{K}\big{\\{}\nabla q_{i}[\log(q_{i})+1]\big{\\}}$
$\displaystyle=\sum_{i=1}^{K}\big{\\{}\frac{{[\sum_{j=1}^{K}{\ell_{j}(\theta)}]\nabla\ell_{i}(\theta)-\ell_{i}(\theta)[\sum_{j=1}^{K}{\nabla\ell_{j}(\theta)}]}}{[\sum_{j=1}^{K}{\ell_{j}(\theta)}]^{2}}[\log(q_{i})+1]\big{\\}}$
$\displaystyle\propto\sum_{i=1}^{K}\nabla\ell_{i}(\theta)\Big{\\{}\log(q_{i})\big{[}\sum_{j=1}^{K}\ell_{j}(\theta)\big{]}-\sum_{j=1}^{K}\log(q_{j})\ell_{j}(\theta)\Big{\\}}$

##  Appendix B Experiment details and additional empirical results

In this section, we first describe the datasets and models used in Appendix
B.1 along with the detailed training configuration for our method, then
provide additional results for the experiment in Appendix B.2. Example images
and data statistics are presented in Table 8, 9, 10 and 11.

###  B.1 Implementation details

Hyper-parameter We mainly examine the proposed method on ResNet50 [35], ViT
B/16 [23] and different variants of OpenAI’s
CLIP111https://github.com/openai/CLIP model. The hyperparameters selected for
each experiment are given in Table 7. Unless stated otherwise, results of
baselines are taken from original papers and [19, 82, 64, 90, 41], which
provide standard evaluation protocols for different backbones and datasets.
Performance of ERM and GroupDRO [68] at different amount of training data and
their training curves are obtained from the released codebase of
GroupDRO222https://github.com/kohpangwei/group_DRO using their provided
commands.

Dataset | Architecture | Learning Rate | Weight Decay | Batch Size | # Epochs | Prompt length
---|---|---|---|---|---|---
Waterbirds | ViT B/16 | 0.1 | 0.001 | 64 | 100 | 10
Waterbirds | Last layers CLIP-RN50 | 0.003 | 0.01 | 32 | 100 | -
Waterbirds | Prompt CLIP-RN50 | 0.03 | 0.01 | 32 | 100 | 16
Waterbirds | Last layer CLIP-ViT-L/14 | 0.01 | 0.01 | 32 | 100 | -
Waterbirds | Prompt CLIP-ViT-L/14 | 0.001 | 0.01 | 32 | 100 | 16
CelebA | ViT B/16 | 0.01 | 0.001 | 128 | 100 | 10
MetaShift | ResNet50 | 0.003 | 0.01 | 16 | 100 | -
MetaShift | ViT B/16 | 0.01 | 0.1 | 16 | 100 | 5
ISIC | ViT B/16 | 0.003 | 0.01 | 16 | 100 | 5

Table 7: Hyperparameter for different experiments throughout our paper. We
report the hyper-parameters selected for our proposed method after performing
grid-search.

Classifier design While we train a classification head from scratch for
ResNet50 or ViT, CLIP allows us to utilize its meaningful multimodality
embeddings as a zero-shot classifier. In particular, given an image embedding
$\mathbf{e}_{i}$ and a text class embedding $\mathbf{e}^{k}_{t}$ for some
class index $k\in\\{1,2,\ldots,\mathbf{N_{c}}\\}$, where $\mathbf{N}_{c}$ is
the number of categories. The probability of the given image belonging to
$k$-th class is computed as: p(y=k ∣e_i)=exp(<ei, ekt>/ τ)∑j=1Ncexp(< ei,
ejt>/ τ) where $<\cdot,\cdot>$ denotes the cosine similarity and $\tau$ is the
temperature parameter.

Dataset statistics We evaluate CPT on Waterbirds [68] (Table 8), CelebA [56]
(Table 9), MetaShift [52] (Table 10) and ISIC [15] (Table 11), following
previous work [68, 19, 82]. While the objective of those image classification
tasks is predicting the categories of input images, those target attributes
are often correlated well with spurious features (please refer to each table
for detailed descriptions of spurious features and group partitions for each
dataset).

Image |  |  |  |
---|---|---|---|---
Group | 0 | 1 | 2 | 3
Label | 0 (landbird) | 0 (landbird) | 1 (waterbird) | 1 (waterbird)
Spurious feature | 0 (land) | 1 (water) | 0 (land) | 1 (water)
Description | landbird on land | landbird on water | waterbird on land | waterbird on water
# Train data | $3,498(73\%)$ | $184(4\%)$ | $56(1\%)$ | $1,057(22\%)$
# Val data | $467$ | $466$ | $133$ | $133$
# Test data | $2,255$ | $2,255$ | $642$ | $642$

Table 8: Example images of Waterbirds [68].

Image |  |  |  |
---|---|---|---|---
Group | 0 | 1 | 2 | 3
Label | 0 (non-blond) | 0 (non-blond) | 1 (blond) | 1 (blond)
Spurious feature | 0 (woman) | 1 (man) | 0 (woman) | 1 (man)
Description | non-blond woman | non-blond man | blond woman | blond man
# Train data | $71,629(44\%)$ | $66,874(41\%)$ | $22,880(14\%)$ | $1,387(1\%)$
# Val data | $8,535$ | $8,276$ | $2,874$ | $182$
# Test data | $9,767$ | $7,535$ | $2,480$ | $180$

Table 9: Example images of CelebA [56].

Image |  |  |  |  |  |
---|---|---|---|---|---|---
Group | 0 | 1 | 2 | 3 | 4 | 5
Label | 0 (cat) | 0 (cat) |  $1$(dog) |  $1$ (dog) | 0 (cat) |  $1$ (dog)
Spurious feature | 0 (sofa) | 1 (bed) | 2 (bench) | 3 (bike) | 4 (shelf) | 4 (shelf)
# Train data | 231 | 380 | 145 | 367 | - | -
# Val data (OOD) | - | - | - | - | 34 | 47
# Test data | - | - | - | - | 201 | 259

Table 10: Example images of MetaShift [52].

Image |  |  |  | $\ldots$ |  |
---|---|---|---|---|---|---
Label | 0 (benign) | 0 (benign) | 1 (malignant) | $\ldots$ | 0 (benign) | 1 (malignant)
Spurious feature | hair | dark corner | gel bubbles | $\ldots$ | ink | ruler
# Train data: | 1,826
# Val data: | 154
# Test data: | 618

Table 11: Example images of ISIC [15].

###  B.2 Additional experiments

Prompt length Thanks to the design of the prompting component, the
computational burden during training is modest: only a small portion of the
model parameters are updated through back-propagation. Here, we vary the value
of prompt length $L$ to observe its effect on the model performance in Table
12. Results of ERM, GroupDRO, the second-best method in this dataset DFR [42]
and a recent baseline in this line of research PDE [19] are also included for
a general comparison. While increasing the prompt length can improve the
performance of the model, we find that just pretending one token per
transformer layer ($L=1$) can outperform the second-best method that requires
$\times 2200$ more trainable parameters via better training. Last, it is
noteworthy to highlight that $L=16$ can help CPT match the average accuracy
score of ERM in this scenario, which demonstrates the large modeling capacity
of our proposed method.

Table 12: Ablation results for different prompt length $N$ on ViT B/16 on
Waterbirds.

$N$ | $1$ | $2$ | $4$ | $8$ | $10$ | $16$ | ERM | GroupDRO | PDE | DFR
---|---|---|---|---|---|---|---|---|---|---
Worst | $92.9_{\pm 0.8}$ | $93.3_{\pm 0.9}$ | $93.5_{\pm 0.6}$ | $93.8_{\pm 0.5}$ | $93.5_{\pm 0.4}$ | $\mathbf{93.9}_{\pm 0.7}$ | $70.0_{\pm 2.3}$ | $86.7_{\pm 0.6}$ | $90.3_{\pm 0.3}$ | $92.9_{\pm 0.2}$
Average | $95.8_{\pm 0.9}$ | $95.9_{\pm 0.9}$ | $96.4_{\pm 0.1}$ | $96.5_{\pm 0.0}$ | $96.3_{\pm 0.1}$ | $\mathbf{97.1}_{\pm 0.3}$ | $\mathbf{97.1}_{\pm 0.1}$ | $93.2_{\pm 0.5}$ | $92.4_{\pm 0.8}$ | $94.2_{\pm 0.2}$
Mean | $94.9_{\pm 0.8}$ | $95.2_{\pm 0.0}$ | $95.6_{\pm 0.4}$ | $\mathbf{96.0}_{\pm 0.8}$ | $95.7_{\pm 0.2}$ | $95.7_{\pm 0.3}$ | N/A | N/A | N/A | N/A
# Params | 10754 | 19970 | 38402 | 75266 | 93698 | 148994 | 23512130 | 23512130 | 23512130 | 23512130

Performance at different number of training samples In order to
comprehensively assess the performance of our proposed model, we conduct a
series of experiments to analyze its behavior across varying sizes of training
data. Those experiments were designed to investigate the impact of data size
on the model performance of comparative methods. We keep group ratios
preserved while subsampling a part of the training data. The x-axis of Figure
7 denotes the percentages of the Waterbirds and CelebA training set used in
each sub-experiment. It can be seen that CPT is better than GroupDRO or ERM in
lower data regimes. Remarkably, by using $20\%$ and $10\%$ data, our proposed
method can still outperform other baselines when using the full training set.

Figure 7: Data Efficiency on Waterbirds (left) and CelebA (right). Error bars
correspond to standard deviations over three trials.

Similarly, we run the above experiment but replace the main backbone with
CLIP-RN50 and plot the performance of different baselines. Note that we freeze
both image and text encoders and employ prompt tuning on their text encoder
only. The text prompt used for training CLIP models is: “a type of bird, a
photo of a waterbird/landbird” While the pretrained CLIP model obtains
$90.9\%$ average accuracy on Waterbirds (Table 2 in the main paper), it
severely suffers from unintended bias when attaining only $44.9\%$ worst-group
accuracy. This motivates the need for debiasing methods to remove such bias
and mitigate the reliance on spurious features when making predictions of
foundation models. Figure 8 shows that while using more data can help enhance
the pre-trained CLIP model better for all methods, CPT obtains the best worst
group accuracy for every amount of data and surpasses the current state-of-
the-art method [85].

Figure 8: Data Efficiency of CLIP-RN50. CPT with $75\%$ data can outperform
the current state-of-the-art method for debiasing CLIP [85] which uses $\times
1800$ trainable parameters.

Running time Figure 9 summarizes the total runtime and the time for solving
the linear programming problem on each epoch. The introduction of the
optimization (7) in the main paper leads to a small overhead in terms of
training time. However, this small sacrifice significantly boosts the
performance of different backbones, especially for CLIP and prompt-based
models. For example, prompt-tuning on CLIP variants takes only $0.3$ second
per epoch since it does not have to update the entire network or even the
classification head.

Figure 9:  Total running time and the time for solving the linear programming
(LP) problem on epoch with Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz and
NVIDIA A100-SXM4-80GB GPU. Results are averaged over 5 epochs. Table 13:
Comparison of different backbones on Waterbirds

Model | Average Acc. | Worst-Group Acc. | # Params
---|---|---|---
BiT-M-R50x1 | $92.05_{\pm 0.05}$ | $75.10_{\pm 0.62}$ | 23.5M
BiT-M-R50x3 | $94.90_{\pm 0.05}$ | $80.51_{\pm 1.02}$ | 211M
BiT-M-R101x1 | $94.05_{\pm 0.07}$ | $77.50_{\pm 0.50}$ | 42.5M
ViT-Ti/16 | $89.50_{\pm 0.05}$ | $71.65_{\pm 0.16}$ | 5.6M
ViT-S/16 | $96.30_{\pm 0.51}$ | $85.45_{\pm 1.16}$ | 21.8M
ViT-B/16 | ${96.75}_{\pm{0.05}}$ | ${89.30}_{\pm{1.95}}$ | 86.1M
CPT | ${{96.33}}_{\pm 0.15}$ | ${\mathbf{93.51}}_{\pm 0.36}$ |  $\mathbf{94}$k

Debiasing ViT For the experiment of debiasing Vision Transformer, we compare
CPT against different ViT and Big Transfer (BiT) models introduced in [43,
29]. More detailed results are given in Table 13, from which we can see that
CPT not only obtains the best worst-group accuracy score but also performs on
par with the second-best method in terms of average accuracy while using much
fewer parameters. Note that other models scale up the resolution to $384\times
384$ while CPT keeps using the $224\times 224$ resolution.

Saliency maps We here provide more GradCAM visual explanations for ERM,
GroupDRO and CPT on Waterbirds in Figure 10 and CelebA Figure 11. From those
images, CPT shows its ability to identify causal features for making
predictions, compared to ERM and GroupDRO. Hence, our balancing mechanism
helps prevent the model from learning spurious features that occur frequently
during training.

Figure 10: Waterbirds GradCAM. Figure 11: CelebA GradCAM.