# Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing
Label Bias in Foundation Models

Beier Zhu1 Kaihua Tang1 Qianru Sun2 Hanwang Zhang1
1Nanyang Technological University 2Singapore Management University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Foundation models like CLIP allow zero-shot transfer on various tasks without
additional training data. Yet, the zero-shot performance is less competitive
than a fully supervised one. Thus, to enhance the performance, fine-tuning and
ensembling are also commonly adopted to better fit the downstream tasks.
However, we argue that such prior work has overlooked the inherent biases in
foundation models. Due to the highly imbalanced Web-scale training set, these
foundation models are inevitably skewed toward frequent semantics, and thus
the subsequent fine-tuning or ensembling is still biased. In this study, we
systematically examine the biases in foundation models and demonstrate the
efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that
bias estimation in foundation models is challenging, as most pre-train data
cannot be explicitly accessed like in traditional long-tailed classification
tasks. To this end, GLA has an optimization-based bias estimation approach for
debiasing foundation models. As our work resolves a fundamental flaw in the
pre-training, the proposed GLA demonstrates significant improvements across a
diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, an
large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains
on long-tailed classification. Codes are in https://github.com/BeierZhu/GLA.

##  1 Introduction

Thanks to the Web-scale data and self-supervised strategies, foundation models
like CLIP [40] empower zero-shot transfer to a wide variety of domains [7, 1,
52]. However, the zero-shot performance is still weak on several domain-
specific tasks such as differentiating models of cars, species of flowers, and
variants of aircraft [40, 7]. Therefore, it is a common practice to improve
the downstream performance via supervised fine-tuning on labeled data, _e.g._
, linear probing, prompt tuning [54, 55], and end-to-end fine-tuning.

However, fine-tuned models are easily biased: they are adept in exploiting
spurious correlations that only hold on the downstream distribution [40, 39,
48, 55]. To improve the robustness, several studies [48, 55, 56] propose to
combine fine-tuned models with zero-shot models. For example, WiSE-FT [48]
ensembles the fine-tuned and zero-shot models in weight space and ProGrad [55]
uses zero-shot predictions to regularize the fine-tuning gradient. The
underlying assumption lies in that the zero-shot models are robust to
distribution shifts [40], and their predictions are complementary to those of
fine-tuned models [48].

Despite these methods exhibiting performance gains on both in-distribution and
out-of-distribution evaluations, they all overlook the inherent bias
originating from the foundation models. Specifically, the Web-scale data for
pre-training foundation models exhibit a highly skewed distribution due to
Zipf’s law of nature [42]. The resulting foundation models develop a biased
decision boundary that leads to a poor zero-shot performance on rare classes.
As evidenced in Figure 1(a) and (b), the purple line encounters a dramatic
drop, and the zero-shot performance of tail classes is significantly lower
than that of head classes ($57.2\%$ _vs_. $78.0\%$). Existing ensemble methods
like WiSE-FT [48] overlook the label bias, resulting in an improvement in
top-1 ($+0.5\%$) and head accuracy ($+1.7\%$) while a noticeable degradation
on the tail performances ($-0.9\%$) in Figure 1(b). Another evidence is that
the orange line (WiSE-FT) is below the blue line (fine-tuned models) for rare
classes in Figure 1(a).

Figure 1: (a) Per class accuracy of CLIP-ViT/B16 on ImageNet. Class index are
sorted using the estimated pre-training label prior. Curves are smoothed for
better visualization. (b) Beak-down performance of different models on
ImageNet. We equally divide the ImageNet classes into three subgroups,
according to the class index. Existing ensemble methods like WiSE-FT [48]
exhibits a clear performance loss on tail classes, while our GLA stands out
for all three subgroups.

We propose Generalized Logit Adjustment (GLA), a simple post-hoc method
consisting of two steps: 1) removing the label bias of zero-shot model via
estimating the label distribution in the pre-training dataset; 2) ensembling
the fine-tuned and debiased zero-shot models. As illustrated in Figure 1 (b),
our GLA achieves consistent improvement across all three subgroups,
particularly showing a significant gain on tail classes ($+1.5\%$). Despite
its simplicity, our GLA has a firm statistical grounding: it is the Bayes
optimal classifier given the fine-tuned and zero-shot models, thus consistent
for minimizing the error on a class-balanced target distribution (Section
4.2). It is worth noting that removing the bias of foundation models is
challenging since the label distribution is often inaccessible due to privacy
or copyright concerns. In this work, we only use the downstream labeled data
and the zero-shot model to estimate the foundation label bias. Specifically,
we formulate the problem by adjusting the margin of the zero-shot models such
that the lowest error is achieved on the downstream dataset. This grounding
translates into strong empirical performance on real-world datasets, covering
few-shot, many-shot, and long-tail learning (Section 5).

The contributions and novelties of this work are summarized as follows:

  * •

We point out the overlooked label bias in foundation models, which originates
from the skewness of pre-training distribution that affects the performance of
downstream tasks.

  * •

We formalize the estimation of the label bias as a constrained optimization
problem (Section 3.2) with theoretical justification (Section 4.3). The entire
process does not require access to the pre-training dataset, making it
practical for fine-tuning scenarios. We present Generalized Logit Adjustment
(GLA) method, which ensembles the debiased zero-shot and fine-tuned models,
and demonstrate its superiority over conventional fine-tuning and ensembling
by proving it’s a Bayes optimal classifier (Section 4.2).

  * •

We build a comprehensive benchmark for evaluation, which considers three real-
world settings and three fine-tuning paradigms. The settings are: 1) many-shot
learning with abundant data 2) few-shot learning; and 3) long-tail
classification, representing a more challenging scenario that combines many-
shot and few-shot data (Section 5). The three fine-tuning paradigms include:
1) end-to-end fine-tuning; 2) linear probing; and 3) prompt tuning (Section
3.1).

  * •

We demonstrate the efficacy of our proposed method GLA by conducting extensive
experiments across various settings and fine-tuning paradigms. We observe 1 to
1.5 pp accuracy gains on ImageNet, large averaged improvement (1.4 to 4.6 pp)
on 11 few-shot datasets and 2.4 pp averaged accuracy gains on long-tail
datasets. (Section 5).

##  2 Related Work

Image-text foundation models. Foundation models pre-trained by contrastive
objectives have set impressive milestones for image and text representation
learning, with CLIP [40], ALIGN [23], CoCa [52] and Flamingo [1] being the
exemplars. Such models exhibit impressive prompt-based zero-shot performance
on various image recognition downstream tasks. Our method aims to reduce
foundation model biases to boost performance in downstream tasks. While [2]
also addresses word frequency bias, we differ in two key areas: Firstly, we
debias zero-shot models using fixed prompts, whereas [2] refines the prompting
process. Secondly, our GLA doesn’t require access to a subset of the pre-
training data.

Ensembles. Ensemble methods aim to boost performances by combining multiple
networks, which can be either implemented by aggregating model outputs [12, 4,
28, 14, 27, 51], weight-space ensembling [48, 22], or ensemble distillation
[19, 29]. For the adaptation of foundation models, several work propose to
ensemble the fine-tuned and zero-shot models for better performance: Wortsman
et al. [48] ensembles them in weight space; ProGrad and ProReg [55, 56]
propose to fuse them via knowledge distillation. Our GLA is orthogonal to
these approaches, as it concentrates on mitigating the biases in foundation
models that are detrimental to ensemble models.

Logit adjustment. Logit adjustment [35, 45, 24, 49, 20] is a post-hoc
technique to adjust the biased output of classification networks. Kang  _et
al_ [24] proposes an element-wise scaling adjustment for the classifier
weight. Tang  _et al_ [45] removes the projection of features on a global
biased direction. Menon [35] derives the theoretically optimal adjustment from
the training distribution. Unlike the those approaches which rely on a
transparent training data or class distribution, our GLA can eliminate the
class bias without accessing to the pre-training statistics.

##  3 Methods

###  3.1 Setup

Task. Consider a classification problem with instances
$\mathbf{x}\in\mathcal{X}$ and labels $y\in\mathcal{Y}=[K]=\\{1,...,K\\}$. We
have a zero-shot model $f_{\mathsf{zs}}$ (given below), a downstream dataset
$\mathcal{D}_{s}=\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{N_{s}}$ drawn from source
distribution $P_{s}$ and a fine-tuned model $f_{\mathsf{ft}}$ (given below)
trained on the dataset $\mathcal{D}_{s}$. Give a model
$f:\mathcal{X}\rightarrow\mathbb{R}^{K}$ that outputs prediction score, we
define the risk of $f$ on the target distribution $P_{t}$ as the mis-
classification rate: $\mathcal{R}_{t}(f)=\mathbb{E}_{\mathbf{x},y\sim
P_{t}}[y\neq\operatorname*{argmax}_{i}f(\mathbf{x})_{i}]$. Our goal is to
learn a model $f_{\mathsf{gla}}$ that best leverages $f_{\mathsf{zs}}$ and
$f_{\mathsf{ft}}$ that minimizes the risk $\mathcal{R}_{t}$.

Zero-shot models. We primarily explore CLIP [40] for zero-shot models. CLIP
consists of a visual encoder $\Phi_{\mathsf{v}}(\mathbf{x})$ and a text
encoder $\Phi_{\mathsf{t}}(\mathbf{t})$, producing $l_{2}$-normalized features
from an image $\mathbf{x}$ and a text $\mathbf{t}$ respectively. Zero-shot
model $f_{\mathsf{zs}}$ for $K$ classes is enabled by matching image features
$\mathbf{v}=\Phi_{\mathsf{v}}(\mathbf{x})$ with classification weights
$\mathbf{w}_{k}=\Phi_{\mathsf{t}}(\mathbf{t}_{k})$, where $\mathbf{t}_{k}$ is
obtained by extending the class name $\\{c_{k}\\}$ to a pre-defined prompt,
_e.g._ , “a photo of a {$c_{k}$}.”. Additional details are provided in
Appendix B.2. The probability of $\mathbf{x}$ being classified as $y$ is
defined as:

$P(y|\mathbf{x})=\mathrm{softmax}(f_{\mathsf{zs}}(\mathbf{x}))_{y}=\frac{\exp(\mathbf{v}^{T}\mathbf{w}_{y})}{\sum_{k=1}^{K}\exp(\mathbf{v}^{T}\mathbf{w}_{k})}.$
(1)

Fine-tuned models. Standard fine-tuning initializes the model
$f_{\mathsf{ft}}$ with the pre-trained parameters and then solve
$f_{\mathsf{ft}}=\operatorname*{argmin}_{f}\mathcal{R}_{s}(f)$ to minimize the
risk on downstream dataset. We consider three common variants of fine-tuning:
(1) end-to-end, where all parameters of $\Phi_{\mathsf{v}}$ and
$\mathbf{w}_{k}$ are updated; (2) linear probing, where only $\mathbf{w}_{k}$
is modified while $\Phi_{\mathsf{v}}$ is fixed; (3) prompt tuning, where the
text input $\mathbf{t}_{k}$ is learned, while keeping $\Phi_{\mathsf{v}}$ and
$\Phi_{\mathsf{t}}$ freezed. See Appendix B.3 for details on fine-tuning
methods.

Notation. Let $P_{p}(y)$, $P_{s}(y)$ and $P_{t}(y)$ be the marginal
probability of class $y\in[K]$ for pre-training, source (training) and target
(test) distribution, respectively. Let $\pi_{p}$ and $\pi_{s}$ denote the log
probabilities of class for pre-training and training distribution, _i.e._ ,
$\pi_{p}(y)=\log P_{p}(y)$ and $\pi_{s}(y)=\log P_{s}(y)$.

###  3.2 Generalized Logit Adjustment Framework

Fine-tuned models often yield significant gains compared to zero-shot models,
and ensembling them can further improve performance. This leads to a natural
question: How should we best leverage the zero-shot and fine-tuned models for
the prediction tasks? We attempt to answer by proposing generalized logit
adjustment in Definition 1.

######  Definition 1.

(GLA) The Generalized Logit Adjustment (GLA) model $f_{\mathsf{gla}}$ is
defined as follows:

$f_{\mathsf{gla}}(\mathbf{x})=f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p}.$
(2)

In Section 4.2, we prove that given the zero-shot and fine-tuned models, our
GLA model is the Bayes optimal classifier and no other combination of the two
models can outperform it. Here remains one important question: how could we
obtain $\pi_{p}$ as we have no access to the pre-training statistics? We
provide the estimation process of $\pi_{p}$ in Eq (4) and postpone the
justification in Section 4.3. The entire GLA algorithm consists of two steps
which is given as follows:

Figure 2: Illustration of debiasing process on ImageNet validation set. (a)
The original distribution of zero-shot outputs; (b) the estimated pre-train
distribution $\mathbf{q}$ based on our algorithm; (c) the distribution of
debiased zero-shot outputs using estimated $\mathbf{q}$.

Step 1: Estimation of $\pi_{p}$. Let $\mathbf{q}$ be an arbitrary probability
simplex over $K$ classes. Given the validation data from $P_{t}$ or the
balanced training data, we can estimate the $\hat{\pi}_{p}=\log\mathbf{q}^{*}$
as the constrained optimization problem (proof in Section 4.3):

$\displaystyle\mathbf{q}^{*}=\operatorname*{argmin}_{\mathbf{q}}\mathcal{R}_{t}(f_{\mathsf{zs}}-\log\mathbf{q})$
$\displaystyle\mathrm{s.t.}\ \mathbf{q}_{i}\geq 0,\mathrm{for}\ i\in[K],$
$\displaystyle\quad\sum_{i\in[K]}\mathbf{q}_{i}=1.$ (3)

We constrain the sum of $\mathbf{q}$ to be 1 and ensure that each element is
non-negative, guaranteeing that it forms a valid probability distribution. We
solve the following Lagrangian problem to find optimal $\mathbf{q}^{*}$:

$\min_{\mathbf{q}}\max_{\lambda_{i}\geq
0,\upsilon}\mathcal{R}_{t}(f_{\mathsf{zs}}-\log\mathbf{q})-\sum_{i}\lambda_{i}\mathbf{q}_{i}+\upsilon(1-\sum_{i\in[K]}\mathbf{q}_{i})$
(4)

Step 2: GLA ensembling. Given the estimated $\hat{\pi}_{p}$ and the known
downstream training $\pi_{s}$, we ensemble the zero-shot model
$f_{\mathsf{zs}}$ and the fine-tuned model $f_{\mathsf{ft}}$ to get our GLA
model $f_{\mathsf{gla}}$ via Eq. (2). We can regard
$f_{\mathsf{zs}}-\hat{\pi}_{p}$ and $f_{\mathsf{ft}}-\pi_{s}$ as the debiased
zero-shot model (Figure 2(c)) and debiased fine-tuned models, respectively.
Our GLA is actually ensembling two debiased models. Note that, different from
[55, 48], we do not require a hyper-parameter to adjust the contribution of
the two models, the optimal solution is to combine them equally (see Section
4.2 for justification).

##  4 Theoretical Analysis

In this section, we explain why our GLA model best ensembles the zero-shot and
fine-tuned models (Section 4.2) and justify the estimation process of the pre-
training label distribution $\pi_{p}$ (Section 4.3). We start with some
preliminaries on Bayes optimal classifier.

###  4.1 Preliminaries

Suppose we have a pair $(X,Y)\sim P$ takes values in
$\mathcal{X}\times\mathcal{Y}$, where $Y$ is the class label of input $X$.

######  Definition 2.

The 0-1 error (risk) of a classifier
$\hat{y}:\mathcal{X}\rightarrow\mathcal{Y}$ on distribution $P$ is given by:

$\mathcal{R}(\hat{y})=P(Y\neq\hat{y}(X))$ (5)

However, the 0-1 error is non-smooth, one typically minimizes a surrogate loss
$\ell$, _e.g._ , cross-entropy:
$\ell(f(\mathbf{x}),y)=\log[\sum_{i\in[K]}\exp(f(\mathbf{x})_{i}-f(\mathbf{x})_{y})]$
, where $\hat{y}(\mathbf{x})=\operatorname*{argmax}_{i}f(\mathbf{x})_{i}$. It
is known that the cross-entropy loss is Bayes consistent [53], _i.e._ , a
nearly optimal minimizer of the cross-entropy loss
($\mathbb{E}_{\mathbf{x},y\sim P}[\ell(f(\mathbf{x}),y)$] is also a nearly
optimal optimizer of the mis-classification error
($\mathbb{E}_{\mathbf{x},y\sim P}[y\neq\hat{y}(\mathbf{x})]$).

######  Definition 3.

The Bayes optimal classifier $y^{*}$ for $P$ given input $\mathbf{x}$ is
defined as:

$y^{*}(\mathbf{x})=\operatorname*{argmax}_{y\in\mathcal{Y}}P(y|\mathbf{x})$
(6)

It is called Bayes optimal classifier because on the average no other
classifier using the same hypothesis and prior knowledge can outperform it.

######  Lemma 1.

The Bayes optimal classifier $y^{*}$ for $P$ has lower risk than all
classifiers $\hat{y}:\mathcal{X}\rightarrow\mathcal{Y}$.

$\mathcal{R}(y^{*})\leq\mathcal{R}(\hat{y})$ (7)

###  4.2 Generalized Logit Adjustment Leads to Better Ensembling

Zero-shot and fine-tuned models are complementary. We revisit an empirical
phenomena observed in [48] Section 5.1: After exploring a series of measures
of diversity, covering predictions and features, they find that zero-shot and
fine-tuned models have diverse predictions, despite sharing the same backbone.
As the two data distribution $P_{s}$ and $P_{p}$ is known to be different, the
resulting models leverage different cues to predict: fine-tuned models risk
exploiting spurious correlations and in-domain patterns which only hold for
downstream dataset [46, 3]; On the other hand, zero-shot CLIP models capture
stable correlations across diverse domains and exhibit much higher robustness
[40]. For instance, zero-shot models rely on robust features for decisions
that can achieve high performance on sketch and adversarial samples, while the
fine-tuned models that trained on real images typically fail on these samples,
as they rely on spurious correlations that only hold on real images. We
formulate the phenomena in the following assumption.

######  Assumption 1.

Zero-shot and fine-tuned models have diverse predictions:

$(f_{\mathsf{ft}}(\mathbf{x})\perp f_{\mathsf{zs}}(\mathbf{x}))|y.$ (8)

We derive the conditional probability
$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))$ w.r.t. the
outputs of $f_{\mathsf{zs}}(\mathbf{x})$ and $f_{\mathsf{ft}}(\mathbf{x})$:

######  Lemma 2.

For a balanced target distribution111This lemma can be easily extended to
imbalanced target distributions (proof in Appendix A). Yet, as most test sets
are class-balanced, we focus on the balanced case for brevity., where
$P_{t}(y)=1/K$ for all $y\in[K]$, we have:

$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))=\mathrm{softmax}(\underbrace{f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p}}_{\textstyle{f_{\mathsf{gla}}(\mathbf{x})}})(y)$
(9)

Intuitively, since the zero-shot and fine-tuned models provide diverse
predictions, conditioned on the two predictions is equivalent to adding the
logits in log space. Additionally, as the target distribution is class-
balanced, we need to remove the class bias of two models by subtracting
$\pi_{s}$ and $\pi_{p}$. The formal proof is given in Appendix A.1. Note that
the RHS of Eq. (9) is exactly the softmax output of our GLA model by
Definition 1, which exhibits the following property:

######  Proposition 1.

Let $g:\mathbb{R}^{K}\times\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ be an
arbitrary function that ensemble the outputs of $f_{\mathsf{zs}}$ and
$f_{\mathsf{ft}}$. Our GLA classifier $f_{\mathsf{gla}}$ has lower risk than
any function
$f_{g}(\mathbf{x})=g(f_{\mathsf{zs}}(\mathbf{x}),f_{\mathsf{ft}}(\mathbf{x}))$,
_i.e._

$\mathcal{R}_{t}(f_{\mathsf{gla}})\leq\mathcal{R}_{t}(f_{g}).$ (10)

###### Proof.

From Lemma 2 and Definition 1, we have:

$\operatorname*{argmax}_{y\in\mathcal{Y}}f_{\mathsf{gla}}(\mathbf{x})_{y}=\operatorname*{argmax}_{y\in\mathcal{Y}}\mathrm{softmax}(f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p})_{y}=\operatorname*{argmax}_{y\in\mathcal{Y}}P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x})),$
(11)

which means $f_{\mathsf{gla}}$ is the Bayes optimal classifier (see Definition
3) given $f_{\mathsf{ft}}(\mathbf{x})$ and $f_{\mathsf{zs}}(\mathbf{x})$.
According to Lemma 1, any other classifier
$g(f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))$ must have higher
risk, _i.e._ , $\mathcal{R}_{t}(f_{\mathsf{gla}})\leq\mathcal{R}_{t}(f_{g})$.
∎

Proposition 1 demonstrates that our $f_{\mathsf{gla}}$ model is the best
model, as it has the lowest risk on target distribution. Proposition 1 further
explains the superiority of $f_{\mathsf{gla}}$ over the fine-tuned model
$f_{\mathsf{ft}}$ and the naive ensemble
$f_{\mathsf{ens}}(\mathbf{x})=f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})$:

######  Corollary 1.

$f_{\mathsf{gla}}$ performs better than fine-tuned model $f_{\mathsf{ft}}$ and
naive emsembling $f_{\mathsf{ens}}$:

$\mathcal{R}_{t}(f_{\mathsf{gla}})\leq\mathcal{R}_{t}(f_{\mathsf{ft}}),\
\mathcal{R}_{t}(f_{\mathsf{gla}})\leq\mathcal{R}_{t}(f_{\mathsf{ens}})$ (12)

Discussion: when do the GLA models degenerate? Note that there are two
equality signs in Eq. (12), indicating that the performance of the GLA model
can degenerate to be equivalent to that of the fine-tuned model and naive
ensembling in the following two cases.

Case 1: For the first equality, if zero-shot model
$f_{\mathsf{zs}}(\mathbf{x})$ provides no further information about $y$ given
$f_{\mathsf{ft}}(\mathbf{x})$, _i.e._ , $(y\perp
f_{\mathsf{zs}}(\mathbf{x}))|f_{\mathsf{ft}}(\mathbf{x})$, then
$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))$ degenerates
to $P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}))$ and the first equality applies.
However, in practice, as downstream model and zero-shot model provides diverse
predictions, we usually encounter strict inequality, _i.e._ ,
$\mathcal{R}_{t}(f_{\mathsf{gla}})<\mathcal{R}(f_{\mathsf{ft}})$.

Case 2: The second equality applies when pre-training and downstream training
distribution are both class-balanced. In fact, the pre-training dataset for
foundation models are known to be highly skewed. Therefore, in most cases, we
have $\mathcal{R}_{t}(f_{\mathsf{gla}})<\mathcal{R}_{t}(f_{\mathsf{ens}})$.

In summary, the above two equalities are usually unattainable, which means
that theoretically, our GLA model performs better than both the fine-tuned and
the naive ensemble models.

###  4.3 Estimate the label bias of the pre-training dataset

However, $\pi_{p}$ is usually unknown as we have no access to pre-training
dataset. In this work, we seek to estimate $\pi_{p}$ using the zero-shot
models and the downstream data. Similar to Proposition 1, we have the
following proposition says that $f_{\mathsf{zs}}-\pi_{p}$ has lower error on
target distribution than any other classifiers that use $f_{\mathsf{zs}}$, see
Appendix A.2 for the full proof.

######  Proposition 2.

Let $h:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ be an arbitrary function that
predicts labels using the outputs of the zero-shot model
$f_{\mathsf{zs}}(\mathbf{x})$. Let the derived classifier be denoted as
$f_{h}(\mathbf{x})=h(f_{\mathsf{zs}}(\mathbf{x}))$. The classifier
$f_{\mathsf{zs}}-\pi_{p}$ is better than any $f_{h}(\mathbf{x})$:
$\mathcal{R}_{t}(f_{\mathsf{zs}}-\pi_{p})\leq\mathcal{R}_{t}(f_{h}(\mathbf{x}))$.

Let $\mathbf{q}$ be an arbitrary probability simplex over $K$ classes, then we
have
$\mathcal{R}_{t}(f_{\mathsf{zs}}(\mathbf{x})-\pi_{p})\leq\mathcal{R}_{t}(f_{\mathsf{zs}}(x)-\log\mathbf{q})$.
Therefore, we choose to optimize a probability simplex $\mathbf{q}$ over $K$
classes such that the model $f_{\mathsf{zs}}-\log\mathbf{q}$ achieves the
minimal empirical risk, as formulated in Eq. (3.2) (the Step 1 of GLA
algorithm). Once we obtain the estimated class prior
$\hat{\pi}_{p}=\log\mathbf{q}$, we can easily implement the GLA model by
ensembling
$f_{\mathsf{gla}}(\mathbf{x})=f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\hat{\pi}_{p}$
(the Step 2 of GLA algorithm).

Figure 3: Estimating label bias of CIFAR-10-LT-IB-10.

Toy experiment. We conducted an experiment to show that the estimated label
distribution closely approximates the true one. Specifically, we trained a
model with a ResNet32 backbone on the imbalanced CIFAR-10-LT [10] dataset with
an imbalanced ratio of 10. Subsequently, we used only the test set combined
with our proposed method to estimate the label distribution. This procedure
simulates scenarios where only downstream data is available and the pre-
training data is inaccessible.

Figure 3 reveals a strong alignment between the estimated (orange line) and
the actual distributions (blue line), which is further emphasized by a small
KL-divergence value of 0.00062. The toy experiment validates the effectiveness
of our debiasing method.

Discussion. The $\log\mathbf{q}$ we estimated is not the marginal log-
probability of the entire pre-training distribution but the label bias matches
the downstream distribution. In the above toy experiment, although training
and test sets show different label distributions, their conditional
distribution $P(\mathbf{x}|y)$ remains invariant. In this case, our estimate
will converge to the actual training label bias. For CLIP models, with diverse
pre-training data, some might not align with the downstream domain,
potentially compromising the accuracy of the estimation of the entire pre-
training distribution.

However, we’d like to point out that removing the label bias of entire pre-
training distribution may not optimal for downstream tasks. As a thought
experiment, consider a pre-training dataset "sketch" and "photo" styles for
"dog" and "cat" samples. Suppose the sample size of "dog” and "cat” is equal
but there are more "sketch dogs” than "sketch cats”. This means that even if
the overall distribution is balanced, each style isn’t, resulting in biased
zero-shot predictions. if we aim to deploy models for the "sketch dogs and
cats” domain, adjusting the overall label bias is insufficient. Instead, the
optimal label bias should be estimated on the “sketch” distribution. We also
provide experiments using LAION-400M dataset in Appendix C.2, illustrating the
situation when the downstream data diverges from the pre-training set.

##  5 Experiments

We evaluate our GLA on three real-world scenarios: many-shot (Section 5.1),
few-shot (Section 5.2) and long-tail learning (Section 5.3). We show that our
GLA boosts performance on all three settings.

###  5.1 Many-shot learning

Datasets. We use ImageNet [11] and CIFAR100 [26] for generic object
classification, StanfordCars [25] for fine-grained classification, and SUN397
[50] for scene recognition. See Appendix B.1 for details.

Baselines. We compare GLA against four methods: (1) Zero-shot model, (2)
Linear Probing (LP), (3) End-to-End fine-tuning (E2E), and (4) weight
ensembling method WiSE-FT[48].

Implementation details. We consider two models: CLIP ViT-B/32 and ViT-B/16.
For learning-based models, we fine-tune with AdamW using a cosine annealing
learning rate scheduler. We fine-tune for 10 epochs on ImageNet and 20 epochs
on other datasets. See Appendix B.3 for further details.

Main results. Table 1 compares our GLA with various baselines. We observe that
our GLA can increase the performance of end-to-end fine-tuned models: it
achieves $1.5\%$ gains on ImageNet. Compared to WiSE-FT, GLA gains $1.1\%$
top-1 accuracy boost on ImageNet dataset,. Beyond generic object recognition,
our method also improves accuracy on the fine-grained dataset (Stanford Cars)
and the scene recognition dataset (SUN397), by $0.4\%$ and $0.5\%$,
respectively.

Method | B/32 | B/16
---|---|---
Zero-shot | 63.2 | 68.3
LP | 75.8 | 79.9
E2E | 76.3 | 81.3
WiSE-FT | 76.9 | 81.7
GLA | 77.9 | 82.8
(a) ImageNet

Method | B/32 | B/16
---|---|---
Zero-shot | 64.2 | 67.2
LP | 80.5 | 83.1
E2E | 89.4 | 91.0
WiSE-FT | 90.0 | 91.3
GLA | 90.7 | 91.9
(b) CIFAR100

Method | B/32 | B/16
---|---|---
Zero-shot | 59.7 | 64.4
LP | 77.1 | 83.1
E2E | 83.7 | 90.2
WiSE-FT | 84.5 | 90.7
GLA | 85.0 | 91.1
(c) Stanford Cars

Method | B/32 | B/16
---|---|---
Zero-shot | 62.2 | 64.8
LP | 75.6 | 78.0
E2E | 80.0 | 82.4
WiSE-FT | 80.8 | 82.9
GLA | 81.2 | 83.4
(d) SUN397

Table 1: Accuracy of various methods using CLIP ViT-B/32 and ViT-B/16. LP:
linear probe; E2E: end-to-end fine-tuning. Results were obtained using the
official implementation from WiSE-FT [48].

Breakdown performance analysis. To analyze the impact of pre-training label
bias on fine-tuned and ensemble models, we present the breakdown results on
ImageNet using CLIP ViT-B/16, as shown in Table 2. Specifically, we sort the
class index using the estimated $\pi_{p}$, and assign the top third of the
classes as the head classes, the last third as the tail classes, and the
remaining classes as the medium classes. Due to the label bias, the zero-shot
tail performance is significantly lower than the head one ($57.2\%$ _vs_.
$78.0\%$). The resulting E2E models are also affected by the bias, with the
tail performance being $6.3\%$ lower than the head. Existing ensemble WiSE-FT
overlooks the bias, exhibiting noticeable degradation on the tail performances
($-0.9\%$) compared to E2E model, while our GLA stands out for all three
subgroups.

Method | Head | Med. | Tail | All
---|---|---|---|---
Zero-shot | 78.0 | 69.8 | 57.2 | 68.3
E2E | 83.6 | 83.0 | 77.3 | 81.3
WiSE-FT | 85.3 | 83.7 | 76.4 | 81.7
GLA | 85.2 | 84.3 | 78.8 | 82.8

Table 2: Breakdown results on ImageNet.

Model | Source | Target
---|---|---
ViT-B/32 | ViT-B/16 | ViT-L/14
$f_{\mathsf{zs}}(\mathbf{x})$ | 63.4 | 68.8 | 75.6
$f_{\mathsf{zs}}(\mathbf{x})-\hat{\pi}_{p}$ | 65.4 | 69.3 | 76.3

Table 3: Estimated $\hat{\pi}_{p}$ is transferable across different backbones.
$\hat{\pi}_{p}$ is estimated using CLIP ViT-B/32.

Estimated $\pi_{p}$ is transferable across different zero-shot models. The
estimated $\pi_{p}$ should be transferable across different zero-shot models
if they are trained on the same pre-training dataset. To verify this, we
employed a CLIP ViT-B/32 based zero-shot model to estimate $\pi_{p}$, which is
subsequently used to debias zero-shot models based on CLIP ViT-B/16 and
ViT-L/14. As shown in Table 3, our debiased models outperform the original
zero-shot versions by a clear margin.

Figure 4: Accuracy with mixing coefficient $\alpha$.

Ensembling with mixing coefficient. In Section 5.1, we prove the optimal
solution is to combine the debiased zero-shot and fine-tuned models equally.
We now examine the claim by introducing a mixture coefficient
$\alpha\in[0,1]$. The ensemble predictions are given by:
$f_{\mathsf{gla}}(\mathbf{x},\alpha)=(1-\alpha)\cdot(f_{\mathsf{zs}}(\mathbf{x})-\pi_{p})+\alpha\cdot(f_{\mathsf{ft}}(\mathbf{x})-\pi_{s})$.
We compare the GLA and the naive ensembling with mixture $\alpha$ in Figure 4,
where GLA meets its optimal performance at $\alpha=0.5$, which is in line with
our theoretical analysis. We also observe that the debiased zero-shot model
increases accuracy by $2.3\%$ and our GLA consistently outperforms naive
ensembling with various $\alpha$.

###  5.2 Few-shot learning

For few-shot scenarios, we primarily choose prompt tuning for fine-tuning,
since it is empirically more effective than end-to-end fine-tuning and linear
probing [54, 55, 8].

Figure 5: Accuracy (%) of few-shot learning on 11 datasets.

Datasets. We follow CoOp[54] to use 15 datasets: ImageNet [11], Caltech101
[13], OxfordPets [37], StanfordCars [25], Flowers102 [36], Food101 [6],
FGVCAircraft [34], EuroSAT [16], UCF101 [44], DTD [9], SUN397 [50],
ImageNet-{V2 [41], Sketch [47], A [18], R [17]}. We randomly select {1, 2, 4,
8, 16} shots for training and use the original test set for evaluation. See
Appendix B.1 for details.

Method | Source | Target
---|---|---
IN | V2 | S | A | R
CLIP | 59.8 | 52.8 | 35.5 | 22.8 | 60.6
CoOp | 61.9 | 54.3 | 32.5 | 21.8 | 54.2
PLOT | 63.0 | 55.1 | 33.0 | 21.9 | 55.6
ProGrad | 63.5 | 55.4 | 33.1 | 21.3 | 55.2
GLA | 65.6 | 57.1 | 36.4 | 23.0 | 62.1

Table 4: Evaluation on robustness to distribution shift at 16 training shots.

Figure 6: Comparison with naive ensembling.

Baselines. We compare with three prompt tuning methods: (1) CoOp [54]
optimizes prompts via empirical risk minimization; (2) ProGrad [55] prevents
forgetting the general knowledge using zero-shot predictions; (3) PLOT [8]
applies optimal transport to match the vision and text modalities.

Implementation details. We implement our proposed GLA method using CLIP-
ResNet-50 as the foundation model and adopt class-specific prompt tuning from
CoOp. Results are averaged over three seeds, with training configurations
aligned with CoOp. See Appendix B.4 for further information.

Main results. Figure 5 summarizes the few-shot results on 11 datasets. The
detailed accuracy and standard deviation are in Appendix C.1. Overall, our GLA
clearly outperforms the baselines on average performance by a large margin,
_e.g._ , our GLA gains $4.6\%$, $4.8\%$, $3.1\%$, $2.6\%$ and $1.6\%$
performance boost over CoOp at $1,2,4,8,16$ shots. In particular, on ImageNet
dataset, we observe a large improvement, _e.g._ , $3.9\%,2.9\%,1.9\%,2.0\%$
and $2.2\%$ over ProGrad at $1,2,4,8,16$ shots. Furthermore, on OxfordPerts,
Food101 and SUN397 datasets, our method’s performance remains stable and
consistently improves with increasing sample sizes, while the one of the
baseline methods fluctuates significantly. In particular, on Food101, baseline
models even underperform zero-shot models by a large margin, while our GLA
shows clearly better performance than zero-shot models.

Robustness to distribution shifts. Following CoOp, we used the ImageNet at 16
training shots as the source domain and assess the robustness on ImageNet-{V2,
Sketch, A, R} datasets. Table 4 summarizes the results, where the prompt
tuning baselines perform worse on distribution shifted datasets compared to
the zero-shot model, as fine-tuning on limited data misleads the model to
learn in-distribution correlations. In comparison, our GLA approach makes the
best of both fine-tuned and zero-shot models, thus consistently outperforms
other methods on both source and target domains.

GLA improves accuracy over naive ensemble. Figure 6 compares the results
between our GLA and naive ensembling. We rank the absolute improvements over
fine-tuning baseline at 16 training shots. In summary, our GLA demonstrates
superior accuracy gains. It is worth noting that the naive ensembling does not
always lead to improvements, _e.g._ , on EuroSAT, SUN, Caltech, UCF and
Aircraft, naive ensembling even underperforms fine-tuning baseline.

Model | IN. | Caltech. | Pets. | Cars. | Flowers. | Food. | Aircraft. | SUN. | DTD | EuroSAT | UCF. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
$f_{\mathsf{zs}}$ | 59.8 | 87.1 | 85.8 | 55.6 | 65.3 | 77.9 | 17.1 | 60.3 | 42.3 | 37.5 | 61.5 | 59.1
$f_{\mathsf{zs}}-\hat{\pi}_{p}$ | 62.3 | 89.9 | 85.9 | 57.6 | 67.2 | 78.6 | 18.4 | 63.5 | 43.0 | 39.0 | 62.4 | 60.7
$\Delta$ | +2.5 | +2.7 | +0.1 | +2.0 | +1.9 | +0.7 | +1.3 | +3.2 | +0.7 | +1.4 | +0.9 | +1.6
Table 5: Comparision between zero-shot models ($f_{\mathsf{zs}}$) and debiased
zero-shot models ($f_{\mathsf{zs}}-\hat{\pi}_{p}$).

Debiased zero-shot models perform better. We estimate $\pi_{p}$ at 16 shots
and compare the original zero-shot models with the debiased zero-shot models
in Table 5. It is clear that the debiasing leads to improvement on all 11
datasets, showcasing an average accuracy gain of $1.6\%$.

###  5.3 Long-tail learning

Datasets and metrics. We evaluate our method on two standard benchmarks:
Places365-LT and ImageNet-LT [31]. In addition to top-1 accuracy, we report
the accuracy on three test subsets according to the number of samples per
class: many-shot ($>100$ samples), medium-shot ($20\sim 100$ samples), and
few-shot ($<20$ samples). Detailed information is provided in Appendix B.1.

Fine-tuning and long-tail learning baselines. We compare our GLA with the
combinations of fine-tuning protocols and long-tailed recognition methods. We
consider three fine-tuning protocols: 1) Linear Probe (LP) 2) End-to-End (E2E)
fine-tuning; 3) Prompt Tuning (PT). The three fine-tuning paradigms are
introduced in Section 3.1 with details in Appendix B.3. We compare with 5
long-tail learning methods: 1) standard Empirical Risk Minimization (ERM); 2)
Learnable Weight Scaling (LWS) [24]; 3) Logit Adjustment (LA) [35]; 4)
Balanced Softmax (BS) [43], and 5) BALLAD [33], which is designed for VLMs.
See Appendix B.6 for more details on long-tail baselines.

Implementation Details. For all combinations of the fine-tuning and long-tail
learning baselines, visual backbones are initialized from CLIP-ResNet-50 and
classifiers are initialized via zero-shot prompting. We use SGD for 50 epochs
with batch size of 512. See Appendix B.6 for further details.

Results. Table 6 shows that our GLA method consistently surpasses baselines
across all long-tailed datasets. Our approach outperforms PT-based models by
$3.5\%$ and $4.4\%$ on ImageNet-LT and Places365-LT, respectively. Against E2E
approaches, GLA exceeds not just the WiSE-FT but also the current SOTA method
BALLAD, by a large margin, _e.g._ , 1pp gains on ImageNet-LT.

| Method | Many | Med | Few | All
---|---|---|---|---|---
LP | ERM | 64.4 | 39.8 | 20.1 | 46.6
LA | 56.7 | 50.5 | 46.4 | 52.3
BS | 57.7 | 50.5 | 43.9 | 52.4
GLA | 65.5 | 60.8 | 57.7 | 62.2
PT | ERM | 66.6 | 57.7 | 53.8 | 60.6
LA | 67.9 | 60.0 | 58.7 | 62.9
BS | 67.0 | 62.2 | 59.4 | 63.5
GLA | 70.7 | 66.5 | 64.3 | 67.8
E2E | ERM | 75.5 | 56.0 | 36.2 | 60.8
LWS [24] | 70.4 | 62.6 | 56.2 | 64.7
LA [35] | 70.4 | 62.8 | 56.9 | 64.9
BS [43] | 71.2 | 62.5 | 56.8 | 65.1
WiSE-FT [48] | 70.7 | 65.0 | 61.1 | 66.7
BALLAD [33] | 71.0 | 66.3 | 59.5 | 67.2
GLA | 72.1 | 66.4 | 63.4 | 68.2
(a) ImageNet-LT

| Method | Many | Med | Few | All
---|---|---|---|---|---
LP | ERM | 43.9 | 21.1 | 9.0 | 26.9
LA | 39.2 | 33.7 | 29.5 | 34.9
BS | 39.6 | 34.0 | 28.3 | 34.9
GLA | 43.6 | 40.2 | 38.8 | 41.1
PT | ERM | 47.7 | 32.6 | 24.9 | 36.5
LA | 43.9 | 40.6 | 39.4 | 41.5
BS | 43.4 | 42.5 | 42.3 | 42.8
GLA | 47.0 | 47.1 | 47.8 | 47.2
E2E | ERM | 46.3 | 27.4 | 12.5 | 31.3
LWS [24] | 41.2 | 39.0 | 32.8 | 38.6
LA [35] | 42.2 | 39.3 | 34.0 | 39.3
BS [43] | 46.7 | 44.4 | 39.5 | 44.3
WiSE-FT [48] | 46.0 | 44.0 | 43.9 | 44.7
BALLAD [33] | 48.7 | 44.7 | 42.2 | 45.7
GLA | 47.3 | 45.4 | 45.3 | 46.1
(b) Places365-LT

Table 6: The performances on ImageNet-LT and Places365-LT.

##  6 Conclusion and Limitation

In this paper, we identify the label bias in foundation models and underscore
its adverse effects on downstream task performance. We propose the Generalized
Logit Adjustment (GLA) framework for fine-tuning foundation models, which
boosts the performance by effectively eliminating label bias and combining
diverse predictions from zero-shot and fine-tuned models. We prove that when
presented with zero-shot and fine-tuned models, our GLA is the Bayes optimal
classifier for downstream task. Extensive experiments across a diverse range
of tasks and fine-tuning framework demonstrate the effectiveness of our
approach. We believe that the proposed GLA may partially improve the fairness
and credibility of foundation models.

The first limitation is that we only focus on the label bias while other forms
of model biases, _e.g._ , representation bias [5], cannot be addressed by our
algorithm yet. The second limitation is that we primarily focus on enhancing
the fine-tuning performance for discriminative models. However, applying our
GLA framework to generative models presents challenges. For instance, language
generation operates as a Markov process, meaning each output depends on
previous ones. This implies it’s not straightforward to estimate the
biasedness of a sequence with our GLA, as we only compute the bias in a pre-
defined and independent label space.

## Acknowledgments

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-01-002 and AI
Singapore AISG2-RP-2021-022).

##  Appendix A Proofs

###  A.1 Proof of Lemma 2

###### Restated Lemma 0 (Lemma 2).

Let $\pi_{t}$ denote the log probability of the target label distribution
$\pi_{t}(y)=\log P_{t}(y)$, we have:

$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))=\mathrm{softmax}(f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p}+\pi_{t})(y).$
(13)

In class-balanced target distribution, Eq. (13) simplifies to:

$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))=\mathrm{softmax}(f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p})(y).$
(14)

###### Proof.

Denote the output $\mathbf{e}=f_{\mathsf{ft}}(\mathbf{x})$ and
$\mathbf{z}=f_{\mathsf{zs}}(\mathbf{x})$. We first use the Bayes Rule to
decompose $P_{t}(y|\mathbf{e},\mathbf{z})$ into
$P_{t}(\mathbf{e},\mathbf{z}|y)$, $P_{t}(y)$ and
$P_{t}(\mathbf{e},\mathbf{z})$ in Eq. (15), then rewrite
$P_{t}(\mathbf{e},\mathbf{z}|y)$ in Eq. (16) according to Assumption 1.
Focusing on label shift problem [35, 20, 30] where $P(\mathbf{x}|y)$ does not
change, we derive Eq. (17)

$\displaystyle P_{t}(y|\mathbf{e},\mathbf{z})$
$\displaystyle=\frac{P_{t}(\mathbf{e},\mathbf{z}|y)P_{t}(y)}{P_{t}(\mathbf{e},\mathbf{z})}$
(15)
$\displaystyle=P_{t}(\mathbf{e}|y)P_{t}(\mathbf{z}|y)\frac{P_{t}(y)}{P_{t}(\mathbf{e},\mathbf{z})}$
(16)
$\displaystyle=P_{s}(\mathbf{e}|y)P_{p}(\mathbf{z}|y)\frac{P_{t}(y)}{P_{t}(\mathbf{e},\mathbf{z})}$
(17)
$\displaystyle=\frac{P_{s}(y|\mathbf{e})P_{s}(\mathbf{e})}{P_{s}(y)}\frac{P_{p}(y|\mathbf{z})P_{p}(\mathbf{z})}{P_{p}(y)}\frac{P_{t}(y)}{P_{t}(\mathbf{e},\mathbf{z})}$
(18)
$\displaystyle=\frac{P_{s}(y|\mathbf{e})}{P_{s}(y)}\frac{P_{p}(y|\mathbf{z})}{P_{p}(y)}\frac{P_{s}(\mathbf{e})P_{p}(\mathbf{z})P_{t}(y)}{P_{t}(\mathbf{e},\mathbf{z})}$
(19)

Since $\mathbf{e},\mathbf{z}$ are fixed, we can replace the terms that not
rely on $y$ with a constant $C_{1}$ in Eq. (21). We replace
$P_{s}(y)=\exp(\log P_{s}(y))=\exp(\pi_{s}(y))$, $P_{p}(y)=\exp(\log
P_{p}(y))=\exp(\pi_{p}(y))$ and $P_{t}(y)=\exp(\log
P_{t}(y))=\exp(\pi_{t}(y))$. Suppose the underlying class-probabilities
$P_{s}(y|\mathbf{e})\propto\exp(\mathbf{e}_{y})$ and
$P_{p}(y|\mathbf{z})\propto\exp(\mathbf{z}_{y})$ for $y\in[K]$. Denote the
constants $C_{s}$ and $C_{p}$ for normalizing $\exp(\mathbf{e}_{y})$ and
$\exp(\mathbf{z}_{y})$ into probabilities, and merge all constants to
$C=\frac{C_{1}}{C_{s}C_{p}}$, we get Eq. (23)

$\displaystyle P_{t}(y|\mathbf{e},\mathbf{z})$
$\displaystyle=\frac{P_{s}(y|\mathbf{e})}{P_{s}(y)}\frac{P_{p}(y|\mathbf{z})}{P_{p}(y)}P_{t}(y)C_{1}$
(21)
$\displaystyle=\exp(\mathbf{e}+\mathbf{z}-\pi_{s}-\pi_{p}+\pi_{t})(y)\frac{C_{1}}{C_{s}C_{p}}$
(22)
$\displaystyle=C\cdot\exp(\mathbf{e}+\mathbf{z}-\pi_{s}-\pi_{p}+\pi_{t})(y)$
(23)

Because the summation of $P_{t}(y|\mathbf{e},\mathbf{z})$ is 1,
$C=1/\sum_{i\in[K]}\exp(\mathbf{e}+\mathbf{z}-\pi_{s}-\pi_{p}+\pi_{t})(i)$.
Therefore, we have:

$\displaystyle
P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))$
$\displaystyle=P_{t}(y|\mathbf{e},\mathbf{z})$ (24)
$\displaystyle=\frac{\exp(\mathbf{e}+\mathbf{z}-\pi_{s}-\pi_{p})_{y}}{\sum_{i\in[K]}\exp(\mathbf{e}+\mathbf{z}-\pi_{s}-\pi_{p}+\pi_{t})_{i}}$
(25)
$\displaystyle=\mathrm{softmax}(f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p}+\pi_{t})_{y}$
(26)

In class-balanced target distribution case, $\pi_{t}=\log\frac{1}{K}$ is
constant. Since the softmax function is invariant to constant offsets, Eq.
(26) simplifies to:

$P_{t}(y|f_{\mathsf{ft}}(\mathbf{x}),f_{\mathsf{zs}}(\mathbf{x}))=\mathrm{softmax}(f_{\mathsf{ft}}(\mathbf{x})+f_{\mathsf{zs}}(\mathbf{x})-\pi_{s}-\pi_{p})_{y}$
(27)

∎

###  A.2 Proof of Proposition 2

###### Restated Proposition 0 (Proposition 2).

Suppose that the target distribution $P_{p}$ is class-balanced. Let
$h:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ be an arbitrary function that
predicts labels using the outputs of the zero-shot model
$f_{\mathsf{zs}}(\mathbf{x})$. Let the derived classifier be denoted as
$f_{h}(\mathbf{x})=h(f_{\mathsf{zs}}(\mathbf{x}))$. The classifier
$f_{\mathsf{zs}}-\pi_{p}$ is better than any $f_{h}(\mathbf{x})$:
$\mathcal{R}_{t}(f_{\mathsf{zs}}-\pi_{p})\leq\mathcal{R}_{t}(f_{h}(\mathbf{x}))$.

###### Proof.

Denote the output $\mathbf{z}=f_{\mathsf{zs}}(\mathbf{x})$. Similar to Eq.
(15)-Eq. (26), we have

$\displaystyle P_{t}(y|\mathbf{z})$
$\displaystyle=\frac{P_{t}(\mathbf{z}|y)P_{t}(y)}{P_{t}(\mathbf{z})}$ (28)
$\displaystyle=\frac{P_{p}(\mathbf{z}|y)P_{t}(y)}{P_{t}(\mathbf{z})}$ (29)
$\displaystyle=\frac{P_{p}(y|\mathbf{z})}{P_{p}(y)}\frac{P_{t}(y)}{P_{t}(\mathbf{z})}$
(30)
$\displaystyle=\exp(\mathbf{z}-\pi_{p})(y)/\sum_{i\in[K]}\exp((\mathbf{z}-\pi_{p})(i))$
(31)
$\displaystyle=\mathrm{softmax}(\mathbf{z}-\pi_{p})=\mathrm{softmax}(f_{\mathsf{zs}}(\mathbf{x})-\pi_{p})$
(32)

Therefore, we have:

$\operatorname*{argmax}_{y\in\mathcal{Y}}(f_{\mathsf{zs}}(\mathbf{x})-\pi_{p})_{y}=\operatorname*{argmax}_{y\in\mathcal{Y}}\mathrm{softmax}(f_{\mathsf{zs}}(\mathbf{x})-\pi_{p})_{y}=\operatorname*{argmax}_{y\in\mathcal{Y}}P_{t}(y|f_{\mathsf{zs}}(\mathbf{x}))$
(33)

Again, using Lemma 1, any other classifier $f_{h}(\mathbf{x})$ has higher risk
than $f_{\mathsf{zs}}(\mathbf{x})-\pi_{p}$, _i.e._ ,
$\mathcal{R}_{t}(f_{\mathsf{zs}}-\pi_{p})\leq\mathcal{R}_{t}(f_{h}(\mathbf{x}))$.
∎

##  Appendix B Experimental Details

###  B.1 Dataset details

Many-shot and few-shot datasets. For many-shot learning, we use ImageNet,
CIFAR100, StanfordCars and SUN397 datasets. For few-shot learning, we evaluate
models on 15 datasets. The details of each dataset are presented in Table 7.

Dataset | Classes | Train size | Test size | Task
---|---|---|---|---
ImageNet | 1,000 | 1.28M | 50,000 | Object-level
CIFAR100 | 100 | 50,000 | 10,000 | Object-level
Caltech101 | 100 | 4,128 | 2,465 | Object-level
DTD | 47 | 2,820 | 1,692 | Textures
EuroSAT | 10 | 13,500 | 8,100 | Satellite images
FGVCAircraft | 100 | 3,334 | 3,333 | Fine-grained aircraft
Flowers102 | 102 | 4,093 | 2,463 | Fine-grained flowers
Food101 | 101 | 50,500 | 30,300 | Fine-grained food
OxfordPets | 37 | 2,944 | 3,669 | Fine-grained pets
StanfordCars | 196 | 6,509 | 8,041 | Fine-grained car
SUN397 | 397 | 15,880 | 19,850 | Scene-level
UCF101 | 101 | 7,639 | 3,783 | Action
ImageNetV2 | 1,000 | - | 10,000 | Robustness to collocation
ImageNet-Sketch | 1000 | - | 50,889 | Robustness to sketch domain
ImageNet-A | 200 | - | 7,500 | Robustness to adversarial attack
ImageNet-R | 200 | - | 30,000 | Robustness to multi-domains
Table 7: The detailed statistics of datasets for many-shot and few-shot
learning.

Long-tail datasets. We use two standard long-tail benchmarks: Places365-LT and
ImageNet-LT [31]. The skewness of a long-tailed training set is typically
represented by imbalanced ratio, which is defined as $N_{\max}/N_{\min}$.
$N_{\max}$ ($N_{\min}$) denotes the largest (smallest) number of instances per
class. A larger imbalanced ratio means a more imbalanced training set. The
test sets are divided into three splits: many-shot subset contains classes
with $>100$ images, medium-shot subset includes classes with $\geq 20$ & $\leq
100$ images, and few-shot subset covers classes with $<20$ images. Details are
listed in Table 8.

Dataset |  | Size of
---
all classes
| Size of
---
many classes
| Size of
---
medium classes
| Size of
---
few classes
| Size of
---
training samples
| Imbalanced
---
ratio
Places365-LT | 365 | 131 | 163 | 71 | 62.5K | 996
ImageNet-LT | 1000 | 385 | 479 | 136 | 186K | 256
Table 8: Details of long-tailed datasets.

###  B.2 CLIP zero-shot

We use prompt ensembling of 80 prompts provided by CLIP [48] for ImageNet,
CIFAR100, and Caltech101 to improve performance, _i.e._ , averaging the text
embedding of many captions, _e.g._., “a photo of a {$c_{k}$}.” and “an image
of a {$c_{k}$}.”. For OxfordPets, StanfordCars, Flowers102, Food101,
FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, we use the pre-defined prompt
from CoOp [54].

###  B.3 Fine-tuned models

End-to-end and linear probe fine-tuning. We follow WiSE-FT [48] to implement
fine-tuning. We initialize the classifier with the zero-shot classifier and
the output of the image encoder $\Phi_{\mathsf{v}}$ is normalized during fine-
tuning. We fine-tune for a total of 10 epochs using AdamW [32] optimizer with
default hyper-parameters $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$ and
weight decay $0.1$. We choose a batch size of 512. We use the same data
augmentation and cosine-annealing learning rate schedule as [48].

###  B.4 Prompt tuning.

Prompt tuning like CoOp [54] automates prompt engineering by learning the
prompt given few samples from downstream tasks. CoOp provides two options of
prompt design: unified prompt that is shared among all classes and class-
specific prompt that is different for each class. In this paper, we adopt the
class-specific prompt design as the fine-tuned model to implement GLA . In
specific, given the word embedding $\mathbf{t}_{k}^{0}$ initialized by zero-
shot prompts, we aim to learn a collection of class-specific word embedding
$\mathbf{R}=\\{\mathbf{r}_{k}\\}_{k=1}^{K}$, such that the text input
$\mathbf{t}_{k}=\mathbf{t}_{k}^{0}+\mathbf{r}_{k}$ minimizes the empirical
risk:
$\mathbf{R}^{*}=\operatorname*{argmin}_{\mathbf{R}}\mathbb{E}_{\mathbf{x},y}[y\neq\operatorname*{argmax}_{i}f(x;\mathbf{R})_{i}]$.

We adhere CoOp to use CLIP ResNet-50 as image encoder for few-shot
classification. The word embedding $\mathbf{R}$ is initialized from zeros. For
the $m$ few-shot classification setting (where $m\in\\{1,2,4,8,16\\}$), we
randomly sample $m$ training and $m$ validation points from the respective
full datasets. For all few-shot datasets except ImageNet, the training epoch
is set to 200 for 16/8 shots, 100 for 4/2 shots, and 50 for 1 shot. For
ImageNet, the epoch is set to 50 for all shots. We fine-tune the prompt with
SGD optimizer decayed by the cosine annrealing rule. The base initial learning
rate and batch size are set to $10^{-4}$ and $32$. When given an $m$-shot
sample setting, we increase the learning rate and batch size by $m$ times
simultaneously to accelerate the training speed.

###  B.5 Estimation of the class prior

To estimate the log-probability of the pre-training distribution
$\hat{\pi}_{s}=\log\mathbf{q}$, we utilize the optimization toolkit Cooper
[15] from https://github.com/cooper-org/cooper. $\mathbf{q}$ is initialized as
a uniformed distribution, $\mathbf{q}(y)=\frac{1}{K}$ for all $y\in[K]$. We
use the standard SGD as the primal and dual optimizers for 2000 steps.

###  B.6 Long-tail learning baselines and training details

We compared with 5 long-tailed classification methods:

  * 1.

Standard ERM: We learn the model by standard empirical risk minimization on
the long-tailed data.

  * 2.

Learnable Weight Scaling (LWS) [24]: We first learn the model by standard ERM,
then fix the model and learn to re-scale the magnitude of the classifier using
class-balanced sampling.

  * 3.

Logit Adjustment (LA) [35]: We first learn the model by standard ERM, then
compensates the long-tailed distribution by subtracting a class-dependent
offset to the model outputs.

  * 4.

Balanced Softmax (BS) [43] modifies the Softmax cross-entropy loss which
explicitly accommodate the label distribution shift during optimization.

  * 5.

BALLAD [33] first fine-tunes the vision-language models via contrastive loss
on long-tailed data, then freezes the backbone and finally employs an adapter
to enhance the representations of tail classes with re-sampling strategies.

For all combinations of the fine-tuning baselines and long-tailed learning
methods, visual backbones are initialized from CLIP-ResNet-50 and all
classifiers are initialized by feeding prompt with class names to the text
encoder. We use SGD for all experiments with a momentum of 0.9 for 50 epochs
with batch size of 512. The initial learning rate is set to $1.6\times
10^{-3}$ which is decayed by the cosine annealing rule. To mitigate explosive
gradients, we use the warmup learning rate equals to $10^{-5}$ during the
first epoch. For the sake of fairness in comparison, all hyper-parameters of
baselines are carefully searched using grid search on the validation set.

##  Appendix C Additional Experiments

###  C.1 Few-shot learning accuracy

We provide mean and standard deviation in Table 9 in for {1, 2, 4, 8, 16}
shots on all 11 few-shot learning datasets.

Dataset | 1 shot | 2 shots | 4 shots | 8 shots | 16 shots
---|---|---|---|---|---
ImageNet |  61.65 $\pm$ 0.15 |  62.64 $\pm$ 0.01 |  63.32 $\pm$ 0.07 |  64.51 $\pm$ 0.09 |  65.61 $\pm$ 0.03
Caltech101 |  89.08 $\pm$ 0.09 |  90.25 $\pm$ 0.25 |  90.98 $\pm$ 0.43 |  91.90 $\pm$ 0.21 |  92.58 $\pm$ 0.42
OxfordPets |  87.79 $\pm$ 0.15 |  87.86 $\pm$ 0.21 |  88.22 $\pm$ 0.21 |  88.09 $\pm$ 0.27 |  89.53 $\pm$ 0.16
StanfordCars |  60.00 $\pm$ 0.14 |  63.10 $\pm$ 0.42 |  66.25 $\pm$ 0.19 |  69.87 $\pm$ 0.09 |  73.95 $\pm$ 0.11
Flowers102 |  73.45 $\pm$ 0.60 |  81.00 $\pm$ 0.46 |  88.31 $\pm$ 0.65 |  92.89 $\pm$ 0.46 |  95.41 $\pm$ 0.32
Food101 |  78.41 $\pm$ 0.07 |  78.62 $\pm$ 0.07 |  78.68 $\pm$ 0.06 |  78.85 $\pm$ 0.19 |  79.54 $\pm$ 0.47
FGVCAircraft |  20.22 $\pm$ 0.59 |  22.09 $\pm$ 0.37 |  24.65 $\pm$ 0.85 |  28.23 $\pm$ 0.44 |  31.99 $\pm$ 0.50
SUN397 |  64.29 $\pm$ 0.19 |  66.32 $\pm$ 0.16 |  68.01 $\pm$ 0.08 |  69.99 $\pm$ 0.18 |  71.64 $\pm$ 0.21
DTD |  47.38 $\pm$ 1.23 |  50.75 $\pm$ 1.46 |  56.90 $\pm$ 0.20 |  62.73 $\pm$ 0.80 |  65.78 $\pm$ 0.49
EuroSAT |  56.50 $\pm$ 1.34 |  67.26 $\pm$ 3.58 |  72.40 $\pm$ 2.43 |  77.59 $\pm$ 1.84 |  84.93 $\pm$ 1.89
UCF101 |  65.32 $\pm$ 0.17 |  68.42 $\pm$ 0.81 |  70.88 $\pm$ 0.50 |  74.23 $\pm$ 0.24 |  76.07 $\pm$ 0.03
Table 9: GLA Accuracy (%) with standard deviation of few-shot learning on 11
datasets.

###  C.2 Experiments on LAION-400M

To support our thought experiment in the discussion of Section 4.3, we use the
Open-CLIP ViT-B/16 [21], the first 20k image in LAION-400M datasets and the
bias estimation method proposed by [2] to estimate the expected logits across
8 classes: “dog”, “cat”, “squirrel”, “tiger”, “elephant”, “horse”, “pig” and
“bird”. The bias estimation proposed by [2] provides a good estimation of
$\log P(\mathbf{x})$ over the labels under pre-training distribution. Our GLA
estimates the label bias matches the downstream domain, we consider two
downstream domain styles, _i.e._ , “photo” and “sketch” from DomainNet [38]
dataset. For each domain, we randomly sampled 50 images for each class.

Method | dog | cat | squirrel | tiger | elephant | horse | pig | bird
---|---|---|---|---|---|---|---|---
expected logits by [2] | 0.059 | 0.039 | 0.043 | 0.053 | 0.043 | 0.062 | 0.061 | 0.028
$\pi_{p}$ by “photo” | 0.059 | 0.041 | 0.047 | 0.055 | 0.048 | 0.062 | 0.060 | 0.033
$\pi_{p}$ by “sketch” | 0.059 | 0.043 | 0.063 | 0.061 | 0.067 | 0.042 | 0.047 | 0.056
Table 10: Comparison among different bias estimation using Open-CLIP-ViT-B/16. Method | Sketch | Real Photo
---|---|---
Zero-shot Open-CLIP | 92.25 | 97.00
Debiased by [2] | 89.50 | 97.50
Debiased by GLA | 93.00 | 97.75
Table 11: Performance on sketch and real photo domain.

We present the expected logits estimated by [2], along with the one calculated
from “photo” and “sketch” downstream domain data in Table 10. Since the
softmax is invariant to constant offsets, _i.e._ ,
$\mathrm{softmax}(\mathbf{x}+c)=\mathrm{softmax}(\mathbf{x})$, we align the
three label bias to yield the same logits on “dog” class by subtracting
constant values. We observe that the label bias estimated by the "photo"
domain align closely with [2] due to its close resemblance to the pre-trained
domain. Conversely, the "sketch" image style, which significantly differs from
the pre-training domain, results in a more pronounced deviation in the pre-
trained label bias.

Additionally, we apply the three estimated label biases to debias the zero-
shot model and evaluate the classification performance. The results are shown
in Table 11, where the superiority of our method becomes evident on “sketch”
domain (93.00% vs 92.25%). Applying the label bias from [2] on the "sketch"
domain degrades the model’s performance (from 92.25% to 89.50%). This is
attributed to the overall pre-training label bias does not adequately reflect
the bias specific to the“sketch” domain.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  NeurIPS, 2022.
  * [2] James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan.  A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models.  In ICML, 2023.
  * [3] Martin Arjovsky.  Out of distribution generalization in machine learning.  PhD thesis, New York University, 2020.
  * [4] Eric Bauer and Ron Kohavi.  An empirical comparison of voting classification algorithms: Bagging, boosting, and variants.  Machine learning, 1999.
  * [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  Technical Report, 2021.
  * [6] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In ECCV, 2014.
  * [7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [8] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang.  Prompt learning with optimal transport for vision-language models.  ICLR, 2023.
  * [9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In CVPR, 2014.
  * [10] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In CVPR, 2019.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [12] Thomas G Dietterich.  Ensemble methods in machine learning.  In Multiple Classifier Systems: First International Workshop, 2000\.
  * [13] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In CVPRW, 2004.
  * [14] Yoav Freund and Robert E Schapire.  A decision-theoretic generalization of on-line learning and an application to boosting.  Journal of computer and system sciences, 1997.
  * [15] Jose Gallego-Posada and Juan Ramirez.  Cooper: a toolkit for Lagrangian-based constrained optimization.  https://github.com/cooper-org/cooper, 2022.
  * [16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [17] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In CVPR, 2021.
  * [18] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In CVPR, 2021.
  * [19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  NeurIPS Workshop, 2014.
  * [20] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang.  Disentangling label distribution for long-tailed visual recognition.  In CVPR, 2021.
  * [21] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip.  Technical report, Zenodo, jul 2021.
  * [22] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  UAI, 2018.
  * [23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [24] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  In ICLR, 2020.
  * [25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In CVPRW, 2013.
  * [26] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [27] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan.  Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift.  In UAI. PMLR, 2022.
  * [28] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  NeurIPS, 2017.
  * [29] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi.  Ensemble distillation for robust model fusion in federated learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NeurIPS, 2020.
  * [30] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola.  Detecting and correcting for label shift with black box predictors.  In ICML, 2018.
  * [31] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, 2019.
  * [32] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [33] Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  A simple long-tailed recognition baseline via vision-language model.  arXiv preprint arXiv:2111.14745, 2021.
  * [34] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [35] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.  Long-tail learning via logit adjustment.  In ICLR, 2021.
  * [36] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
  * [37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [38] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In ICCV, 2019.
  * [39] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al.  Combined scaling for zero-shot transfer learning.  arXiv preprint arXiv:2111.10050, 2021.
  * [40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [41] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In ICML, 2019.
  * [42] William J Reed.  The pareto, zipf and other power laws.  Economics letters, 2001.
  * [43] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al.  Balanced meta-softmax for long-tailed visual recognition.  NeurIPS, 2020.
  * [44] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [45] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang.  Long-tailed classification by keeping the good and removing the bad momentum causal effect.  NeurIPS, 2020.
  * [46] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.  Measuring robustness to natural distribution shifts in image classification.  NeurIPS, 2020.
  * [47] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  NeurIPS, 2019.
  * [48] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In CVPR, 2022.  https://arxiv.org/abs/2109.01903.
  * [49] Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin.  Adversarial robustness under long-tailed distribution.  CVPR, 2021.
  * [50] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In CVPR, 2010.
  * [51] Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, and Hanwang Zhang.  Invariant training 2d-3d joint hard samples for few-shot point cloud recognition.  In ICCV, 2023.
  * [52] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  TMLR, 2022.
  * [53] Tong Zhang.  Statistical behavior and consistency of classification methods based on convex risk minimization.  The Annals of Statistics, 2004.
  * [54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  IJCV, 2022.
  * [55] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang.  Prompt-aligned gradient for prompt tuning.  In ICCV, 2023.
  * [56] Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang.  Debiased fine-tuning for vision-language models by prompt regularization.  AAAI, 2023.


# AILAB-Udine@SMM4H’22:
Limits of Transformers and BERT Ensembles

Beatrice Portelli
<EMAIL_ADDRESS>University of Udine, Italy University of
Naples Federico II , Italy Simone Scaboro
<EMAIL_ADDRESS>University of Udine, Italy Emmanuele Chersoni
<EMAIL_ADDRESS>The Hong Kong Polytechnic University, Hong
Kong Enrico Santus
<EMAIL_ADDRESS>DSIG - Bayer Pharmaceuticals, New Jersey, USA Giuseppe Serra
<EMAIL_ADDRESS>University of Udine, Italy

###### Abstract

This paper describes the models developed by the AILAB-Udine team for the
SMM4H’22 Shared Task. We explored the limits of Transformer based models on
text classification, entity extraction and entity normalization, tackling
Tasks 1, 2, 5, 6 and 10. The main takeaways we got from participating in
different tasks are: the overwhelming positive effects of combining different
architectures when using ensemble learning, and the great potential of
generative models for term normalization.

##  1 Introduction

Transformer-based models are the backbone of state-of-the-art solutions for a
lot of NLP tasks. The real strength of these models (like BERT Devlin et al.,
2018, GPT Radford et al., 2019, and their variants Joshi et al., 2019; Gu et
al., 2020) stands in the pre-training phase which permits them to have
extensive language knowledge. This is particularly helpful in tasks where the
amount of training data is restricted, like the ones addressed in this
workshop (Weissenbacher et al., 2022).

In this work we used a variety of pretrained Transformers models for the
tasks. We refer to Table 1 for a summary of their names in the Huggingface
library and the shorthand version of their name used in this report.

Short name |  Model name in the Huggingface library | Reference
---|---|---
GPT-2 |  gpt2 | (Radford et al., 2019)
BERTEng |  bert-base-uncased | (Devlin et al., 2018)
BERTMul |  bert-base-multilingual-uncased | (Devlin et al., 2018)
BERTMed |  microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract | (Gu et al., 2020)
BERTSpan |  SpanBERT/spanbert-base-cased | (Joshi et al., 2019)
RoBERTaTwi |  cardiffnlp/twitter-roberta-base-sentiment | (Barbieri et al., 2020)
RoBERTaXML |  xlm-roberta-base | (Conneau et al., 2019)
Table 1: List of pretrained models used for the tasks and the abbreviations
used in this report.

##  2 Simple Classification (Task 5 / 6)

Task 5 and 6 both entail the simple classification of tweets (binary or
ternary) in a class-unbalanced setting. Task 5 consists in the ternary
classification of Spanish tweets about COVID-19 symptoms as: containing
literature/news reports (News, 60%), containing personal reports (Pers, 16%)
or reporting about someone else’s symptoms (Non-Pers, 24%). Task 6 consists in
the binary classification of English tweets regarding COVID-19 vaccinations
as: general vaccine chatter (Chatter, 89%) or personal reports confirming the
vaccination status of the user (Pers, 11%). For both tasks the text
preprocessing consisted in replacing all usernames with “user” and all URLs
with “(see url)” or “(ver url)” (“(see url)” in Spanish).

Figure 1: Summary of the model architectures used in the different tasks.

###  2.1 Models

We tackled the tasks using a simple Transformer-based model with a
classification head, that is a linear layer applied to the output embedding of
the [CLS] token (see Figure 1a). This layer maps the embedding to either two
or three classes depending on the task. The kind of pretrained model was
chosen based on the characteristics of the input text, such as language. For
each task we selected two kinds of pretrained models with different
characteristics to try and combine them, and ensembled their predictions. The
selected models were: two BERTMul and one RoBERTaXML for Task 5, two BERTMul
and one BERTEng for Task 6. They were chosen by training and evaluating 5
models of each kind locally on a 70-30 random split of the training data, and
selecting the ones with the higher performance on their respective test fold.
In Task 5, BERTMul models were trained for 10 epochs while RoBERTaXML models
for 15 epochs. In Task 6, BERTMul models were trained for 5 epochs while
BERTEng models for 4 epochs. The final label for the task was chosen via
majority vote.

###  2.2 Results

Table 2 shows the results of the base models and their ensemble for both tasks
on the validation set.

Task | Model | P | R | F1
---|---|---|---|---
5 | BERTMul (1) | 0.826 | 0.826 | 0.826
5 | BERTMul (2) | 0.833 | 0.833 | 0.833
5 | RoBERTaXML | 0.823 | 0.823 | 0.823
5 | Ensemble | 0.838 | 0.838 | 0.838
6 | BERTMul (1) | 0.875 | 0.734 | 0.799
6 | BERTMul (2) | 0.946 | 0.748 | 0.835
6 | BERTEng | 0.928 | 0.715 | 0.807
6 | Ensemble | 0.954 | 0.741 | 0.834
Table 2: Task 5 and Task 6 results on the validation set.

Ensembling different model typologies had a positive effect on Task 5, as the
overall performance is higher than any model on its own. Looking at the
confusion matrices in Figure 2, we see that most of the improvements come from
a better accuracy in classifying Pers samples and distinguishing them from
Non-Pers ones. The precision on Pers class goes from 0.68 (single models) to
0.72 (Ensemble) and the percentage of Pers samples classified as Non-Pers
lowers from 0.27 to 0.23.

BERTMul (1)

BERTMul (2)

RoBERTaXML

Ensemble

Agreement

Figure 2: Task 5. Confusion matrices for the three separate models and their
ensemble, and agreement matrix.

As regards Task 6, the Ensemble has a higher precision than each individual
model, but a lower recall. BERTMul (2) had significantly higher metrics
compared to the other two models, and the majority vote might have favored the
most frequent (incorrect) prediction of the other two models. Looking at the
confusion matrices in Figure 3, we can see that the Ensemble model has a
higher precision on the most frequent class (Chatter) compared to the single
models, but the two weaker models severely hampered the performance of BERTMul
(2).

BERTMul (1) BERTEng

BERTMul (2) Ensemble

Agreement

Figure 3: Task 6. Confusion matrices for the three separate models and their
ensemble, and agreement matrix.

The main differences between the models ensembled for Task 5 and Task 6 is
that the models used for Task 5 had different base architectures (BERT vs
RoBERTa), while the ones for Task 6 were all based on BERT. If we calculate
the agreements between the models using Cohen’s Kappa Cohen, 1960, we see that
the models used for Task 6 had higher agreement than the ones in Task 5
(compare the agreement matrices in Figures 2 and 3). The lower agreement for
Task 5 is likely caused by the use of different model architectures, and it
might lead to higher performance when combining the predictions.

##  3 Multitask Classification (Task 2a+2b)

Task 2a and 2b (Davydova and Tutubalina, 2022) consist in the classification
of English tweets containing opinions about mandates during the COVID-19
pandemic. The tweets can deal with three topics: Face Masks (M), Stay At Home
Orders (H) and School Closures (S). Task 2a is a ternary stance classification
(Against, None, Favor), while Task 2b is a binary premise classification (1,
0) to determine whether the tweet is argumentative or not.

###  3.1 Models

We use the same architecture to solve both tasks, reformulating them as a
single 6-way classification with labels: Against-1, Against-0, None-1, None-0,
Favor-1 and Favor-0. We use a simple Transformer-based model with a
classification head on top of the [CLS] embedding. The output of the
classification head is a probability distribution over the six labels. At
inference time, the probabilities are aggregated in three or two classes
according to the task (e.g., Against=Against-1 + Against-0 for Task 2a or 1 =
Against-1 + None-1 + Favor-1 for Task 2b). This process is illustrated in
Figure 1b.

The text preprocessing is the same as Task 6. The input for the models was
formatted as “About CLAIM. [SEP] TWEET_TEXT”, where CLAIM is one of the three
topics. We finetuned a BERTEng model for 4 epochs on all training data. To
increase the robustness of the model for Task 2a, we also finetuned two
RoBERTaTwi models for 5 epochs on the three-way classification
Against/None/Favor, leveraging the model’s pretrained weights for sentiment
classification on Twitter. We then ensemble the predictions of the two
RoBERTaTwi models and the BERTEng model for Task 2a (similarly to Task 5/6).

###  3.2 Results

Table 3 reports the metrics for Task 2a and 2b on the validation set. The
Ensemble for Task 2a achieves higher metrics compared to the single models in
two out of the three topics (M and H), as well as on the overall F1 score. The
F1 score of the single models differ up to 7-9 points between each other, yet
their interaction leads to a higher score overall. Figure 4 shows the
agreement between the three models, which is even lower than the one recorded
for Task 5. This further strengthens the hypothesis that using different
architectures and models with high disagreements leads to an ensemble with
higher performance.

Task | Model | F1M | F1S | F1H | F1
---|---|---|---|---|---
2a | RoBERTaTwi (1) | 0.840 | 0.677 | 0.819 | 0.779
2a | RoBERTaTwi (2) | 0.816 | 0.700 | 0.821 | 0.779
2a | BERTEng | 0.749 | 0.610 | 0.742 | 0.700
2a | Ensemble | 0.855 | 0.722 | 0.807 | 0.795
2b | BERTEng | 0.786 | 0.818 | 0.814 | 0.806
Table 3: Task 2a and 2b results on the validation set. Agreement  Figure 4:
Task 2a. Agreement matrix for the three models and their ensemble.

##  4 Disease Extraction (Task 10)

Task 10 (Gasco et al., 2022) consists in extracting disease mentions from
Spanish tweets.

###  4.1 Models

We solved this task using a simple Transformer-based model with a token-
classification head, that is a linear layer applied to the output embedding of
each token (see Figure 1c) with three output classes. These represent the BIO
tagging scheme (Begin-Inside-Outside), commonly used to mark the presence of
Named Entities in NER tasks. This straightforward method has been previously
used in SMM4H Tasks for ADE extraction (Portelli et al., 2022), and we were
interested in testing if it was possible to adapt it to Disease Extraction
with minimal changes. The text preprocessing is the same as Task 5. We used a
BERTMul model (with multilingual pretraining) and trained it for 5 epochs on
the training data of SocialDisNER without using additional resources.

###  4.2 Results

Table 4 reports the strict and relaxed metrics on the validation set. There is
a gap of 40 points between the two, which is almost double what is usually
reported in ADE extraction tasks. This means that the model was able to
identify the broad area of text containing the disease, but not to pinpoint
it. This goes to show that a Disease extraction system needs more mechanisms
in place to precisely extract the relevant text.

Task | Model | P | R | F1
---|---|---|---|---
10 | BERTMul (Strict) | 0.543 | 0.498 | 0.520
10 | BERTMul (Relaxed) | 0.946 | 0.865 | 0.904
Table 4: Task 10 results on the validation set.

##  5 ADE Normalization (Task 1c)

Task 1c consists in mapping ADE mentions from English tweets to their
corresponding MedDRA terms (formal medical terms). The dataset (Magge et al.,
2021) was developed with three sub-tasks in mind, so to perform Task 1c it is
necessary to complete the two preliminary Tasks 1a (binary classification
ADE/noADE) and 1b (ADE extraction).

###  5.1 Models

Task 1a and Task 1b were not the main focus of our work, so we tackled them
with simple and effective strategies seen in other tasks. Task 1a was solved
with a BERTMed model with a binary classification head, trained as seen in
Task 6, without model ensembling. For Task 1b we used a model previously
developed for the same task the SMM4H’19 Shared Task Portelli et al. (2021,
2022). It consists of a BERTSpan for token classification (see model for Task
10) combined with a Conditional Random Field (CRF) module (see Figure 1c).

For Task 1c, we used a GPT-2 model, trained to take as input a ADE and
generate the string corresponding to the correct MedDRA term (e.g., “feel like
crap” → “malaise”). GPT-2 was trained on the whole training set for 15 epochs.

###  5.2 Results

Table 5 reports the results of the models on the validation set. The models
for Tasks 1a and 1b achieve average performance. We report the metrics for
Task 1c in two ways: calculating them on the output of BERTSpan (same
procedures used on the blind test set); and using an oracle for Task 1b (that
is, giving as input to GPT-2 only the correct ADEs). Using the oracle, we see
that the model developed for Task 1c has a very high accuracy (0.759) if given
the correct ADEs. Applying GPT-2 to the predictions of BERTSpan leads to lower
metrics due to the low quality of the preceding steps. The proposed model for
Task 1c also performed extremely well on the blind test set, where it achieved
results well over the average despite the low performance reached on Task 1b
(see Table 6).

Task | Model | P | R | F1
---|---|---|---|---
1a | BERTMed | 0.663 | 0.477 | 0.544
1b | BERTSpan | 0.295 | 0.851 | 0.438
1c | GPT-2 (from BERTSpan) | 0.219 | 0.632 | 0.325
1c | GPT-2 (from oracle) | 0.759 | 0.759 | 0.759
Table 5: Task 1 results on the validation set.

##  6 Results on the Test Set

The following table reports the metrics of all presented models on the test
set, together with the reference scores supplied by organizers. Models were
not re-trained using validation data, with the exception of Tasks 1a and 1b.

|  | Strict | Relaxed
---|---|---|---
Task | Model | P | R | F1 | P | R | F1
1a | Our | .607 | .386 | .472 |  |  |
1a | Average | .646 | .497 | .562 |  |  |
1b | Our | .360 | .254 | .298 | .489 | .344 | .404
1b | Average | .344 | .339 | .341 | .539 | .517 | .527
1c | Our | .243 | .171 | .201 | .294 | .207 | .243
1c | Average | .085 | .082 | .083 | .120 | .112 | .116
2a | Our |  |  | .529 |  |  |
2a | Average |  |  | .491 |  |  |
2a | Median |  |  | .550 |  |  |
2b | Our |  |  | .649 |  |  |
2b | Average |  |  | .574 |  |  |
2b | Median |  |  | .647 |  |  |
5 | Our | .840 | .840 | .840 |  |  |
5 | Median | .840 | .840 | .840 |  |  |
5 | Baseline | .900 | .900 | .900 |  |  |
6 | Our | .930 | .750 | .830 |  |  |
6 | Median | .900 | .680 | .770 |  |  |
6 | Baseline | .900 | .770 | .830 |  |  |
10 | Our | .504 | .461 | .481 |  |  |
10 | Average | .680 | .677 | .675 |  |  |
10 | Median | .758 | .780 | .761 |  |  |
Table 6: Results for all tasks on the blind test set.

##  7 Conclusions

We explored the use of simple Transformer-based architectures for several
tasks proposed by SMM4H’22. The most noticeable phenomena we encountered were:
the collaborative effect of different architectures (e.g., BERT and RoBERTa)
when used in ensemble learning (Task 2 and 5, as opposed to Task 6); the
efficacy of generative models for term normalization (Task 1c); and the low
transferability of methods developed for ADE extraction to Disease detection
(Task 10).

## References

  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  TweetEval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650, Online. Association for Computational Linguistics.
  * Cohen (1960) Jacob Cohen. 1960.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ , 20(1):37–46.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised Cross-lingual Representation Learning at Scale.  _CoRR_ , abs/1911.02116.
  * Davydova and Tutubalina (2022) Vera Davydova and Elena Tutubalina. 2022.  Smm4h 2022 task 2: Dataset for stance and premise detection in tweets about health mandates related to covid-19.  In _Proceedings of the Seventh Social Media Mining for Health Applications (#SMM4H) Workshop & Shared Task_, pages –.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  * Gasco et al. (2022) Luis Gasco, Darryl Estrada-Zavala, Eulàlia Farré-Maduell, Salvador Lima-López, Antonio Miranda-Escalada, and Martin Krallinger. 2022.  Overview of the SocialDisNER shared task on detection of diseases mentions from healthcare related and patient generated social media content: methods, evaluation and corpora.  In _Proceedings of the Seventh Social Media Mining for Health (#SMM4H) Workshop and Shared Task_.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.
  * Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.
  * Magge et al. (2021) Arjun Magge, Elena Tutubalina, Zulfat Miftahutdinov, Ilseyar Alimova, Anne Dirkson, Suzan Verberne, Davy Weissenbacher, and Graciela Gonzalez-Hernandez. 2021\.  DeepADEMiner: a deep learning pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on Twitter.  _Journal of the American Medical Informatics Association_ , 28(10):2184–2192.
  * Portelli et al. (2021) Beatrice Portelli, Edoardo Lenzi, Emmanuele Chersoni, Giuseppe Serra, and Enrico Santus. 2021.  Bert prescriptions to avoid unwanted headaches: a comparison of transformer architectures for adverse drug event detection.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1740–1747.
  * Portelli et al. (2022) Beatrice Portelli, Daniele Passabì, Edoardo Lenzi, Giuseppe Serra, Enrico Santus, and Emmanuele Chersoni. 2022.  _Improving Adverse Drug Event Extraction with SpanBERT on Different Text Typologies_, pages 87–99. Springer International Publishing, Cham.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Weissenbacher et al. (2022) Davy Weissenbacher, Ari Z. Klein, Luis Gascó, Darryl Estrada-Zavala, Martin Krallinger, Yuting Guo, Yao Ge, Abeed Sarker, Ana Lucia Schmidt, Raul Rodriguez-Esteban, Mathias Leddin, Arjun Magge, Juan M. Banda, Vera Davydova, Elena Tutubalina, and Graciela Gonzalez-Hernandez. 2022.  Overview of the seventh social media mining for health applications #smm4h shared tasks at coling 2022.  In _Proceedings of the Seventh Social Media Mining for Health Applications (#SMM4H) Workshop & Shared Task_, pages –.