# Distribution Normalization: An “Effortless” Test-Time Augmentation for
Contrastively Learned Visual-language Models

Yifei Zhou  Juntao Ren  Fengyu Li  Ramin Zabih  Ser-Nam Lim

###### Abstract

Advances in the field of visual-language contrastive learning have made it
possible for many downstream applications to be carried out efficiently and
accurately by simply taking the dot product between image and text
representations. One of the most representative approaches proposed recently
known as CLIP (Radford et al., 2021) has quickly garnered widespread adoption
due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into
account both positive and negative samples to help learn a much more robust
representation space. This paper however reveals that the common downstream
practice of taking a dot product is only a zeroth-order approximation of the
optimization goal, resulting in a loss of information during test-time.
Intuitively, since the model has been optimized based on the InfoNCE loss,
test-time procedures should ideally also be in alignment. The question lies in
how one can retrieve any semblance of negative samples information during
inference. We propose Distribution Normalization (DN), where we approximate
the mean representation of a batch of test samples and use such a mean to
represent what would be analogous to negative samples in the InfoNCE loss. DN
requires no retraining or fine-tuning and can be effortlessly applied during
inference. Extensive experiments on a wide variety of downstream tasks exhibit
a clear advantage of DN over the dot product. Our code is available at
https://github.com/fengyuli2002/distribution-normalization.

Machine Learning, ICML, Contrastive Learning, Multi-modal Learning

##  1 Introduction

Contrastive Learning (Jaiswal et al., 2020; Chen et al., 2020a; Tian et al.,
2020; Chen et al., 2020d; Robinson et al., 2020) has become an important area
of research in recent years due to its wild success in self-supervised
representation learning. The key idea behind contrastive learning is to group
positive samples that share similar semantics (e.g., images resulting from
applying different augmentations to the same image), while separating
semantically different negative samples (e.g., different images with different
data augmentations).

On the heels of this progress, contrastive learning has also been shown to be
very effective for creating a joint representation space between images and
text (Radford et al., 2021; Patashnik et al., 2021; Zhou et al., 2022;
Dhariwal & Nichol, 2021). The CLIP approach proposed in (Radford et al., 2021)
is at the forefront of this research effort and has been shown to be the
state-of-the-art in multiple downstream tasks. The model is generally pre-
trained on a large-scale dataset of image-text pairs that form the positive
samples, while the negative samples are composed from random pairing of images
and text. After the model is trained, it becomes fairly straightforward to
perform cross modal retrieval: given an image, its representation from the
model can be used to retrieve text representations from the same model and
vice versa. Consequently, such a paradigm has similarly opened the doors to
state-of-the-art performance in zero-shot classification (Zhai et al., 2021;
Jia et al., 2021), cross-modal retrieval (Yu et al., 2022; Lu et al., 2021; Li
et al., 2020b, 2019), and evaluation of machine-generated captions (Hessel et
al., 2021; Jiang et al., 2019; Lee et al., 2020).

Figure 1: An illustration of how our proposed Distribution Normalization
better aligns with the contrastive training objective. Image encoder and Text
encoder are omitted for simplicity.

With all the success mentioned, we however observe a “mismatch” between the
pre-training objective of such cross-modal representation models and their
downstream usage. Most if not all of these models are trained taking into
considering of both positive and negative samples, of which one of the most
popular optimization objectives is the InfoNCE loss (Sohn, 2016; van den Oord
et al., 2018; Zhang et al., 2020). The InfoNCE loss considers negative samples
from the data distribution per positive sample and has been shown to be very
effective for cross-modal representation learning. In contrast, during test-
time, the retrieval is often conducted using the dot product between the query
and the gallery, which does not utilize any information in regards to the data
distribution.

This paper presents our findings that rectifying such misalignment can boost
performance consistently across a variety of downstream tasks. We particularly
analyze the InfoNCE objective and first show that the common practice of
taking a dot product is only a zeroth-order approximation. We further show
that by approximating the mean of the testing distribution and incorporating
it into the similarity measure better approximates the InfoNCE objective used
during training. We call our proposed approach Distribution Normalization (DN)
– DN is “effortless” to implement and does not require any retraining, fine-
tuning or labeled data.

To investigate how this simple modification affects the performance of CLIP
and its state-of-the-art variants in practice, we conduct extensive
experiments on a wide variety of tasks involving measuring cross-modal
similarities, including zeroshot classification, cross-modal retrieval, and
evaluation of machine-generated captions. In all the settings that we have
studied, DN displays a clear benefit over direct dot product. Our
contributions are: (1) We present an insightful analysis of why the dot
product as a similarity measure is only a zeroth-order approximation; (2) We
propose DN to overcome this shortcoming by utilizing an approximation of the
test-time distribution, and also provide guidance on how DN can be
effortlessly implemented in practice; (3) We provide extensive supporting
empirical evidence as well as ablation studies that demonstrate the fidelity
of DN for practical use.

##  2 Background & Related Works

###  2.1 Contrastive Learning

Contrastive learning has become a popular method for self-supervised
representation learning, where the goal is to learn embedding functions such
that semantically similar samples are closer in embeddings space while
semantically dissimilar samples are embedded further apart (Tian et al., 2019;
Wang & Isola, 2020; Peng et al., 2022; You et al., 2021; Yuan et al., 2021;
Grill et al., 2020). Among various contrastive learning methods, this paper
specifically focuses on the InfoNCE loss, which optimizes the probability of
correctly identifying the positive sample (van den Oord et al., 2018), as it
has been one of the most frequently used objectives in recent contrastive
learning tasks (He et al., 2019; Chen et al., 2020c, a, b).

###  2.2 Learning via Natural Language Supervision

Several recent studies (Chen et al., 2019; Gan et al., 2020; Li et al., 2020a,
b) leverage large-scale datasets of image-text pairs to supervise vision-
language pre-training (VLP) with a contrastive objective. CLIP (Radford et
al., 2021) is one such example that is contrastively trained to create a joint
visual-language representation space, which has produced empirical success in
downstream tasks such as zero-shot classification and cross-modal retrieval
over previous state-of-the-art models (Lu et al., 2021; Wang et al., 2020;
Yang et al., 2022; Li et al., 2021). Since CLIP’s introduction, several other
works have utilized a similar contrastive objective to create good visual-
language representations and achieve state-of-the-art performance on many
downstream tasks involving datasets that the model was not pre-trained on (Li
et al., 2021; Jia et al., 2021; Chen et al., 2019; Faghri et al., 2017; Kim et
al., 2021). For example, ALIGN scales joint pre-training to minimally-filtered
datasets (Jia et al., 2021), while ALBEF (Li et al., 2021) and TCL (Yang et
al., 2022) focus on learning more semantically meaningful multi-modal
interactions. It is noteworthy that these SOTA models all leverage the InfoNCE
loss as their key training objective, and apply a simple dot product in
downstream tasks.

###  2.3 Image Caption Evaluation

Image caption evaluation is the task of automatically scoring the quality of a
caption given the corresponding image and optionally reference captions. The
quality of image caption metrics are evaluated with respect to its correlation
with human ratings. While traditional image caption metrics such as BLEU-4
(Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee & Lavie, 2005),
CIDEr (Vedantam et al., 2014), and SPICE (Anderson et al., 2016) evaluate the
candidate caption by measuring the n-gram overlap of the candidate with the
references, more contemporary neural metrics (Jiang et al., 2019; Yi et al.,
2020; Kane et al., 2020; Lee et al., 2020; Cui et al., 2018) judge the
candidate caption by comparing the neural embeddings of the caption with that
of the image and references, and are much more flexible. Among them, the one
most related to our work is CLIPScore (Hessel et al., 2021). By simply passing
test image and caption pairs through the pre-trained CLIP image and text
encoders and taking the dot product, CLIPScore achieves state-of-the-art
performance using only the candidate caption and the corresponding image
without the need for references.

##  3 Methodology

This section is organized as follows: in Section 3.1, we show that a dot
product is only a zeroth-order approximation of the pre-training objective and
how a better approximation arises naturally from the InfoNCE objective, and in
Section 3.2, we explain how this idea can be effortlessly implemented in
practice.

###  3.1 Analysis of InfoNCE Loss

We start from the original form of InfoNCE loss (Sohn, 2016; van den Oord et
al., 2018; Zhang et al., 2020):

$\displaystyle\mathcal{L}_{NCE}=\mathbb{E}_{x_{0},y_{0}\sim D}$
$\displaystyle[\mathbb{E}_{y_{1},...,y_{n}\sim
D}-\log\frac{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}+\sum_{i=1}^{n}e^{\phi(x_{0})^{\intercal}\psi(y_{i})/\tau}}$
$\displaystyle-\mathbb{E}_{x_{1},...,x_{n}\sim
D}\log\frac{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}+\sum_{i=1}^{n}e^{\phi(x_{i})^{\intercal}\psi(y_{0})/\tau}}],$
(1)

where $D$ is the training distribution, $x_{0},y_{0}$ are images and text
belonging to the same pair, $x_{1},...,x_{n}$ and $y_{1},...,y_{n}$ are random
images and text in the same batch, $\phi,\psi$ are image and text encoders
respectively, and $\tau$ is a temperature constant.

InfoNCE loss is equivalent to:

$\displaystyle\mathcal{L}_{NCE}=\mathbb{E}_{x_{0},y_{0}\sim
D}[\mathbb{E}_{y_{1},...,y_{n}\sim
D}\log\frac{1}{1+\frac{\sum_{i=1}^{n}e^{\phi(x_{0})^{\intercal}\psi(y_{i})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}}$
$\displaystyle+\mathbb{E}_{x_{1},...,x_{n}\sim
D}\log\frac{1}{1+\frac{\sum_{i=1}^{n}e^{\phi(x_{i})^{\intercal}\psi(y_{0})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}}].$
(2)

Since
$\frac{\sum_{i=1}^{n}e^{\phi(x_{i})^{\intercal}\psi(y_{0})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}$
is usually quite small ($<0.1$) due to the exponentiation, we can taylor
expand the above term around 0 and discard 2 or higher order terms, resulting
in

$\displaystyle\mathcal{L}_{NCE}\approx$
$\displaystyle\mathbb{E}_{x_{0},y_{0}\sim D}[\mathbb{E}_{y_{1},...,y_{n}\sim
D}\frac{\sum_{i=1}^{n}e^{\phi(x_{0})^{\intercal}\psi(y_{i})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}$
$\displaystyle+\mathbb{E}_{x_{1},...,x_{n}\sim
D}\frac{\sum_{i=1}^{n}e^{\phi(x_{i})^{\intercal}\psi(y_{0})/\tau}}{e^{\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}}],$
(3)

which can be simplified to

$\displaystyle\mathcal{L}_{NCE}\approx$ $\displaystyle
n\mathbb{E}_{x_{0},y_{0}\sim D}[\mathbb{E}_{y_{1}\sim
D}e^{\phi(x_{0})^{\intercal}[\psi(y_{1})-\psi(y_{0})]/\tau}$
$\displaystyle+\mathbb{E}_{x_{1}\sim
D}e^{[\phi(x_{1})-\phi(x_{0})]^{\intercal}\psi(y_{0})/\tau}].$ (4)

####  3.1.1 Reduction to Dot Product

We argue that Eqn. 4 can be reduced to straightforwardly comparing the dot
products. When we have no knowledge of the distribution $D$, the only thing
that we can do is to perform a zeroth-order approximation of Eqn. 4. The most
intuitive way of doing so is to reduce the distribution of $\phi(x_{1})$ and
$\psi(y_{1})$ to be 0 deterministically, so that:

$\displaystyle\mathcal{L}_{NCE}^{(0)}=2n\mathbb{E}_{x_{0},y_{0}\sim
D}e^{-\phi(x_{0})^{\intercal}\psi(y_{0})/\tau}.$ (5)

This shows that we can essentially just output
$e^{-\phi(x_{0})^{\intercal}\psi(y_{0})}$ as the distance between $x_{0}$ and
$y_{0}$. Since in most applications, only relative comparisons of the
distances matter (e.g. finding the best caption describing the image), it is
the same as the negative dot product without the exponentiation.

####  3.1.2 Efficient Approximation

The common practice of taking a naive zeroth-order approximation can result in
a loss of some of the important information. On the other hand, computing Eqn.
4 for each pair $x_{0},y_{0}$ involves iterating over all samples in a
distribution $D$, and is computationally inefficient. We therefore turn to
more efficient approximations using statistics of higher-order moments of the
distribution. In our experiments, we found that this intuitive modification
maintains most of the useful information as shown in Appendix 4.7.

If we know the first moment of the distribution, i.e., the mean of the
unlabeled distribution $\mu_{x}$ and $\mu_{y}$ for $\phi(x_{1})$ and
$\psi(y_{1})$, we can perform a first-order approximation of the distribution
with $\widehat{P}(\phi(x_{1}))=\mathbb{I}\\{\phi(x_{1})=\mu_{x}\\}$ and
$\widehat{P}(\psi(y_{1}))=\mathbb{I}\\{\psi(y_{1})=\mu_{y}\\}$, so that
$\widehat{P}(\phi(x_{1})),\widehat{P}(\psi(y_{1}))$ matches the true
distribution in terms of the first-order moment. This simplifies Eqn. 4 to be:

$\displaystyle\mathcal{L}_{NCE}^{(1)}=$
$\displaystyle\mathbb{E}_{x_{0},y_{0}\sim
D}[e^{\phi(x_{0})^{\intercal}(\mu_{y}-\psi(y_{0}))/\tau}+e^{(\mu_{x}-\phi(x_{0}))^{\intercal}\psi(y_{0})/\tau}].$
(6)

Therefore, to evaluate the distance of any $x_{0},y_{0}$, we can simply output
$e^{\phi(x_{0})^{\intercal}(\mu_{y}-\psi(y_{0}))/\tau}+e^{(\mu_{x}-\phi(x_{0}))^{\intercal}\psi(y_{0})/\tau}$
without having to calculate an expectation by iterating through all samples in
the distribution.

###  3.2 Effortless Implementation as Distribution Normalization

In our preliminary experiments, we found that
$e^{\phi(x_{0})^{\intercal}(\mu_{y}-\psi(y_{0}))/\tau}+e^{(\mu_{x}-\phi(x_{0}))^{\intercal}\psi(y_{0})/\tau}$
correlates well with the simpler form
$-(\phi(x_{0})-\frac{1}{2}\mu_{x})^{\intercal}(\psi(y_{0})-\frac{1}{2}\mu_{y})$
after removing the exponentiation, which now introduces a constant coefficient
of 0.5 to the means. For more flexibility, we replace it with $\lambda$ so
that it can be set empirically, and define the new similarity measure as:

$\displaystyle
S(x_{0},y_{0})=(\phi(x_{0})-\lambda\mu_{x})^{\intercal}(\psi(y_{0})-\lambda\mu_{y}).$
(7)

In all our experiments, we fix $\lambda=0.25$ as we found that to be the
optimal value empirically. Note that the new similarity measure is equivalent
to subtracting the means immediately after the original representations are
calculated, a method we call Distribution Normalization (DN), after which the
similarity is just their dot product. Clearly, our proposal is extremely easy
to implement when access to the mean of the distribution of interest is
available.

Fortunately, we observed that the mean of the distribution can usually be
robustly estimated from very few unlabeled samples as shown in Section 4.6.1.
This is easily accessible in all the downstream tasks that we have considered
in this paper and is much cheaper than fine-tuning where not only more data is
required but also the data needs to be annotated. For example, in the task of
cross-modal retrieval, the mean can be estimated from the gallery data and
past queries (without knowing the correct answer to the queries); in the task
of zero-shot classification, the mean can be estimated from a small and
unlabeled validation set; and in the task of image caption metrics, the mean
can be estimated from the images to be captioned and the references provided.

##  4 Experiments

Our experiments are designed to answer the following questions: 1) whether our
proposed DN can uniformly improve a wide range of cross-modal alignment tasks
for different kinds of cross-modal representation models, 2) how robust DN is
when only scarce, unlabeled data is available to estimate the mean from, and
3) whether DN can improve the performance of fine-tuned models in addition to
pre-trained models.

###  4.1 Downstream Tasks

Cross-modal Retrieval includes two subtasks where images are used to query
corresponding text and vice versa. Following (Yang et al., 2022; Li et al.,
2021), for each subtask, we consider both the zero-shot setting and the fine-
tuning setting on COCO (Lin et al., 2014) and Flickr30K (Plummer et al.,
2015). In the zero-shot setting, we directly apply the pre-trained model on
the test set of each dataset while in the fine-tuned setting we first fine-
tune each pre-trained model on the training set of each dataset with the
InfoNCE loss and apply the fine-tuned model to the test set. For each setting,
we report Recall@k (k = 1,5) whereby the retrieval is considered a success if
its corresponding image/text is in the top-k.

Zero-shot Classification is the task where we directly apply the pre-trained
models to predict the category of the images in the test set. Each class is
represented with a fixed textual template of “a photo of {class_name}.”, and
the corresponding representation is obtained by encoding those sentences with
the text encoder of the pre-trained model. For each image, we encode it with
the image encoder and output the top-k most similar class representations as
predictions. We benchmark all methods with top-1 and top-5 accuracy on
ImageNet1K (Deng et al., 2009), Cifar100 (Krizhevsky, 2009), SUN397 (Xiao et
al., 2010), and Stanford Cars (Krause et al., 2013) datasets.

Image Captioning Metric is the task that measures the correlation between
human ratings and similarity metrics on machine-generated captions. Following
(Hessel et al., 2021), we consider both the reference-free setting and the
reference-based setting. In the reference-free setting, the metric does not
have access to human-written example captions (references) and has to give a
score based on the source image and the generated caption alone while in the
reference-based setting the metric can additionally take advantage of the
information from the provided references.

###  4.2 Base Representation Models

This paper examines the effect of a Distribution Normalization layer on the
following state-of-the-art open-sourced cross-modal representation models:

CLIP (Radford et al., 2021) is one of the earliest pioneers that successfully
created a joint representation space of images and text through large-scale
contrastive learning. It is trained on over 400M image-caption pairs collected
by the authors. They have open-sourced multiple versions of pretrained models
and this paper examines the commonly used version ”ViT-B/32”.

ALBEF (Li et al., 2021) is one of the later variants of CLIP that adds a
multi-modal encoder to capture the interplay between images and text by
predicting if they are paired samples or hard negatives. Additional momentum
distillation loss and masked language modeling loss are also added to improve
the model performance. Two versions of pre-trained models trained on dataset
consisting of 4M and 14M unique images are released and this work uses the
latter. To focus on the effect of DN on cross-modal representations, we only
keep the image and text encoder component of ALBEF and do not use the multi-
modal encoder component.

TCL (Yang et al., 2022) is another state-of-the-art CLIP variant. It inherits
the same model architecture as ALBEF, but adds triplet contrastive loss
including Cross-modal Alignment, Intra-modal Contrastive, and Local MI
Maximization. Their model is trained on 4M unique images. As with ALBEF, we
only keep the image and the text encoder component.

###  4.3 Cross-modal Retrieval Results

We conducted experiments on both image-to-text and text-to-image retrieval on
Microsoft COCO (Lin et al., 2014) and Flicker30K (Plummer et al., 2015) (see
Appendix A.1 for datasets details). For all the retrieval tasks, we estimated
the mean with 100 random unlabeled samples from the validation set and
calculated standard deviations and average recalls with 5 random seeds.

The results for both datasets are presented in Table 1 for the zero-shot
setting. For all three base models, adding DN almost always improves recall
for both zero-shot image-to-text and text-to-image retrieval tasks, with an
average 1.1% increase in top-1 recall. This provides substantial evidence that
DN is effective for both tasks even in datasets with diverse images like
MSCOCO. The improvement can be as large as a 2.2% increase in top-1 image-to-
text recall as we see for CLIP on Flickr30K. We also observe that, in general,
adding DN brings a larger improvement for top-1 recall than for top-5/10
recall, suggesting that DN is particularly suitable for distinguishing similar
samples. We additionally report DN’s effectiveness when we fine-tune the base
models on target datasets as an ablation in section 4.6.2.

| MSCOCO (5K test set) | Flickr30K (1K test set)
---|---|---
| Image $\rightarrow$ Text | Text $\rightarrow$ Image | Image $\rightarrow$ Text | Text $\rightarrow$ Image
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
CLIP (Radford et al., 2021) | $52.4$ | $76.0$ | $84.5$ | $30.2$ | $55.1$ | $66.4$ | $81.3$ | $95.0$ | $98.5$ | $62.7$ | $86.0$ | $92.0$
CLIP + DN | $\textbf{52.9}\tiny{\pm 0.1}$ | $\textbf{76.4}\tiny{\pm 0.1}$ | $\textbf{84.9}\tiny{\pm 0.1}$ | $\textbf{32.1}\tiny{\pm 0.1}$ | $\textbf{57.4}\tiny{\pm 0.0}$ | $\textbf{68.3}\tiny{\pm 0.1}$ | $\textbf{83.5}\tiny{\pm 0.1}$ | $\textbf{96.2}\tiny{\pm 0.0}$ | $98.5\tiny{\pm 0.1}$ | $\textbf{64.8}\tiny{\pm 0.2}$ | $\textbf{87.5}\tiny{\pm 0.1}$ | $\textbf{93.1}\tiny{\pm 0.0}$
TCL (Yang et al., 2022) | $57.6$ | $84.3$ | $91.8$ | $41.8$ | $70.6$ | $80.6$ | $73.8$ | $93.3$ | $96.9$ | $59.1$ | $84.6$ | $91.1$
TCL + DN | $\textbf{59.5}\tiny{\pm 0.1}$ | $\textbf{85.2}\tiny{\pm 0.0}$ | $\textbf{92.2}\tiny{\pm 0.1}$ | $\textbf{42.7}\tiny{\pm 0.0}$ | $\textbf{71.5}\tiny{\pm 0.0}$ | $\textbf{81.3}\tiny{\pm 0.0}$ | $\textbf{75.5}\tiny{\pm 0.0}$ | $\textbf{94.4}\tiny{\pm 0.1}$ | $96.9\tiny{\pm 0.1}$ | $\textbf{60.0}\tiny{\pm 0.1}$ | $\textbf{85.1}\tiny{\pm 0.0}$ | $91.1\tiny{\pm 0.0}$
ALBEF (Li et al., 2021) | $62.5$ | $85.9$ | $92.2$ | $40.2$ | $68.4$ | $78.9$ | $78.2$ | $95.5$ | $97.9$ | $59.9$ | $84.8$ | $90.6$
ALBEF +DN | $\textbf{63.0}\tiny{\pm 0.1}$ | $\textbf{86.0}\tiny{\pm 0.0}$ | $\textbf{92.5}\tiny{\pm 0.1}$ | $\textbf{42.8}\tiny{\pm 0.1}$ | $\textbf{70.8}\tiny{\pm 0.0}$ | $\textbf{80.7}\tiny{\pm 0.0}$ | $\textbf{79.2}\tiny{\pm 0.1}$ | $\textbf{96.2}\tiny{\pm 0.0}$ | $\textbf{98.0}\tiny{\pm 0.0}$ | $\textbf{62.4}\tiny{\pm 0.1}$ | $\textbf{86.1}\tiny{\pm 0.1}$ | $\textbf{91.9}\tiny{\pm 0.1}$

Table 1: Cross-modal retrieval performance on MSCOCO and Flickr30K in the
zero-shot setting. Means for DN are estimated using 100 random unlabeled
validation samples. Average recalls and standard deviations are calculated
with 5 random seeds.

###  4.4 Zero-shot Classification Results

Table 2 gives the zero-shot classification results for all three base models
and their variants with DN on ImageNet1K (Deng et al., 2009), Cifar100
(Krizhevsky, 2009), SUN397 (Xiao et al., 2010), and Stanford Cars (Krause et
al., 2013) (see Appendix A.2 for datasets details). In these experiments, we
perform DN before the final norm normalization layer of image/text encoders.
DN boosted the performance of all three base models on all four datasets, with
an average 1.9% boost on top-1 classification accuracy. Our DN method displays
great robustness as it brings performance boost in 11 out of all 12 model-
dataset combinations in Table 2. Our results show that DN is effective in not
only general-purpose vision datasets such as the ImageNet but also special-
purpose datasets like the Stanford Cars, where the models still get (on
average) 0.5% increase in accuracy. Notably, TCL saw the largest gain of 2.95%
on average in top-1 accuracy and 3.3% on average in top-5 accuracy. On
ImageNet1K particularly, TCL’s top-1 accuracy is improved from 20.0% to 25.1%
by adding DN. We believe the fact that TCL and ALBEF are pre-trained on a
smaller dataset (4M & 14M images) than CLIP (400M images) resulted in TCL and
ALBEF getting a larger improvement than CLIP, as DN may have helped ease the
instability when the model is not adequately trained. All these improvements
are achieved with a single layer that incurs minimal computational overhead.

| ImageNet1K | Cifar100 | SUN397 | Stanford Cars
---|---|---|---|---
| Acc@1 | Acc@5 | Acc@1 | Acc@5 | Acc@1 | Acc@5 | Acc@1 | Acc@5
CLIP (Radford et al., 2021) | $61.0$ | $87.4$ | $63.9$ | $88.7$ | $56.1$ | $89.4$ | $58.6$ | 90.9
CLIP + DN | $\textbf{61.7}\pm 0.1$ | $\textbf{87.8}\pm 0.0$ | $\textbf{65.1}\pm 0.1$ | $\textbf{89.4}\pm 0.0$ | $\textbf{57.3}\pm 0.0$ | $\textbf{90.2}\pm 0.1$ | $58.6\pm 0.1$ | $90.7\pm 0.0$
TCL (Yang et al., 2022) | $20.0$ | $42.9$ | $39.1$ | $68.2$ | $28.6$ | $63.4$ | $2.0$ | $8.7$
TCL + DN | $\textbf{25.1}\pm 0.1$ | $\textbf{48.9}\pm 0.2$ | $\textbf{42.2}\pm 0.0$ | $\textbf{71.1}\pm 0.1$ | $\textbf{31.8}\pm 0.1$ | $\textbf{66.8}\pm 0.2$ | $\textbf{2.4}\pm 0.0$ | $\textbf{9.5}\pm 0.1$
ALBEF (Li et al., 2021) | $37.3$ | $65.8$ | $38.5$ | $65.7$ | $45.6$ | $81.5$ | $25.0$ | $61.0$
ALBEF + DN | $\textbf{38.7}\pm 0.1$ | $\textbf{67.3}\pm 0.0$ | $\textbf{44.8}\pm 0.2$ | $\textbf{72.0}\pm 0.2$ | $\textbf{46.3}\pm 0.1$ | $\textbf{82.0}\pm 0.0$ | $\textbf{26.0}\pm 0.1$ | $\textbf{61.7}\pm 0.2$

Table 2: Zero-shot classification performance on ImageNet1K, Cifar100, SUN397
and Stanford Cars. Means for DN are estimated using 100 random unlabeled
validation samples. Average accuracies and standard deviations are calculated
with 5 random seeds.

###  4.5 Image Captioning Metric Results

Inspired by the success of CLIPScore (Hessel et al., 2021) as a metric for
image captioning, we conduct experiments to test whether DN can help various
cross-modal representation models align better with human judgments. To
compare the performance of different methods, we report the Kendall’s
$\tau_{b}$ or $\tau_{c}$ coefficients against human judgments on
Flickr8k-expert, Flickr8k-cf (Hodosh et al., 2013) and THumb (Kasai et al.,
2021) in Table 3, following the standard practice in (Hessel et al., 2021). We
also report the resulting accuracy for caption-caption preferences on
Pascal-50S (Rashtchian et al., 2010) in Table 4. Details about these datasets
are given in appendix A.3. In the reference-based setting, for CLIP-ref, we
follow (Hessel et al., 2021) in using an averaged score given by:

$\displaystyle\text{CLIP-ref}(x_{0},y_{0})=$
$\displaystyle\text{H-mean}(\phi(x_{0})^{\intercal}\psi(y_{0}),\max_{c\in
R}\psi(c)^{\intercal}\psi(y_{0})),$

where H-mean is the harmonic mean and $R$ is the set of provided references.
On the other hand, for a distribution normalized CLIP + DN-ref, we use a
weighted algorithmic mean instead:

$\displaystyle\text{CLIP + DN-ref}(x_{0},y_{0})=$
$\displaystyle\text{A-mean}(S(x_{0},y_{0}),0.25*\max_{c\in
R}\psi(c)^{\intercal}\psi(y_{0})),$

where A-mean is the algorithmic mean, $R$ is the set of provided references,
and $S$ is defined in Eqn. 7.

In Table 3, we report the results on Flickr8K-Expert, Flicker8K-CF, and THumb
for all three base models. Some methods, such as the BLEU score, are not
applicable in the reference-free setting and are thus only compared with in
the reference-based setting. The results strongly support that adding DN
significantly improves the $\tau$ correlation for all three base models on all
dataset in the reference-free setting. In particular, CLIP + DN improves CLIP
by 2.2% on Flickr8k-expert, 0.8% on Flickr8k-cf and 2.3 % on THumb. In the
reference-based setting, to give a better context, we also provide the
performance of other reference-based SOTA metrics that are not contrastively
trained and found that CLIP and CLIP-ref outperform all of them. However, by
simply adding a DN layer, CLIP-ref + DN again yields an average 1.1% gain
across the board. A similar pattern is also observed in our experiment results
on Pascal-50S in Table 4. Our main results on cross-modal retrieval, zero-shot
classification, and image caption metrics provide extensive evidence that our
proposed DN is a widely applicable modification that can improve the
performance of cross-modal representation models across a variety of
downstream applications.

Figure 2: Comparison of the effects of fine-tuning between CLIP and CLIP + DN
for Acc@1. Results on MSCOCO’s 5k test set are shown to the left, and
Flickr30K’s 1k test set are shown to the right. For each set, image-to-text
retrieval (left) and image-to-text retrieval (right) are reported. For each of
the 5 checkpoints we trained, we find its average accuracy and standard
deviation with another 5 iterations random sampling for mean estimation, and
plot the mean of these 5 accuracies and standard deviations from 5
independently fine-tuned checkpoints.

|  | Flickr8k-expert | Flickr8k-cf | THumb
---|---|---|---|---
|  | $\tau_{c}$ | $\tau_{b}$ | $\tau_{c}$
Ref-free | CLIP (Hessel et al., 2021) | 51.4 | 34.3 | 19.9
CLIP + DN | 53.2 | 35.1 | 22.2
TCL (Yang et al., 2022) | 31.0 | 20.6 | 8.1
TCL + DN | 36.0 | 23.3 | 11.1
| ALBEF (Li et al., 2021) | 24.9 | 15.4 | 0.9
| ALBEF + DN | 29.2 | 18.1 | 2.5
Ref-based | BLEU-1 | 32.3 | 17.9 | 11.1
BLEU-4 | 30.8 | 16.9 | 6.9
CIDEr (Vedantam et al., 2014) | 43.9 | 24.6 | 13.8
NUBIA* (Kane et al., 2020) | 49.5 | - | -
ViLBERTScore-F (Yi et al., 2020) | 50.1 | - | -
CLIP-ref (Hessel et al., 2021) | 53.0 | 36.4 | 24.7
CLIP + DN-ref | 54.3 | 36.9 | 26.2

Table 3: Image captioning metric results on Flickr8k-Expert, Flickr8k-CF, and
THumb.

| HC | HI | HM | MM | Mean
---|---|---|---|---|---
CLIP (Hessel et al., 2021) | 55.0 | 99.2 | 96.9 | 71.8 | 80.7
CLIP + DN | 55.6 | 99.3 | 97.3 | 73.6 | 81.4
TCL (Yang et al., 2022) | 52.4 | 91.8 | 54.7 | 63.4 | 65.6
TCL + DN | 52.7 | 93.8 | 59.0 | 64.3 | 67.4
ALBEF (Li et al., 2021) | 56.9 | 98.1 | 82.5 | 67.2 | 76.2
ALBEF + DN | 56.9 | 98.1 | 81.7 | 67.8 | 76.1
BLEU-4 | 60.4 | 90.6 | 84.9 | 54.7 | 72.6
CIDEr (Vedantam et al., 2014) | 65.1 | 98.1 | 90.5 | 64.8 | 79.6
ViLBERTScore-F (Yi et al., 2020) | 49.9 | 99.6 | 93.1 | 75.8 | 79.6
CLIP -ref (Hessel et al., 2021) | 62.4 | 99.7 | 96.7 | 73.0 | 83.0
CLIP + DN -ref | 61.1 | 99.7 | 97.3 | 74.8 | 83.2

Table 4: Accuracy results on Pascal-50S given different categories of caption-
caption pairs. HC means two correct human-generated captions. HI means two
human-generated captions with one incorrect. HM means a human-generated and a
machine-generated caption. MM means two machine-generated captions.

###  4.6 Ablation Study

####  4.6.1 Number of Samples for Mean Estimation

We study the performance of DN when different numbers of samples are used to
estimate the mean. We observe that DN consistently improves performance for
CLIP, TCL, and ALBEF even when using as few as 10 unlabeled samples. Using the
ImageNet1K, Cifar100, SUN397, and Stanford Cars datasets, we first test the
retrieval accuracy of DN by estimating the mean with the entire test set. We
use the performance achieved here as an upper bound, because the mean is most
reliable if it is directly calculated on the unlabeled testing samples. We
compare it with the estimated means using 1, 10, and 100 random samples from
the training set, and present our results in Table 5.

As shown in the table, decreasing the number of samples to just 100 from the
training set results in an accuracy drop of no more than 0.2% compared to the
upperbound, with an average of only 0.09%. We find that even decreasing the
number of samples to as few as 10 does not signficantly hurt the result. For
10 samples, there is an average drop in accuracy of 0.14%, with the greatest
difference of no more than 0.4%. Moreover, for CLIP, TCL, and ALBEF, we find
that estimating the mean with as few as 10 samples still improved accuracy
over the base model by an average of 1.96% for Acc@1, and 1.92% for Acc@5.
Furthermore, our results are generally stable despite the low number of sample
points used to estimate the mean. Across 5 random samplings, the standard
deviation of our accuracy is on average 0.08% for 100 training set samples,
and 0.22% for 10 training set samples. Even in the pathological case of just 1
sample, we still observe a slight improvement of 1.15% over the base model
averaged across Acc@1 and Acc@5, but with a slightly larger standard deviation
of 0.77%. These ablation results show that DN can generalize very well even in
the setting of extremely scarce data, possibly because DN does not overfit to
the unlabeled data.

| ImageNet1K | Cifar100 | SUN397 | Stanford Cars
---|---|---|---|---
| Acc@1 | Acc@5 | Acc@1 | Acc@5 | Acc@1 | Acc@5 | Acc@1 | Acc@5
CLIP (Radford et al., 2021) | 61.0 | 87.4 | 63.9 | 88.7 | 56.1 | 89.4 | 58.6 | 90.9
CLIP + DN (test) | 61.8 | 87.9 | 65.1 | 89.4 | 57.3 | 90.1 | 58.7 | 90.7
CLIP + DN (100) | 61.7 $\pm$ 0.0 | 87.8 $\pm$ 0.0 | 65.1 $\pm$ 0.1 | 89.4 $\pm$ 0.0 | 57.3 $\pm$ 0.0 | 90.2 $\pm$ 0.1 | 58.6 $\pm$ 0.1 | 90.7 $\pm$ 0.0
CLIP + DN (10) | 61.6 $\pm$ 0.1 | 87.8 $\pm$ 0.1 | 65.1 $\pm$ 0.2 | 89.2 $\pm$ 0.1 | 57.1 $\pm$ 0.1 | 90.1 $\pm$ 0.1 | 58.3 $\pm$ 0.1 | 90.7 $\pm$ 0.1
CLIP + DN (1) | 60.6 $\pm$ 0.4 | 87.2 $\pm$ 0.2 | 64.1 $\pm$ 0.6 | 88.4 $\pm$ 0.6 | 56.9 $\pm$ 0.5 | 89.7 $\pm$ 0.2 | 56.5 $\pm$ 0.4 | 89.7 $\pm$ 0.5
TCL (Yang et al., 2022) | 20.0 | 42.9 | 39.1 | 68.2 | 28.6 | 63.4 | 2.0 | 8.7
TCL + DN (test) | 25.4 | 49.4 | 42.2 | 71.1 | 31.9 | 66.9 | 2.4 | 9.5
TCL + DN (100) | 25.1 $\pm$ 0.1 | 48.9 $\pm$ 0.2 | 42.2 $\pm$ 0.0 | 71.1 $\pm$ 0.1 | 31.8 $\pm$ 0.1 | 66.8 $\pm$ 0.2 | 2.4 $\pm$ 0.0 | 9.5 $\pm$ 0.1
TCL + DN (10) | 25.3 $\pm$ 0.1 | 49.1 $\pm$ 0.2 | 42.2 $\pm$ 0.2 | 70.9 $\pm$ 0.1 | 31.9 $\pm$ 0.4 | 66.8 $\pm$ 0.5 | 2.3 $\pm$ 0.1 | 9.5 $\pm$ 0.0
TCL + DN (1) | 25.5 $\pm$ 0.3 | 49.2$\pm$ 0.3 | 40.8 $\pm$ 0.7 | 69.4 $\pm$ 0.9 | 31.4 $\pm$ 0.5 | 65.8 $\pm$ 0.9 | 2.3 $\pm$ 0.1 | 9.5 $\pm$ 0.3
ALBEF (Li et al., 2021) | 37.3 | 65.8 | 38.5 | 65.7 | 45.6 | 81.5 | 25.0 | 61.0
ALBEF + DN (test) | 38.7 | 67.4 | 45.0 | 72.1 | 46.2 | 81.9 | 26.0 | 61.8
ALBEF + DN (100) | 38.7 $\pm$ 0.1 | 67.3 $\pm$ 0.0 | 44.8 $\pm$ 0.2 | 72.0 $\pm$ 0.2 | 46.3 $\pm$ 0.1 | 82.0 $\pm$ 0.0 | 26.0 $\pm$ 0.1 | 61.7 $\pm$ 0.2
ALBEF + DN (10) | 38.7 $\pm$ 0.1 | 67.3 $\pm$ 0.1 | 45.1 $\pm$ 0.6 | 72.2 $\pm$ 0.5 | 45.9 $\pm$ 0.3 | 81.7 $\pm$ 0.2 | 25.8 $\pm$ 0.3 | 61.3 $\pm$ 0.6
ALBEF + DN (1) | 37.3 $\pm$ 1.2 | 65.6 $\pm$ 1.4 | 44.3 $\pm$ 2.8 | 71.1 $\pm$ 3.1 | 45.4 $\pm$ 0.3 | 81.4 $\pm$ 0.6 | 25.0 $\pm$ 0.6 | 60.1 $\pm$ 1.2

Table 5: Ablation study on the effect of the number of samples to estimate the
mean for ImageNet1K, Cifar100, SUN397 and Stanford Cars. Parenthesized is the
number of unlabeled samples from the training set. (test) means that DN is
conducted on the entire unlabeled test set. Mean and standard deviation are
calculated based on 5 random samplings.

####  4.6.2 Effect of fine-tuning on the results

We also perform ablations on the effects of DN on fine-tuned models. We fine-
tune CLIP on the MSCOCO training set, for a total of 10 epochs on 4 Nvidia
2080Ti. We use the Adam optimizer with a learning rate of 1e-5 and a weight
decay of 0.1. We then evaluated each checkpoint on the same MSCOCO test split
used in Table 1, and estimate the mean for CLIP + DN using 100 randomly
sampled validation points. A total of 5 checkpoints were trained, and we
evaluated each checkpoint for 5 iterations. For each checkpoint, we calculate
the average accuracy and standard deviation, then plot the mean of the
accuracies and standard deviations over all 5 checkpoints. We repeat this
process for Flickr30K, using the same number of epochs, checkpoints, and model
of GPUs. We again use the Adam optimizer with a learning rate of 1e-5, but
with a weight decay of 0.02. We record accuracy scores at 1, 5, and 10 for
both image-to-text retrieval and text-to-image retrieval on the two datasets.
We present Acc@1 here, and leave Acc@5 and Acc@10 in Apendix B.1.

The following analysis focuses specifically on the case of Acc@1 for MSCOCO,
but the general trend is also present in Flickr30K. As seen in Figure 2, fine-
tuning allows for an improvement in accuracy over the zero-shot setting
initially by 8.06% and 9.47% for CLIP + DN and CLIP respectively in the text-
to-image case, and 6.02% and 4.88% in the image-to-text case in the first
epochs. But the fine-tuning performance gradually decays after that due to
overfitting to the training set. At their highest peak, CLIP + DN achieves
43.06% accuracy compared to CLIP’s 42.39% for text-to-image retrieval, and
59.82% compared to 59.57% for image-to-text retrieval. We observe for both
image-to-text retrieval and text-to-image retrieval that CLIP + DN performs
better than CLIP across all fine-tuning epochs by an average of 0.80% for
text-to-image, and 0.60% for image-to-text case.

A similar pattern is observed in Flickr30K. At their respective peaks, CLIP +
DN achieves 72.80% accuracy compared to CLIP’s 72.64% for text-to-image
retrieval, and 87.88% compared to 87.10% for image-to-text retrieval. On
average, CLIP + DN performs better than CLIP across all fine-tuning epochs by
0.33% for text-to-image, and 1.02% for image-to-text case. Our experiments on
the fine-tuning setting provide an affirmative answer to the last question we
seek to answer, that DN provides a non-trivial performance boost to both pre-
trained and fine-tuned cross-modal representation models.

###  4.7 Comparison of Simplified Equation and Full Equation

To investigate how much information might be lost in the process of our first-
moment approximation of Eqn. 4, we carry out experiments to compare DN and its
original form in Eqn. 4 and present our results in Table 6 in terms of their
correlation with human judgments on image captioning datasets. Recall that in
Section 3.2, we have set $\lambda=0.25$ based on empirical observations. To be
fair, we need to also incorporate that into Eqn. 4, resulting in:

$\displaystyle S(x_{0},y_{0})=$ $\displaystyle\mathbb{E}_{y_{1}\sim
D}e^{\phi(x_{0})^{\intercal}[0.5*\psi(y_{1})-\psi(y_{0})]/\tau}$
$\displaystyle+\mathbb{E}_{x_{1}\sim
D}e^{[0.5*\phi(x_{1})-\phi(x_{0})]^{\intercal}\psi(y_{0})/\tau}.$ (8)

Note that this objective is extremely inefficient in practice because to
calculate the similarity between every pair $x_{0},y_{0}$, Eqn. 8 requires
iterating over all validation samples (images and references in this case)
while DN only needs to compute the mean once and does not need to access the
validation samples after that. Surprisingly, in all the base models and
datasets that we have studied, we did not notice any significant difference
between DN and Eqn. 8 (difference $<0.1\%$). This shows that higher-order
information contributes negligibly to the downstream applications compared to
only taking the mean, and DN provides equivalent performance without incurring
expensive computational costs.

| Flickr8k-expert | Flickr8k-cf | THumb
---|---|---|---
| $\tau_{c}$ | $\tau_{b}$ | $\tau_{c}$
CLIP (Radford et al., 2021; Hessel et al., 2021) | 51.4 | 34.3 | 19.9
CLIP + DN | 53.2 | 35.1 | 22.2
CLIP + DN (Eqn.8) | 53.2 | 35.1 | 22.2
TCL (Yang et al., 2022) | 31.0 | 20.6 | 8.1
TCL + DN | 36.0 | 23.3 | 11.1
TCL + DN (Eqn.8) | 36.0 | 23.3 | 11.1
ALBEF (Li et al., 2021) | 24.9 | 15.4 | 0.9
ALBEF + DN | 29.2 | 18.1 | 2.5
ALBEF + DN (Eqn.8) | 29.2 | 18.1 | 2.5

Table 6: Abaltion study on comparison with distribution normalization and full
Eqn.4 on Flickr8k-Expert, Flickr8k-CF, and THumb.

##  5 Conclusion

In this paper, we identify a mismatch between the pre-training objective of
cross-modal representation models like CLIP and its downstream use with a
direct dot product. To address this problem, we propose Distribution
Normalization (DN) that not only displays a significant advantage over direct
dot product in a wide variety of settings but is also straightforward to
implement. However, though being extremely sample efficient, DN still requires
estimating a mean separately for each distribution and it can be of interest
if a universal mean can be found so that DN is more easily applied to a wider
range of downstream applications. Future research can also explore the
implications of DN to the contrastive training process of cross-modal
representation models.

## References

  * Anderson et al. (2016) Anderson, P., Fernando, B., Johnson, M., and Gould, S.  SPICE: semantic propositional image caption evaluation.  _CoRR_ , abs/1607.08822, 2016.  URL http://arxiv.org/abs/1607.08822.
  * Banerjee & Lavie (2005) Banerjee, S. and Lavie, A.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pp. 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.  URL https://aclanthology.org/W05-0909.
  * Chen et al. (2020a) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E.  A simple framework for contrastive learning of visual representations.  _CoRR_ , abs/2002.05709, 2020a.  URL https://arxiv.org/abs/2002.05709.
  * Chen et al. (2020b) Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E.  Big self-supervised models are strong semi-supervised learners.  _CoRR_ , abs/2006.10029, 2020b.  URL https://arxiv.org/abs/2006.10029.
  * Chen et al. (2020c) Chen, X., Fan, H., Girshick, R. B., and He, K.  Improved baselines with momentum contrastive learning.  _CoRR_ , abs/2003.04297, 2020c.  URL https://arxiv.org/abs/2003.04297.
  * Chen et al. (2020d) Chen, X., Fan, H., Girshick, R. B., and He, K.  Improved baselines with momentum contrastive learning.  _CoRR_ , abs/2003.04297, 2020d.  URL https://arxiv.org/abs/2003.04297.
  * Chen et al. (2019) Chen, Y., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J.  UNITER: learning universal image-text representations.  _CoRR_ , abs/1909.11740, 2019.  URL http://arxiv.org/abs/1909.11740.
  * Cui et al. (2018) Cui, Y., Yang, G., Veit, A., Huang, X., and Belongie, S. J.  Learning to evaluate image captioning.  _CoRR_ , abs/1806.06422, 2018.  URL http://arxiv.org/abs/1806.06422.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 8780–8794. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf.
  * Faghri et al. (2017) Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S.  VSE++: improved visual-semantic embeddings.  _CoRR_ , abs/1707.05612, 2017.  URL http://arxiv.org/abs/1707.05612.
  * Gan et al. (2020) Gan, Z., Chen, Y., Li, L., Zhu, C., Cheng, Y., and Liu, J.  Large-scale adversarial training for vision-and-language representation learning.  _CoRR_ , abs/2006.06195, 2020.  URL https://arxiv.org/abs/2006.06195.
  * Grill et al. (2020) Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. Á., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M.  Bootstrap your own latent: A new approach to self-supervised learning.  _CoRR_ , abs/2006.07733, 2020.  URL https://arxiv.org/abs/2006.07733.
  * He et al. (2019) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B.  Momentum contrast for unsupervised visual representation learning.  _CoRR_ , abs/1911.05722, 2019.  URL http://arxiv.org/abs/1911.05722.
  * Hessel et al. (2021) Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y.  Clipscore: A reference-free evaluation metric for image captioning.  _CoRR_ , abs/2104.08718, 2021.  URL https://arxiv.org/abs/2104.08718.
  * Hodosh et al. (2013) Hodosh, M., Young, P., and Hockenmaier, J.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ , 47:853–899, 2013.
  * Jaiswal et al. (2020) Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D., and Makedon, F.  A survey on contrastive self-supervised learning.  _CoRR_ , abs/2011.00362, 2020.  URL https://arxiv.org/abs/2011.00362.
  * Jia et al. (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T.  Scaling up visual and vision-language representation learning with noisy text supervision.  _CoRR_ , abs/2102.05918, 2021.  URL https://arxiv.org/abs/2102.05918.
  * Jiang et al. (2019) Jiang, M., Huang, Q., Zhang, L., Wang, X., Zhang, P., Gan, Z., Diesner, J., and Gao, J.  Tiger: Text-to-image grounding for image caption evaluation.  _CoRR_ , abs/1909.02050, 2019.  URL http://arxiv.org/abs/1909.02050.
  * Kane et al. (2020) Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., and Coulibali, M.  NUBIA: NeUral based interchangeability assessor for text generation.  In _Proceedings of the 1st Workshop on Evaluating NLG Evaluation_ , pp. 28–37, Online (Dublin, Ireland), December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.evalnlgeval-1.4.
  * Kasai et al. (2021) Kasai, J., Sakaguchi, K., Dunagan, L., Morrison, J., Bras, R. L., Choi, Y., and Smith, N. A.  Transparent human evaluation for image captioning.  _arXiv preprint arXiv:2111.08940_ , 2021.
  * Kim et al. (2021) Kim, W., Son, B., and Kim, I.  Vilt: Vision-and-language transformer without convolution or region supervision, 2021.  URL https://arxiv.org/abs/2102.03334.
  * Krause et al. (2013) Krause, J., Stark, M., Deng, J., and Fei-Fei, L.  3d object representations for fine-grained categorization.  In _4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)_ , Sydney, Australia, 2013.
  * Krizhevsky (2009) Krizhevsky, A.  Learning multiple layers of features from tiny images.  2009\.
  * Lee et al. (2020) Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., and Jung, K.  ViLBERTScore: Evaluating image caption using vision-and-language BERT.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pp. 34–39, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.eval4nlp-1.4.  URL https://aclanthology.org/2020.eval4nlp-1.4.
  * Li et al. (2019) Li, G., Duan, N., Fang, Y., Jiang, D., and Zhou, M.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  _CoRR_ , abs/1908.06066, 2019.  URL http://arxiv.org/abs/1908.06066.
  * Li et al. (2021) Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S. R., Xiong, C., and Hoi, S. C. H.  Align before fuse: Vision and language representation learning with momentum distillation.  _CoRR_ , abs/2107.07651, 2021.  URL https://arxiv.org/abs/2107.07651.
  * Li et al. (2020a) Li, W., Gao, C., Niu, G., Xiao, X., Liu, H., Liu, J., Wu, H., and Wang, H.  UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning.  _CoRR_ , abs/2012.15409, 2020a.  URL https://arxiv.org/abs/2012.15409.
  * Li et al. (2020b) Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., and Gao, J.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  _CoRR_ , abs/2004.06165, 2020b.  URL https://arxiv.org/abs/2004.06165.
  * Lin (2004) Lin, C.-Y.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Lin et al. (2014) Lin, T., Maire, M., Belongie, S. J., Bourdev, L. D., Girshick, R. B., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312, 2014.  URL http://arxiv.org/abs/1405.0312.
  * Lu et al. (2021) Lu, X., Zhao, T., and Lee, K.  Visualsparta: Sparse transformer fragment-level matching for large-scale text-to-image search.  _CoRR_ , abs/2101.00265, 2021.  URL https://arxiv.org/abs/2101.00265.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Patashnik et al. (2021) Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 2085–2094, October 2021.
  * Peng et al. (2022) Peng, X., Wang, K., Zhu, Z., and You, Y.  Crafting better contrastive views for siamese representation learning.  _CoRR_ , abs/2202.03278, 2022.  URL https://arxiv.org/abs/2202.03278.
  * Plummer et al. (2015) Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  _CoRR_ , abs/1505.04870, 2015.  URL http://arxiv.org/abs/1505.04870.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision.  _CoRR_ , abs/2103.00020, 2021.  URL https://arxiv.org/abs/2103.00020.
  * Rashtchian et al. (2010) Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J.  Collecting image annotations using amazon’s mechanical turk.  In _Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk_ , pp. 139–147, 2010\.
  * Robinson et al. (2020) Robinson, J., Chuang, C., Sra, S., and Jegelka, S.  Contrastive learning with hard negative samples.  _CoRR_ , abs/2010.04592, 2020.  URL https://arxiv.org/abs/2010.04592.
  * Sohn (2016) Sohn, K.  Improved deep metric learning with multi-class n-pair loss objective.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper/2016/file/6b180037abbebea991d8b1232f8a8ca9-Paper.pdf.
  * Tian et al. (2019) Tian, Y., Krishnan, D., and Isola, P.  Contrastive multiview coding.  _CoRR_ , abs/1906.05849, 2019.  URL http://arxiv.org/abs/1906.05849.
  * Tian et al. (2020) Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P.  What makes for good views for contrastive learning?  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 6827–6839. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/4c2e5eaae9152079b9e95845750bb9ab-Paper.pdf.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.  URL http://arxiv.org/abs/1807.03748.
  * Vedantam et al. (2014) Vedantam, R., Zitnick, C. L., and Parikh, D.  Cider: Consensus-based image description evaluation.  _CoRR_ , abs/1411.5726, 2014.  URL http://arxiv.org/abs/1411.5726.
  * Wang et al. (2020) Wang, H., Zhang, Y., Ji, Z., Pang, Y., and Ma, L.  Consensus-aware visual-semantic embedding for image-text matching.  _CoRR_ , abs/2007.08883, 2020.  URL https://arxiv.org/abs/2007.08883.
  * Wang & Isola (2020) Wang, T. and Isola, P.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  _CoRR_ , abs/2005.10242, 2020.  URL https://arxiv.org/abs/2005.10242.
  * Xiao et al. (2010) Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 3485–3492, 2010.  doi: 10.1109/CVPR.2010.5539970.
  * Yang et al. (2022) Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., and Huang, J.  Vision-language pre-training with triple contrastive learning, 2022.  URL https://arxiv.org/abs/2202.10401.
  * Yi et al. (2020) Yi, Y., Deng, H., and Hu, J.  Improving image captioning evaluation by considering inter references variance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 985–994, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.93.  URL https://aclanthology.org/2020.acl-main.93.
  * You et al. (2021) You, C., Zhao, R., Staib, L. H., and Duncan, J. S.  Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation.  _CoRR_ , abs/2105.07059, 2021.  URL https://arxiv.org/abs/2105.07059.
  * Yu et al. (2022) Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y.  Coca: Contrastive captioners are image-text foundation models, 2022.  URL https://arxiv.org/abs/2205.01917.
  * Yuan et al. (2021) Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., and Faieta, B.  Multimodal contrastive training for visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 6995–7004, June 2021.
  * Zhai et al. (2021) Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L.  Lit: Zero-shot transfer with locked-image text tuning.  _CoRR_ , abs/2111.07991, 2021.  URL https://arxiv.org/abs/2111.07991.
  * Zhang et al. (2020) Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P.  Contrastive learning of medical visual representations from paired images and text.  _CoRR_ , abs/2010.00747, 2020.  URL https://arxiv.org/abs/2010.00747.
  * Zhou et al. (2022) Zhou, K., Yang, J., Loy, C. C., and Liu, Z.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022.

##  Appendix A Datasets Details

There is a total of 10 datasets mentioned in this paper, as described below.

###  A.1 Cross-modal Retrieval Datasets

MSCOCO (Lin et al., 2014) is a multi-purpose dataset known for its rich
compatibility with object detection, segmentation, and image captioning tasks.
It contains 118K images in its training split and more than 5K images in its
test split.

Flicker30K (Plummer et al., 2015) is a popular benchmark for sentence-based
picture portrayal. It contains 31K images with each image having five
reference sentences generated by human annotators. We took a train-test split
following (Yang et al., 2022) and (Li et al., 2021), which involves a
selection of 30K images for fine-tuning and 1K images for testing.

###  A.2 Zeroshot Classification Datasets

ImageNet1K (Deng et al., 2009) is the most well-known dataset for image
classification that contains more than 1200K images in its training set and
100K images in its test set and covers 1000 categories of objects. Accuracy on
ImageNet1K is especially widely used for evaluating state-of-the-art image
classification models.

Cifar100 (Krizhevsky, 2009) is a dataset containing 100 classes of objects,
with each class having 500 training images and 100 test images.

SUN397 (Xiao et al., 2010), the Scene UNderstanding database, contains 899
categories and about 130K images. 397 well-sampled categories are available
for benchmarking state-of-the-art algorithms for scene recognition tasks.

Stanford Cars (Krause et al., 2013) is an image classification dataset
dedicated for cars. It collects about 16K images for 196 types of cars and
adopts a 50-50 train-test split. Results on Stanford Cars test our method’s
effectiveness in improving special-purpose classification models.

###  A.3 Image Captions Evaluation Datasets

Flickr8K-Expert (Hodosh et al., 2013) is a curated subset of the Flickr8K
dataset that contains 17K human ratings of image-caption pairs. Each human
score corresponds to one pair and is from 1 to 4 (1 means the caption is
irrelevant, 4 means the caption describes the image fully correctly.)

Flicker8K-CF (Hodosh et al., 2013) is a similar dataset gathered from
CloudFlower that contains 48K image-caption pairs and 145K binary ratings of
these pairs.

THumb (Kasai et al., 2021) is a dataset containing some machine- and human-
generated captions from the MSCOCO dataset. It has 500 images, each with 5
candidates captions that are evaluated by human annotators.

Pascal-50S (Rashtchian et al., 2010) contains 4K caption-caption pairs with
each pair describing the same image. For all pairs, annotators give a
preference on which caption in the pair provides better description of the
image. Caption-caption pairs are categorized based on the origins of the
captions (see Table 4 for details about categories.)

##  Appendix B Additional Results

###  B.1 Fine-tuning Epochs on MSCOCO and Flickr30K

In Figure 3, we present the remaining two plots of fine-tuning ablations for
MSCOCO. CLIP + DN does better than CLIP by an average of 0.52% for Acc@5, and
0.37% for Acc@10. Both graphs observe a peak in accuracy around 4-5 epochs.
For Acc@5, we observe an initial improvement of 9.10% for CLIP + DN after just
1 epoch, and 11.10% for CLIP on text-to-image retrieval. These respective
numbers are 5.17% and 4.39% for image-to-text retrieval. For Acc@10, we
observe an initial improvement of 8.53% for CLIP + DN and 10.03% for CLIP on
text-to-image retrieval, and 3.14% and 3.14% respectively for image-to-text
retrieval.

Figure 3: Comparison of the effects of fine-tuning between CLIP and CLIP + DN
on MSCOCO’s 5k test set. We report text-to-image retrieval (left) and image-
to-text retrieval (right) for Acc@5 and Acc@10. The average accuracy over 5
checkpoints trained with 5 random seeds is plotted. For each of the 5
checkpoints we trained, we find its average accuracy and standard deviation
with another 5 iterations random sampling for mean estimation, and plot the
mean of these 5 accuracies and standard deviations from 5 independently fine-
tuned checkpoints.

In Figure 4, we present the effects of finetuning on Acc@5 and Acc@10 for
Flickr30K. Again, CLIP+DN does better than CLIP by an average of 0.22% for
Acc@5 and 0.14% for Acc@10. All results yield a similar trend in support of
the conclusion we have made in Section 4.6.2.

Figure 4: Comparison of the effects of fine-tuning between CLIP and CLIP + DN
on Flickr30K’s 1k test set. We report text-to-image retrieval (left) and
image-to-text retrieval (right) for all Acc@5 and Acc@10. The average accuracy
over 5 checkpoints trained with 5 random seeds is plotted. For each of the 5
checkpoints we trained, we find its average accuracy and standard deviation
with another 5 iterations random sampling for mean estimation, and plot the
mean of these 5 accuracies and standard deviations from 5 independently fine-
tuned checkpoints.