# Quality Not Quantity: On the Interaction between
Dataset Design and Robustness of CLIP

Thao Nguyen University of Washington {thaottn, gamaga, mitchnw, sewoong,
<EMAIL_ADDRESS>Gabriel Ilharco11footnotemark: 1 Mitchell
Wortsman11footnotemark: 1 Sewoong Oh11footnotemark: 1 Ludwig
Schmidt11footnotemark: 1 ,
Allen Institute for Artificial Intelligence

###### Abstract

Web-crawled datasets have enabled remarkable generalization capabilities in
recent image-text models such as CLIP (Contrastive Language-Image pre-
training) or Flamingo, but little is known about the dataset creation
processes. In this work, we introduce a testbed of six publicly available data
sources—YFCC, LAION, Conceptual Captions, WIT, RedCaps, Shutterstock—to
investigate how pre-training distributions induce robustness in CLIP. We find
that the performance of the pre-training data varies substantially across
distribution shifts, with no single data source dominating. Moreover, we
systematically study the interactions between these data sources and find that
combining multiple sources does not necessarily yield better models, but
rather dilutes the robustness of the best individual data source. We
complement our empirical findings with theoretical insights from a simple
setting, where combining the training data also results in diluted robustness.
In addition, our theoretical model provides a candidate explanation for the
success of the CLIP-based data filtering technique recently employed in the
LAION dataset. Overall our results demonstrate that simply gathering a large
amount of data from the web is not the most effective way to build a pre-
training dataset for robust generalization, necessitating further study into
dataset design. Code is available at
https://github.com/mlfoundations/clip_quality_not_quantity.

.tocmtsection

##  1 Introduction

Large models pre-trained on web-scale datasets are becoming a cornerstone of
machine learning. For instance, the past two years have witnessed the arrival
of several new models such as GPT-3 [12], Chinchilla [37], and PaLM [15] for
natural language processing, or CLIP [61], BASIC [58], and Flamingo [3] for
computer vision. These models exhibit unprecedented generalization
capabilities in zero-shot inference, in-context learning, and robustness to
distribution shift. A key ingredient enabling their generalization performance
are the large and diverse pre-training corpora that exceed previous datasets
by multiple orders of magnitude. For instance, the training set of BASIC [58]
contains 6.6 billion images, which is more than 1,000 times larger than the
widely used ImageNet [19] training set from the 2012 competition containing
1.2 million images.

Despite the central role datasets play for pre-trained models, little is known
about them, especially for image-text models. The aforementioned CLIP [61],
BASIC [58], and Flamingo [3] all rely on datasets internal to the respective
organizations, which is also the case for other models such as DALL-E [63],
Florence [86], and ALIGN [39]. In addition, research publications often
provide little details on the data collection processes, e.g., the data
sources or data filtering mechanisms. Beyond clear issues such as
reproducibility and the potential presence of harmful content, the opaque
dataset creation practices also make it hard to identify effective methods for
assembling pre-training datasets. As a result, researchers cannot build on
each other’s dataset innovations, which obstructs the incremental research
process that has successfully accumulated algorithm and architecture
improvements in machine learning models. A more principled understanding of
dataset creation will likely enable further progress in the generalization
capabilities of pre-trained models.

A basic approach to dataset creation would be to simply train on _all_
available data of a given type. While scaling up training sets has indeed been
integral to the recent progress in large models, advances in weak and self-
supervision [14, 33, 34, 12, 21] have led to an abundance of potential
training data. For instance, the large LAION-5B dataset [2] of 5 billion
image-text pairs is itself a subset of about 50 billion images from Common
Crawl. This abundance is already exceeding the amount of data models can
currently be trained on within a reasonable time,111Recall that the training
set for Google’s largest image-text model ALIGN contains “only” 6.6 billion
images, making it about eight times smaller than the source dataset for
LAION-5B. making the aforementioned baseline of using all available data
infeasible. Consequently, deciding _what_ data to train on is becoming
increasingly important.

In this paper, we take a step towards a better understanding of pre-training
data and investigate the impact of dataset design on the generalization
capabilities of image-text models. Specifically, we focus on CLIP, the first
large model of this kind that demonstrated remarkable robustness to multiple
challenging distribution shifts. The main questions we ask are: ($i$) How much
do different web data sources vary in their induced robustness? ($ii$) Do
dataset combinations lead to better robustness? ($iii$) Can filtering with an
existing image-text model improve data quality? We address these questions
from both an experimental and theoretical perspective.

On the experimental side, we begin by assembling a corpus of six different
datasets from the web, spanning a variety of sources including Flickr,
Shutterstock, Wikipedia, Common Crawl, and Reddit. We then measure the
robustness of CLIP models trained on each dataset. In particular, we compare
the zero-shot accuracy of these models on ImageNet and a set of canonical
ImageNet distribution shifts. We find that the robustness induced by each pre-
training dataset varies widely, and that sources with careful curation such as
Wikipedia do not necessarily outperform those with minimal filtering.

In addition, the datasets cannot be compared along a single dimension:
different datasets help with robustness to different distribution shifts. This
in turn motivates studying how combining datasets affects robustness. We find
that a model trained on two datasets does not inherit the robustness
properties of both. Rather, while the model is exposed to both distributions,
its robustness interpolates between that of the individual datasets. This
indicates that dataset designers must carefully combine the pre-training data
in order to preserve and enhance the robustness of the resulting models.

Building on our empirical results, we introduce a theoretical model to better
understand our experimental findings. Our model is simple enough to allow for
mathematical analysis and can still capture some phenomena of real-world, web-
crawled datasets. Specifically, our theoretical model also shows that
combining multiple datasets dilutes the robustness of the better data
distribution. Moreover, our model provides a candidate explanation for another
interesting phenomenon in the curation of pre-training data: the LAION-400M
experiment [68] and follow-up CLIP reproductions [2] demonstrated that
filtering a noisy source (Common Crawl) with a CLIP model results in a dataset
on which newly trained CLIP models exhibit higher robustness to some
distribution shifts.

#### Paper outline.

We first briefly review related work and the relevant background on robustness
to distribution shift (Section 2), before discussing our experimental setup in
Section 3. Sections 4 and 5 then measure the robustness of CLIP induced by
individual data sources and their combinations. To support these empirical
results, Section 6 presents our theoretical analysis. We conclude with future
research directions in Section 7.

##  2 Background & Related Work

#### Vision-language models.

Large vision-language models like CLIP and ALIGN have become an active area of
research owing to their success on various computer vision tasks [61, 39].
Existing work expands their capabilities by either increasing model and
dataset size [58], or by using additional supervision as in DeCLIP [46], SLIP
[54], and FILIP [84]. In contrast, we study the effect of _pre-training data
composition_ on task performance with a focus on robustness.

In a recent work, Fang et al. [24] showed that CLIP’s robustness primarily
stems from its diverse pre-training distribution and not training set size,
language supervision, or contrastive loss functions. However, Fang et al. [24]
only conducted experiments with two datasets (YFCC-15M and ImageNet-Captions),
one of which contains less than one million images. We take the insights of
Fang et al. [24] as our starting point and expand the range of pre-training
datasets to six different sources, each containing at least five million
images. This enables us to study the robustness induced by various pre-
training sources, and how dataset combinations affect robustness.

Figure 1: Models pre-trained on LAION exhibit _effective robustness_ [75]
compared to standard models trained on ImageNet. Effective robustness is
defined as movement towards a classifier which is robust to distribution
shift. A classifier is more robust the closer it is to the $y=x$ line. A
classifier on the $y=x$ line is not affected by the distribution shift.

#### Distribution shift.

Robustness to distribution shift is a long-standing issue in machine learning
[78, 60] and has recently received renewed attention as researchers scrutinize
the generalization performance of neural networks in greater detail [74, 8, 9,
5, 66, 31, 16, 44]. Similar to CLIP [61], we focus on robustness to natural
distribution shifts, where the corresponding test sets contain only unmodified
images and are not intentionally perturbed by adversarial examples or
synthetic corruptions (such as Gaussian noise or blur patterns) [35].
Specifically, we test robustness on ImageNetV2 [65], ImageNet-R [36],
ImageNet-Sketch [81], and ObjectNet [6] because prior work has established
many baselines for these distribution shifts [75]. Moreover, models robust to
these shifts also show improved robustness on other out-of-distribution
benchmarks such as WILDS [42, 82].

The robustness literature usually discusses distribution shift in terms of
“in-distribution” and “out-of-distribution” data. This terminology is natural
when data from the same distribution as the “in-distribution” test set is used
for training or fine-tuning. However, it is unclear what counts as “in-
distribution” when models are trained on large-scale, generic pre-training
datasets that aim to improve performance on a wide variety of tasks. To
address this issue, we follow the more flexible definition of distribution
shift employed in prior work [75, 52] and measure robustness as accuracy
difference between two related but distinct test distributions (e.g., ImageNet
and ImageNet-Sketch). The expectation is that the shift between the two test
distributions should not affect an ideal robust model, for instance because
the shift does not affect the accuracy of humans labelers [69]. Taori et al.
[75] defined this notion of robustness as _effective robustness_. Radford et
al. [61] adopted the effective robustness framework in the evaluations of
their CLIP model. Effective robustness is illustrated in Figure 1 and measures
movement towards a perfectly robust classifier which is not affected by the
shift between two test distributions.

Prior work [52] has evaluated several hundred models and demonstrated
experimentally that changes to model architecture, training set size, training
algorithm, and other model-related factors do _not_ change effective
robustness in most cases. In contrast, changes to the pre-training
_distribution_ can improve the effective robustness of a model. This makes
effective robustness a useful metric for evaluating the influence of pre-
training data sources on robust generalization, since it removes confounders
stemming from model hyperparameters and the number of training samples. The
effect of the pre-training distribution on effective robustness is
particularly pronounced when models are evaluated in a zero-shot setting:
while training on a target distribution can improve accuracy on that
distribution [82, 28, 87, 88, 83], this process can deteriorate robustness to
distribution shift [61, 82, 4, 58].

The existence of a universal linear trend for accuracies on a pair of test
sets has been analytically studied in [52]. For a simple binary classification
model similar to our Assumption 1 (see Section 6), convergence to a linear
trend is shown with the deviation from the line scaling as $O(1/\sqrt{d})$.
However, the analysis in [52] is restricted to a specific stochastic
distribution shift for the out-of-distribution test data and a fixed
classifier independent of the training data. Hence the dependence of the slope
on the training set size, variations in the training methods, and properties
of the training distribution cannot be described in their model. We provide
significantly more fine-grained analyses (Theorem 1) that captures all such
tradeoffs, which allows us to draw novel insights into how data mixing
(Theorem 2) and filtering (Theorem 3) affect model robustness.

##  3 Experiment Setup

Model. We focus on the CLIP model [61] which has demonstrated unprecedented
zero-shot performance on a wide range of downstream tasks, as well as
robustness to various distribution shifts [52]. Given an image-text pair, CLIP
is trained to maximize the cosine similarity between the embedding of the text
and that of the image, relative to the similarity of unconnected image-text
pairs. We use the CLIP implementation from the OpenCLIP GitHub repository
[38], with ResNet-50 [32] as the image encoder architecture. We vary the pre-
training set size and hyperparameters such as number of epochs to obtain
different accuracies on each data distribution. Due to compute constraints,
the total training set size is at most 15M samples for most of our
experiments. Appendix B contains further training details.

Data. To study the effects of training distributions on robust generalization,
we collect several datasets from publicly available sources. Most of these
have been studied in the context of various vision and language tasks in
previous work:

  * •

YFCC: We experiment with the 15M subset of the YFCC100M dataset [76] that the
original CLIP paper [61] used for dataset ablation studies. The images and
captions are collected from Flickr.

  * •

LAION [68]: The images and corresponding alt-texts come from web pages
collected by Common Crawl [1] between 2014 and 2021. We randomly select a
subset of 15M samples to experiment with, and ensure that the accompanying
NSFW tags of all chosen images are ‘UNLIKELY’.

  * •

Conceptual Captions [13]: We use CC-12M for our experiments, which consists of
images and HTML alt-text from an unspecified set of web pages.

  * •

RedCaps [20]: This dataset contains 12M examples, obtained from 350 manually
curated subreddits between 2008 and 2020. The subreddits are selected to
contain a large number of image posts that are mostly photographs and not
images of people.

  * •

Shutterstock: 15M images and captions were crawled from the Shutterstock
website in 2021.

  * •

WIT [71]: Image-text pairs come from Wikipedia pages. We use reference
description as the source of text data and obtain 5M examples in total after
filtering to include only English language examples.

Appendix A.1 contains an analysis of image and text statistics, as well as
randomly selected data samples from each source.

Evaluation. Similar to Taori et al. [75] and Radford et al. [61], we choose
ImageNet as the reference distribution and evaluate CLIP on four natural
distribution shifts derived from ImageNet:

  * •

ImageNet-V2 [65]: A reproduction of the ImageNet validation set closely
following the original dataset creation process.

  * •

ImageNet-R [36]: Renditions (e.g., sculptures, paintings, etc.) for 200
ImageNet classes.

  * •

ImageNet-Sketch [81]: Sketches of ImageNet class objects.

  * •

ObjectNet [6]: A test set of objects in novel backgrounds, rotations, and
viewpoints with 113 classes overlapping with ImageNet

Visualizations of random samples from each distribution shift can be found in
Appendix A.2.

##  4 Individual Pre-training Data Sources

Figure 2: Performance of the six pre-training data sources under various
distribution shifts. We find that the behavior—both in terms of accuracy and
the slope of the linear trend—of the pre-training data varies substantially
across distribution shifts, with no single data source dominating. Most shifts
help highlight the strengths and weaknesses of different data sources, except
for ImageNet-V2, where the linear trends produced by individual sources are
highly correlated with one another.

We first investigate how well a CLIP model trained on each data source would
perform under different distribution shifts of interest. As seen from Figure
2, while all sources yield the same linear trend on ImageNet-V2, some display
clear advantages when evaluated on other test distributions. For example,
Shutterstock offers the best out-of-distribution performance on ImageNet-
Sketch, while RedCaps displays the most effective robustness on ObjectNet. On
ImageNet-R, LAION, CC12M and Shutterstock seem to do much better than the
rest. Overall no pre-training data distribution is consistently the most
robust across all evaluation settings.

We also measure the data efficiency of each source, i.e., how much the
performance would change with more samples from the same source, see Appendix
C. Similar to the previous observation, the six pre-training sources display
vastly different data efficiency depending on the distribution shifts of
interest. Although LAION and CC-12M exhibit similar effective robustness in
all evaluation settings in Figure 2, this analysis reveals subtle differences
between these two training distributions in the low-data regimes.

##  5 Combining Data Sources

In the previous section, we demonstrate the variability in behavior of
different data sources based on the distribution shift at test time. A natural
question then arises from this observation: does combining multiple sources
help improve robustness across some, if not all, test distributions of
interest? We investigate two common approaches of aggregating information from
different training sets—input mixing and output ensembling. In the subsequent
discussion, we focus on distribution shifts that bring out significant
differences in behavior across the pre-training data sources (e.g., ImageNet-R
and ImageNet-Sketch); the full results on all distribution shifts can be found
in Appendices D and E.

###  5.1 Input Mixing

In input mixing, we randomly select and combine samples from multiple data
sources to build the pre-training dataset. Our findings indicate that this
exposure to more training distributions doesn’t help CLIP take advantage of
the complimentary strengths of each source. Rather, the effective robustness
of the resulting model is less than that of training on the best individual
source for each distribution shift.

Figure 3: Combining YFCC and LAION training data in equal ratios produces
models with intermediate robustness. Given a fixed data budget of 15M samples,
the linear trend produced by training CLIP on a YFCC-LAION data mixture, with
7.5M datapoints from each source (cyan line), lies between that of training
CLIP on YFCC (blue line) and LAION (green line) entirely. Even when we
increase the total training set size (30M) and use all data available from
both sources (orange line), the same pattern persists.  Figure 4: Varying the
sample contributions of YFCC and LAION to the input data mixture produces a
smooth interpolation of the linear trend between those of training on YFCC and
LAION separately. Keeping the total number of training samples fixed at 15M,
as we vary the contribution of YFCC to the dataset mixture from 15M (i.e.,
only training on YFCC) to 0M (i.e., only training on LAION), the resulting
linear trend gradually shifts from that of YFCC-15M (blue line) to that of
LAION-15M (green line).

We start with combining data from two largest data sources in the testbed—YFCC
and LAION. To remove dataset size as a confounder, we fix the total amount of
training data at 15M samples, and sample 7.5M image-text pairs from each
source. This data mixture yields a linear trend that lies in between the
trends obtained from training on 15M YFCC and 15M LAION datapoints separately
(Figure 3). Even when we remove the constraint on the training set size and
use all the data available from these two sources (i.e., 30M samples), the
same observation on robustness holds, and the resulting linear trend is highly
correlated with that of training on the YFCC-7.5M + LAION-7.5M mixture.

The previous set of experiments combines YFCC and LAION data with a 50:50
ratio. In Figure 4, we find that when this ratio is varied within a fixed
budget of 15M datapoints, the robustness linear trends of the corresponding
mixtures form a smooth interpolation between the linear trends of training on
15M YFCC and 15M LAION samples separately. Experiments with different
combinations of sources, as well as mixtures of a larger number of sources,
can be found in Appendix D. There we also include results for combining data
from CIFAR-10 and CINIC-10 distributions to train ResNets on image
classification tasks, where we observe the same interpolation pattern. We
provide a theoretical justification for this phenomenon in Section 6.2.

###  5.2 Output Mixing

Figure 5: Ensemble outputs of CLIP models trained on YFCC and LAION separately
share the same linear trend as a single model trained on the combined data
mixture (where each source contributed equally). We ensemble the logit
predictions of YFCC-trained (blue line) and LAION-trained (green line) models
taken from the same epoch, with varying ensemble weights between 0 and 1 (red
dashed line). When the outputs are combined with equal weights (red markers),
the resulting test accuracies closely track the linear trend produced by pre-
training CLIP on a data mixture with equal number of samples from each source
(purple line).  Figure 6: Using ensemble outputs to predict the linear trend
of input mixing without retraining CLIP from scratch. A generalization of the
observation made in Figure 5 is that given an existing pre-training dataset
that could be a mixture (e.g., YFCC-5M + LAION-5M, green line) and a new data
source (e.g., CC-5M, orange line), we could use the ensemble outputs (blue
markers) of two CLIP models that have been trained separately on these two
data distributions, to estimate where the linear trend for a CLIP model
trained on all the data would lie (purple line). This removes the need to
actually train CLIP from scratch on the now bigger 3-source mixture.

Another common approach to take advantage of different training sets is to
combine them at model output level (i.e., ensemble) [22, 45, 29, 25, 7, 11,
26, 56]. Here, we train a CLIP model on each pre-training distribution of
interest and combine the logit predictions of all the resulting models with
equal weights. In experiments with mixing YFCC and LAION (Figure 5), the
ensembles of 2 single-source-trained CLIP models taken from the same epoch of
training, lie on the linear trend of a single CLIP model trained on the
combined data. This holds across different distribution shifts that we
consider. The same observation also applies when we ensemble 6 CLIP models
trained separately on the 6 data sources collected, with the contribution of
each model being weighted equally. Refer to Appendix E for more details.

We next show that this phenomenon extends to more complex mixtures. The fact
that output mixing (i.e., ensembling the predictions of CLIPs trained on
individual sources) is predictive of the linear trend produced by input mixing
(i.e., training on all the data from these sources) presents an opportunity to
estimate model robustness given new data sources, without having to train the
model from scratch on the new, potentially much larger, combined dataset. For
example, in Figure 6, assuming access to a CLIP model trained on the YFCC-5M +
LAION-5M mixture, and another one trained on CC-5M, the ensembles of these two
models across different stages of training share the same linear trend as a
CLIP model trained on the YFCC-5M + LAION-5M + CC-5M mixture. In Appendix E,
we show the results of ensembling 6 CLIP models trained with the 6 data
sources we collected, as well as the CINIC-10 + CIFAR-10 ensemble, a uni-modal
image classification setting where output mixing accuracies are also
predictive of the linear trend of input mixing.

##  6 Analysis under Simple Binary Classification Models

Empirically we observe, for e.g., in Figure 2, that for a pair of test
datasets $({\cal D}_{1},{\cal D}_{2})$, the corresponding test accuracies
after probit transform achieved by various trained models (including different
architectures and training algorithms) on the same training dataset ${\cal D}$
all lie on the same line. Furthermore, this line includes models trained on
only a subset of ${\cal D}$ [52]. In other words, there is a universal line
that is determined only by the training data distribution and the two test
distributions, and is independent of which architecture we use, how we train
the model, or how many samples we use from the training set. To explain this
phenomenon, we study the following class of trained models parametrized by
$(\theta\in{\mathbb{R}}^{d},\rho\in{\mathbb{R}}_{+})$ representing the
training distribution, with $n$ representing the training data size, and
$\xi\in{\mathbb{R}}_{+}$ representing the variations due to the training
algorithm.

###  6.1 Universality of Accuracy on the Line for Binary Classification

We analyze a simple but canonical binary classification example where each
training data is parametrized by its ground-truth linear classifier
$\theta\in{\mathbb{R}}^{d}$ and its Signal-to-Noise Ratio (SNR)
$\rho^{2}\in{\mathbb{R}}_{+}$. Concretely, a training dataset ${\cal
D}_{n,\theta,\rho}=\\{(x_{i}\in{\mathbb{R}}^{d},y_{i}\in{\pm 1})\\}_{i=1}^{n}$
is a set of $n$ i.i.d. paired samples from a joint distribution
$P_{\theta,\rho}$ defined as follows.

######  Assumption 1 (Data distribution).

We define a joint distribution $(x_{i},y_{i})\sim P_{\theta,\rho}$ as $(i)$
$y_{i}=\pm 1$ uniformly at random, and $(ii)$
$x_{i}=y_{i}\theta+(\|\theta\|/\rho)z_{i}$ where the noise $z_{i}$ is zero-
mean and has independent entries with variance one. For an observation
$(x_{i},y_{i})$, we refer to $\|\theta\|^{2}$ as the signal power and
$\|\theta\|^{2}/\rho^{2}$ as the corresponding noise power with SNR
$\rho^{2}$.

######  Assumption 2 (Trained model distribution).

We consider a (random) linear model parameter
$\hat{\theta}_{n,\xi}\in{\mathbb{R}}^{d}$ that predicts a binary label ${\rm
sign}(\langle x,\hat{\theta}_{n,\xi}\rangle)$ for a test example
$x\in{\mathbb{R}}^{d}$. We assume that the random model
$\hat{\theta}_{n,\xi}=\theta+(\xi\|\theta\|/(\rho\sqrt{n}))z\in{\mathbb{R}}^{d}$,
trained on ${\cal D}_{n,\theta,\rho}$ is unbiased,
${\mathbb{E}}[\hat{\theta}_{n,\xi}]=\theta$, and that $z$ has independent
entries with variance one each. The randomness comes from the training data as
well as any internal randomness in the training algorithm. The parameter
$\xi\in{\mathbb{R}}_{+}$ captures the variations in the resulting model
distribution due to changes the training algorithm.

Concretely, one canonical example of a trained model is
$\hat{\theta}_{n,\xi}=(1/n)\sum_{i=1}^{n}y_{i}x_{i}$. It follows that
$\hat{\theta}_{n}=\theta+(\|\theta\|/(\rho\sqrt{n}))z$ with $\xi=1$. Hence,
$\xi$ measures the randomness of the trained model relative to this simple
training algorithm.

We analyze the resulting accuracy when evaluated on two test distributions
$P_{\theta_{1},\rho_{1}}$ and $P_{\theta_{2},\rho_{2}}$ as defined above, with
${\rm Acc}_{\theta_{1},\rho_{1}}:=P_{\theta_{1},\rho_{1}}\\{{\rm sign}(\langle
X,\hat{\theta}_{n,\xi}\rangle)=Y\\}$, and ${\rm Acc}_{\theta_{2},\rho_{2}}$
defined similarly. In particular, we are interested in how the accuracy pair
$(\Phi^{-1}({\rm Acc}_{\theta_{1},\rho_{1}}),\Phi^{-1}({\rm
Acc}_{\theta_{2},\rho_{2}}))$ behaves as we vary the sample size $n$ and as we
vary the training algorithm represented by $\xi$. Here,
$\Phi^{-1}:[0,1]\to{\mathbb{R}}$ is the inverse of the CDF of a standard
Gaussian distribution, defined as $\Phi(t)={\mathbb{P}}(z\leq t)$ where
$z\sim{\cal N}(0,1)$. $\Phi^{-1}$ is also called the probit function. This
choice of mapping the accuracy with the probit function is critical in getting
the linear relation, which we will explain in Remark 1.

######  Theorem 1 (Universality of accuracy on the line).

Under Assumptions 1 and 2, asymptotically as the dimension $d$ grows linearly
in the sample size $n$ such that $\lim_{d\to\infty}n/d=\alpha^{2}$, we have

$\displaystyle\lim_{d\to\infty}{\rm
Acc}_{\theta_{1},\rho_{1}}=\Phi\Big{(}\,\cos(\theta_{1},\theta)\frac{\rho_{1}\rho\alpha}{\xi}\,\Big{)}\;,\text{
and }\lim_{d\to\infty}{\rm
Acc}_{\theta_{2},\rho_{2}}=\Phi\Big{(}\cos(\theta_{2},\theta)\frac{\rho_{2}\rho\alpha}{\xi}\,\Big{)}\;.$
(1)

Further, under Assumption 5 in the Appendix, for some universal constant
$c>0$,

$\displaystyle\Phi^{-1}({\rm
Acc}_{\theta_{2},\rho_{2}})=\frac{\cos(\theta_{2},\theta)\rho_{2}}{\cos(\theta_{1},\theta)\rho_{1}}\Phi^{-1}({\rm
Acc}_{\theta_{1},\rho_{1}})+O\Big{(}\frac{e^{\frac{cn}{d}}}{\sqrt{n}}\Big{)}\;.$
(2)

We provide a proof in Appendix F.1. This analysis implies that for any
training sample size $n$ and any (variation due to the) training algorithm
$\xi$, the resulting accuracy pair after probit transform lies on a universal
line determined by Eq. (2), and the slope only depends on the two test
distributions $(P_{\theta_{1},\rho_{1}},P_{\theta_{2},\rho_{2}})$ and the
training distribution $P_{\theta,\rho}$. The similarity between the training
data and each test dataset is captured by the angles:
$\cos(\theta_{1},\theta)$ and $\cos(\theta_{2},\theta)$. More similar training
data achieves a larger test accuracy. The hardness of the test distribution is
captured by the SNR of each test dataset: $\rho_{1}$ and $\rho_{2}$. We
emphasize that this line is universal in the sense that the slope does not
depend on the sample size $n$ and training algorithm parameter $\xi$. We are
interested in the regime where $n$ scales linearly in $d$ and both are large
such that the second term in the above equation is negligible.

Remark 1. Why do we get the accuracy-on-the-line phenomenon? The prediction of
the trained (random) linear model on a random test data point $X$ involves the
inner product, which performs a natural spatial averaging over the $d$
coordinates. Since the noise across the coordinates is independent and has
bounded variance, the central limit theorem applies. The resulting error has a
Gaussian tail. Hence, the probit mapping, $\Phi^{-1}({\rm
Acc}_{\theta_{1}\rho_{1}})$, is critical in translating accuracy into the
relevant mean to standard deviation ratio:
$\cos(\theta_{1},\theta)\rho_{1}\rho\sqrt{n}/(\xi\sqrt{d})$. This is
consequently important for getting the universal linear relation, because
irrelevant parameters, $n$ and $\xi$, cancel out in the slope of
$\Phi^{-1}({\rm Acc}_{\theta_{2}\rho_{2}})/\Phi^{-1}({\rm
Acc}_{\theta_{1}\rho_{1}})$. Refer to the proof of Theorem 1 (Appendix F.1)
for more details.

Remark 2. Intersection at the random guess: Previous work [52] that considered
models with a wide range of accuracies has found that all lines intersect at a
point corresponding to “random guess", which in this binary example is
$(\Phi^{-1}(1/2),\Phi^{-1}(1/2))=(0,0)$. Our theoretical analysis is
consistent with this empirical observation: all universal lines intersect at
$(0,0)$ up to a small additive error scaling as $O(1/\sqrt{n})$.

###  6.2 Input Mixing Yields an Intermediate Slope

Figures 3 and 4 show that when a model is trained on samples combined from two
datasets, the resulting robustness trend lies in between the trends achieved
by individual datasets. We show that this finding is universally true under
our current setup in Theorem 2. Note that for the linear models we consider,
input mixing (combining training sets) and output mixing (combining model
outputs) are equivalent.

######  Assumption 3.

Consider two training datasets ${\cal
D}_{n_{1},\tilde{\theta}_{1},\tilde{\rho}_{1}}$ and ${\cal
D}_{n_{2},\tilde{\theta}_{2},\tilde{\rho}_{2}}$ of sizes $n_{1}$ and $n_{2}$
from distributions $P_{\tilde{\theta}_{1},\tilde{\rho}_{1}}$ and
$P_{\tilde{\theta}_{2},\tilde{\rho}_{2}}$ as defined in Assumption 1.
Separately training on individual datasets gives two models
$\hat{\theta}_{n_{1},\xi_{1}}({\cal
D}_{n_{1},\tilde{\theta}_{1},\tilde{\rho}_{1}})$ and
$\hat{\theta}_{n_{2},\xi_{2}}({\cal
D}_{n_{2},\tilde{\theta}_{2},\tilde{\rho}_{2}})$ as defined in Assumption 2. A
model trained on a combined (and possibly subsampled) training dataset ${\cal
D}_{n_{1}^{\prime},\tilde{\theta}_{1},\tilde{\rho}_{1}}\cup{\cal
D}_{n_{2}^{\prime},\tilde{\theta}_{2},\tilde{\rho}_{2}}$ is represented by
$\hat{\theta}_{n_{1}^{\prime}+n_{2}^{\prime},\xi}({\cal
D}_{n_{1}^{\prime},\tilde{\theta}_{2},\tilde{\rho}_{2}}\cup{\cal
D}_{n_{2}^{\prime}\tilde{\theta}_{2},\tilde{\rho}_{2}})=\bar{\theta}+(\xi\|\bar{\theta}\|/(\bar{\rho}\sqrt{n_{1}^{\prime}+n_{2}^{\prime}}))z$
where
$\bar{\theta}=(n_{1}^{\prime}\tilde{\theta}_{1}+n_{2}^{\prime}\tilde{\theta}_{2})/(n_{1}^{\prime}+n_{2}^{\prime})$,
$\bar{\rho}=\|\bar{\theta}\|/\sqrt{(n_{1}^{\prime}\|\tilde{\theta}_{1}\|^{2}/\rho_{1}^{2})+(n_{2}^{\prime}\|\tilde{\theta}_{2}\|^{2}/\rho_{2}^{2})}$,
and $z\in{\mathbb{R}}^{d}$ is a zero-mean random vector with independent
entries each with variance one.

Again, a canonical example of a trained model is
$\hat{\theta}_{n_{1}^{\prime}+n_{2}^{\prime},\xi}=(1/(n_{1}^{\prime}+n_{2}^{\prime}))\big{(}\sum_{(x_{i},y_{i})\in{\cal
D}_{n_{1}^{\prime},\tilde{\theta}_{1},\tilde{\rho}_{1}}}y_{i}x_{i}+\sum_{(x_{i},y_{i})\in{\cal
D}_{n_{2}^{\prime},\tilde{\theta}_{2},\tilde{\rho}_{2}}}y_{i}x_{i}\big{)}$. We
then have
$\hat{\theta}_{n_{1}^{\prime}+n_{2}^{\prime},\xi}=\bar{\theta}+(\|\bar{\theta}\|/(\bar{\rho}\sqrt{n_{1}^{\prime}+n_{2}^{\prime}}))z$
with $\xi=1$.
It follows from applying Theorem 1 to the model trained on the combined data
(Assumption 3) that the resulting linear trend achieves

$\displaystyle{\rm
Slope}(\hat{\theta}_{n_{1}^{\prime}+n_{2}^{\prime},\xi}({\cal
D}_{n_{1}^{\prime},\tilde{\theta}_{1},\tilde{\rho}_{1}}\cup{\cal
D}_{n_{2}^{\prime},\tilde{\theta}_{2},\tilde{\rho}_{2}}))\;\;:=\;\;\frac{\cos(\theta_{2},\bar{\theta})\rho_{2}}{\cos(\theta_{1},\bar{\theta})\rho_{1}}\;,$
(3)

We show that the slope obtained from training on the combined dataset lies
between those obtained from training on the two datasets separately. The proof
could be found in Appendix F.2. Let ${\rm Slope}_{1}:={\rm
Slope}(\hat{\theta}_{n_{1}}({\cal
D}_{n_{1},\tilde{\theta}_{1},\tilde{\rho}_{1}}))$ and ${\rm Slope}_{2}$ be
defined similarly.

######  Theorem 2.

Under Assumption 3, ${\rm Slope}_{1}\leq{\rm
Slope}\big{(}\,\hat{\theta}_{n_{1}^{\prime}+n_{2}^{\prime}}({\cal
D}_{n_{1}^{\prime},\tilde{\theta}_{1},\tilde{\rho}_{1}}\cup{\cal
D}_{n_{2}^{\prime},\tilde{\theta}_{2},\tilde{\rho}_{2}})\,\big{)}\leq{\rm
Slope}_{2}$.

###  6.3 Filtering Data to Improve Robustness

The input mixing analysis in Section 6.2 suggests a filtering strategy to
improve robustness.

######  Assumption 4 (Gaussian distributions for filtering).

Consider a training dataset ${\cal D}_{n,\theta_{\rm train},\rho}$ of size $n$
draw i.i.d. from $(X,Y)\sim P_{\theta_{\rm train},\rho}$ where $Y=\pm 1$
uniformly at random and $X|Y\sim{\cal N}(Y\theta_{\rm train},(\|\theta_{\rm
train}\|/\rho)^{2}{\bf I})$. A model trained on an unfiltered ${\cal
D}_{n,\theta_{\rm train},\rho}$ is denoted by $\hat{\theta}_{\rm
unfiltered}=(1/n)\sum_{(x_{i},y_{i})\in{\cal D}_{n,\theta_{\rm
train},\rho}}x_{i}y_{i}$. Note that $\hat{\theta}_{\rm unfiltered}\sim{\cal
N}(\theta_{\rm train},(\|\theta_{\rm train}\|/(\rho\sqrt{n}))^{2}{\bf I})$.

The model is evaluated on two test datasets: in-distribution (ID) and out-of-
distribution (OOD), each with an isotropic Gaussian noise: $(X,Y)\sim
P_{\theta_{\rm ID},\rho_{\rm ID}}$, where $Y=\pm 1$ uniformly at random and
$X|Y\sim{\cal N}(Y\theta_{\rm ID},(\|\theta_{\rm ID}\|/\rho_{\rm ID})^{2}{\bf
I})$, and $P_{\theta_{\rm OOD},\rho_{\rm OOD}}$ is defined similarly. Let
${\rm Slope}(\theta):=\frac{\cos(\theta_{\rm OOD},\theta)\rho_{\rm
OOD}}{\cos(\theta_{\rm ID},\theta)\rho_{\rm ID}}\;$and assume that all models
achieve better ID accuracy than OOD accuracy, i.e., Slope$(\theta)<1$. A model
is more robust if the slope is closer to one.

Suppose we have access to a pre-trained model $\hat{\theta}_{\rm pretrained}$
that achieves better robustness than the model $\hat{\theta}_{\rm unfiltered}$
trained on the unfiltered data, i.e., ${\rm Slope}(\hat{\theta}_{\rm
unfiltered})<{\rm Slope}(\hat{\theta}_{\rm pretrained})\leq 1$. We want to use
the pre-trained model to filter the data ${\cal D}_{n,\theta_{\rm
train},\rho}$ and achieve better robustness. We consider a generic family of
filtering strategies that subsamples each data point $(x_{i},y_{i})\in{\cal
D}_{n,\theta_{\rm train},\rho}$ with probability that is monotonically non-
decreasing with its correlation to the pre-trained model, i.e.,
${\mathbb{P}}((x_{i},y_{i})\text{ passes the filter})\propto h(\langle
x_{i}y_{i},\hat{\theta}_{\rm pretrained}\rangle)$ for some monotonic scalar
function $h:{\mathbb{R}}\to{\mathbb{R}}_{+}$. We analyze the model
$\hat{\theta}_{\rm filtered}=(1/m)\sum_{(x_{i},y_{i})\in\text{filtered
dataset}}x_{i}y_{i}$. The next theorem shows that any such filtering strategy
improves robustness, and the full proof can be found in Appendix F.3.

######  Theorem 3.

Under Assumption 4, if ${\rm Slope}(\hat{\theta}_{\rm unfiltered})<{\rm
Slope}(\hat{\theta}_{\rm pretrained})\leq 1$ then any model $\hat{\theta}_{\rm
filtered}$ that is trained on the above family of filtered datasets achieves
better robustness than the model trained on unfiltered data: ${\rm
Slope}(\hat{\theta}_{\rm unfiltered})<{\rm
Slope}({\mathbb{E}}[\hat{\theta}_{\rm filtered}])\leq{\rm
Slope}(\hat{\theta}_{\rm pretrained})$, where ${\rm
Slope}({\mathbb{E}}[\hat{\theta}_{\rm filtered}])$ denotes the slope of the
linear trend of a model trained on the filtered data.

Remark 3. Relevance to empirical practice: The data filtering approach
currently employed for the LAION dataset [68] motivates our theoretical setup.
The authors of LAION first retrieved a large number of image-text pairs from
Common Crawl [1] and then computed the cosine similarity between the image and
text embeddings using one of the CLIP models originally introduced by Radford
et al. [61]. Next, the authors removed all pairs with similarity less than 0.3
from the dataset. While it was initially unclear if filtering the noisy Common
Crawl source with an existing CLIP model would lead to a dataset with good
generalization properties, experiments with medium-scale models on LAION so
far demonstrate that the resulting models are comparable to the original CLIP
models [2].

Our theorem 3 does not provide a full justification for the filtering process
described above, but it offers theoretical evidence that filtering a noisy
data source with an existing robust model can indeed improve the effective
robustness of the resulting dataset. It remains to be shown empirically that
the slope of the linear trend exhibited by the CLIP model from [61] is larger
than that of a model trained on the Common Crawl data pool without filtering.
If this assumption holds, we can expect that training on data with better
text-image alignment according to the original CLIP model, will improve
robustness compared to training on unfiltered data from Common Crawl.

##  7 Conclusion

Motivated by CLIP’s robustness to distribution shift, we introduced a testbed
of six image-text datasets to study the influence of pre-training data on the
robustness of multimodal models such as CLIP. We analyzed the interactions of
these datasets through both experiments and theoretical analyses, finding that
simply combining multiple datasets dilutes the robustness of the best-
performing one. Moreover, we offer an explanation for the potential benefits
of the data filtering process currently used for LAION. Our findings suggest
that training on a carefully selected subset of data may provide more
robustness than training on simply more data, for example, obtained by
combining multiple data sources.

To summarize, our results lead to the following recommendations for dataset
design:

  * •

On the evaluation front: building a pre-training dataset requires multiple
robustness test sets, as we have shown that different pre-training datasets
exhibit substantially different behaviors across distribution shifts.

  * •

On the dataset design front: each candidate data source for the overall pre-
training dataset should be evaluated separately for its generalization and
robustness properties. The final pre-training dataset should then contain more
samples from the higher-quality pre-training sources. Creating a more
“diverse” pre-training dataset by randomly sampling from all candidate sources
does not result in a better dataset.

  * •

Assuming access to models separately trained on each source of interest,
output ensemble is a good predictor of the effective robustness obtained from
input mixing, and hence can significantly reduce the time to search for the
right mixing strategy.

  * •

Filtering a noisy data source with an existing robust model can improve the
generalization properties of the resulting dataset.

The broad societal implications of image-text models and web-crawled
multimodal datasets have been extensively discussed by Radford et al. [61] and
Birhane et al. [10] respectively. As these models continue to grow in size,
full-scale experiments are becoming prohibitively expensive. This necessitates
small, systematic experiments that allow reliable extrapolation to larger
scales. Since effective robustness is independent of model size and quantity
of training data, we hope that our testbed and results can serve as a first
step towards building multimodal pre-training datasets in a principled manner.

###  7.1 Limitations

The pre-training datasets we experiment with range from 5M to 15M samples.
While this lags behind current state-of-the-art settings that large image-text
models are often trained on, we believe our findings also hold for models with
higher accuracy for the following reasons:

  * •

The original CLIP paper [61] offers evidence showing that CLIP’s performance
exhibits reliable linear trends across compute scales (Figure 9 of [61], which
includes the ResNet50-encoder that our paper works with) and performance on
natural distribution shifts (Figure 14 of [61]). In particular, the linear
trends in this Figure 14 are analogous to the linear trends in our scatter
plots.

  * •

In the OpenCLIP GitHub repository [38] that seeks to match the performance of
the original CLIP paper, the authors there also note that their models that
were trained on 15M samples or fewer share the same effective robustness trend
as OpenAI’s CLIP models (trained on 400M samples). Refer to their Why are low-
accuracy CLIP models interesting? remark for more details.

  * •

Previous work [52] has experimented with training on randomly sampled subsets
of the training dataset to produce models covering a wide range of accuracies,
and found that these models lie on the same effective robustness trend. This
includes models trained on only 1% of the original data, which matches the
scale of our experiments (5M-15M samples) relative to the scale of the
original CLIP paper (400M samples).

The focus of our work is to study the behavior of a diverse set of data
sources, and performing full-scale experiments on 6 datasets (and their
combinations) would be prohibitively expensive. To provide some perspectives
on the amount of resources involved in full-scale experiments: ($i$) One of
the larger datasets that are publicly available, LAION-5B, is a substantial
effort that involves 14 people actively working on it [2], ($ii$) Training a
ResNet50-encoder CLIP on the full dataset of 400M samples took 18 days on 592
V100 GPUs, according to the original paper [61]. This prompts us to leverage
publicly available web-crawled datasets, and study the behavior of our trained
models through the effective robustness framework, which has been shown to be
scale-invariant.

###  7.2 Future Research Directions

Our findings motivate a number of interesting future directions, which we
summarize below:

#### Data sources

  * •

_Comparing more data sources_ : Our textbed of six image-text datasets is only
a first step toward mapping the landscape of pre-training datasets for vision-
language models. There are many more web sources from which pre-training
datasets could be assembled for interesting comparisons, e.g., web searches,
further social media sites (Pinterest, Tumblr, etc.), or captioned videos from
YouTube, to name a few. In addition, there are also more domain-specific data
sources such as iNaturalist [79], BigEarthNet [73], and Patch Camelyon [80]
that likely offer complementary robustness profiles. An interesting outcome of
this research direction would be a “dataset atlas” that allows practitioners
to quickly identify relevant pre-training datasets for their task of interest.

  * •

_Synthetic pre-training data_ : The datasets in our testbed contain images and
text initially created by humans for other purposes and then assembled into a
dataset for model training. An interesting alternative is to create training
data specifically to improve the generalization of image-text models. There
are at least three potential avenues: (i) Rendering virtual 3D environments
[30, 27, 50], (ii) text-to-image generative models [67, 64, 85, 55], and (iii)
image captioning models [53, 89, 72]. All three avenues offer more fine-
grained control over the training data, but in all three cases it is not yet
well understood how synthetic data affects robustness.

  * •

_Data efficiency_ : Our focus has been on the impact of datasets on
generalization through the lens of effective robustness, which depends only on
the training _distribution_ , not the number of samples drawn from the
distribution [52]. A natural complementary question is how the datasets in our
testbed compare in terms of their scaling behavior with training set size. For
instance, it may be the case that some datasets offer less effective
robustness, but the absolute accuracy numbers may improve more quickly with
increasing dataset size than for other data sources.

This direction is closely related to empirical scaling laws for neural
networks [59, 40, 12, 23], which have recently gained attention because the
laws aim to predict model performance as model size, compute, or amount of
training data increases. We provide initial experiments on data scaling for
our testbed with minimal hyperparameter tuning in Appendix C (Figure 17).
Understanding these trends in more detail and testing their reliability (e.g.,
under hyperparameter variations) may serve as a starting point for future
work.

#### Understanding and improving dataset curation

  * •

_Combining data sources at a more fine-grained level_ : We have shown both
empirically and in our theoretical model that simply taking the union of two
datasets dilutes the robustness of the better dataset. This suggests that
improving robust generalization may require more sophisticated methods to
combine datasets, e.g., by selecting individual samples or subsets from each
dataset with complementary generalization properties. We posit that this
approach could take inspiration from the subset selection literature [18, 57,
41, 70].

  * •

_Data filtering techniques_ : Some of the datasets in our testbed such as
LAION and YFCC-15M do not contain “raw” unfiltered data from the web. Instead,
they are already the result of an initial filtering process. In both cases,
this filtering process reduces the size of the dataset by a factor of seven to
eight, and the impact of this filtering on generalization is currently not
well understood. In addition, the filtering approaches can vary substantially:
OpenAI built YFCC-15M from hand-crafted rules while LAION relied on CLIP
models for their filtering process. Experimentally exploring the effect of
these filtering processes is likely an important direction for future work
since the filtering step removes a large number of potential training
examples. Two concrete questions would be whether using higher-accuracy models
in the context of LAION’s filtering process leads to better datasets, and
comparing the LAION dataset to an unfiltered sample from the same source
(Common Crawl).

#### Theory

  * •

_More general theoretical models_ : Our theoretical model could be extended so
it is more aligned with standard experimental settings, e.g., by increasing
the number of classes beyond two, or by including more complex classifiers
beyond the linear model we currently examine. Furthermore, Miller et al. [52]
have shown that non-isotropic covariance shifts in the data distribution lead
to a deviation from the linear trend, at least for fixed classifiers
independent of the training data. In future work, we would like to develop
similar theoretical analyses involving covariance shifts, but for classifiers
with explicit dependence on the training distribution and training set size,
similar to what we have studied in this paper.

  * •

_Low-accuracy regime_ : The emergence of linear trends relating in-
distribution to out-of-distribution accuracy in our theoretical models is
consistent with our empirical observations for classifiers that reach at least
10% accuracy on ImageNet. On the other hand, in a lower accuracy regime, our
theoretical analyses predict that any trained model approaches the in-
distribution and out-of-distribution accuracy of random guessing (see Remark 2
in Section 1). Future work should empirically explore models with lower zero-
shot accuracies (e.g., by reducing the pre-training dataset size) to see how
the empirical trends behave in a lower accuracy regime. If the intercept is
not at the accuracy of random guessing, this suggests an area of improvement
for the theoretical analysis to better fit these new empirical observations in
the low-accuracy regime.

#### Further directions

  * •

_Connections to large language models_ : Our focus in this paper has been on
vision-language models. Another area where large pre-training sets are
essential is unimodal language models [21, 12, 37, 77, 47, 62]. Many of the
questions we address in this paper also arise in the context of language
models. A first step in this direction would be to assemble a comprehensive
robustness testbed for NLP tasks to measure the extent to which the “accuracy
on the line” phenomenon uncovered by Miller et al. [52] also holds in NLP. One
recent set of experiments indicates that this may be the case [51], but they
do not yet include recent large language models such as GPT-3 [12].

  * •

_Dataset design for fine-tuning_ : We have studied the robustness properties
of _zero-shot_ models, but the models can often also be fine-tuned on task-
specific data to increase performance on a target task. It remains an open
question how the pre-training dataset affects the generalization properties of
models after an extra fine-tuning step.

We look forward to addressing these topics in our future work.

## Acknowledgements

We would like to thank Tiffany Cai, Nicholas Carlini, Tatsunori Hashimoto,
Jong Wook Kim, Hongseok Namkoong, Alec Radford, Rohan Taori, and Thomas Wolf
for helpful discussions while working on this paper. We also thank Tiffany Cai
and Jack Gallagher for their help with the dataset collection. HuggingFace has
provided generous assistance in the form of compute resources that made many
of our experiments possible. In particular, we would like to thank Leandro von
Werra and Victor Sanh for their help with setting up the compute
infrastructure and their constructive feedback on the dataset testbed. This
work is supported in part by Open Philanthropy, the Allen Institute for AI,
and NSF grants CNS-2002664, IIS-1929955, DMS-2134012, and CCF-2019844 as a
part of NSF Institute for Foundations of Machine Learning (IFML).

## References

  * [1] Common crawl.  https://commoncrawl.org/.  Accessed: 2022-05-18.
  * [2] Laion-5b: A new era of open large-scale multi-modal datasets.  https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/.  Accessed: 2022-05-18.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arXiv:2204.14198_ , 2022.
  * Andreassen et al. [2021] Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs.  The evolution of out-of-distribution robustness throughout fine-tuning, 2021.  https://arxiv.org/abs/2106.15831.
  * Arjovsky et al. [2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization, 2019.  https://arxiv.org/abs/1907.02893.
  * Barbu et al. [2019] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  _Advances in neural information processing systems_ , 32, 2019.
  * Bauer and Kohavi [1999] Eric Bauer and Ron Kohavi.  An empirical comparison of voting classification algorithms: Bagging, boosting, and variants.  _Machine learning_ , 1999.  https://link.springer.com/article/10.1023/A:1007515423169.
  * Biggio and Roli [2018] Battista Biggio and Fabio Roli.  Wild patterns: Ten years after the rise of adversarial machine learning.  _Pattern Recognition_ , 84:317–331, 2018.
  * Biggio et al. [2013] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pages 387–402. Springer, 2013.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  _arXiv preprint arXiv:2110.01963_ , 2021.
  * Breiman [1996] Leo Breiman.  Bagging predictors.  _Machine learning_ , 1996.  https://link.springer.com/article/10.1007/BF00058655.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568, 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * D’Amour et al. [2020] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al.  Underspecification presents challenges for credibility in modern machine learning, 2020.  https://arxiv.org/abs/2011.03395.
  * Darlow et al. [2018] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey.  Cinic-10 is not imagenet or cifar-10.  _arXiv preprint arXiv:1810.03505_ , 2018.
  * Dasgupta et al. [2019] Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, and Xiaojin Zhu.  Teaching a black-box learner.  In _International Conference on Machine Learning_ , pages 1547–1555. PMLR, 2019.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Conference on Computer Vision and Pattern Recognition_ , 2009\.  https://ieeexplore.ieee.org/document/5206848.
  * Desai et al. [2021] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson.  Redcaps: Web-curated image-text data created by the people, for the people.  _arXiv preprint arXiv:2111.11431_ , 2021.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dietterich [2000] Thomas G Dietterich.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , 2000\.  https://link.springer.com/chapter/10.1007/3-540-45014-9_1.
  * Djolonga et al. [2021] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al.  On robustness and transferability of convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16458–16468, 2021.
  * Fang et al. [2022] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  _arXiv preprint arXiv:2205.01397_ , 2022.
  * Fort et al. [2019] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective, 2019.  https://arxiv.org/abs/1912.02757.
  * Freund and Schapire [1997] Yoav Freund and Robert E Schapire.  A decision-theoretic generalization of on-line learning and an application to boosting.  _Journal of Computer and System Sciences_ , 1997.  https://www.sciencedirect.com/science/article/pii/S002200009791504X.
  * Fu et al. [2021] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.  3d-future: 3d furniture shape with texture.  _International Journal of Computer Vision_ , pages 1–25, 2021.
  * Gao et al. [2021] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters, 2021\.  https://arxiv.org/abs/2110.04544.
  * Gontijo-Lopes et al. [2021] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk.  No one representation to rule them all: Overlapping features of training methods, 2021.  https://arxiv.org/abs/2007.01434.
  * Greff et al. [2022] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al.  Kubric: A scalable dataset generator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3749–3761, 2022.
  * Gulrajani and Lopez-Paz [2020] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  _International Conference on Learning Representations (ICLR)_ , 2020\.  https://arxiv.org/abs/2007.01434.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _International Conference on Learning Representations (ICLR)_ , 2019\.  https://arxiv.org/abs/1903.12261.
  * Hendrycks et al. [2021] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349, 2021.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.  If you use this software, please cite it as below.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR, 2021.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Killamsetty et al. [2021] Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer.  Grad-match: Gradient matching based data subset selection for efficient deep model training.  In _International Conference on Machine Learning_ , pages 5464–5474. PMLR, 2021.
  * Koh et al. [2021] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR, 2021.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Krueger et al. [2021] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex), 2021\.  https://arxiv.org/abs/2003.00688.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.  https://arxiv.org/abs/1612.01474.
  * Li et al. [2021] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  _arXiv preprint arXiv:2110.05208_ , 2021.
  * Lieber et al. [2021] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham.  Jurassic-1: Technical details and evaluation.  _White Paper. AI21 Labs_ , 2021.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * McCormac et al. [2016] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison.  Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth.  _arXiv preprint arXiv:1612.05079_ , 2016.
  * Miller et al. [2020] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt.  The effect of natural distribution shift on question answering models.  In _International Conference on Machine Learning_ , pages 6905–6916. PMLR, 2020.
  * Miller et al. [2021] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt.  Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.  In _International Conference on Machine Learning_ , pages 7721–7735. PMLR, 2021.
  * Mokady et al. [2021] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  _arXiv preprint arXiv:2111.09734_ , 2021.
  * Mu et al. [2021] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  _arXiv preprint arXiv:2112.12750_ , 2021.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nixon et al. [2020] Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran.  Why are bootstrapped deep ensembles not better?  In _”I Can’t Believe It’s Not Better!” NeurIPS 2020 workshop_ , 2020\.  URL https://openreview.net/forum?id=dTCir0ceyv0.
  * Paul et al. [2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  _Advances in Neural Information Processing Systems_ , 34:20596–20607, 2021.
  * Pham et al. [2021] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le.  Combined scaling for zero-shot transfer learning.  _arXiv preprint arXiv:2111.10050_ , 2021.
  * Prato et al. [2021] Gabriele Prato, Simon Guiroy, Ethan Caballero, Irina Rish, and Sarath Chandar.  Scaling laws for the few-shot adaptation of pre-trained image classifiers.  _arXiv preprint arXiv:2110.06990_ , 2021.
  * Quiñonero-Candela et al. [2008] Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  _Dataset shift in machine learning_.  Mit Press, 2008.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Rae et al. [2021] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_ , 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International Conference on Machine Learning_ , pages 5389–5400. PMLR, 2019.
  * Sagawa et al. [2019] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2019.  https://arxiv.org/abs/1911.08731.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Shankar et al. [2020] Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt.  Evaluating machine accuracy on imagenet.  In _International Conference on Machine Learning (ICML)_ , 2020.  http://proceedings.mlr.press/v119/shankar20c/shankar20c.pdf.
  * Sorscher et al. [2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  _arXiv preprint arXiv:2206.14486_ , 2022.
  * Srinivasan et al. [2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2443–2449, 2021\.
  * Stefanini et al. [2021] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.  From show to tell: A survey on image captioning.  _arXiv preprint arXiv:2107.06912_ , 2021.
  * Sumbul et al. [2019] Gencer Sumbul, Marcela Charfuelan, Beg"um Demir, and Volker Markl.  Bigearthnet: A large-scale benchmark archive for remote sensing image understanding.  _CoRR_ , abs/1902.06148, 2019.
  * Szegedy et al. [2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Taori et al. [2020] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.  Measuring robustness to natural distribution shifts in image classification.  _Advances in Neural Information Processing Systems_ , 33:18583–18599, 2020.
  * Thomee et al. [2016] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73, 2016\.
  * Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Torralba and Efros [2011] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In _CVPR 2011_ , pages 1521–1528. IEEE, 2011.
  * Van Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8769–8778, 2018.
  * Veeling et al. [2018] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant cnns for digital pathology.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 210–218. Springer, 2018.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wortsman et al. [2021] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.  Robust fine-tuning of zero-shot models.  _arXiv preprint arXiv:2109.01903_ , 2021.  https://arxiv.org/abs/2109.01903.
  * Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  _International Conference on Machine Learning (ICML)_ , 2022.  https://arxiv.org/abs/2203.05482.
  * Yao et al. [2021] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  _arXiv preprint arXiv:2111.07783_ , 2021.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2022.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zhang et al. [2021] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling, 2021.  https://arxiv.org/abs/2111.03930.
  * Zhou et al. [2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022\.  https://arxiv.org/abs/2109.01134.
  * Zhou et al. [2020] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 13041–13049, 2020.

.tocmtappendix

##  Appendix A Dataset Details

###  A.1 Pre-training Datasets

Dataset | Source | Total Size
---|---|---
YFCC | Flickr | 14,826,000
LAION | Common Crawl | 15,504,742
CC-12M | Unspecified web pages | 9,594,338
RedCaps | Reddit | 11,882,403
WIT | Wikipedia | 5,038,295
ShutterStock | ShutterStock | 15,540,452
Table 1: Origin and total number of samples for each of the datasets we used
in our experiments.

To get a better understanding of the diversity of different data sources, we
analyze the distributions of caption lengths, image sizes and image aspect
ratios for a set of 10,000 samples randomly selected from each source:

Figure 7: Distributions of caption lengths for each data source. Figure 8:
Distributions of image sizes for each data source. Figure 9: Distributions of
image aspect ratios for each data source.

Below we also show some examples of image-caption pairs randomly selected from
each data source:

Figure 10: Random training samples from YFCC. Figure 11: Random training
samples from LAION. Figure 12: Random training samples from Conceptual
Captions (CC-12M). Figure 13: Random training samples from RedCaps. Figure 14:
Random training samples from WIT. Figure 15: Random training samples from
ShutterStock.

###  A.2 Test Distributions

Figure 16 illustrates the four distribution shifts that we use for evaluating
the quality of CLIP features after pre-training on different data sources.

Figure 16: Distribution shifts at test time. We visualize samples of the class
“broom” from the reference distribution ImageNet [19], and the four
distribution shifts derived from ImageNet: ImageNet-V2 [65], ImageNet-R [36],
ImageNet-Sketch [81] and ObjectNet [6].

##  Appendix B Training Details

Our implementation closely follows the training code from OpenCLIP GitHub
repository [38]. When training CLIP from scratch on each of the pre-training
datasets, unless otherwise mentioned, we use AdamW optimizer [49] with default
PyTorch parameters $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$, (per GPU)
batch size 128 and weight decay of 0.1. For learning rate, we start with a
learning rate of $10^{-3}$ and apply a cosine-annealing learning rate schedule
[48] with 5,000 steps. We use the same data augmentations as in [61]. Models
then undergo distributed training on 8 A40 or A100 GPUs for 16 epochs.

##  Appendix C Behavior of Individual Data Sources

Figure 17: Data efficiency of the six pre-training data sources on different
test sets. For each source, we randomly sample various subsets of data with
sizes ranging from 1M to a maximum of 15M samples, and measure the zero-shot
classification error of a CLIP model trained on the subset, on ImageNet and
the four shifted test sets (i.e., ImageNet-V2, ImageNet-R, ImageNet-Sketch,
ObjectNet). Plotted error values are log-transformed and averaged over 3
random seeds. We find that the data efficiency (i.e., how fast the error would
decrease with more samples) of the six data sources varies significantly based
on the evaluation setting.

##  Appendix D Input Mixing

###  D.1 More Experiments with CLIP Pre-training Data Sources

Figure 18: Full plot for Figure 3 with all distribution shifts. Combining YFCC
and LAION training data in equal ratios results in a CLIP model with
intermediate robustness.  Figure 19: Full plot for Figure 4 with all
distribution shifts. Varying the sample contributions of YFCC and LAION to the
input data mixture produces a smooth interpolation of the linear trend between
the trends of training on YFCC and LAION separately.  Figure 20: Input mixing
results for YFCC and RedCaps data sources. Similar to previous observations
(Figure 4), combining YFCC and RedCaps data in the pre-training dataset with
different ratios yields different linear trends that all lie between that of
training on YFCC and that of training on RedCaps alone.  Figure 21: Input
mixing results for all six data sources. We combine data from all six sources
in the testbed with equal ratios (i.e., taking 2.7M samples from each), and
find that the resulting robustness of CLIP trained on this data mixture (black
line), is less than that of training only on the best-performing data source
for each distribution shift setting.

###  D.2 Experiments on CIFAR-10 & CINIC-10

We also investigate the phenomenon that mixing data sources resulting in
diluted robustness (Section 5.1) in smaller-scale, uni-modal classification
settings. Here, we experiment with mixing CIFAR-10 [43] and CINIC-10 [17]
sources, each having 50K samples in total. CINIC-10 is itself a mixture of
CIFAR-10 images and images selected and downsampled from the ImageNet database
(for the same 10 classes). We use three architectures—ResNet-18, ResNet-34 and
ResNet-50 [32]—and vary the number of epochs of training to obtain models of
different accuracies. Models are evaluated on both CIFAR-10 and CINIC-10
standard test sets, and their performances are plotted along the axes of a
scatter plot. Similar to previous input mixing results for CLIP, we observe in
Figure 22 that ResNets trained on a 50K-sample dataset made up of both
CIFAR-10 and CINIC-10 data, produce a linear trend that lies in between the
trends of training models separately on just 50K CIFAR-10 images and just 50K
CINIC-10 images.

Figure 22: Mixing inputs from CIFAR-10 and CINIC-10 distributions also
produces models with intermediate robustness. Similar to our findings from the
multimodal setting with CLIP pre-training, we also observe that for standard
image classification tasks like CIFAR-10 and CINIC-10, combining data samples
from these two distributions with varying ratios ends up diluting the
robustness of the original sources. The training set size is fixed at 50K
samples for all linear trends displayed in this plot.

##  Appendix E Output Mixing

###  E.1 More Experiments with CLIP Pre-training Data Sources

Figure 23: Full plot for Figure 5 with all distribution shifts. Ensemble
outputs of two CLIP models trained on YFCC and LAION separately share the same
linear trend as a single model trained on the combined data mixture (with
equal sample contribution from each source).  Figure 24: Full plot for Figure
6 with all distribution shifts. Given an existing pre-training dataset that
could be a mixture (e.g., YFCC-5M + LAION-5M, green line) and a new data
source (e.g., CC-5M, orange line), we could use the ensemble outputs (blue
markers) of two CLIP models that have been trained separately on these two
data distributions, to estimate the linear trend for models that would be
trained on all the data (purple line).  Figure 25:  Output mixing results for
two CLIP models trained on YFCC-3M + CC-3M mixture and ShutterStock-3M
respectively. We repeat the experiment in Figure 24 for a different set of
data sources (YFCC, ShutterStock, Conceptual Captions), taking 3M samples from
each. The same output mixing phenomenon applies: the ensemble outputs of CLIPs
trained on different data sources and dataset sizes (purple and orange lines),
taken from the same epoch, lie on the linear trend of training a single model
on the combined dataset made up of these three sources (cyan line). The two
models’ logit predictions are ensembled with equal weights (blue markers).
Figure 26:  Output mixing results for two CLIP models trained on YFCC-3M +
CC-3M mixture and RedCaps-3M respectively. Ensemble outputs of CLIPs trained
on different data sources and dataset sizes (red and orange lines), taken from
the same stage of training (i.e., epoch), lie on the linear trend of training
a single model on the combined dataset made up of these three sources (cyan
line), when the two models’ logit predictions are ensembled with equal weights
(blue markers).  Figure 27:  Ensemble outputs of CLIPs trained separately on
each of the data sources of interest share the same linear trend as a single
CLIP model trained on the 6-source data mixture. Following the input mixing
setup in Figure 21, when we ensemble the logit predictions of six CLIP models,
each trained on 2.7M samples randomly selected from a single data source, with
equal weights, we find that the ensemble outputs are also predictive of the
linear trend of training CLIP models on a single data mixture made up of 2.7M
samples from each source.

###  E.2 Experiments on CIFAR-10 & CINIC-10

Figure 28: Ensembling outputs of two models trained separately on CIFAR10 and
CINIC10 lie on the same linear trend as training from scratch on the combined
data mixture (where each source contributed equally). We combine the logit
predictions of CINIC10-trained and CIFAR10-trained models that have the same
architecture (e.g., ResNet-18, ResNet-34 and ResNet-50 in this case) with
varying ensemble weights between 0 and 1 (dashed lines). Similar to our
findings from the multimodal setting with CLIP, we also observe that when the
predictions are combined with equal weights (markers on the dashed lines), the
resulting test accuracies on the two corresponding test sets lie on the linear
trend produced by training ResNets on a CIFAR10 + CINIC10 data mixture with
equal number of samples from each source.

##  Appendix F Proofs of the Analyses

We provide proofs of main theoretical claims in Section 6.

###  F.1 Proof of Theorem 1

######  Assumption 5.

Under the hypotheses of Theorem 1, suppose there exists a positive constant
$c$ such that the third moments are bounded by ${\mathbb{E}}_{(X,Y)\sim
P_{\theta_{1},\rho_{1}}}[(X_{i}Y-\theta_{1,i})^{3}]\leq
c{\mathbb{E}}_{(X,Y)\sim
P_{\theta_{1},\rho_{1}}}[(X_{i}Y-\theta_{1,i})^{2}]^{3/2}$,
${\mathbb{E}}_{(X,Y)\sim
P_{\theta_{2},\rho_{2}}}[(X_{i}Y-\theta_{1,i})^{3}]\leq
c{\mathbb{E}}_{(X,Y)\sim
P_{\theta_{2},\rho_{2}}}[(X_{i}Y-\theta_{1,i})^{2}]^{3/2}$, and
${\mathbb{E}}[(\hat{\theta}_{n,i}-\theta_{i})^{3}]\leq
c{\mathbb{E}}[(\hat{\theta}_{n,i}-\theta_{i})^{2}]^{3/2}$ for all $i\in[d]$.

Under this assumption, we show that

$\displaystyle\Phi^{-1}({\rm Acc}_{\theta_{1},\rho_{1}})$ $\displaystyle=$
$\displaystyle\cos(\theta_{1},\theta)\rho_{1}\frac{\rho}{\xi}\sqrt{\frac{n}{d}}\,+\,O\Big{(}\,\frac{\exp({\frac{\rho_{1}^{2}\rho^{2}n}{2\xi^{2}d}})}{\sqrt{n}}\,\Big{)}\;,\text{
and }$ (4) $\displaystyle\Phi^{-1}({\rm Acc}_{\theta_{2},\rho_{2}})$
$\displaystyle=$
$\displaystyle\cos(\theta_{2},\theta)\rho_{2}\frac{\rho}{\xi}\sqrt{\frac{n}{d}}\,+\,O\Big{(}\,\frac{\exp({\frac{\rho_{2}^{2}\rho^{2}n}{2\xi^{2}d}})}{\sqrt{n}}\,\Big{)}\;,$
(5)

as it will make the first and second claims straightforward. For
$(X_{1},Y_{1})\sim P_{\theta_{1},\rho_{1}}$, the first error event is $\\{{\rm
sign}(\langle X_{1},\hat{\theta}_{n,\xi}\rangle)\neq Y_{1}\\}=\\{\langle
X_{1},\hat{\theta}_{n,\xi}\rangle Y_{1}\leq
0\\}=\\{\langle\theta+(\xi\|\theta\|/\rho\sqrt{n})z,\theta_{1}+(\|\theta_{1}\|/\rho_{1})z_{1}\rangle\leq
0\\}$, where we used the fact that
$\hat{\theta}_{n,\xi}=\theta+(\xi\|\theta\|/\rho\sqrt{n})z$ and
$XY\overset{d}{=}\theta_{1}+(\|\theta_{1}\|/\rho_{1})z_{1}$. Since the third
moments are bounded, applying Berry-Esseen theorem, we get that the
probability of error is bounded by
$\Phi(-(\langle\theta,\theta_{1}\rangle\rho_{1}\rho\sqrt{n}/(\xi\|\theta_{1}\|\|\theta\|\sqrt{d})))+O(1/\sqrt{d})$.
This gives ${\rm
Acc_{\theta_{1},\rho_{1}}}=\Phi(\langle\theta,\theta_{1}\rangle\rho_{1}\rho\sqrt{n}/(\xi\|\theta_{1}\|\|\theta\|\sqrt{d}))+O(1/\sqrt{d})$,
and consequently

$\displaystyle\Phi^{-1}({\rm Acc_{\theta_{1},\rho_{1}}})$ $\displaystyle=$
$\displaystyle\cos(\theta_{1},\theta)\,\rho_{1}\,\frac{\rho}{\xi}\sqrt{\frac{n}{d}}+O\Big{(}\,\frac{e^{\frac{\cos(\theta_{1},\theta)^{2}\rho_{1}^{2}\rho^{2}n}{2\xi^{2}d}}}{\sqrt{n}}\,\Big{)}\;.$
(6)

This proves the desired claim.

###  F.2 Proof of Theorem 2

Recall that ${\rm Slope}(\hat{\theta}_{n}({\cal
D}_{n,\theta,\rho}))=c\langle\theta_{2},\theta\rangle/\langle\theta_{1},\theta\rangle$
for a positive constant $c>0$ that does not depend on the training data.
Although the slope only depends on the training data and algorithm through
$\theta$, we write all the parameters including the sample size $n$ and the
training SNR $\rho$ to make it explicit that the results hold for all
variations of the sample size and the training algorithm within the class that
we assume. It is sufficient to show that this is a monotonic function over
$\theta$ when we linearly traverse from $\tilde{\theta}_{1}$ to
$\tilde{\theta}_{2}$, i.e.
$\theta(\alpha)=\alpha\tilde{\theta}_{1}+(1-\alpha)\tilde{\theta}_{2}$ for
$\alpha\in[0,1]$. Note that
$f(\alpha)=c\langle\theta_{2},\theta(\alpha)\rangle/\langle\theta_{1},\theta(\alpha)\rangle=c_{1}+c_{2}/\langle\theta_{1},\theta(\alpha)\rangle$
for some $c_{1}$ and $c_{2}$ that do not depend on $\alpha$. The monotonicity
follows from the fact that the derivative is

$\frac{\partial
f(\alpha)}{\partial\alpha}\;=\;-c_{2}\frac{\langle\theta_{1},\tilde{\theta}_{1}-\tilde{\theta}_{2}\rangle}{\langle\theta_{1},\theta(\alpha)\rangle^{2}}\;,$

whose sign does not change for any $\alpha$.

###  F.3 Proof of Theorem 3

The train data distribution satisfies $x_{i}y_{i}\sim{\cal N}(\theta_{\rm
train},(\|\theta_{\rm train}\|/\rho)^{2}{\bf I})$. Note that filtering does
not change the distribution in $d-1$ dimensional subspace orthogonal to
$\hat{\theta}_{\rm pretrained}$, due to rotation invariance of a spherical
Gaussian distribution. This implies that ${\mathbb{E}}[{\cal
P}_{\perp}(\hat{\theta}_{\rm filtered})]={\cal P}_{\perp}(\theta_{\rm
train})$, where ${\cal P}_{\perp}$ denotes the projection operator to the
$d-1$ dimensional subspace orthogonal to $\hat{\theta}_{\rm pretrained}$. On
the other hand, on the direction of $\hat{\theta}_{\rm pretrained}$, the
filtering increases the correlation in expectation: $|{\mathbb{E}}[{\cal
P}_{\parallel}(\hat{\theta}_{\rm filtered})]-{\cal
P}_{\parallel}(\hat{\theta}_{\rm pretrained})|\leq|{\cal
P}_{\parallel}(\theta_{\rm train})-{\cal P}_{\parallel}(\hat{\theta}_{\rm
pretrained})|$, where ${\cal P}_{\parallel}$ denotes the projection operator
to the one dimensional subspace spanned by $\hat{\theta}_{\rm pretrained}$.
This implies that ${\mathbb{E}}[\hat{\theta}_{\rm filtered}]=\theta_{\rm
train}+c\hat{\theta}_{\rm pretrained}$ for some positive $c$. It follows that
${\mathbb{E}}[\hat{\theta}_{\rm filtered}]$ is a convex interpolation between
two vectors, each in the direction of $\theta_{\rm trian}$ and
$\hat{\theta}_{\rm pretrained}$, respectively. We can apply Theorem 2 which
gives that

${\rm Slope}(\hat{\theta}_{\rm unfiltered})<{\rm
Slope}({\mathbb{E}}[\hat{\theta}_{\rm filtered}])\leq{\rm
Slope}(\hat{\theta}_{\rm pretrained})\;,$

when ${\rm Slope}(\hat{\theta}_{\rm unfiltered})<{\rm Slope}(\hat{\theta}_{\rm
pretrained})$.