00footnotetext: Equal contribution, randomly ordered. Correspondence to
<EMAIL_ADDRESS>1University of Washington 2Columbia University 3Tel Aviv
University 4Apple 5UT Austin 6LAION 7AI2 8Juelich Supercomputing Center,
Research Center Juelich 9University of Illinois Urbana-Champaign 10Graz
University of Technology 11Hebrew University 12Google Research 13Snorkel AI

#  DataComp:
In search of the next generation of multimodal datasets

Samir Yitzhak Gadre*2, Gabriel Ilharco*1, Alex Fang*1, Jonathan Hayase1,
Georgios Smyrnis5, Thao Nguyen1, Ryan Marten7,9, Mitchell Wortsman1,
Dhruba Ghosh1, Jieyu Zhang1, Eyal Orgad3, Rahim Entezari10, Giannis Daras5,
Sarah Pratt1, Vivek Ramanujan1, Yonatan Bitton11, Kalyani Marathe1,
Stephen Mussmann1, Richard Vencu6, Mehdi Cherti6,8, Ranjay Krishna1,
Pang Wei Koh1,12, Olga Saukh10, Alexander Ratner1,13, Shuran Song2,
Hannaneh Hajishirzi1,7, Ali Farhadi1, Romain Beaumont6,
Sewoong Oh1, Alex Dimakis5, Jenia Jitsev6,8,
Yair Carmon3, Vaishaal Shankar4, Ludwig Schmidt1,6,7

###### Abstract

Multimodal datasets are a critical component in recent breakthroughs such as
CLIP, Stable Diffusion and GPT-4, yet their design does not receive the same
research attention as model architectures or training algorithms. To address
this shortcoming in the machine learning ecosystem, we introduce DataComp, a
testbed for dataset experiments centered around a new candidate pool of 12.8
billion image-text pairs from Common Crawl. Participants in our benchmark
design new filtering techniques or curate new data sources and then evaluate
their new dataset by running our standardized CLIP training code and testing
the resulting model on 38 downstream test sets. Our benchmark consists of
multiple compute scales spanning four orders of magnitude, which enables the
study of scaling trends and makes the benchmark accessible to researchers with
varying resources. Our baseline experiments show that the DataComp workflow
leads to better training sets. Our best baseline, DataComp-1B, enables
training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet,
outperforming OpenAI’s CLIP ViT-L/14 by 3.7 percentage points while using the
same training procedure and compute. We release DataComp and all accompanying
code at www.datacomp.ai.

###  1 Introduction

Recent advances in multimodal learning such as CLIP [111], DALL-E [115, 116],
Stable Diffusion [123], Flamingo [8], and GPT-4 [103] offer unprecedented
generalization capabilities in zero-shot classification, image generation, and
in-context learning. While these advances use different algorithmic
techniques, e.g., contrastive learning, diffusion, or auto-regressive
modeling, they all rest on a common foundation: large datasets containing
paired image-text examples. For instance, CLIP’s training set contains 400
million image-text pairs, and Stable Diffusion was trained on the two billion
examples from LAION-2B [129]. This new generation of image-text datasets is
1,000 times larger than previous datasets such as ImageNet, which contains
1.2M images [37, 126].

Table 1: Zero-shot performance of CLIP models trained on different datasets. DataComp-1B, assembled with a simple filtering procedure on image-text pairs from Common Crawl, leads to a model with higher accuracy than previous results while using the same number of multiply-accumulate operations (MACs) or less during training. See Section 3.5 for details on the evaluation datasets.  |  | # samples |  | Train compute | ImageNet
---|---|---|---|---|---
Dataset | Dataset size | seen | Architecture | (MACs) | accuracy
OpenAI’s WIT [111] | 0.4B | 13B | ViT-L/14 | $1.1\times 10^{21}$ | 75.5
LAION-400M [128, 28] | 0.4B | 13B | ViT-L/14 | $1.1\times 10^{21}$ | 72.8
LAION-2B [129, 28] | 2.3B | 13B | ViT-L/14 | $1.1\times 10^{21}$ | 73.1
LAION-2B [129, 28] | 2.3B | 34B | ViT-H/14 | $6.5\times 10^{21}$ | 78.0
LAION-2B [129, 28] | 2.3B | 34B | ViT-g/14 | $9.9\times 10^{21}$ | 78.5
DataComp-1B (ours) | 1.4B | 13B | ViT-L/14 | $1.1\times 10^{21}$ | 79.2

Despite the central role of image-text datasets, little is known about them.
Many state-of-the-art datasets are proprietary, and even for public datasets
such as LAION-2B [129], it is unclear how design choices such as the data
source or filtering techniques affect the resulting models. While there are
thousands of ablation studies for algorithmic design choices (loss function,
model architecture, etc.), datasets are often treated as monolithic artifacts
without detailed investigation. Moreover, datasets currently lack the
benchmark-driven development process that has enabled a steady stream of
improvements on the model side and isolates data enhancements from changes to
the model. These issues impede further progress in multimodal learning, as
evidenced by recent work showing that public datasets currently do not match
the scaling behavior of proprietary alternatives [28].

In this paper, we take a step towards a more rigorous dataset development
process. Our first and central contribution is DataComp, a new benchmark for
multimodal dataset design. DataComp flips the traditional benchmarking
paradigm in machine learning where the dataset is fixed and researchers
propose new training algorithms. Instead, we hold the entire training code and
computational budget constant so that participants innovate by proposing new
training sets. To evaluate the quality of a training set, we score the
resulting model with a testbed of 38 classification and retrieval tasks such
as ImageNet [37], ImageNetV2 [121], DTD [30], EuroSAT [63], SUN-397 [146], and
MSCOCO [26].

DataComp focuses on two key challenges that arise when assembling large
training datasets: what data sources to train on, and how to filter a given
data source. Each challenge corresponds to one track in our benchmark. To
facilitate the _filtering track_ , our second contribution is CommonPool, a
dataset of 12.8B image-text pairs collected from Common Crawl and currently
the largest public image-text dataset. We release CommonPool as an index of
image url-text pairs under a CC-BY-4.0 license, and apply content checks in
its construction to remove unsafe or unwanted content. In the _filtering
track_ , the goal of participants is to find the best subset of CommonPool to
train on. In the second track, _Bring Your Own Data_ (BYOD), participants may
leverage any data source, as long as it does not overlap with our evaluation
testbed.

Our third contribution is an investigation of scaling trends for dataset
design. In particular, DataComp contains _four_ scales, where we vary the
training budget and the candidate pool size from 12.8M to 12.8B samples (see
Table 2). Expressed in GPU hours, the cost of a single training run ranges
from 4 to 40,000 GPU hours on the A100 cluster we used for development. The
different scales enable researchers with different resources to participate in
our benchmark. Moreover, our results show that the ranking of filtering
approaches is largely consistent across scale.

Our fourth contribution is over three hundred baseline experiments, including
techniques such as querying captions for relevant keywords, filtering based on
image embeddings, and applying a threshold on CLIP scores. A key result from
our baselines experiments is that smaller, more stringently filtered datasets
can lead to models that generalize _better_ than larger datasets coming from
the same pool. At the 12.8B scale, our best filtering baseline increases
ImageNet zero-shot accuracy by 6.9 percentage points (pp) relative to the
unfiltered pool (see Table 3). For the BYOD track, our initial experiments
show that 109M additional data points (less than 1% of the 12.8B pool) improve
the CLIP-filtered subsets of CommonPool by up to 1.2 pp ImageNet accuracy (see
Table 18).

Finally, our fifth contribution is DataComp-1B, a new state-of-the-art
multimodal dataset. We obtain DataComp-1B by combining our two most promising
filtering baselines. DataComp-1B enables training a CLIP ViT-L/14 model to an
ImageNet zero-shot accuracy of 79.2% (see Table 1), corresponding to a
$9\times$ computational cost reduction when compared to a larger CLIP ViT-g/14
model trained on LAION-2B for about $3\times$ longer. Moreover, our model
outperforms OpenAI’s original CLIP ViT-L/14 by 3.7 percentage points, while
using the same compute budget.

To make DataComp a shared environment for controlled dataset experiments, we
publicly release our candidate pool url index, our tooling for assembling
these pools, our filtering baselines, and our code for training and evaluating
models at www.datacomp.ai. We believe that our infrastructure will help put
research on dataset design on rigorous empirical foundations, draw attention
to this understudied research area, and lead to the next generation of
multimodal datasets.

Figure 1: DataComp participant workflow. A) Choose a scale based on resource
constraints. B) Design a dataset, in either the filtering or BYOD track. C)
Train a CLIP model on the designed dataset using a fixed architecture and
hyperparameters (Section 3.4). D) Evaluate the trained model on a suite of
diverse downstream tasks (Section 3.5).

###  2 Related Work

We review the most closely related work and include additional related work in
Appendix C.

###### The effects of data curation.

Classical work considers dataset cleaning and outlier removal [74, 152, 124,
125] to discard samples that may lead to undesirable model bias. A related
line of work develops coreset selection algorithms [61, 7, 46, 11, 94, 145,
32], which aim to select data subsets that lead to the same performance as
training on the entire dataset. These techniques appear to scale poorly to
larger data regimes [51, 6]. More recent efforts in subset selection often
operate on already curated datasets [98, 141, 130, 16, 33, 106] (e.g.,
CIFAR-10, ImageNet) or on smaller data regimes (e.g., YFCC-15M [111, 140]).
These settings often do not reflect newer training paradigms that involve (1)
_noisy_ image-text pairs instead of category labeled images and (2) large
scale datasets (e.g., billions of samples). While data-centric investigations
have led to community competitions like dcbench [43] and DataPerf [97],
existing benchmarks have likewise operated at small data scales [100] compared
to datasets like LAION-2B [129], which contains over two billion images.
DataComp bridges this gap by aligning data-centric investigation with large
scale image-text training.

There has also been renewed interest in dataset pruning and deduplication.
Sorscher et al. [135] show that data pruning can improve traditional scaling
trends on ImageNet, but do not consider image-text training or larger
datasets. Raffel et al. [113] remove sentence redundancies when creating the
C4 corpus. Subsequent work further demonstrated the benefits of deduplication
for better language modeling [90]. Radenovic et al. [110] introduce CAT
filtering for image-text datasets—a rule-based system to retain high quality
samples. Abbas et al. [6] propose SemDeDup, which starts with the CAT-filtered
LAION-440M subset, further employing clustering to remove semantic duplicates.
DataComp facilitates data-centric investigation at an even larger scale (i.e.,
12.8B sample scale) and provides a common experimental setting for fair
comparison amongst dataset creation algorithms.

###### Large-scale multimodal datasets.

Datasets have been instrumental to building multimodal models like CLIP [111],
Flamingo [8], Stable Diffusion [123], DALL-E [115, 116] and GPT-4 [103]. These
methods succeeded by training on large, heterogeneous datasets rather than
solely through advanced modelling techniques. For example, OpenAI’s CLIP
trains on 400M image-text pairs from the web, roughly $300\times$ the size of
ImageNet [37]. Prior work on scaling image-text datasets also provides
promising trends with respect to zero-shot model performance [73, 107].
Additional large scale datasets like FILIP-300M [149], FLD-900M [153], and
PaLI-10B [25] were constructed to train multimodal models. However, many
datasets used to train such models (including the dataset for OpenAI’s CLIP)
are proprietary, making it hard to conduct data-centric investigations.

Even for public image-text datasets like SBU [104], Flickr30k [151], MS-COCO
[26], TaiSu [92], Conceptual Captions [131], CC12M [24], RedCaps [38], WIT
[136], Shutterstock [101], YFCC-100M [140], COYO-700M [20], LAION-400M [128],
or LAION-2B [129] little is known about what constitutes a good image-text
dataset. Preliminary analysis suggests that different image-text data sources
lead to CLIP models with different properties [101]. However, previous work is
limited to smaller scale data (10-15M examples). Birhane et al. [15] examine
LAION-400M and find NSFW imagery and racial slurs, centering the dangers in
web-scale multimodal datasets. To combat toxicity, we preprocess our pool to
remove NSFW content and blur human faces detected in images. For more details
on our safety preprocessing see Section 3.2, Appendices E and G.

###  3 The DataComp benchmark

DataComp is meant to facilitate data-centric experimentation. While
traditional benchmarks emphasize model design, DataComp is centered around
dataset development, where the resulting datasets can be used to train high
accuracy models. We focus on large image-text datasets and quantify a dataset
submission by training a CLIP model on it from scratch [111] and evaluating on
38 downstream image classification and retrieval tasks. We additionally have
three secret test sets, which will be released after a year, to guard against
overfitting. To facilitate such investigations, we provide a candidate pool of
uncurated image-text pairs sourced from the public internet. Our benchmark
offers two tracks: one where participants must filter samples from the pools
we provide, and another where participants can use external data. Moreover,
DataComp is structured to accommodate participants with diverse levels of
computational resources: each track is broken down into four scales with
varying compute requirements. We now discuss high-level design decisions,
construction of a 12.8B image-text data pool to facilitate the competition,
benchmark tracks, model training, and evaluation.

####  3.1 Competition design

###### Overview.

In many areas of machine learning, larger datasets lead to better performing
models [87, 79, 73, 107, 66, 28, 19, 111, 112]. Hence comparing only datasets
with the same size is a natural starting point. However, this approach is
flawed as controlling the dataset size ignores critical curation constraints:
candidate pool size (i.e., number of image-text pairs to harvest) and training
compute. For instance, assembling a dataset like LAION-2B consists of
identifying _data sources_ (e.g., Common Crawl or Reddit) and _filtering_ the
data source. Notably, _the final dataset size is a design choice_ and is only
upper-bounded by the data sources. Hence, the true data constraint is the size
of the reservoir of samples: _candidate pool_ to be filtered. To make DataComp
a realistic benchmark, we therefore fix the candidate pool in the filtering
track, but give participants control over the training set size.

Compute cost is another relevant constraint. To put datasets of different size
on equal footing, we specify the total _number of training samples seen_.
Consider the 12.8B compute scale and filtered datasets $A$ and $B$, with 6.4B
and 3.2B image-text pairs respectively. At this scale, we train by making two
passes over $A$, while making four passes over $B$. A key result from our
experiments is that smaller, more stringently filtered datasets can lead to
models that generalize _better_.

###### Competition tracks.

Two key procedures in assembling a training dataset are filtering a data
source [128, 129, 20] and aggregating data sources [36, 37]. To reflect this
structure, DataComp has two tracks: filtering, where participants select a
subset of the samples from CommonPool, and Bring Your Own Data (BYOD), where
participants can use any source of data. Key decisions for each tracks are
described in Sections 3.2 and 3.3, respectively. For full competition track
rules see Appendix A.

Table 2: Experimental configurations, with compute in multiply-accumulate operations (MACs).  Scale | Model | Train compute (MACs) | Pool size and # samples seen
---|---|---|---
small | ViT-B/32 | $9.5\times 10^{16}$ | 12.8M
medium | ViT-B/32 | $9.5\times 10^{17}$ | 128M
large | ViT-B/16 | $2.6\times 10^{19}$ | 1.28B
xlarge | ViT-L/14 | $1.1\times 10^{21}$ | 12.8B

###### Competition compute scales.

To facilitate study of scaling trends and accommodate participants with
various computational resources, we structure DataComp using four scales of
compute: small, medium, large and xlarge. Each new scale increases the number
of samples seen during training by 10$\times$ (from 12.8M to 12.8B samples
seen), and the pool we provide by the same factor (from 12.8M samples to 12.8B
samples). Table 2 gives the experimental configuration used for each scale.
For the small scale, our runs took 4 hours on an A100 GPU, and for the xlarge
scale 81 hours on 512 GPUs.

####  3.2 CommonPool generation, for the filtering track

We construct a large-scale pool of image-text pairs, CommonPool, from Common
Crawl [3]. CommonPool is distributed as an image url-text pair index under a
CC-BY-4.0 license. Our pool construction pipeline has four steps: url
extraction and data download, NSFW detection, evaluation set deduplication,
and face blurring. We additionally provide per sample metadata (e.g., CLIP
features). Starting from the xlarge CommonPool, we take successive random
subsets to create large, medium, and small CommonPool (e.g., medium is a
subset of large).

###### Extracting urls and dowloading data.

We first use cc2dataset [1], which utilizes Apache Spark [155], to extract
pairs of image urls and nonempty alt-text from all Common Crawl snapshots from
2014 to 2022. We then deduplicate the url-text pairs and randomly shuffle.
This step results in $\sim$88B possible samples. Not all samples are
downloadable; other samples are not suitable due to NSFW content or overlap
with our evaluation sets. We attempt to download $\sim$40B samples using
img2dataset [5] resulting in $\sim$16.8B image-text pairs. For more details,
see Appendix D.

###### Safety preprocessing.

Since Common Crawl is a snapshot of the internet, we require strict
preprocessing to remove unsafe content. We use Detoxify [60] to prune samples
that contain unsafe text (e.g., obscene, sexually explicit, or threatening
language). We also discard samples with explicit visual content. To do so, we
train a classifier on CLIP ViT-L/14 [111] features, using the NSFW dataset
used in LAION-5B [129]. We validate our classifier against the Google
commercial image safety API. See Appendix E for details. Around 19% of image-
text pairs are considered NSFW, taking the pool of $\sim$16.8B downloads to
$\sim$13.6B samples.

###### Evaluation set deduplication.

To prevent accidental overfitting to certain test sets in our evaluation
suite, we perform a thorough near-duplicate removal between the candidate pool
and our evaluation sets, using a state-of-the-art image deduplication model
[150]. Appendix F contains additional details. The model flags $\sim$3% of the
16.8B images as near-duplicates, reducing the $\sim$13.6B pool to $\sim$13.1B
samples. From here we select a random subset to get the xlarge pool of 12.8B
samples.

###### Face detection & blurring.

To protect the privacy of individuals, we detect and blur faces from images in
our pool using a face detector [53]. As observed by Yang et al. [148],
obfuscating faces has little impact on model performance, as we also observe
in our experiments (Appendix G).

###### Pool metadata.

To bootstrap participants we distribute metadata for each sample in CommonPool
(e.g., image url, alt-text, original image resolution, CLIP features, and CLIP
similarity scores). Following Carlini et al. [22], we release SHA256 hashes
for each image to guard against data poisoning in subsequent CommonPool
downloads. For additional details see Appendix H. We open-source our metadata
processing pipeline as dataset2metadata [4].

####  3.3 The bring your own data (BYOD) track

While CommonPool can be used to study different filtering techniques, state-
of-the-art models often train on data from different sources. For instance,
the Flamingo model [8] uses both multimodal massive web (M3W) and ALIGN
datasets [73]. To facilitate non-proprietary research on curating data from
many sources, we instantiate a separate DataComp track to allow participants
to combine multiple data streams. For example, participants could construct a
training set from CC12M [24], YFCC100M [140], and data sources they label
themselves. In Section 4.2 and Appendix P.2 we describe our exploration using
existing public, image-text datasets. These datasets are acquired from their
respective sources and are not re-release as part of DataComp.

####  3.4 Training

We create a common experimental setting that enables comparable experiments by
fixing the training procedure. We closely follow the CLIP training recipe
proposed by Radford et al. [111]: training models from scratch with a
contrastive objective over images and captions. Given a set of image-caption
pairs, we train an image encoder and a text encoder such that the similarity
between the representations of images and their corresponding text is
maximized relative to unaligned pairs.111More precisely, given a batch of data
$\\{(x_{1},y_{1}),...,(x_{B},y_{B})\\}$ with images $x$ and captions $y$, we
train the image encoder $g$ and text encoder $v$ with the loss
$\ell=\frac{1}{2}\sum_{i=1}^{B}\frac{\sigma_{ii}}{\sum_{j=1}^{B}\sigma_{ij}}+\frac{1}{2}\sum_{i=1}^{B}\frac{\sigma_{ii}}{\sum_{j=1}^{B}\sigma_{ji}}$,
where $\sigma_{ij}=\exp{\langle g(x_{i}),h(y_{j})\rangle}$. We also use a
learnable temperature parameter as in Radford et al. [111]. For each scale, we
fix the model architecture and hyperparameters (see Table 2). We pick Vision
Transformers (ViTs) [39] as the image encoder, considering the better scaling
trends observed by Radford et al. [111] compared to ResNets [62]. Models are
trained for a fixed number of steps determined by the scale (Table 2), using
the OpenCLIP repository [69]. See Appendix N for details.

####  3.5 Evaluation

We evaluate on a suite of 38 image classification and retrieval tasks. We also
study two additional fairness tasks, detailed in Section 5 and Appendix Q. As
discussed in Section 3.2, we remove test set images from DataComp to avoid
contamination. Image classification datasets range from satellite imagery
recognition to classifying metastatic tissues. In total we have (with some
overlap): 22 of the datasets evaluated in Radford et al. [111], 6 ImageNet
distribution shifts (i.e., ImageNet-Sketch [143], ImageNet-V2 [121],
ImageNet-A [65], ImageNet-O [65], ImageNet-R [64], and ObjectNet [13]), 13
datasets from VTAB [156], and 3 datasets from WILDS [83, 127]. Retrieval
datasets include Flickr30k [151], MSCOCO [26], and the WinoGAViL commonsense
association task [17]. To aggregate results over all evaluation tasks, we
average the preferred metric for each task.

DataComp adopts a zero-shot evaluation protocol: models are tested without
training on the evaluation tasks. This approach is computationally efficient
and measures a model’s ability to perform well without any additional
training. We find a strong rank correlation (${>}0.99$) between performance in
linear probe zero-shot settings (Appendix Figure 16). Additional details are
in Appendix O.

###  4 Baselines

####  4.1 Filtering baselines

We study six simple filtering methods for the filtering track; see Section P.1
for further details.

###### No filtering.

We simply use the entire pool as the subset, without any filtering. Since each
pool size is equal to the sample budget, training consists of one pass over
the data.

###### Random subsets.

To isolate the effects of increasing the compute budget from increasing the
dataset size, we form subsets consisting of 1%, 10%, 25%, 50% and 75% of the
pool chosen at random.

###### Basic filtering.

We consider many simple filtering operations inspired by Schuhmann et al.
[128] and Byeon et al. [20]: filtering by _language_ (English captions, using
either fasttext [77] or cld3 [2]); filtering by _caption length_ (over two
words and five characters); and filtering by _image size_ (smaller dimension
above 200 pixels and aspect ratio below three). We also experiment with
combining language and caption length filtering and combining language,
caption length, image size fitering. Unless otherwise specified, “basic”
refers fasttext English, caption length, and image size filtering.

###### CLIP score and LAION filtering.

We experiment with CLIP score filtering (also employed by LAION), where we
take only examples having cosine similarity scores between CLIP image and text
embeddings that exceed a pre-defined threshold. We investigate a range of
thresholds and two OpenAI CLIP models for computing the scores: the ViT-B/32
model (as in LAION) and the larger ViT-L/14. We also combine CLIP score
thresholds and cld3 English filtering to reproduce the LAION-2B filtering
scheme. Table 16 in Section P.1 summarizes the different CLIP score
configurations.

###### Text-based filtering.

We select examples that contain text overlapping with ImageNet class names,
which serve as a proxy for relevance to downstream tasks. Specifically, we
select English captions (according to fasttext) that contain words from
ImageNet-21K or ImageNet-1K [37] class synsets.

###### Image-based filtering.

We select a subset of examples whose visual content overlaps with ImageNet
classes. After applying English language (fasttext) and caption length
filtering, we cluster the image embeddings extracted by the OpenAI ViT-L/14
model for each image into 100K groups using Faiss [75]. We then find the
nearest neighbor group for every ImageNet training example, and keep examples
belonging to these groups. We apply this procedure using either ImageNet-21K
(14M images) or ImageNet-1K (1.2M images), forming two subsets.

####  4.2 BYOD baselines

We experiment with multiple external data sources, including four moderately
sized datasets (10 to 58M samples) studied by Nguyen et al. [101]—CC12M [24],
YFCC15M [140, 111], RedCaps [38] and Shutterstock [101]—and the larger
LAION-2B [129]. Additional experiments, along with more details about the data
sources are provided in Appendix P.2. We consider these data sources as they
are and do not perform additional preprocessing. We also present experiments
combining some of the data sources (using only the external datasets, or in
addition to data from our pool).

Table 3: Zero-shot performance for select baselines in the filtering track. On
all scales, filtering strategies lead to better performance than using the
entire, unfiltered pool. The intersection between imaged-based and CLIP score
strategies performs well on most tasks and scales. For all metrics, higher is
better (see Appendix O for details). $\cap$ denotes the intersection of
filtering strategies.

|  | Dataset | Samples |  | ImageNet |  |  | Average over
---|---|---|---|---|---|---|---|---
Scale | Filtering strategy | size | seen | ImageNet | dist. shifts | VTAB | Retrieval | 38 datasets
| No filtering | 12.8M | 12.8M | 0.025 | 0.033 | 0.145 | 0.114 | 0.132
| Basic filtering | 3M | 12.8M | 0.038 | 0.043 | 0.150 | 0.118 | 0.142
| Text-based | 3.2M | 12.8M | 0.046 | 0.052 | 0.169 | 0.125 | 0.157
| Image-based | 3M | 12.8M | 0.043 | 0.047 | 0.178 | 0.121 | 0.159
| LAION-2B filtering | 1.3M | 12.8M | 0.031 | 0.040 | 0.136 | 0.092 | 0.133
| CLIP score (L/14 30%) | 3.8M | 12.8M | 0.051 | 0.055 | 0.190 | 0.119 | 0.173
small | Image-based $\cap$ CLIP score (L/14 30%) | 1.4M | 12.8M | 0.039 | 0.045 | 0.162 | 0.094 | 0.144
| No filtering | 128M | 128M | 0.176 | 0.152 | 0.259 | 0.219 | 0.258
| Basic filtering | 30M | 128M | 0.226 | 0.193 | 0.284 | 0.251 | 0.285
| Text-based | 31M | 128M | 0.255 | 0.215 | 0.328 | 0.249 | 0.307
| Image-based | 29M | 128M | 0.268 | 0.213 | 0.319 | 0.256 | 0.312
| LAION-2B filtering | 13M | 128M | 0.230 | 0.198 | 0.307 | 0.233 | 0.292
| CLIP score (L/14 30%) | 38M | 128M | 0.273 | 0.230 | 0.338 | 0.251 | 0.328
medium | Image-based $\cap$ CLIP score (L/14 30%) | 14M | 128M | 0.297 | 0.239 | 0.346 | 0.231 | 0.328
| No filtering | 1.28B | 1.28B | 0.459 | 0.378 | 0.426 | 0.419 | 0.437
| Basic filtering | 298M | 1.28B | 0.516 | 0.423 | 0.446 | 0.480 | 0.458
| Text-based | 317M | 1.28B | 0.561 | 0.465 | 0.465 | 0.352 | 0.466
| Image-based | 293M | 1.28B | 0.572 | 0.454 | 0.483 | 0.479 | 0.476
| LAION-2B filtering | 130M | 1.28B | 0.553 | 0.453 | 0.510 | 0.495 | 0.501
| CLIP score (L/14 30%) | 384M | 1.28B | 0.578 | 0.474 | 0.538 | 0.466 | 0.529
large | Image-based $\cap$ CLIP score (L/14 30%) | 140M | 1.28B | 0.631 | 0.508 | 0.546 | 0.498 | 0.537
| No filtering | 12.8B | 12.8B | 0.723 | 0.612 | 0.611 | 0.569 | 0.621
| LAION-2B filtering | 1.3B | 12.8B | 0.755 | 0.637 | 0.624 | 0.620 | 0.636
| CLIP score (L/14 30%) | 3.8B | 12.8B | 0.764 | 0.655 | 0.643 | 0.588 | 0.650
xlarge | Image-based $\cap$ CLIP score (L/14 30%) | 1.4B | 12.8B | 0.792 | 0.679 | 0.652 | 0.608 | 0.663

###  5 Results and discussion

####  5.1 Building better datasets

###### Main results.

Our key results are in Table 3. Most notably, the intersection between image-
based filtering and CLIP score filtering excels on most tasks. The exception
is at the small scale and for retrieval datasets.222Cherti et al. [28] also
observe that models rank differently on classification and retrieval tasks.
Furthermore, other filtering strategies like basic, CLIP score, image-based,
text-based filtering show better downstream performance when compared to no
filtering. A much larger suite of experiment results can be found in Appendix
R.

######  DataComp leads to better image-text datasets.

We hope DataComp catalyzes the search for the next generation of multimodal
datasets. We contribute DataComp-1B, which is the output of the Image-based
$\cap$ CLIP score (L/14 30%) baseline filter at the xlarge scale of the
filtering track. Our dataset is comprised of 1.4B samples, which not only is
_smaller_ than the LAION-2B dataset with 2.3B samples, but also comes from a
smaller pool. Nevertheless, a CLIP L/14 trained on DataComp-1B outperforms the
LAION-2B competitor by 6.1 percentage points on ImageNet (see Table 1).
Moreover, training on DataComp-1B improves ImageNet accuracy by 3.7 percentage
points over OpenAI’s ViT-L/14 trained with the same compute budget.
Additionally, even if we restrict ourselves to 400M samples, we can still find
a subset of DataComp-1B that outperforms OpenAI’s ViT-L/14, as seen in Table
24. These results demonstrate the impact that DataComp can make and provide a
foundation upon which participants can build.

###### External data sources can improve performance.

Appendix P.2 Table 18 shows results for several baselines in the BYOD track.
We find several instances where adding external data sources improves
performance over using just data from CommonPool. For example, at the large
scale, combining CLIP-filtered data from CommonPool with external data from
CC12M [24], YFCC15M [140, 111], RedCaps [38] and Shutterstock [101] boosts
ImageNet accuracy by 4.3 percentage points. See Appendix P.2 for more
experiments and details.

Figure 2: Performance of random subsets (dotted line) and CLIP score filtering
(solid line) when varying the subset size. When taking random subsets, larger
subsets are always better. For CLIP score filtering, subsets with intermediate
size perform best.

###### Trade-off between data diversity and repetition.

In Figure 2, we see that randomly selecting subsets of the pool has little
effect and degrades performance substantially when only small fractions are
used. When filtering with CLIP scores, the optimal training set comes from
selecting $\sim$30% of the pool with the highest scores. The difference in
performance trends between random subsets and CLIP score filtering highlights
the importance of filtering strategies for selecting samples.

####  5.2 DataComp design analyses

######  CommonPool and LAION are comparable with the same filtering.

To validate our pool construction, we show that we can build datasets
comparable to LAION-2B by employing their filtering technique on our pool.
LAION-2B selects all samples where the caption is in English and the cosine
similarity score from a trained ViT-B/32 CLIP model is above 0.28. We compare
this filtering approach on our pool using the same number samples, 130M
samples at the large scale. We find that the different data sources perform
comparably: 55.3% vs 55.7% accuracy on ImageNet, and 0.501 vs 0.489 average
performance over our evaluation sets using our pool and LAION-2B,
respectively.

###### Consistency across scales.

We find that the ranking between filtering strategies is typically consistent
across different scales. This is illustrated in Figure 3, which shows that the
baselines at small and medium scales are positively correlated. Moreover, as
shown in Appendix Table 22, the rank correlations of performance is high,
between 0.71 and 0.90 for different scale pairs.

Figure 3: Correlation between small and medium scale baselines. Smaller scales
can serve as useful guides for larger scales. Results for additional scales
are shown in Appendix Figure 22.

###### Consistency across training changes.

DataComp fixes the training procedure, so a natural question is whether better
datasets from DataComp are better outside of DataComp. While DataComp-1B is
trained at the xlarge scale, we show in Appendix Table 23 that even when
substituting the ViT-L/14 for a ViT-B/16 or ViT-B/32, training on DataComp-1B
outperforms training on OpenAI’s WIT and LAION-2B. Additionally, we found that
modifying hyperparameters such as training steps and batch size minimally
affects the relative ordering of different data curation methods on downstream
performance. Details on hyperparameter ablations are in Appendix L.

####  5.3 Evaluation trends

###### ImageNet accuracy is indicative, but not the complete picture.

Similarly to Kornblith et al. [84], in Appendix Figure 25 we find that
ImageNet performance is highly correlated with the average performance across
all datasets we study, with an overall correlation of 0.99. 333Note that
unlike Kornblith et al. [84] we evaluate zero-shot performance rather than
transfer learning. However, ImageNet performance is not representative of all
evaluation tasks, as the correlation between ImageNet accuracy and accuracy on
other individual datasets varies substantially, in some cases even exhibiting
a negative correlation, as discussed in Appendix R.

###### Robustness and fairness.

While typical models trained on a target task suffer large performance drops
under data distribution shift, zero-shot CLIP models are known to exhibit
strong performance across many distributions [111]. In Appendix Figure 26, we
show that CLIP models trained with data from our pool are more robust to
distribution shift than ImageNet-trained models from Taori et al. [139]’s
testbed. Examining geographic diversity, we find that our models are better
than ImageNet-trained models, but fall short of models fine-tuned on diverse
curated datasets (see Appendix Figure 21). We also perform a face
classification analysis and identify demographic biases in our models:
notably, the BYOD datasets we consider can increase the risk of
misclassification. See Appendix Q for more fairness and diversity analyses.

###  6 Limitations and conclusion

In terms of societal risks, creating an index of image-text pairs from the
public internet can be problematic. The internet contains unsafe, toxic, and
sensitive content, which ideally should not percolate into machine learning
datasets. Though we take steps to remove NSFW content and blur human faces to
protect privacy, we hope future work will further explore the biases and risks
from CommonPool and DataComp-1B. We see several additional directions for
future work, including 1) Curating more data sources. 2) Improved data
filtering algorithms. 3) Further supervision signals (e.g., image captions
coming from captioning models). 4) Additional input modalities (e.g., video,
3D objects). 5) Broader evaluations for vision-and-language and robotics
tasks.

Overall, we see DataComp as a first step towards improving training datasets,
and hope our new benchmark will foster further research. By providing a
controlled experimental setting, DataComp enables researchers to iterate on
dataset design on rigorous empirical foundations. We open-source all of our
code, data, and infrastructure, and hope these resources will help the
community build the next generation of multimodal datasets.

### Acknowledgements

SYG and JH are supported by NSF Graduate Research Fellowships. GS is supported
by the Onassis Foundation - Scholarship ID: F ZS 056-1/2022-2023. GD has been
supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/2022-2023),
the Bodossaki Fellowship and the Leventis Fellowship. This research has been
supported by NSF Grants AF 1901292, CNS 2148141, DMS 2134012, TRIPODS II-DMS
2023166, Tripods CCF 1934932, IFML CCF 2019844 and research gifts by Western
Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, the Len
Blavatnik and the Blavatnik Family Foundation, the Stanly P. Finch Centennial
Professorship in Engineering, Open Philanthropy, Google, Microsoft, and the
Allen Institute for AI.

We would like to thank Amro Abbas, Danny Bickson, Alper Canberk, Jessie
Chapman, Brian Cheung, Tim Dettmers, Joshua Gardner, Nancy Garland, Sachin
Goyal, Huy Ha, Zaid Harchaoui, Ari Holtzman, Andrew Hundt, Andy Jones, Adam
Klivans, Ronak Mehta, Sachit Menon, Ari Morcos, Raviteja Mullapudi, Jonathon
Shlens, Brandon McKinzie, Alexander Toshev, David Grangier, Navdeep Jaitly,
Kentrell Owens, Marco Tulio Ribeiro, Shiori Sagawa, Christoph Schuhmann,
Matthew Wallingford, and Ross Wightman for helpful feedback at various stages
of the project. We are particularly grateful to Daniel Levy and Alec Radford
for early encouragement to pursue this project and feedback on the
experimental design.

We thank Stability AI and the Gauss Centre for Supercomputing
e.V.444https://gauss-centre.eu for providing us with compute resources to
train models. We are thankful for the compute time provided through the John
von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS
Booster [78] at Jülich Supercomputing Centre (JSC), and for storage resources
on JUST [50] granted and operated by JSC, as well as computing and storage
resources from the Helmholtz Data Federation (HDF).

### References

  * cc [2] cc2dataset.  https://github.com/rom1504/cc2dataset.
  * [2] CLD3.  https://github.com/google/cld3.
  * [3] Common Crawl.  https://commoncrawl.org.
  * [4] dataset2metadata.  https://github.com/mlfoundations/dataset2metadata.
  * [5] img2dataset.  https://github.com/rom1504/img2dataset.
  * Abbas et al. [2023] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.  https://arxiv.org/abs/2303.09540.
  * Agarwal et al. [2004] Pankaj K. Agarwal, Sariel Har-Peled, and Kasturi R. Varadarajan.  Approximating extent measures of points.  _Journal of the ACM (JACM)_ , 2004.  https://doi.org/10.1145/1008731.1008736.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://openreview.net/forum?id=EbMuimAbPbs.
  * Awasthi et al. [2020] Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi.  Learning from rules generalizing labeled exemplars.  In _International Conference on Learning Representations (ICLR)_ , 2020.  https://openreview.net/forum?id=SkeuexBtDr.
  * Bach et al. [2019] Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin.  Snorkel drybell: A case study in deploying weak supervision at industrial scale.  In _Special Interest Group on Management of Data (SIGMOD)_ , 2019\.  https://arxiv.org/abs/1812.00417.
  * Bachem et al. [2015] Olivier Bachem, Mario Lucic, and Andreas Krause.  Coresets for nonparametric estimation - the case of dp-means.  In _International Conference on Machine Learning (ICML)_ , 2015.  https://proceedings.mlr.press/v37/bachem15.html.
  * Bandi et al. [2018] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al.  From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.  _IEEE Transactions on Medical Imaging_ , 2018.  https://pubmed.ncbi.nlm.nih.gov/30716025/.
  * Barbu et al. [2019] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 32. Curran Associates, Inc., 2019.  https://proceedings.neurips.cc/paper/2019/file/97af07a14cacba681feacf3012730892-Paper.pdf.
  * Beery et al. [2020] Sara Beery, Elijah Cole, and Arvi Gjoka.  The iwildcam 2020 competition dataset, 2020.  https://arxiv.org/abs/2004.10340.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes, 2021.  https://arxiv.org/abs/2110.01963.
  * Birodkar et al. [2019] Vighnesh Birodkar, Hossein Mobahi, and Samy Bengio.  Semantic redundancies in image-classification datasets: The 10% you don’t need.  _arXiv preprint arXiv:1901.11409_ , 2019.  https://arxiv.org/abs/1901.11409.
  * Bitton et al. [2022] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz.  WinoGAViL: Gamified association benchmark to challenge vision-and-language models, 2022.  https://arxiv.org/abs/2207.12576.
  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _European Conference on Computer Vision (ECCV)_ , 2014.  https://link.springer.com/chapter/10.1007/978-3-319-10599-4_29.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Caballero et al. [2023] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger.  Broken neural scaling laws.  _International Conference on Learning Representations (ICLR)_ , 2023\.  https://arxiv.org/abs/2210.14891.
  * Carlini et al. [2023] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr.  Poisoning web-scale training datasets is practical, 2023.  https://arxiv.org/abs/2302.10149.
  * Chan et al. [2022] Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill.  Data distributional properties drive emergent in-context learning in transformers.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://arxiv.org/abs/2205.05055.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.  https://arxiv.org/abs/2102.08981.
  * Chen et al. [2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  In _International Conference on Learning Representations (ICLR)_ , 2022.  https://arxiv.org/abs/2209.06794.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO captions: Data collection and evaluation server, 2015\.  https://arxiv.org/abs/1504.00325.
  * Cheng et al. [2017] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  _Proceedings of the Institute of Electrical and Electronics Engineers (IEEE)_ , 2017.  https://ieeexplore.ieee.org/abstract/document/7891544.
  * Cherti et al. [2022] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning, 2022\.  https://arxiv.org/abs/2212.07143.
  * Christie et al. [2018] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.  https://arxiv.org/abs/1711.07846.
  * Cimpoi et al. [2014] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014.  https://openaccess.thecvf.com/content_cvpr_2014/html/Cimpoi_Describing_Textures_in_2014_CVPR_paper.html.
  * Coates et al. [2011] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2011.  https://proceedings.mlr.press/v15/coates11a.html.
  * Cohen et al. [2017] Michael B. Cohen, Cameron Musco, and Christopher Musco.  Input sparsity time low-rank approximation via ridge leverage score sampling.  In _ACM-SIAM Symposium on Discrete Algorithms_ , 2017.  https://dl.acm.org/doi/10.5555/3039686.3039801.
  * Coleman et al. [2020] C Coleman, C Yeh, S Mussmann, B Mirzasoleiman, P Bailis, P Liang, J Leskovec, and M Zaharia.  Selection via proxy: Efficient data selection for deep learning.  In _International Conference on Learning Representations (ICLR)_ , 2020.  https://arxiv.org/abs/1906.11829.
  * Conneau et al. [2019] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2019.  https://arxiv.org/abs/1911.02116.
  * Cook [1977] R Dennis Cook.  Detection of influential observation in linear.  _Technometrics_ , 19(1):15–18, 1977.
  * Dave et al. [2020] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan.  Tao: A large-scale benchmark for tracking any object.  In _European Conference on Computer Vision (ECCV)_ , 2020.  https://arxiv.org/abs/2005.10356.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.  https://ieeexplore.ieee.org/abstract/document/5206848.
  * Desai et al. [2021] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson.  Redcaps: Web-curated image-text data created by the people, for the people, 2021.  https://arxiv.org/abs/2111.11431.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations (ICLR)_ , 2021.  https://openreview.net/forum?id=YicbFdNTTy.
  * Douze et al. [2021] Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, Ondrej Chum, and Cristian Canton-Ferrer.  The 2021 image similarity dataset and challenge, 2021.  https://arxiv.org/abs/2106.09672.
  * Ethayarajh et al. [2022] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.  Understanding dataset difficulty with v-usable information.  In _International Conference on Machine Learning (ICML)_ , 2022.  https://arxiv.org/abs/2110.08420.
  * Everingham et al. [2007] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results, 2007.  http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  * Eyuboglu et al. [2022] Sabri Eyuboglu, Bojan Karlaš, Christopher Ré, Ce Zhang, and James Zou.  dcbench: a benchmark for data-centric ai systems.  In _Proceedings of the Sixth Workshop on Data Management for End-To-End Machine Learning_ , 2022.  https://dl.acm.org/doi/abs/10.1145/3533028.3533310.
  * Fang et al. [2022] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  In _International Conference on Machine Learning (ICML)_ , 2022.  https://arxiv.org/abs/2205.01397.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories.  _Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2004.  https://ieeexplore.ieee.org/document/1384978.
  * Feldman et al. [2011] Dan Feldman, Matthew Faulkner, and Andreas Krause.  Scalable training of mixture models via coresets.  In _Advances in Neural Information Processing Systems (NeuIPS)_ , 2011\.  https://proceedings.neurips.cc/paper_files/paper/2011/file/2b6d65b9a9445c4271ab9076ead5605a-Paper.pdf.
  * Fu et al. [2020] Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré.  Fast and three-rious: Speeding up weak supervision with triplet methods.  In _International Conference on Machine Learning (ICML)_ , 2020.  https://arxiv.org/abs/2002.11955.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.  https://ieeexplore.ieee.org/abstract/document/6248074.
  * Ghorbani & Zou [2019] Amirata Ghorbani and James Zou.  Data shapley: Equitable valuation of data for machine learning.  In _International Conference on Machine Learning_ , pp. 2242–2251. PMLR, 2019.
  * Graf & Mextorf [2021] Stephan Graf and Olaf Mextorf.  Just: Large-scale multi-tier storage infrastructure at the jülich supercomputing centre.  _Journal of large-scale research facilities JLSRF_ , 2021.  https://jlsrf.org/index.php/lsf/article/view/180.
  * Guo et al. [2022] Chengcheng Guo, Bo Zhao, and Yanbing Bai.  Deepcore: A comprehensive library for coreset selection in deep learning, 2022.  https://arxiv.org/abs/2204.08499.
  * Guo et al. [2020] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong.  Fastif: Scalable influence functions for efficient model interpretation and debugging, 2020.  https://arxiv.org/abs/2012.15781.
  * Guo et al. [2021] Jia Guo, Jiankang Deng, Alexandros Lattas, and Stefanos Zafeiriou.  Sample and computation redistribution for efficient face detection.  In _International Conference on Learning Representations (ICLR)_ , 2021.  https://arxiv.org/abs/2105.04714.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  LVIS: A dataset for large vocabulary instance segmentation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Gururangan et al. [2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith.  Annotation artifacts in natural language inference data.  In _Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2018.  https://aclanthology.org/N18-2017.
  * Guu et al. [2023] Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi.  Simfluence: Modeling the influence of individual training examples by simulating training runs, 2023.  https://arxiv.org/abs/2303.08114.
  * Hampel [1974] Frank R Hampel.  The influence curve and its role in robust estimation.  _Journal of the american statistical association_ , 1974.  https://www.jstor.org/stable/2285666.
  * Han et al. [2020] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov.  Explaining black box predictions and unveiling data artifacts through influence functions, 2020.  https://arxiv.org/abs/2005.06676.
  * Hanna et al. [2020] A. Hanna, Emily L. Denton, Andrew Smart, and Jamila Smith-Loud.  Towards a critical race methodology in algorithmic fairness.  In _Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2020.  https://arxiv.org/abs/1912.03593.
  * Hanu & Unitary team [2020] Laura Hanu and Unitary team.  Detoxify, 2020.  https://github.com/unitaryai/detoxify.
  * Har-Peled & Mazumdar [2004] Sariel Har-Peled and Soham Mazumdar.  On coresets for k-means and k-median clustering.  In _Symposium on Theory of Computing (STOC)_ , 2004.  https://doi.org/10.1145/1007352.1007400.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.  https://arxiv.org/abs/1512.03385.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.  https://arxiv.org/abs/1709.00029.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021a.  https://arxiv.org/abs/2006.16241.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021b.  https://arxiv.org/abs/1907.07174.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models, 2022.  https://arxiv.org/abs/2203.15556.
  * Hoffmann et al. [2011] Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2011.  https://aclanthology.org/P11-1055.
  * Hundt et al. [2022] Andrew Hundt, William Agnew, Vicky Zeng, Severin Kacianka, and Matthew Gombolay.  Robots enact malignant stereotypes.  In _Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2022.  https://arxiv.org/abs/2207.11569.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  OpenCLIP, July 2021.  https://doi.org/10.5281/zenodo.5143773.
  * Ilharco et al. [2022] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt.  Patching open-vocabulary models by interpolating weights.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://arXiv.org/abs/2208.05592.
  * Ilyas et al. [2022] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Predicting predictions from training data, 2022.  https://arxiv.org/abs/2202.00622.
  * Jain et al. [2019] Tanuj Jain, Christopher Lennan, Zubin John, and Dat Tran.  Imagededup, 2019.  https://github.com/idealo/imagededup.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning (ICML)_ , 2021.  https://arxiv.org/abs/2102.05918.
  * Jiang et al. [2001] Mon-Fong Jiang, Shian-Shyong Tseng, and Chih-Ming Su.  Two-phase clustering process for outliers detection.  _Pattern recognition letters_ , 2001.  https://www.sciencedirect.com/science/article/abs/pii/S0167865500001318.
  * Johnson et al. [2019] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 2019.  https://arxiv.org/abs/1702.08734.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017\.  https://arxiv.org/abs/1612.06890.
  * Joulin et al. [2017] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  In _Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , 2017.  https://arxiv.org/abs/1607.01759.
  * Juelich Supercomputing Center [2020] Juelich Supercomputing Center.  JUWELS Booster Supercomputer, 2020.  https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html#hardware-configuration-of-the-system-name-booster-module.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models, 2020.  https://arxiv.org/abs/2001.08361.
  * Karkkainen & Joo [2021] Kimmo Karkkainen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.  https://arxiv.org/abs/1908.04913.
  * Koh & Liang [2017] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _International Conference on Machine Learning (ICML)_ , 2017.  https://arxiv.org/abs/1703.04730.
  * Koh et al. [2019] Pang Wei Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang.  On the accuracy of influence functions for measuring group effects.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.  https://arxiv.org/abs/1905.13289.
  * Koh et al. [2021] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.  WILDS: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning (ICML)_ , 2021.  https://arxiv.org/abs/2012.07421.
  * Kornblith et al. [2019] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.  https://arxiv.org/abs/1805.08974.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _International Conference on Computer Vision Workshops (ICML)_ , 2013.  https://www.cv-foundation.org/openaccess/content_iccv_workshops_2013/W19/html/Krause_3D_Object_Representations_2013_ICCV_paper.html.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.  https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2012.  https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  * Le Bras et al. [2020] Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi.  Adversarial filters of dataset biases.  In _International Conference on Machine Learning (ICML)_ , 2020.  https://arxiv.org/abs/2002.04108.
  * LeCun [1998] Yann LeCun.  The MNIST database of handwritten digits, 1998.  http://yann.lecun.com/exdb/mnist/.
  * Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2021.  https://arxiv.org/abs/2107.06499.
  * Li & Vasconcelos [2019] Yi Li and Nuno Vasconcelos.  Repair: Removing representation bias by dataset resampling.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.  https://arxiv.org/abs/1904.07911.
  * Liu et al. [2022a] Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang.  Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022a.  https://proceedings.neurips.cc/paper_files/paper/2022/file/6a386d703b50f1cf1f61ab02a15967bb-Paper-Datasets_and_Benchmarks.pdf.
  * Liu et al. [2022b] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022b.  https://arxiv.org/abs/2201.03545.
  * Lucic et al. [2018] Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman.  Training gaussian mixture models at scale via coresets.  _Journal of Machine Learning Research (JMLR)_ , 2018.  http://jmlr.org/papers/v18/15-506.html.
  * Maji et al. [2013] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft, 2013.  https://arxiv.org/abs/1306.5151.
  * Mann & McCallum [2010] Gideon S Mann and Andrew McCallum.  Generalized expectation criteria for semi-supervised learning with weakly labeled data.  _Journal of Machine Learning Research (JMLR)_ , 2010.  https://www.jmlr.org/papers/v11/mann10a.html.
  * Mazumder et al. [2022] Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi.  Dataperf: Benchmarks for data-centric ai development, 2022.  https://arxiv.org/abs/2207.10062.
  * Mirzasoleiman et al. [2020] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec.  Coresets for data-efficient training of machine learning models.  In _International Conference on Machine Learning (ICML)_ , 2020.  https://arxiv.org/abs/1906.01827.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In _Advances in Neural Information Processing Systems (NeurIPS) Workshops_ , 2011.  https://storage.googleapis.com/pub-tools-public-publication-data/pdf/37648.pdf.
  * Ng et al. [2021] Andrew Ng, Dillon Laird, and Lynn He.  Data-centric ai competition, 2021.  https://https-deeplearning-ai.github.io/data-centric-comp/.
  * Nguyen et al. [2022] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt.  Quality not quantity: On the interaction between dataset design and robustness of clip.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://openreview.net/forum?id=LTCBavFWp5C.
  * Nilsback & Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _Indian Conference on Computer Vision, Graphics and Image Processing_ , 2008.  https://ieeexplore.ieee.org/document/4756141.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.  https://arxiv.org/abs/2303.08774.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.  Im2text: Describing images using 1 million captioned photographs.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2011.  https://papers.nips.cc/paper_files/paper/2011/file/5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf.
  * Parkhi et al. [2012] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.  https://ieeexplore.ieee.org/document/6248092.
  * Paul et al. [2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.  https://arxiv.org/abs/2107.07075.
  * Pham et al. [2021] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le.  Combined scaling for zero-shot transfer learning, 2021.  https://arxiv.org/abs/2111.10050.
  * Prabhu & Birhane [2020] Vinay Uday Prabhu and Abeba Birhane.  Large image datasets: A pyrrhic win for computer vision?  In _Winter Conference on Applications of Computer Vision (WACV)_ , 2020.  https://arxiv.org/abs/2006.16923.
  * Pruthi et al. [2020] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan.  Estimating training data influence by tracing gradient descent.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.  https://arxiv.org/abs/2002.08484.
  * Radenovic et al. [2023] Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan.  Filtering, distillation, and hard negatives for vision-language pre-training.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.  https://arxiv.org/abs/2301.02280.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , 2021.  https://arxiv.org/abs/2103.00020.
  * Radford et al. [2022] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision, 2022.  https://arxiv.org/abs/2212.04356.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research (JMLR)_ , 2020.  https://arxiv.org/abs/1910.10683.
  * Ramaswamy et al. [2023] Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky.  Beyond web-scraping: Crowd-sourcing a geodiverse datase, 2023.  https://arxiv.org/abs/2301.02560.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning (ICML)_ , 2021.  https://arxiv.org/abs/2102.12092.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.  https://arxiv.org/abs/2204.06125.
  * Ratner et al. [2019] A. J. Ratner, B. Hancock, J. Dunnmon, F. Sala, S. Pandey, and C. Ré.  Training complex models with multi-task weak supervision.  In _Association for the Advancement of Artificial Intelligence (AAAI)_ , 2019.  https://arxiv.org/abs/1810.02840.
  * Ratner et al. [2016] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré.  Data programming: Creating large training sets, quickly.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016.  https://arxiv.org/abs/1605.07723.
  * Ratner et al. [2017] Alexander J Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré.  Snorkel: Rapid training data creation with weak supervision.  In _Very Large Data Bases Conference (VLDB)_ , 2017.  https://arxiv.org/abs/1711.10160.
  * Ré [2020] Christopher Ré.  Overton: A data system for monitoring and improving machine-learned products.  In _10th Conference on Innovative Data Systems Research, CIDR 2020, Amsterdam, The Netherlands, January 12-15, 2020, Online Proceedings_. www.cidrdb.org, 2020.  URL http://cidrdb.org/cidr2020/papers/p33-re-cidr20.pdf.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do ImageNet classifiers generalize to ImageNet?  In _International Conference on Machine Learning (ICML)_ , 2019.  http://proceedings.mlr.press/v97/recht19a.html.
  * Rojas et al. [2022] William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman.  The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world.  In _Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track_ , 2022.  https://openreview.net/forum?id=qnfYsave0U4.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.  https://arxiv.org/abs/2112.10752.
  * Rousseeuw & Hubert [2011] Peter J Rousseeuw and Mia Hubert.  Robust statistics for outlier detection.  _Wiley interdisciplinary reviews: Data mining and knowledge discovery_ , 2011.  http://i2pc.es/coss/Docencia/SignalProcessingReviews/Rousseeuw2011.pdf.
  * Rousseeuw & Hubert [2018] Peter J Rousseeuw and Mia Hubert.  Anomaly detection by robust statistics.  _Wiley interdisciplinary reviews: Data mining and knowledge discovery_ , 2018.  https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/widm.1236.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 2015.  https://arxiv.org/abs/1409.0575.
  * Sagawa et al. [2022] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang.  Extending the wilds benchmark for unsupervised adaptation.  In _International Conference on Learning Representations (ICLR)_ , 2022.  https://arxiv.org/abs/2112.05090.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open dataset of clip-filtered 400 million image-text pairs, 2021.  https://arxiv.org/abs/2111.02114.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track_ , 2022.  https://openreview.net/forum?id=M3Y74vmsMcY.
  * Sener & Savarese [2018] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations (ICLR)_ , 2018.  https://openreview.net/forum?id=H1aIuk-RW.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2018.  https://aclanthology.org/P18-1238/.
  * Shen et al. [2021] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer.  How much can clip benefit vision-and-language tasks?, 2021.  https://arxiv.org/abs/2107.06383.
  * Shin et al. [2022] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala.  Universalizing weak supervision.  In _International Conference on Learning Representations (ICLR)_ , 2022.  https://openreview.net/forum?id=YpPiNigTzMT.
  * Song et al. [2022] Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei.  CLIP models are few-shot learners: Empirical studies on VQA and visual entailment.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2022.  https://aclanthology.org/2022.acl-long.421.
  * Sorscher et al. [2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://openreview.net/forum?id=UmvSlP-PyV.
  * Srinivasan et al. [2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In _44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2021.  https://arxiv.org/abs/2103.01913.
  * Stallkamp et al. [2011] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.  The german traffic sign recognition benchmark: a multi-class classification competition.  In _International Joint Conference on Neural Networks (IJCNN)_ , 2011\.  https://ieeexplore.ieee.org/document/6033395.
  * Swayamdipta et al. [2020] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020.  https://aclanthology.org/2020.emnlp-main.746.
  * Taori et al. [2020] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.  Measuring robustness to natural distribution shifts in image classification.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.  https://dl.acm.org/doi/abs/10.5555/3495724.3497285.
  * Thomee et al. [2016] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  YFCC100M: The new data in multimedia research.  _Communications of the ACM_ , 2016.  https://arxiv.org/abs/1503.01817.
  * Toneva et al. [2018] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon.  An empirical study of example forgetting during deep neural network learning.  In _International Conference on Learning Representations (ICLR)_ , 2018.  https://arxiv.org/abs/1812.05159.
  * Veeling et al. [2018] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant CNNs for digital pathology, 2018.  https://arxiv.org/abs/1806.03962.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.  https://arxiv.org/abs/1905.13549.
  * Webster et al. [2023] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie.  On the de-duplication of laion-2b, 2023.  https://arxiv.org/abs/2303.12733.
  * Wei et al. [2015] Kai Wei, Rishabh Iyer, and Jeff Bilmes.  Submodularity in data subset selection and active learning.  In _International Conference on Machine Learning (ICML)_ , 2015.  https://proceedings.mlr.press/v37/wei15.html.
  * Xiao et al. [2016] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.  Sun database: Exploring a large collection of scene categories.  _International Journal of Computer Vision (IJCV)_ , 2016.  https://link.springer.com/article/10.1007/s11263-014-0748-y.
  * Yang et al. [2020] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky.  Towards fairer datasets: filtering and balancing the distribution of the people subtree in the imagenet hierarchy.  In _Conference on Fairness, Accountability, and Transparency (FAccT)_ , 2020.  https://arxiv.org/abs/1912.07726.
  * Yang et al. [2022] Kaiyu Yang, Jacqueline H Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky.  A study of face obfuscation in ImageNet.  In _International Conference on Machine Learning (ICML)_ , 2022.  https://arxiv.org/abs/2103.06191.
  * Yao et al. [2022] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  In _International Conference on Learning Representations (ICLR)_ , 2022.  https://arxiv.org/abs/2111.07783.
  * Yokoo [2021] Shuhei Yokoo.  Contrastive learning with large memory bank and negative embedding subtraction for accurate copy detection, 2021.  https://arxiv.org/abs/2112.04323.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2014\.  https://aclanthology.org/Q14-1006/.
  * Yu et al. [2002] Dantong Yu, Gholamhosein Sheikholeslami, and Aidong Zhang.  Findout: Finding outliers in very large datasets.  _Knowledge and information Systems_ , 2002.  https://link.springer.com/article/10.1007/s101150200013.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision, 2021.  https://arxiv.org/abs/2111.11432.
  * Yuen et al. [2011] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung.  A survey of crowdsourcing systems.  In _SocialCom_. IEEE, 2011.  https://ieeexplore.ieee.org/document/6113213.
  * Zaharia et al. [2016] Matei Zaharia, Reynold S Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J Franklin, et al.  Apache spark: a unified engine for big data processing.  _Communications of the ACM_ , 2016.  https://dl.acm.org/doi/10.1145/2934664.
  * Zhai et al. [2019] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  The visual task adaptation benchmark, 2019.  http://arxiv.org/abs/1910.04867.
  * Zhang et al. [2021] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner.  WRENCH: A comprehensive benchmark for weak supervision.  In _NeurIPS_ , 2021.  URL https://openreview.net/forum?id=Q9SKS5k8io.
  * Zhang et al. [2022] Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner.  A survey on programmatic weak supervision, 2022.  https://arxiv.org/abs/2202.05433.
  * Zhang et al. [2017] Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.  https://arxiv.org/abs/1702.08423.
  * Zhou et al. [2022] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _European Conference on Computer Vision (ECCV)_ , 2022.  https://arxiv.org/abs/2201.02605.

## Appendix

###### Contents

  1. 1 Introduction
  2. 2 Related Work
  3. 3 The DataComp benchmark
    1. 3.1 Competition design
    2. 3.2 CommonPool generation, for the filtering track
    3. 3.3 The bring your own data (BYOD) track
    4. 3.4 Training
    5. 3.5 Evaluation
  4. 4 Baselines
    1. 4.1 Filtering baselines
    2. 4.2 BYOD baselines
  5. 5 Results and discussion
    1. 5.1 Building better datasets
    2. 5.2 DataComp design analyses
    3. 5.3 Evaluation trends
  6. 6 Limitations and conclusion
  7. A Benchmark rules
    1. A.1 Filtering track rules
    2. A.2 Bring your own data track: amendments
  8. B Contributions
    1. B.1 Candidate pool
    2. B.2 Participant tooling
    3. B.3 Baselines
    4. B.4 Leadership and Advising
  9. C Additional related work
  10. D Parsing Common Crawl
  11. E Not safe for work (NSFW) filtering
  12. F Deduplication against evaluation sets
  13. G Face blurring
  14. H DataComp CommonPool creation pipeline
  15. I CommonPool statistics
  16. J Efficient training on data subsets
  17. K Effect of duplicates in the training data
  18. L Hyperparameter ablations
    1. L.1 Batch size
    2. L.2 Model architecture
    3. L.3 Number of training steps
  19. M Detector-based baselines
  20. N Training details
  21. O Evaluation details
    1. O.1 Visual Question Answering
  22. P Baseline details
    1. P.1 Filtering track
    2. P.2 BYOD track
      1. P.2.1 Additional results
  23. Q Fairness and biases
    1. Q.1 Diversity
    2. Q.2 Fairness
  24. R Extra figures and tables
  25. S Datasheet
    1. S.1 Motivation
    2. S.2 Composition
    3. S.3 Collection Process
    4. S.4 Preprocessing, Cleaning, and/or Labeling
    5. S.5 Uses
    6. S.6 Distribution
    7. S.7 Maintenance

###  Appendix A Benchmark rules

We provide concrete rules below for the two competition tracks that comprise
DataComp: filtering and BYOD. Additionally, we provide a checklist, which
encourages participants to specify design decisions, which allows for more
granular comparison between submissions.

####  A.1 Filtering track rules

  * •

Participants can enter submissions for one or many different scales: small,
medium, large or xlarge, which represent the raw number of image-text pairs in
CommonPool that should be filtered.

  * •

After choosing a scale, participants generate a list of uids, where each uid
refers to a CommonPool sample. The list of uids is used to recover image-text
pairs from the pool, which is used for downstream CLIP training.

  * •

Duplicate uids are allowed.

  * •

Participants are _not_ allowed to modify the training procedure. Hence,
changing hyperparameters, model architecture, optimizer, compute budget, or
number of training steps is not allowed. Changing any other training details
is also not allowed.

  * •

Participants are strongly encouraged to submit and open-source both the list
of uids and the code used to generate this list; however, this is not
required.

  * •

To avoid overfitting, we do not permit running any code or algorithmic
dependence on the test images of the evaluation tasks. However, use of other
images associated with these tasks (e.g., supervised training sets) is
permitted.

  * •

Participants can use templates or class labels from the downstream tasks in
their filtering algorithms.

For clarity, we include some examples of permitted and forbidden uses:

  * ✓

We permit using the ImageNet class label “triceratops” in a filtering
algorithm.

  * $\times$

We forbid examining individual or aggregate predictions on the test sets of
the evaluation tasks.

####  A.2 Bring your own data track: amendments

To facilitate more open-ended exploration, we provide amendments to the Track
1 competition to allow for more diverse submissions in Track 2.

  * •

Participants are allowed to augment CommonPool data with existing datasets, so
long as these data sources do not contain test images from the evaluation
tasks. Participants can use data from any CommonPool; however, they are not
required to do so.

  * •

Assembling one’s own dataset is allowed; however, test images from the
evaluation tasks can neither be contained nor otherwise used to construct said
dataset. We encourage releasing the image urls or the images themselves in
addition to the text for each image. We also encourage rigorous documentation
of face-blurring and other data safety checks (see Section 3.2 for more
details). We reserve the right to run our own safety code on participant
provided data and disqualify entries that do not meet adequate safety
standards.

###### Checklist.

The following checklist provides the basis for more fine-grained comparison
between submissions.

  * $\square$

Images from the evaluation tasks are included in my submission. If yes, please
specify which datasets.

  * $\square$

I used an existing datasets (e.g., YFCC100M [140]) in my submission. If yes,
please specify which datasets. (Note: applies to BYOD only)

  * $\square$

I curated my own data. If yes, please provide (1) image data or urls, (2) text
for each image, (3) list of safety steps taken including but not limited to
face blurring, explicit content image and text filtering. (Note: applies to
BYOD only)

###  Appendix B Contributions

For this section, contributors are ordered alphabetically.

####  B.1 Candidate pool

###### Candidate pool lead.

Vaishaal Shankar

###### Data collection.

Romain Beaumont, Vaishaal Shankar

###### Pre-processing and metadata.

Giannis Daras, Alex Fang (content filtering lead), Samir Yitzhak Gadre
(metadata lead), Ryan Marten (deduplication lead), Vivek Ramanujan, Vaishaal
Shankar, George Smyrnis (face blurring lead)

####  B.2 Participant tooling

###### Participant tooling lead.

Gabriel Ilharco

###### Resharder.

Romain Beaumont, Yair Carmon, Alex Fang, Jonathan Hayase (lead), Gabriel
Ilharco, Vivek Ramanujan, Vaishaal Shankar, Georgios Smyrnis

###### Training.

Mehdi Cherti, Gabriel Ilharco, Jenia Jitsev, Vivek Ramanujan, Georgios
Smyrnis, Mitchell Wortsman (lead)

###### Evaluation.

Romain Beaumont, Yonatan Bitton, Mehdi Cherti, Dhruba Ghosh (lead), Gabriel
Ilharco

###### Additional infrastructure.

Stephen Mussmann, Sarah Pratt

####  B.3 Baselines

###### Baselines lead.

Yair Carmon

###### Filtering track.

Yair Carmon, Rahim Enterazi, Alex Fang, Samir Yitzhak Gadre, Gabriel Ilharco,
Kalyani Marathe, Thao Nguyen, Eyal Orgad (co-lead), Georgios Smyrnis, Mitchell
Wortsman, Jieyu Zhang (co-lead)

###### BYOD track.

Gabriel Ilharco, Thao Nguyen

###### Experiment babysitting.

Alex Fang, Gabriel Ilharco, Samir Yitzhak Gadre

####  B.4 Leadership and Advising

###### Advising.

Romain Beaumont, Yair Carmon, Alexandros G. Dimakis, Ali Farhadi, Hannaneh
Hajishirzi, Jenia Jitsev, Pang Wei Koh, Ranjay Krishna, Stephen Mussmann,
Sewoong Oh, Alexander Ratner, Olga Saukh, Ludwig Schmidt, Vaishaal Shankar,
Shuran Song, Richard Vencu

###### Leadership.

Yair Carmon, Alexandros G. Dimakis, Jenia Jitsev, Sewoong Oh, Ludwig Schmidt,
Vaishaal Shankar

###### Overall project lead.

Ludwig Schmidt

###  Appendix C Additional related work

Here we expand on the related work described in Section 2.

Image dataset safety is an active area of research, especially in the context
of large-scale dataset construction. In addition to Birhane et al. [15], who
study problematic content in LAION-400M, Yang et al. [147] study the ImageNet
dataset and reveal limitations associated with the ImageNet curation
strategy—with negative implications for downstream model fairness. Prabhu &
Birhane [108] also study the ImageNet dataset and find pornographic content.
Both Birhane et al. [15] and Prabhu & Birhane [108] survey ethical conundrums
and harms that are borne out of improper dataset curation. In an effort to
combat dataset toxicity, we conduct NSFW preprocessing (Section 3.2, Appendix
E) and blur detected faces (Section 3.2, Appendix G) during pool construction.
We also conduct preliminary fairness evaluations (Section 5.3, Appendix Q) for
models trained on our data. We hope CommonPool will serve as a research
artifact for future work examining dataset safety.

Beyond data selection, Chan et al. [23] investigate the effects of dataset
distribution on emergent properties of transformers, while Fang et al. [44]
look at the relationship between data and model robustness to distribution
shifts. We hope our extensive evaluation suite comprised of 38 diverse tasks
will facilitate similar studies when training multimodal models at large
scale.

Others study how to reduce the burdens of training data annotation in the
curation process. Classic approaches include distant supervision [67], crowd-
sourced labels [154], heuristic rules [9] and feature annotation [96], among
others. A recent line of work known as data programming or programmatic weak
supervision [118, 119, 157, 158] attempts to reduce annotation cost and is
found in many industry applications [10, 120]. In data programming, developers
write programmatic labeling functions to automatically label a large amount of
unlabeled data. The labeling functions could produce noisy and conflicting
labels, so researchers have developed methods to aggregate noisy votes to
produce the final training labels [117, 47, 133].

Previous literature also studies methods for training data attribution, which
seek to link a model’s behavior (e.g., its accuracy on a particular task or
subset of data) to particular subsets of its training data. Such methods
include influence functions, a classic technique from robust statistics [57,
35] that uses a second-order Taylor expansion to approximate the effect of
removing a training point on the learned model parameters [81, 82, 58, 52], as
well as methods that fit attribution functions directly to the dynamics of
repeated training runs [49, 109, 71, 56]. Training data attribution methods
assume that we have already trained a model, though they can be subsequently
used to refine the training data (e.g., by identifying potentially mislabeled
training points [81]). Our focus in this paper is instead on data curation
methods—that is, methods for selecting a subset of the training data to train
a model in the first place.

In the context of natural language processing, Swayamdipta et al. [138]
proposes a tool for characterizing samples in a dataset based on training
dynamics, labelling instances as ambiguous, easy to learn or hard to learn.
Previous literature such as work by Le Bras et al. [88], Li & Vasconcelos
[91], Gururangan et al. [55] advocate for removing easy instances from the
training data. Ethayarajh et al. [41] propose a measure of how difficult a
dataset is to learn, $\mathcal{V}$-usable information. Such techniques could
be promising directions of further exploration in the context of our
benchmark.

Finally, another related line of work is studying scaling trends. In addition
to Sorscher et al. [135], researchers have investigated how model performance
changes as a function of compute budget, model size, and number of training
samples [79, 66, 21, 28]. However, this line of work does not consider how
dataset design may affects scaling trends. Beyond dataset size, we measure the
effects of different dataset sources and filtering strategies. While scaling
trends are central to our investigations, the purpose of our benchmark is to
search for the next generation of large multimodal datasets to facilitate more
accurate and reliable models.

###  Appendix D Parsing Common Crawl

Common Crawl releases metadata files for the websites that they index (i.e.,
WAT files). They release these files approximately once a month. We consider
all files available from 2014 through November of 2022. We first parse these
files, utilizing Apache Spark [155] to extract image urls and corresponding
alt-text. We map each url, text pair to a uid hash and remove duplicates. This
results in 88 billion url, text pairs, which are randomized via a distributed
shuffle. Note, we do not consider image content when running uid deduplication
at this step. Hence, two identical images with different urls and the same
caption would both be retained.

###  Appendix E Not safe for work (NSFW) filtering

Our data is sourced from Common Crawl, which contains snapshots of the web.
Therefore, we apply multiple layers of NSFW content filtering to remove
problematic images and captions from CommonPool.

First, we filter our captions with Detoxify [60], a language model for toxic
comment classification. Specifically, we use the multilingual XLM-RoBERTa [34]
variant. The model outputs scores between zero and one for the following
categories: toxicity, severe toxicity, obscene, identity attack, insult,
threat, and sexually explicit. As we had no ground truth for our data, we
manually spot check a 1 million random subset of CommonPool at varying
thresholds. We found that a threshold of 0.1 provided good coverage of
filtering out NSFW text. If any of the detoxify category scores exceeds the
threshold, the sample is discarded. Qualitatively, we found that the model
struggled with multilingual content, acronyms, and innuendo. Even at 0.1, we
noticed there are some captions that are NSFW. However, lowering the threshold
further heavily affected false positives. We therefore use a 0.1 threshold for
all NSFW categories, which on a random subset of one million captions achieves
positive rates shown in Table 4.

Table 4: Detoxify positive rates by threshold on 1 million caption subset of
Common Crawl.

Threshold | Toxicity | Severe Toxicity | Obscene | Identity Attack | Insult | Threat | Sexual Explicit
---|---|---|---|---|---|---|---
0.01 | 9.5% | 1.0% | 33.4% | 1.8% | 35.0% | 1.3% | 2.0%
0.1 | 3.6% | 0.1% | 0.8% | 0.3% | 1.4% | 0.1% | 1.0%

Second, on the vision side, we use a modified version of LAION-5B’s [129]
CLIP-based binary classification NSFW model, which takes CLIP ViT-L/14 visual
embeddings as input. We remove the initial multi-category encoder from the
model, and retrain on the same data with an initial normalization layer
followed by a 4-layer multilayer perceptron. Our retrained model matches the
performance of the original model on their manually annotated testset.
Specifically, we achieve 97.4% classification accuracy on a held out test set
compared to 96.1% for the original LAION NSFW image filtering model.
Additional details about the training data can be found in Appendix C.5 of the
LAION-5B paper. In brief, the training data contains 682K images that is
roughly balanced with images from safe for work and NSFW categories.

Table 5: Comparing LAION-2B CLIP based NSFW filtering model to Google Vision
API Safe Search adult category on a 40,000 random subset of Common Crawl.

| False Positive Rate | True Positives |  |
---|---|---|---|---
Threshold | (Relative to Google) | (Manual Review) | Model Positive Rate | Google API Positive Rate
0.1 | 3.6% | 2 | 14.4% | 3.5%
0.2 | 0.6% | 2 | 9.1% | 3.5%
0.3 | 0.3% | 3 | 7.2% | 3.5%

To evaluate our model and determine a threshold, we used Google Vision API’s
SafeSearch explicit content detector to generate labels for an 40,000 random
subset of our candidate pool. Specifically, an image is NSFW if SafeSearch
classifies it as likely or very likely adult (i.e., sexually explicit). As
shown in Table 5, we found that by thresholding at 0.1 we achieve high recall
relative to SafeSearch and very few true positives after manual review. We
also manually reviewed images classified by SafeSearch as likely or very
likely racy and found that the images were either benign, subjectively
suggestive but not explicit, or already found in the set of images labeled as
adult.

###  Appendix F Deduplication against evaluation sets

Figure 4:  Candidate images (top) that are detected as duplicates against
images in the evaluation sets (bottom) are removed from the pool. In addition
to exact duplicate images, near-duplicates with variable aspect ratios, JPEG
compression, overlays, color adjustment, and artistic rendering are also
detected.

To prevent data leakage, we filter CommonPool by removing duplicate and near-
duplicate matches of evaluation set images. See Figure 4 for example query
images from Common Crawl and corresponding near-duplicates in our evaluations
sets. We consider images as duplicates when the cosine similarity between a
query (Common Crawl image) feature and a reference (evaluation image) feature
is higher than a fixed threshold. We employ the deduplication model proposed
by Yokoo [150], which earned 1st place in the Facebook AI Image Similarity
Challenge (ISC) [40]. We choose a cosine similarity threshold of 0.604169 to
maximize the true duplicates detected, without removing too many false
duplicates from the pool. We compare against OpenAI’s CLIP ViT-B/32 as a
baseline on ISC. We find that for our threshold, the ISC model achieves
precision 0.9 and recall 0.8. At a threshold of 0.96, CLIP achieves the same
precision 0.9, but a significantly worse recall of 0.02. Approximately 2.8% of
downloaded samples are flagged as evaluation set near-duplicates.

Figure 5: Analysis of different de-duplication strategies across a variety of
image transformations. We see that the model introduced by Yokoo [150] is
better in almost every transformation, with the exception of very aggressive
aspect ratio modification.

To verify the performance of our de-duplication models with greater
granularity, we modify the evaluation procedure in Douze et al. [40] to
include transformations which are representative of naturally-occurring
duplications on the Internet. Specifically, we study: 1) jpeg compression
(encoding), 2) image flips, 3) image rotations, 4) aspect ratio modifications,
and 5) grayscaling. To do this, we sample 20% of the images from each of our
evaluation datasets uniformly at random to serve as a reference set of about
140,000 images. Next we sample 560,000 images uniformly at random from
LAION-2B to serve as distractors, for a 4-to-1 distractor to reference ratio.
Finally, we apply each of the augmentations above and use threshold filtering
to determine duplicates. Figure 5 shows the results from the deduplication
model [150] compared with OpenAI’s CLIP ViT-L/14. At high recall values, we
see that CLIP filtering results in removing over 2$\times$ the data as that of
the deduplication model from Yokoo [150].

###  Appendix G Face blurring

As an extra step to safeguard against issues of privacy that may arise from
the use of data scraped from the web, we include face blurring as part of our
pool creation. To create face metadata, we use the SCRFD face detector [53] to
extract bounding boxes for the faces in our images. These bounding boxes are
included as part of the image metadata in our pool. We make use of the
pretrained SCRFD-10G model. We use the same preprocessing as the one described
in the official repository of the paper, with the exception of providing
$224\times 224$ input images (by padding each image to square and then
resizing) to limit computation costs. Invoking this model provides us with
bounding boxes along with an associated score, which we then compare against a
threshold of $0.3$ to keep or discard this bounding box. This threshold is the
default one used in the repository of SCRFD for the visualization of bounding
boxes, and we found it to perform well on our data as discussed next.

In Table 6 we can see the result of face detection on a set of 3293 images
from CommonPool. We evaluate the detection on whether the image has visible
faces or not (where images such as cartoon drawings of non-real human faces
are not considered as positives), and whether the detector has detected these
visible faces. We considered an image as a true positive if all the clearly
visible faces in the image were detected, based on the above thresholding
process. We did not do extensive box labeling. True positives are instead
determined by human inspection. We compare the quality of these detections
with the Amazon Rekognition system, which is the one upon which the face
detections on ImageNet were based [148]. Note that in this scenario, the
recall of the detectors is more important than precision (as detecting a few
more bounding boxes across our pool does not affect privacy).

Table 6: Face detection performance on a set of 3293 random images from CommonPool. | SCRFD-10G | Amazon Rekognition
---|---|---
Accuracy | 93.87 | 96.57
Precision | 75.87 | 86.09
Recall | 90.53 | 93.75

To utilize these bounding boxes on our data, we apply a standard blurring
pipeline, as proposed by Yang et al. [148]. The result of this process is an
image where the faces is blurred and there is a smooth transition from blurred
to clean parts of the image. In Figure 6 we see the distribution of faces for
the small CommonPool. Note that the majority of images do not contain faces.

Figure 6: Frequency of predicted number of faces in the small CommonPool.

As part of our competition pipeline, images are by default blurred during the
download process. In Table 7 we can see the results of training on a set of
images with the size of our medium scale after filtering with each method,
with and without the application of face blurring as provided by our detector.
We can see that the difference in performance is small, which suggests that
the application of face blurring does not significantly affect the performance
on our downstream tasks. However, we note that this design decision may be
more detrimental in generative settings, especially when a generative model
needs to output faces. Our competition is primarily focused on discriminative
tasks, and as such when designing our dataset, we wished to prioritize the
safety and privacy of individuals through blurring faces in our download
tooling by default.

Finally, we evaluated the detector we used for potential biases. More
specifically, we used the detector on the validation set of the FairFace
dataset [80]. We found that the central face of the image was detected in all
the images of the validation set, regardless of subgroup annotate in the
dataset.

Table 7: Effect of face blurring on zero-shot performance. Face blurring
improves the privacy preservation of our dataset, while affecting model
performance negligibly. Results shown for training on a set of images with the
size of our medium scale, after filtering with each method.

Filtering | Face blurring | ImageNet acc. | Avg. performance
---|---|---|---
| $\times$ | 0.209 | 0.246
CLIP score (B/32, thresh. 0.3) + English filtering | ✓ | 0.196 | 0.243
| $\times$ | 0.287 | 0.301
CLIP score (B/32, 30%) | ✓ | 0.282 | 0.298

###  Appendix H DataComp CommonPool creation pipeline

Figure 7: Data funnel from potential samples in Common Crawl to 13.1B image-text pairs that were suitable for CommonPool. We sampled uniformly 12.8B datapoints for the xlarge CommonPool. Table 8: Provided metadata for CommonPool. Generation Time | Label | Additional notes
---|---|---
| uid |
| url | Link to the image.
| text | Image caption.
| original_width |
| original_height |
Step 2 | sha256 | Safeguard for data poisoning.
| clip_b32_similarity_score |
| clip_b32_image_features | In separate file.
| clip_b32_text_features | In separate file.
| clip_l14_similarity_score |
| clip_l14_image_features | In separate file.
| clip_l14_text_features | In separate file.
Step 1 | face_bboxes |
| nsfw_image_score |
| nsfw_text_score |
Step 2, dropped during Step 3 | dedup_score |

Creating CommonPool was a multistep process, which involved (1) parsing image
urls and alt-text from Common Crawl dumps and downloading these images, (2)
tagging images with metadata and (3) conducting safety content filtering and
evaluation set duplication. In this section we provide an overview of the data
pipeline used to create CommonPool. For an overview of our “data funnel” see
Figure 7.

  1. 1.

For the first step, we use parse Common Crawl metadata files to harvest image-
text pairs (Section D). We use img2dataset [5] to obtain $\sim$16.8B
downloaded samples. This is the first, unfiltered version of CommonPool, and
contains only basic information for our images (i.e., the original image
height, width, and alt-text caption). During this step we also resize images
such that their largest dimension does not exceed 512 pixels. This eases
storage requirements for large images, but is still larger than the 224 pixel
resolution used for later training stages.

  2. 2.

For the second step, we process our unfiltered pool and create richer metadata
for each image-text pair. We generate the following for each sample:

     * •

CLIP ViT-B/32 and CLIP ViT-L/14 image and text features, with their associated
similarities.

     * •

NSFW scores for the image and the text, using the analysis described in
Appendix E.

     * •

Deduplication score for the image, as described in Appendix F.

     * •

Bounding boxes for faces detected in the image, using the method described in
Appendix G.

  3. 3.

For the third and final step, we filter our image-text pairs based on the
metadata generated during the second stage. We filter out image-text pairs
where the NSFW and deduplication scores exceed the respective thresholds
(Section E). From the images that pass through this filtering, we keep only
the desired amount (e.g., 12.8B images from the xlarge CommonPool). Smaller
pools are telescoping subsets of larger pools. We package the metadata and
image urls, which is made publicly available to the participants. Note, we do
not release raw image data but rather image urls pointing to images.

A summary of the metadata for each sample is found in Table 8. To validate our
pipeline for duplication and CLIP feature correctness, we also take ImageNet
train though metadata generation as a unit test. Using the deduplication
features, we detect that 100% of the images are in fact duplicates.
Additionally using the CLIP ViT-B/32 and CLIP ViT-L/14 image features and
corresponding text features from OpenAI’s 80-prompt ensemble, we achieve
63.36% and 75.54% top-1 accuracies, which match the performance reported in
the CLIP paper [111].

When creating pools of different scale (i.e., number of samples), we ensure
that smaller pools are subsets of larger pools. For instance, the small
CommonPool is a subset of the xlarge CommonPool.

After CommonPool is created, the participants can then download the final
image-text pairs using the provided files via img2dataset. To further ease the
computational burden on participants, we additionally provide metadata for
each sample in CommonPool. Note that when downloading, our img2dataset
configuration automatically blurs faces. Hence this is an automatic step on
not something participants must do ad hoc.

###  Appendix I CommonPool statistics

To provide more information about the kinds of samples in our CommonPool, we
conduct additional analysis on the small pool, which is an i.i.d. sample of
downloaded data and a subset of the larger pools.

In Figure 8 we show CLIP similarity similarity scores between images and their
corresponding text. We notice a flatter distribution of CLIP ViT-L/14 scores
than corresponding B/32 scores.

Turning our attention to images in CommonPool, in Figure 9, we visualize the
aspect ratios and sizes of original images (i.e., before they are downloaded
and resized). In Figure 10, we display a distribution of image height and
width after _download_ resizing. Notice that the majority of images are around
$224\times 224$ pixels, which is the final resized resolution used for
training.

Analysing the textual component of each sample, we visualize frequency of the
number of CLIP BPE tokens in the captions (Figure 11) and most common
languages (Figure 12). Token counts follow a long-tailed distribution with
much more mass in the short sequence range, while English is the predominant
language in CommonPool according to fasttext and cld3.

We also look at url statistics. In Figure 13 we see common domain names in
CommonPool (e.g., wordpress domains) and common suffixes (e.g., .com or .net).

Figure 8: Image-text similarity score distributions using CLIP ViT-B/32
_(left)_ and ViT-L/14 _(right)_ models. We plot samples from the small
CommonPool, which are an i.i.d. sample of the xlarge CommonPool. Figure 9:
Statistics for images in the small CommonPool, before applying resizing.
Figure 10: Image pixel heatmap. Each entry in the above heatmap represents the
estimated probability that a pixel is occupied. The center entry has a value
of 1.0 as every image has a center pixel. We compute the heatmap over the
small CommonPool. Note that image sizes are bounded as we resize all images
such that their max dimension does not exceed 512 pixels during dataset
download. Figure 11: Distribution of token length for alt-text in the small
CommonPool. The CLIP BPE tokenizer is used for tokenization. Figure 12: Counts
for the top 25 most frequent languages in the small CommonPool, as predicted
by fasttext _(left)_ and cld3 (_right_). Figure 13: Counts for the top 25 most
frequent domains _(left)_ and suffixes (_right_) in the small CommonPool.

###  Appendix J Efficient training on data subsets

When training at large scale, it is important to use efficient access patterns
to load training data. This typically means that data must be loaded using
large sequential reads instead of random reads in order to maximize
throughput. In DataComp, this is facilitated by the
WebDataset555https://github.com/webdataset/webdataset format which stores the
training examples in tar files (called “shards”) and WebDataLoader which makes
it easy to load data stored in this format.

Given an arbitrary subset of a pool, we would like to efficiently train on
that subset. Because WebDataset format does not permit efficient random access
(a feature inherited from tar), we must read through the entire pool to select
the required images. There are two ways to implement this filtering:

  1. 1.

Filter during training: we apply a predicate during training data loading that
discards data not present in the subset.

  2. 2.

Filter before training: we iterate over the pool, selecting the images in the
subset, and write them to a new WebDataset.

After some profiling, we concluded that option 1 had too much overhead in the
case where the subset is much smaller than the pool. To see why, note that if
the subset is an $p$-fraction of the pool size, then we would end up reading a
$1/p$ factor more data than needed for training. Instead, we give an
implementation of option 2, which performs at most twice as many reads as
needed for training.666Since in DataComp, the number of examples seen is equal
to the pool size.

Our tool, called the resharder, reads a set of uids in NumPy array format,
scans through the pool, selecting those examples, and writes them to a new
WebDataset. The resharder uses multiprocessing to make good use of hardware
and can be distributed over many computers to further increase throughput. The
resharder also supports streaming data to and from cloud storage such as
Amazon S3. The resharder is provided to participants as part of the
competition tooling.

###  Appendix K Effect of duplicates in the training data

Given that CommonPool was constructed by scraping the web for image and text
pairs, there is a likelihood that some of our images are duplicates of each
other, even if they originated from different web sources and have different
captions. Here we examine the effect of removing such duplicates. We used the
technique proposed by Webster et al. [144], where CLIP image features are
first compressed and then used to do an approximate nearest neighbor search.
After this process, two images $x$ and $y$ are considered duplicates if
$\frac{|d_{ADC}(x,x)-d_{ADC}(x,y)|}{d_{ADC}(x,x)}<T_{ADC}$, where $T_{ADC}$ is
some threshold and $d_{ADC}(x,x)$ is the distance of a vector with its
quantized version used for approximate nearest neighbor search. For each
image, we search duplicates across its $1000$ nearest neighbors, and keep it
if it’s the one with the highest CLIP ViT-L/14 similarity score across its
duplicates. Results can be seen in Table 9, both when this technique is used
by itself and in conjunction with ViT-B/32 filtering. We can see that results
are similar to when only using CLIP filtering.

Table 9: Effect of deduplication of training set for the medium size CommonPool. The filtering performed here is CLIP B32 score top 30% (see Table 26). Higher threshold values lead to more samples being labeled as duplicates. Subset | Training dataset size | ImageNet accuracy | Average performance
---|---|---|---
$T_{ADC}=0.1$, without filtering | 99.8M | 0.195 | 0.275
$T_{ADC}=0.2$, without filtering | 85.9M | 0.200 | 0.277
$T_{ADC}=0.5$, without filtering | 29.6M | 0.227 | 0.295
$T_{ADC}=0.1$, with filtering | 33.5M | 0.288 | 0.337
$T_{ADC}=0.2$, with filtering | 30.6M | 0.289 | 0.337
$T_{ADC}=0.5$, with filtering | 15.5M | 0.252 | 0.311

###  Appendix L Hyperparameter ablations

Recall that in DataComp, we freeze the training procedure and hyperparameters
to focus the competition on dataset curation. However, this leads to the
natural question: do “better” datasets (i.e., datasets that lead to higher
accuracy models on zero-shot downstream tasks) remain consistent when training
is modified. Hence we ablate key experimental choices: batch size, model
architecture, and number of training steps.

####  L.1 Batch size

We ablate over the batch size hyperparameter, doubling the batch size at the
medium scale, but holding all other hyperparameters constant. As see in Table
10, we find that the delta rankings are largely consistent, for both ImageNet
and Average performance, with rankings changing by at most plus or minus one
position. More specifically, rank correlation before and after doubling batch
size is 0.96 for ImageNet and 0.98 for the Average over 38 datasets metric.

####  L.2 Model architecture

We choose to use the ViT architecture [39] because of favorable CLIP scaling
trends over vanilla ResNets [62] as reported by Radford et al. [111]. However,
we still hope that better datasets for downstream ViT performance will lead to
better datasets to train convolutional architectures. We look at the medium
scale, swapping the ViT-B/32 architecture with a ConvNeXt model [93] with
matched giga multiplier–accumulate operations (GMACs). Looking at Table 11, we
see that ranking of different filtering methods is again relatively consistent
(i.e., 1.0 rank correlation for ImageNet and 0.87 rank correlation for the
average metric). We conclude that improvements in dataset filtering have
potential to improve more than just CLIP ViT model performance.

####  L.3 Number of training steps

Recall that one of our major design decisions for DataComp is to fix the
hyperparameters associated with model training, following closely
hyperparameters from prior work [111]. We choose to fix hyperparameters to
place emphasis on data curation and remove confounders arising from
hyperparameter differences between participants. Here we ablate our
hyperparameter configuration by training small baselines for 10$\times$ more
steps. In Figure 14 we see positive correlation for ImageNet accuracy for the
ablated and original hyperparameter configurations. We see similar correlation
for average performance. See Table 12 for specific values.

Table 10: Batch size ablation at the medium scale. We compare the standard
DataComp medium configuration, with batch size 4096 against an ablated
configuration with batch size 8192 (medium: batch size 2x). We find that the
rankings of the baseline filtering strategies are relatively consistent. More
precisely, the rank correlation is 0.96 on ImageNet and 0.98 for the Average
over 38 datasets.

|  | Dataset | Samples | ImageNet | Average over | Delta ranking | Delta ranking
---|---|---|---|---|---|---|---
Scale | Filtering strategy | size | seen |  | 38 datasets | ImageNet | Average
| No filtering | 128M | 128M | 0.176 | 0.258 | - | -
| Basic filtering | 30M | 128M | 0.226 | 0.285 | - | -
| Text-based | 31M | 128M | 0.255 | 0.307 | - | -
| Image-based | 29M | 128M | 0.268 | 0.312 | - | -
| LAION-2B filtering | 13M | 128M | 0.230 | 0.292 | - | -
| CLIP score (L/14 30%) | 38M | 128M | 0.273 | 0.328 | - | -
medium | Image-based $\cap$ CLIP score (L/14 30%) | 14M | 128M | 0.297 | 0.328 | - | -
| No filtering | 128M | 128M | 0.171 | 0.258 | 0 | 0
| Basic filtering | 30M | 128M | 0.219 | 0.277 | +1 (worse) | 0
| Text-based | 31M | 128M | 0.251 | 0.299 | 0 | -1 (better)
| Image-based | 29M | 128M | 0.260 | 0.299 | 0 | 0
| LAION-2B filtering | 13M | 128M | 0.215 | 0.288 | -1 (better) | 0
| CLIP score (L/14 30%) | 38M | 128M | 0.271 | 0.324 | 0 | 0
medium: batch size 2x | Image-based $\cap$ CLIP score (L/14 30%) | 14M | 128M | 0.276 | 0.311 | 0 | +1 (worse)

Table 11: Architure ablation at the medium scale. We compare the standard
DataComp medium configuration, with a ViT-B/32 model against an ablated
configuration (medium: ConvNeXt), which uses a ConvNeXt model with the same
number of multiply-accumulate operations as the ViT. We find that the rankings
of the baseline filtering strategies are relatively consistent. More
precisely, the rank correlation is 1.0 on ImageNet and 0.87 for the Average
over 38 datasets.

|  | Dataset | Samples | ImageNet | Average over | Delta ranking | Delta ranking
---|---|---|---|---|---|---|---
Scale | Filtering strategy | size | seen |  | 38 datasets | ImageNet | Average
| No filtering | 128M | 128M | 0.176 | 0.254 | - | -
| Basic filtering | 30M | 128M | 0.226 | 0.280 | - | -
| Text-based | 31M | 128M | 0.255 | 0.301 | - | -
| Image-based | 29M | 128M | 0.268 | 0.307 | - | -
| LAION-2B filtering | 13M | 128M | 0.230 | 0.287 | - | -
| CLIP score (L/14 30%) | 38M | 128M | 0.273 | 0.323 | - | -
medium | Image-based $\cap$ CLIP score (L/14 30%) | 14M | 128M | 0.297 | 0.323 | - | -
| No filtering | 128M | 128M | 0.178 | 0.255 | 0 | 0
| Basic filtering | 30M | 128M | 0.232 | 0.272 | 0 | 0
| Text-based | 31M | 128M | 0.255 | 0.298 | 0 | 0
| Image-based | 29M | 128M | 0.270 | 0.298 | 0 | +1 (better)
| LAION-2B filtering | 13M | 128M | 0.253 | 0.300 | 0 | -2 (better)
| CLIP score (L/14 30%) | 38M | 128M | 0.279 | 0.326 | 0 | +1 (worse)
medium: ConvNeXt | Image-based $\cap$ CLIP score (L/14 30%) | 14M | 128M | 0.323 | 0.331 | 0 | 0

Figure 14: _(left)_ The effect of training for 10$\times$ steps for for small
filtering track baselines on ImageNet. _(right)_ Similar plot but for Avg.
performance. While the ordering of some methods changes quite drastically, we,
in general, see a positive correlation. Table 12: Experiment details when
extending the number of steps by 10 times the standard amount for that scale.

Scale | Filtering | ImageNet | ImageNet | VTAB | Retrieval | Average over
---|---|---|---|---|---|---
dist. shifts | 38 datasets
| No filtering | 0.102 | 0.093 | 0.204 | 0.147 | 0.196
| Random subset(75%) | 0.078 | 0.072 | 0.182 | 0.129 | 0.178
| Random subset(50%) | 0.045 | 0.049 | 0.161 | 0.104 | 0.150
| Random subset(25%) | 0.023 | 0.029 | 0.134 | 0.075 | 0.119
| Random subset(10%) | 0.010 | 0.018 | 0.119 | 0.069 | 0.101
| Random subset(1%) | 0.002 | 0.006 | 0.097 | 0.056 | 0.082
| Caption length | 0.085 | 0.080 | 0.198 | 0.136 | 0.184
| Image size | 0.066 | 0.064 | 0.153 | 0.115 | 0.158
| English (fasttext) | 0.068 | 0.068 | 0.172 | 0.108 | 0.159
| English (fasttext) and caption length | 0.066 | 0.065 | 0.182 | 0.106 | 0.163
| English (fasttext), caption length, and image size | 0.045 | 0.048 | 0.164 | 0.092 | 0.149
| CLIP B32 score top 10% | 0.035 | 0.046 | 0.162 | 0.079 | 0.139
| CLIP B32 score top 20% | 0.076 | 0.076 | 0.182 | 0.099 | 0.172
| CLIP B32 score top 30% | 0.096 | 0.090 | 0.221 | 0.121 | 0.205
| CLIP B32 score top 40% | 0.081 | 0.077 | 0.200 | 0.124 | 0.193
| CLIP B32 score top 50% | 0.106 | 0.097 | 0.211 | 0.134 | 0.205
| CLIP B32 score top 75% | 0.103 | 0.096 | 0.210 | 0.150 | 0.198
| CLIP B32 score top 90% | 0.105 | 0.096 | 0.212 | 0.152 | 0.202
| CLIP B32 threshold at 0.3 + English filter | 0.029 | 0.036 | 0.152 | 0.078 | 0.134
| CLIP B32 threshold at 0.28 + English filter | 0.035 | 0.041 | 0.168 | 0.086 | 0.145
| CLIP B32 threshold at 0.3 | 0.076 | 0.078 | 0.199 | 0.102 | 0.182
| CLIP L14 score top 10% | 0.026 | 0.037 | 0.130 | 0.073 | 0.123
| CLIP L14 score top 20% | 0.060 | 0.064 | 0.161 | 0.096 | 0.153
| CLIP L14 score top 30% | 0.088 | 0.087 | 0.199 | 0.115 | 0.188
| CLIP L14 score top 40% | 0.100 | 0.096 | 0.217 | 0.122 | 0.207
| CLIP L14 score top 50% | 0.104 | 0.098 | 0.212 | 0.136 | 0.203
| CLIP L14 score top 75% | 0.103 | 0.095 | 0.189 | 0.146 | 0.191
| CLIP L14 score top 90% | 0.105 | 0.095 | 0.203 | 0.145 | 0.198
| Image-based clustering (ImageNet1k) | 0.053 | 0.053 | 0.162 | 0.091 | 0.146
| Image-based clustering (ImageNet21k) | 0.063 | 0.059 | 0.173 | 0.108 | 0.167
| Text-based clustering (ImageNet1k) | 0.012 | 0.018 | 0.120 | 0.062 | 0.104
| Text-based clustering (ImageNet21k) | 0.262 | 0.216 | 0.305 | 0.246 | 0.300
| Intersect IN1k image clustering and CLIP B32 score top 30% | 0.058 | 0.059 | 0.179 | 0.098 | 0.161
| Intersect IN1k image clustering and CLIP L14 score top 30% | 0.049 | 0.051 | 0.171 | 0.090 | 0.150
| Intersect IN21k image clustering and CLIP B32 score top 30% | 0.071 | 0.070 | 0.192 | 0.107 | 0.175
small | Intersect IN21k image clustering and CLIP L14 score top 30% | 0.064 | 0.065 | 0.200 | 0.096 | 0.173
| No filtering | 0.370 | 0.304 | 0.387 | 0.355 | 0.383
| English (fasttext), caption length, and image size | 0.317 | 0.269 | 0.324 | 0.271 | 0.334
| CLIP B32 score top 30% | 0.436 | 0.351 | 0.433 | 0.345 | 0.430
| CLIP B32 score top 40% | 0.434 | 0.353 | 0.448 | 0.365 | 0.442
| CLIP B32 score top 50% | 0.426 | 0.352 | 0.439 | 0.377 | 0.433
| CLIP B32 score top 75% | 0.398 | 0.325 | 0.396 | 0.374 | 0.411
| Image-based clustering (ImageNet1k) | 0.363 | 0.294 | 0.347 | 0.279 | 0.347
| Image-based clustering (ImageNet21k) | 0.374 | 0.303 | 0.372 | 0.318 | 0.372
| Intersect IN1k image clustering and CLIP B32 score top 30% | 0.415 | 0.330 | 0.413 | 0.310 | 0.403
medium | Intersect IN1k image clustering and CLIP L14 score top 30% | 0.405 | 0.325 | 0.399 | 0.295 | 0.387

###  Appendix M Detector-based baselines

While controlling for factors such as class balance is common in the
supervised settings, experimenting with analogous strategies in the context of
multimodal datasets and CLIP training is a pertinent direction. Towards this
end, we use the Detic detector [160] to annotate the medium pool (128M
samples) by extracting bounding boxes and class labels for the 1203 LVIS [54]
objects categories. Following the original Detic paper, we retain predictions
whose confidence score exceeds 0.5. Based on these annotations, we construct
the following five strategies:

  * •

_Object exists_ : Subset for which there exists at least one detection from
the 1203 LVIS categories.

  * •

_Object centered_ : Subset for which there exists at least one detection from
the 1203 LVIS categories with a bounding box center falling in the center grid
cell of a 3x3 grid superimposed on the image.

  * •

Balancing by class: We define 1204 buckets—1203 buckets corresponding to the
LVIS classes and an additional bucket for images that do not have any
detections. For each image in the medium pool, we assign the image to the
bucket(s) corresponding to the detected classes. We then construct a dataset
such that there are an equal number of samples from each bucket and the total
number of samples specified by a particular scale (e.g., 128M samples for
medium scale). Note, for rare classes there can be many repeated samples and
for common classes only a subset of the total samples will be in the dataset.

  * •

Balancing by position: We define 26 buckets—0, 1, …, 24 corresponding to 5x5
grid locations in an image. An image is added to bucket(s) when it contains a
bounding box whose center falls in the bucket’s grid cell. The 25th bucket
contains images for which there are no detections. We again construct a