00footnotetext: ∗ Shared first author; † Shared last author

# DataComp-LM: In search of the next generation of training sets for language
models

Jeffrey Li*1,2 Alex Fang*1,2 Georgios Smyrnis*4 Maor Ivgi*5 Matt Jordan4 Samir
Gadre3,6 Hritik Bansal8 Etash Guha1,15 Sedrick Keh3 Kushal Arora3 Saurabh
Garg13 Rui Xin1 Niklas Muennighoff22 Reinhard Heckel12 Jean Mercat3 Mayee
Chen7 Suchin Gururangan1 Mitchell Wortsman1 Alon Albalak19,20 Yonatan Bitton14
Marianna Nezhurina9,10 Amro Abbas23 Cheng-Yu Hsieh1 Dhruba Ghosh1 Josh
Gardner1 Maciej Kilian17 Hanlin Zhang18 Rulin Shao1 Sarah Pratt1 Sunny Sanyal4
Gabriel Ilharco1 Giannis Daras4 Kalyani Marathe1 Aaron Gokaslan16 Jieyu Zhang1
Khyathi Chandu11 Thao Nguyen1 Igor Vasiljevic3 Sham Kakade18 Shuran Song6,7
Sujay Sanghavi4 Fartash Faghri2 Sewoong Oh1 Luke Zettlemoyer1 Kyle Lo11
Alaaeldin El-Nouby2 Hadi Pouransari2 Alexander Toshev2 Stephanie Wang1 Dirk
Groeneveld11 Luca Soldaini11 Pang Wei Koh1 Jenia Jitsev9,10 Thomas Kollar3
Alexandros G. Dimakis4,21 Yair Carmon5 Achal Dave†3 Ludwig Schmidt†1,7
Vaishaal Shankar†2

1University of Washington, 2Apple, 3Toyota Research Institute, 4UT Austin,
5Tel Aviv University, 6Columbia University, 7Stanford, 8UCLA, 9JSC, 10LAION,
11AI2, 12TUM, 13CMU, 14Hebrew University, 15SambaNova, 16Cornell, 17USC,
18Harvard, 19UCSB, 20SynthLabs, 21Bespokelabs.AI, 22Contextual AI,
23DatologyAI

###### Abstract

We introduce DataComp for Language Models (DCLM), a testbed for controlled
dataset experiments with the goal of improving language models. As part of
DCLM, we provide a standardized corpus of 240T tokens extracted from Common
Crawl, effective pretraining recipes based on the OpenLM framework, and a
broad suite of 53 downstream evaluations. Participants in the DCLM benchmark
can experiment with data curation strategies such as deduplication, filtering,
and data mixing at model scales ranging from 412M to 7B parameters. As a
baseline for DCLM, we conduct extensive experiments and find that model-based
filtering is key to assembling a high-quality training set. The resulting
dataset, DCLM-baseline, enables training a 7B parameter language model from
scratch to $64$% 5-shot accuracy on MMLU with 2.6T training tokens. Compared
to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-
baseline represents a 6.6 percentage point improvement on MMLU while being
trained with 40% less compute. Our baseline model is also comparable to
Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on
an average of 53 natural language understanding tasks while being trained with
$6.6\times$ less compute than Llama 3 8B. Our results highlight the importance
of dataset design for training language models and offer a starting point for
further research on data curation. We release the DCLM benchmark, framework,
models, and datasets at https://datacomp.ai/dclm.

###  Introduction

Large training datasets are an important driver of progress in the recent
language modeling (LM) revolution [84, 131, 150, 59, 123, 155, 172, 64]. As
the cost of training state-of-the-art language models continues to grow,
researchers increasingly focus not only on _scaling_ but also on _improving_
training datasets that enable efficient generalization on a wide range of
downstream tasks. Indeed, there is a growing number of proposals for filtering
data, removing (near-) duplicates, finding new data sources, weighting data
points, generating synthetic data, and so on [113, 88, 8, 177, 96, 69, 91, 2].

Figure 1:  Improving training sets leads to better models that are cheaper to
train. Using DataComp-LM, we develop a high-quality dataset, DCLM-baseline,
which we use to train models with state-of-the-art trade-off between compute
and performance. We compare on both _(left)_ a Core set of tasks and on
_(right)_ MMLU 5-shot. Specifically DCLM-baseline (orange) shows favorable
performance relative to both close-source models (crosses) and other open-
source datasets and models (circles). Models in this figure are from [161,
162, 4, 97, 22, 189, 154, 121, 43, 160, 68, 150, 130, 100, 10, 156].

A key challenge in this emerging research area is a lack of controlled
comparisons. While the aforementioned proposals generally use the same
evaluation datasets, researchers often compare models that are trained with
different architectures, compute, or hyperparameters. Hence, it is often
unclear what data curation strategies work best: Are the results of training
set A better than training set B because training set A is truly better, or
because the model trained on A was combined with a better architecture,
learning rate schedule, or more compute? Disentangling the many factors
influencing the quality of a language model is crucial to understanding which
data curation strategies work best and ultimately building better language
models.

Beyond the lack of standardized benchmarks, another challenge for research on
training data is that details about training sets are becoming increasingly
rare, even for open weight models such as the Llama, Mistral, or Gemma models
[161, 77, 154]. For all of these models, the training sets are not publicly
available, and the corresponding model documentation only provides a coarse
description of the respective training data, if any at all. As a result, it is
currently unclear what ingredients constitute a state-of-the-art training set
for language models.

To address these challenges, we introduce DataComp for Language Models (DCLM),
the first benchmark for language model training data curation. In DCLM,
researchers propose new training sets and data curation algorithms and then
evaluate their datasets by training language models with a _fixed_ training
recipe on their data. By measuring the performance of the resulting model on
downstream tasks, researchers can quantify the strengths and weaknesses of the
corresponding training set.

Figure 2:  The DCLM workflow. _(A)_ A participant first chooses a scale, where
larger scales reflect more training tokens or model parameters. _(B)_ A
participant then filters a pool of data (filtering track) or mixes data of
their own (mixing track) to create a dataset. _(C)_ Using the curated dataset,
a participant trains a language model, with standardized training code and
scale-specific hyperparameters, which is then _(D)_ evaluated on 53 downstream
tasks to judge dataset quality.

To enable DCLM, we contribute a comprehensive experimental testbed. A key
component is DCLM-Pool, a corpus of 240 trillion tokens derived from Common
Crawl [42]. DCLM-Pool is the largest public corpus for language model training
and forms the cornerstone of the DCLM filtering track, where participants aim
to curate the best possible training set out of DCLM-Pool. In addition, we
provide open-source software for processing large datasets with several
filtering approaches.

The high cost of training language models makes it necessary to understand the
performance of training recipes across different compute and data scales.
Hence, our third contribution is an investigation of scaling trends for
dataset design. We find that models as small as 400M parameters can still
provide signal on which training sets perform better at larger scales. Based
on our experiments, we organize DCLM into five compute scales spanning a range
of about $600\times$ in compute from 400M parameter models to over-trained 7B
models. This multi-scale design makes DCLM accessible to researchers with
varying compute budgets.

As a starting point for DCLM, we conduct 416 baseline experiments with
different training sets and compute scales. Our experiments identify model-
based filtering as a key component in an effective data curation pipeline. We
also show that details of the filtering model can have a large impact on
performance, ranging from 35% to 44% accuracy on MMLU 5-shot [71] at the 7B
parameter scale (280B training tokens). Interestingly, a simple bigram
classifier, combined with a carefully selected set of positive and negative
examples, performs best among the classifiers we experimented with. In
addition, we find that human quality judgments have only limited value in
identifying high-quality training data.

Finally, we combine our results into DCLM-baseline, a new state-of-the-art
public training set for language models. When training a 7B parameter language
model on 2.6 trillion tokens using DCLM-baseline, the resulting model achieves
$64$% on MMLU, which is state-of-the-art among open-data models and close to
models such as Mistral-7B-v0.3 (63%) or Llama 3 8B (66%) that are trained with
up to $6.6\times$ more compute (Llama 3 8B). Compared to Llama 2 7B, training
a 7B parameter model on 280B tokens from DCLM-baseline achieves 5 pp higher
MMLU while being trained with $7\times$ less compute. As our 7B model uses a
standard decoder-only Transformer [165, 127, 161], our results also highlight
that a systematic approach to data curation is key to training performant
language models.

We publicly release our DCLM framework, models, and training sets at
https://datacomp.ai/dclm to enable other researchers to participate in DCLM
and to strengthen the empirical foundations for data-centric research on
language models.

###  Related work

We summarize closely related work in this section and provide additional
related work in Appendix B.

###### Data curation for language models.

To collect large datasets for training LMs [30], researchers typically resort
to web crawls, which can contain undesirable content that can be improved via
curation. Most data curation efforts focus on methods for improving model
performance [131, 30, 170, 128, 121, 150], including filtering by language
[44, 131, 180, 86], heuristic-based filtering [59, 128, 121, 34, 150], quality
filtering [170, 49, 99, 178, 139], data deduplication [3, 88] and mixing [177,
148, 6]. While prior work examines a limited set of filters, we conduct the
largest public investigation of data curation, resulting in a strong DCLM-
baseline dataset.

###### Open-source datasets.

As the scale of LMs has increased over the past years [161, 162, 4, 36, 73,
115, 128, 154], the community has curated larger datasets to match. Early
works include the C4 dataset with 160 billion (B) tokens and The Pile [59]
with 300B tokens. More recently, RefinedWeb [121] contains 600B tokens, Dolma
[150] 3 trillion (T) tokens, FineWeb 15T tokens [122], and RedPajama-v2 30T
tokens [43]. There are also large domain-specific datasets, such as the code-
focused StackV2 with 900B tokens [101], as well as high-quality filtered
subsets such as FineWeb-Edu [100] with 1.3T tokens. We include performance
comparisons with various datasets in Figure 1 and examine FineWeb’s LightEval
evaluation framework more closely in Appendix G. We release the largest pool
of raw text data to date with 240T web-crawled tokens. We also release DCLM-
baseline, a high-quality dataset from our pool that yields better models than
prior datasets.

###### Data-centric benchmarks.

Past work on benchmarking data improvements includes dataset distillation
[46], curriculum learning [137], and transfer learning [5, 31]. In DataComp
[57] and DataPerf [106], participants iterate on a dataset with a fixed model
and training recipe for vision, vision-language, and speech tasks. The BabyLM
challenge _Loose_ track [167] focuses on efficient development of LMs with
125M to 220M parameters trained on 10M to 100M tokens. With a 200T token pool
and 7B models, DCLM is the first large-scale data-centric benchmark for
language models.

###  The DataComp for language models (DCLM) benchmark

Table 1:  DCLM competition scales. DCLM contains five competition scales, enabling research in varying compute regimes. Each scale specifies the model size (‘Model parameters’, $N$), the number of tokens seen during training (‘Train tokens’, $D$), and the size of the original pool that can be used for filtering (‘Pool size’). We provide an estimate of the compute required for training (‘Train FLOPs’$=6ND$) and GPU hours (‘Train H100 hours’) using the OpenLM training framework [70]. Scale |  Model parameters | Train tokens | Train FLOPs |  Train H100 hours | Pool size
---|---|---|---|---|---
400M-1x | 412M | 8.2B | 2.0e19 | 26 | 469B
1B-1x | 1.4B | 28.8B | 2.4e20 | 240 | 1.64T
1B-5x | 1.4B | 144B | 1.2e21 | 1,200 | 8.20T
7B-1x | 6.9B | 138B | 5.7e21 | 3,700 | 7.85T
7B-2x | 6.9B | 276B | 1.1e22 | 7,300 | 15.7T

This section describes the main components of DCLM. We start with DCLM-Pool,
the raw text corpus underlying our benchmark (Section 3.1). We then develop
the DCLM workflow, visualized in Figure 2: selecting a competition scale
(Section 3.2), curating a dataset by filtering DCLM-Pool and potentially
mixing in other sources (Section 3.3), training a model with fixed
hyperparameters (Section 3.4), and evaluating the model to score the dataset
(Section 3.5).

####  DCLM-Pool

DCLM-Pool is an unfiltered web-text corpus comprised of all Common Crawl [42]
data prior to 2023. Based on Section 4.2, we re-extract text from HTML using
resiliparse [20, 21] instead of using Common Crawl’s pre-extracted text. DCLM-
Pool contains 200B documents (370TB after gzip compression), resulting in 240T
GPT-NeoX [24] tokens. See Appendix E for additional details.

###### Decontamination.

Test set samples often contaminate language model training sets [48, 181, 51];
however, the effect of such samples on downstream performance remains largely
unclear [88, 150, 115]. To allow researchers to better understand
contamination, we release decontamination tooling instead of decontaminating
DCLM-Pool directly. Our tools are based on Lee et al. [88] and allow
participants to examine their datasets for overlap with our test sets. We ask
all submissions to disclose a decontamination report and avoid using highly-
contaminated data. For the highest scoring submissions, we plan to
specifically evaluate them for contamination. In Section 4.6, we apply our
tools to DCLM-Pool and evaluate whether contamination affects our models.

####  Competition scales: Supporting participants with different compute
constraints

Figure 3: Datasets rank consistently across competition scales in DCLM. This
makes it possible to iterate on data curation at small scales.

To ensure DCLM is accessible to researchers with different compute constraints
and to facilitate the study of scaling trends, we create different competition
scales spanning three orders of compute magnitude (Table 1).Each scale (i.e.,
400M-1x, 1B-1x, 1B-5x, 7B-1x, and 7B-2x) specifies the number of model
parameters (e.g., 7B) and a _Chinchilla multiplier_ (e.g., 1x). The number of
training tokens for each scale is $20$ $\times$ number of parameters $\times$
Chinchilla multiplier so that a multiplier of 1x corresponds to a compute
allocation that Hoffmann et al. [73] found near-optimal.

A potential pitfall in our multi-scale design is that the ranking of data
curation methods may change when increasing the compute scale. To better
understand this concern, in Figure 3, we plot the performance of 10 methods at
the 7B-1x scale as a function of their 400M-1x and 1B-1x performance. We find
high rank correlation between the smaller 400M-1x, 1B-1x results and larger
7B-1x results (Pearson’s $r=0.885$ and $r=0.919$, respectively), suggesting
better curation strategies at smaller scales transfer to larger scales. For
more competition scale ablations, including experiments suggesting dataset
improvements are largely orthogonal to training hyperparameters, see Appendix
H.

####  Benchmark tracks: Filtering and mixing

After choosing a scale, participants choose one of two tracks. (i) In the
_filtering track_ , participants propose algorithms to select training data
from a candidate pool. We start with five pools, one for each scale in Table
1, which are random document subsets of DCLM-Pool. We restrict initial pool
sizes by scale to encourage scalable filtering strategies and reflect
realistic data download and storage constraints. (ii) In the _mixing track_ ,
a submission combines documents from potentially many sources. For instance,
participants can synthesize documents from DCLM-Pool, a custom crawl, Stack
Overflow, and Wikipedia. Appendix C provides detailed rules for each track,
and Appendix D describes our open-source, extensible tooling for executing
filtering and mixing operations.

####  Training

To isolate the effect of dataset interventions, we fix a training recipe at
each scale. Based on prior ablations on model architectures and training [127,
174, 58, 87, 73, 30, 36, 161, 162, 4], we adopt a decoder-only Transformer
(e.g., GPT-2, Llama) [165, 161, 127], implemented in OpenLM [70]. We also
provide unified data processing utilities. Appendix F contains additional
training details.

####  Evaluation

Our full evaluation suite, based on LLM-Foundry [109], contains 53 downstream
tasks suitable for base model evaluation (i.e., without fine-tuning): from
question answering to open-ended generation formats, considering varied
domains like coding, text-book knowledge, and common-sense reasoning. To
evaluate data curation algorithms, we focus on three main performance metrics.
First, we consider _MMLU 5-shot accuracy_ [72], which is widely used to
compare state-of-the-art models like GPT-4 [115] and Llama 3 70B [4]. Second,
we propose _Core centered accuracy_, computed over a subset of 22 tasks (e.g.,
HellaSwag [186] and ARC-E [40]) that provide a low-variance signal even at
small scales, linearly rescaling the accuracy per task so that 0 corresponds
to random guessing and 1 corresponds to perfect accuracy. Finally, we report
_Extended centered accuracy_, which averages the centered performance for all
of our 53 tasks. For more metric details, see Appendix G.

###  Building high-quality training datasets with DCLM

We now show how the DCLM workflow can lead to high-quality datasets and
quantify the effect of data curation methods. This section describes the
process of converting Common Crawl into our dataset, DCLM-baseline, as shown
in Figure 4. We provide ablation experiments for each step along the way. We
first evaluate open-source datasets as a starting point (Section 4.1). Next,
we experiment with alternatives for several key phases of dataset
construction: text extraction (Section 4.2), deduplication (Section 4.3), and
model-based filtering (Section 4.4). We then experiment with mixing in high-
quality sources (Section 4.5) and provide a contamination analysis (Section
4.6). In Section 5, we scale up this approach to train a 7B model for 2T
tokens.

Figure 4: Construction of DCLM-baseline from DCLM-Pool. Before this pipeline,
we extracted DCLM-Pool from Common Crawl with resiliparse. Percentages are
based on the total number of original documents. Figure 4: Construction of
DCLM-baseline from DCLM-Pool. Before this pipeline, we extracted DCLM-Pool
from Common Crawl with resiliparse. Percentages are based on the total number
of original documents.

####  Evaluating existing training datasets

We begin by evaluating several well-known open-source datasets (C4 [130, 48],
RefinedWeb [121], RedPajama [160], and Dolma-V1 [150]) in Table 2. While all
four datasets use various heuristic filters and data cleaning steps, we find
that RefinedWeb performs the best on our Core and Extended metrics at the
7B-1x scale. RefinedWeb applies the following filtering pipeline: Common Crawl
text extraction, heuristic selection rules (e.g., to remove spam), and
deduplication of repeated content. Interestingly, RefinedWeb is solely
filtered from Common Crawl, unlike RedPajama and Dolma-V1, which additionally
mix in curated, “high-quality” sources like Wikipedia. The comparison suggests
the relative strength of filtering, which we explore later in our experiments.

Table 2:  Comparison to existing datasets (7B-1x scale). Despite not mixing high-quality sources, RefinedWeb performs best.  Dataset | Core | Extended
---|---|---
C4 | 34.2 | 18.0
Dolma-V1 | 35.0 | 18.4
RedPajama | 35.3 | 18.2
RefinedWeb | 36.9 | 19.8

Takeaway: For DCLM-baseline and other experiments, we adopt RefinedWeb’s
heuristic filters.

####  Text extraction

Text extraction is a common early processing step that pulls content from raw
HTML. To understand the effect of this step, we compare three text extraction
approaches: resiliparse, trafilatura (used by RefinedWeb), and the Common
Crawl-provided WET files that contain pre-extracted text. We then apply
RefinedWeb’s heuristic quality filters to each of the text extractions. In
Table 3, we find both resiliparse and trafilatura improve Core by at least 2.5
points over the WET extraction. This is significant because most open source
datasets, including C4, RedPajama, and Dolma-V1, use the WET extraction, which
could partially explain their worse performance in Table 2. While resiliparse
and trafilatura have similar downstream performance, resiliparse is $8\times$
faster to run and hence more practical for large-scale processing. For more
analysis, see Appendix J.

Takeaway: For DCLM-Pool and the remaining experiments, we use resiliparse to
extract text.

Table 3: Comparison of text extractors (1B-1x scale). We apply three approaches for text extraction from HTML, process their output using the RefinedWeb heuristic quality filters, and evaluate the quality of models trained on the resulting datasets. We find stricter extractors such as resiliparse and trafilatura are superior to WET files provided by Common Crawl. Text Extraction | Core | Extended
---|---|---
resiliparse | 24.1 | 13.4
trafilatura | 24.5 | 12.5
WET files | 20.7 | 12.2

####  Deduplication

Web-crawled datasets often contain many duplicate or near-duplicate data
strings. Removing these duplicates from a training set serves the dual purpose
of improving performance by reducing memorization [33, 88] and increasing data
diversity. For deduplication, we explore MinHash [28], as part of a suffix
array pipeline [88, 121], and near-duplicate Bloom filtering, which modifies
an exact document and paragraph deduplication scheme [150]. We find that both
approaches provide comparable downstream performance: within 0.2 Core
percentage points at the 7B-2x scale. However, our modified Bloom filter
approach scales more easily to datasets surpassing 10TB. We provide additional
analysis in Appendix K.

Takeaway: We use a Bloom filter for DCLM-baseline and MinHash for other
experiments.

####  Model-based quality filtering

Recent literature [55, 150, 27] indicates that using learnable models as
quality filters leads to downstream improvements. In this section, we
investigate model-based filtering.

Table 4: Quality filtering comparison (1B-1x scale). We evaluate various choices for model-based quality filters. Training a fastText classifier for filtering performs best. Filter | Core | Extended
---|---|---
RefinedWeb reproduction | 27.5 | 14.6
Top 20% by Pagerank | 26.1 | 12.9
SemDedup [1] | 27.1 | 13.8
Classifier on BGE features [176] | 27.2 | 14.0
AskLLM [139] | 28.6 | 14.3
Perplexity filtering | 29.0 | 15.0
Top-k average logits | 29.2 | 14.7
fastText [81] OH-2.5 +ELI5 | 30.2 | 15.4

###### Comparing model-based filtering approaches.

We compare many strategies: 1)  _PageRank score filtering_ to retain documents
based on how likely they are to be linked to other documents, 2)  _Semantic
Deduplication (SemDedup)_ to remove documents with similar informational
content [1], 3)  _linear classifiers_ fit on pre-trained BGE text embeddings
[176], 4)  _AskLLM_ that prompts an LM to see if a document is helpful [139],
5)  _Perplexity filtering_ where we retain low perplexity sequences following
CCNet [170], 6)  _Top-k average logits_ where we average the top-$k$ model
logits over all words in a document to score how confident a model is that the
correct words are within $k$ reasonable choices, and 7) fastText [81] binary
classifiers to distinguish data quality. For training classifiers, we train on
$\sim 400$k documents split equally between positive and negative classes. We
experiment with different options for positive data and fix negative data as a
sample from RefinedWeb. For the perplexity filtering and the top-k average
logits strategies, we utilize a 154M parameter causal Transformer trained on a
mix of English Wikipedia, the books subset of RedPajama v1, and peS2o [149,
160]. We compare the aforementioned approaches in Table 4 and find that
fastText-based filtering outperforms all other approaches. We next aim to
understand how fastText training recipes affect its effectiveness as a data
filtering network [55].

###### Text classifier ablations.

To better understand the limits of fastText, we train several variants,
exploring different choices for the reference data (i.e., the examples given
positive labels), feature space, and filtering threshold, as shown in Table 5.
For reference positive data, we considered commonly used sources like
Wikipedia [59], OpenWebText2 [59], and RedPajama-books [160], following the
reference data used for GPT-3 [30]. We also try a novel approach, using
instruction-formatted data, drawing examples from OpenHermes 2.5 [157]
(OH-2.5) and high-scoring posts from the r/ExplainLikeImFive (ELI5) subreddit.
Overall, we find, when controlling for other hyperparameters, the fastText
OH-2.5 +ELI5 approach gives a 3.5 percentage point lift on Core compared to
the conventional choices. It is natural to ask whether using OH-2.5 data for
filtering could preclude additional gains from instruction-tuning. In Appendix
P, we show this is not the case, further suggesting the strength and
compatibility of this approach with modern fine-tuning paradigms. Finally, we
observe that using a fairly strict threshold, which keeps the top-10% of
examples, helps over more permissive top-15% and top-20% thresholds. We
further study the unintuitive behavior of dataset filtering and its connection
to human judgment in Appendix M.

Table 5: fastText ablations (7B-1x scale). We ablate choices for the positive data (top) and threshold (bottom). ‘Dataset’ is positive set for fastText, while the negatives are randomly sampled from RefinedWeb. ‘Threshold’ is the percentile used for filtering based on fastText scores. “GPT-3 Approx” refers to a uniform mix of Wikipedia, OpenWebText2, and RPJ Books, as in [30]. Dataset | Threshold | Core | MMLU | Extended
---|---|---|---|---
OH-2.5 \+ ELI5 | 10% | 41.0 | 29.2 | 21.4
Wikipedia | 10% | 35.7 | 27.0 | 19.1
OpenWebText2 | 10% | 34.7 | 25.0 | 18.7
GPT-3 Approx | 10% | 37.5 | 24.4 | 20.0
OH-2.5 \+ ELI5 | 15% | 39.8 | 27.2 | 21.5
OH-2.5 \+ ELI5 | 20% | 38.7 | 24.2 | 20.3

Takeaway: For DCLM-baseline and the remaining experiments, we use fastText
OH-2.5 \+ ELI5 classifier score to keep the top 10% of documents. The result
of this filtering is DCLM-baseline.

####  Dataset mixing

Researchers often combine Common Crawl (CC) with other data sources that are
considered high-quality [160, 59, 65, 162] (e.g., Wikipedia, arXiv, Stack
exchange, and peS2o [149]). Since DCLM participants can include additional
data sources in the bring your own data track, we examined the potential
benefits of adding high-quality sources to training sets derived from Common
Crawl only. We compare a model trained on 100% filtered CC data to models
trained with the mixing proportion from Llama 1 and RedPajama: 67% CC, and 33%
from Wikipedia, Books, Stack exchange, arXiv, and Github. For the CC
component, we consider different variants: a subset of our DCLM-baseline,
RedPajama’s CC portion, RefinedWeb, and C4. The results in Table 6 show that
mixing improves performance for the lower-performing CC subsets (C4,
RedPajama-CC, and RefinedWeb). In the case of DCLM-baseline however, mixing
actually hurts performance on average, which suggests it can be
counterproductive given performant filtering. For additional mixing results,
see Appendix L.

Table 6: Mixing high-quality sources with subsets of CommonCrawl (1B-1x scale). We evaluate the impact of mixing high-quality source (‘RPJ extras’) to various datasets derived from CommonCrawl, using the mixing ratios from Llama and RedPajama. Numbers in parentheses indicate the improvement/degradation due to mixing, compared to using only the base dataset. | Core | Extended
---|---|---
Dataset | Base | w/ RPJ extras | Base | w/ RPJ extras
C4 | 23.7 | 25.9 ($+2.2$) | 12.5 | 13.3 ($+0.8$)
RPJ CC only | 24.0 | 25.7 ($+1.7$) | 12.1 | 13.5 ($+1.4$)
RefinedWeb | 25.1 | 26.5 ($+1.4$) | 12.9 | 13.1 ($+0.2$)
DCLM-baseline 111The DCLM-baseline subset used here relied on an earlier version of our classifier and our pool. | 31.1 | 29.5 ($-1.6$) | 16.0 | 15.3 ($-0.7$)

####  Decontamination

Here, we perform analysis to examine whether contamination of our pretraining
data with our evaluation is an issue that influences our results. We focus on
MMLU as our evaluation set of choice, given its popularity as a metric for
language model performance at this scale.

As an experiment, we also attempt to detect and remove questions from MMLU
that exist in DCLM-baseline. Our strategy is to flag training documents that
contain the last sentence of a question from MMLU along with one of the
corresponding options. For these flagged examples, we then remove all matched
question and option strings. In order to improve recall, we opt to detect only
the last sentence from each question, reducing the chance of missing questions
due to formatting differences. Based on inspection, this also incurs many
false positives. We then train a 7B-2x model with our DCLM-baseline without
the detected MMLU overlap.

The results of this analysis can be seen in Table 7. We see that this removal
of contaminated samples does not lead to a decrease in performance for our
model. As such, we can see that our performance gains in MMLU are not caused
by increased presence of MMLU in our dataset.

Table 7: MMLU overlap removal results. We remove overlaps detected with MMLU, in cases where a question and one of its options are detected in the text. We compare the performance between a model trained with and without this data removed, and see that there is no gain from increased contamination. This experiment is done at the 7B-2x scale. Dataset | MMLU
---|---
DCLM-baseline | 51.8
DCLM-baseline (MMLU removed) | 52.7

We also apply the above removal strategy on Dolma-V1.7 [150] and FineWeb-Edu
[100], to measure what the contamination differences are between DCLM-baseline
and those datasets. The results can be seen in Table 8. We see that our DCLM-
baseline has roughly similar contamination stats as other high performing
datasets according to this analysis. We provide further contamination analysis
that extends to our entire benchmark suite in Appendix N.

Table 8: MMLU overlap removal comparison. We remove overlaps detected with MMLU, in cases where a question and one of its options are detected in the text. For Dolma-V1.7 [150], we sample 1/10th of the dataset for this analysis (roughly 230B tokens). For FineWeb-Edu [100], we use the 10B token subset released by the authors. Note that because our flagging rule prioritizes recall over precision, these numbers are likely to be overestimates of the true contamination rates. Dataset | Percentage of samples flagged
---|---
DCLM-baseline | 0.007%
Dolma-V1.7 | 0.001%
FineWeb-Edu | 0.009%

###  Scaling up DCLM-baseline to the trillion token scale

Here, we test if datasets that perform well on the DCLM benchmark also
maintain their strength with an order of magnitude more compute. To ensure our
trained model is broadly useful, including for math and coding tasks, we
combine our 3.8T DCLM-baseline with the StarCoder [90] and ProofPile2 [14]
data to arrive at a 4.1T token dataset. We train a 7B model for 2.5T tokens on
this dataset with the same hyperparameters as our largest competition scale
except for two separate cool-downs phase for the 200B and 270B tokens on a
modified distribution that was 70% DCLM-baseline with a tighter fastText
threshold, and 30% math datasets (see Appendix P). We then take a “model soup"
of these two separate cool-downs[173]. We then adopt the continual pre-
training methodology from Pouransari et al. [126] for 100B tokens on the same
distribution to increase the context length from 2048 to 8192, we provide more
details on this procedure in Section P.2.

In Table 9, we show that our model outperforms all 7B models trained on public
training sets and approaches closed-data models trained for more tokens such
as Llama-8B, Mistral-7B, and Gemma-7B. Additionally, in Appendix O, we show
that our model achieves strong instruction-tuning performance. After
instruction tuning on publicly available IT datasets, our model maintains most
of its benchmark performance and achieves an AlpacaEval2.0 LC Win-rate of
16.6, which outperforms Gemma-Instruct (10.4), while approaching the strong
performance of Mistral-v0.2-7B (17.1) and Llama3-Instruct (22.9).

Table 9: State-of-the-art comparison (beyond 7B-2x scale). We compare our final model with others in the 7–8B parameter regime. Our DCLM-baseline dataset yields a model that outperforms models trained on open datasets and is competitive with models trained on private datasets. Model | Params | Tokens |  Open dataset? | Core | MMLU | Extended
---|---|---|---|---|---|---
Open weights, closed datasets
Llama2 | 7B | 2T | ✗ | 49.2 | 45.8 | 34.1
DeepSeek | 7B | 2T | ✗ | 50.7 | 48.5 | 35.3
Mistral-0.3 | 7B | ? | ✗ | 57.0 | 62.7 | 45.1
QWEN-2 | 7B | ? | ✗ | 57.5 | 71.9 | 50.5
Llama3 | 8B | 15T | ✗ | 57.6 | 66.2 | 46.3
Gemma | 8B | 6T | ✗ | 57.8 | 64.3 | 44.6
Phi-3 | 7B | ? | ✗ | 61.0 | 69.9 | 57.9
Open weights, open datasets
Falcon | 7B | 1T | ✓ | 44.1 | 27.4 | 25.1
OLMo-1.7 | 7B | 2.1T | ✓ | 47.0 | 54.0 | 34.2
MAP-Neo | 7B | 4.5T | ✓ | 50.2 | 57.1 | 40.4
Models we trained
FineWeb edu | 7B | 0.14T | ✓ | 38.7 | 26.3 | 22.1
FineWeb edu | 7B | 0.28T | ✓ | 41.9 | 37.3 | 24.5
DCLM-baseline | 7B | 0.14T | ✓ | 44.1 | 38.3 | 25.0
DCLM-baseline | 7B | 0.28T | ✓ | 48.9 | 50.8 | 31.8
DCLM-baseline | 7B | 2.6T | ✓ | 57.1 | 63.7 | 45.4

###  Conclusion & limitations

We introduced the DCLM testbed and demonstrated how it leads to new state-of-
the-art training sets. Our exploration of the dataset design space is only the
beginning and has clear limitations. Due to compute constraints, we could only
ablate design dimensions individually and could not test all approaches on
larger scales. Moreover, there are many variations of DCLM-baseline that we
did not explore. For instance, understanding the impact of sharded
deduplication in more detail is important, and there are many more ways of
training filtering models, both in terms of their architecture and training
data. We also conducted most of our experiments with only one tokenizer (GPT-
NeoX), and other tokenizers may perform better on multilingual tasks or math.
Another limitation is that we could not sufficiently explore the run-to-run
variation from different random seeds. Still, we hope that this paper is a
starting point for further research on data curation that pushes the state-of-
the-art beyond DCLM-baseline.

While our models trained on DCLM-baseline are competitive on common language
understanding evaluations, they currently do not perform as well on code and
math. We view this as a consequence of our focus on language understanding in
the first version of DCLM, and not an inherent limitation of our benchmark or
the DCLM-baseline training set. Indeed, prior work has shown that adding
specific training data and post training methods for code and math can
substantially improve performance on those domains [193, 188, 14, 90, 169].
Combining DCLM-baseline with these domain-specific training sets and extending
DCLM to cover code and math are intresting avenues for future work.

There are other important performance dimensions our evaluation suite
currently does not incorporate such as fairness, multilinguality, and safety.
Similarly, studying toxicity or privacy filtering in the context of DCLM would
be fruitful. Expanding DCLM along these dimensions is a fruitful direction for
future work, and we hope that our open and accessible testbed can strengthen
the foundations of data-centric research in these directions as well.

Lastly, we have only trained 7B parameter models as part of DCLM. In contrast,
state-of-the-art language models are now substantially larger. While we are
optimistic that our gains will also extend to larger model scales, future work
still needs to test this experimentally. One possible limitation of the
approach behind DCLM-baseline may be its stringent filtering ratio. After an
exact global deduplication at the document level, DCLM-baseline contains
approximately 2T tokens, and after removing all near-duplicates globally,
about 1T tokens remain. Understanding the interaction between data quality,
filtering ratio, deduplication, and multi-epoch training will be key for
building larger training sets in the future.

###### Acknowledgements.

We would like to thank Lilith Bat-Leah, Loubna Ben Allal, Samy Bengio, Mia
Chiquier, Adrien Gaidon, Lizzy Grant, Tom Gunter, Awni Hannun, Jonathan
Hayase, Mike Lewis, Percy Liang, Ian Magnusson, Yifan Mai, Sewon Min, David
Mizrahi, Praveen Paritosh, Guilherme Penedo, Kyle Richardson, Weijia Shi,
Karanjeet Singh, Joshua Susskind, Oyvind Tafjord, Carl Vondrick, and Elle
Wohlmuth, for helpful feedback at various stages of the project. We would like
to thank Mike Garrison and Romil Shah for help with compute and
infrastructure.

This research was supported by Allen Institute for AI, Open Philanthropy,
Institute for Foundations of Machine Learning (IFML), AFOSR MURI grant
FA9550-22-1-0380, Israeli Science Foundation (ISF) grant no. 2486/21, Alon
Fellowship, Adelis foundation, Israeli Council for Higher Education, Onassis
Foundation - Scholarship ID: F ZS 056-1/2022-2023, NSF Grants AF 1901292, CNS
2148141, Tripods CCF 1934932, IFML CCF 2019844,, research gifts by Western
Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the
Stanly P. Finch Centennial Professorship in Engineering, NSF Graduate Research
Fellowship. MN acknowledges funding by the Federal Ministry of Education and
Research of Germany under grant no. 01IS22094B WestAI - AI Service Center
West.

We gratefully acknowledge compute budget granted by Gauss Centre for
Supercomputing e.V. and by the John von Neumann Institute for Computing (NIC)
on the supercomputers JUWELS Booster and JURECA at Jülich Supercomputing
Centre (JSC)

### References

  * Abbas et al. [2023] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.  URL https://arxiv.org/abs/2303.09540.
  * Abdin et al. [2024] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al.  Phi-3 technical report: A highly capable language model locally on your phone.  _ArXiv preprint_ , abs/2404.14219, 2024.  URL https://arxiv.org/abs/2404.14219.
  * Agarwal et al. [2009] Amit Agarwal, Hema Swetha Koppula, Krishna P. Leela, Krishna Prasad Chitrapura, Sachin Garg, Pavan Kumar GM, Chittaranjan Haty, Anirban Roy, and Amit Sasturkar.  Url normalization for de-duplication of web pages.  In _ACM Conference on Information and Knowledge Management_ , 2009.  https://doi.org/10.1145/1645953.1646283.
  * AI [2024] Meta AI.  Introducing meta llama 3: The most capable openly available llm to date, 2024.  https://ai.meta.com/blog/meta-llama-3/.
  * Albalak et al. [2022] Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang.  FETA: A benchmark for few-sample task transfer in open-domain dialogue.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 10936–10953, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.751.
  * Albalak et al. [2023a] Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang.  Efficient online data mixing for language model pre-training.  _ArXiv preprint_ , abs/2312.02406, 2023a.  URL https://arxiv.org/abs/2312.02406.
  * Albalak et al. [2023b] Alon Albalak, Colin Raffel, and William Yang Wang.  Improving few-shot generalization by exploring and exploiting auxiliary data.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023b.  https://openreview.net/forum?id=JDnLXc4NOn.
  * Albalak et al. [2024] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al.  A survey on data selection for language models.  _ArXiv preprint_ , abs/2402.16827, 2024.  URL https://arxiv.org/abs/2402.16827.
  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  _ArXiv preprint_ , abs/2301.03988, 2023.  URL https://arxiv.org/abs/2301.03988.
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra-Aimée Cojocaru, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of open language models.  _arXiv preprint arXiv:2311.16867_ , 2023.
  * Amini et al. [2019] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.  MathQA: Towards interpretable math word problem solving with operation-based formalisms.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 2357–2367, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1245.  URL https://aclanthology.org/N19-1245.
  * Ankner et al. [2024] Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Mansheej Paul.  Perplexed by perplexity: Perplexity-based data pruning with small reference models.  _ArXiv preprint_ , abs/2405.20541, 2024.  URL https://arxiv.org/abs/2405.20541.
  * Ansel et al. [2024] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, David Berard, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Laurent Kirsch, Michael Lazos, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala.  Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation.  In _International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_ , 2024.  https://pytorch.org/blog/pytorch-2-paper-tutorial.
  * Azerbayev et al. [2023] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.  _ArXiv preprint_ , abs/2310.10631, 2023.  URL https://arxiv.org/abs/2310.10631.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _ArXiv preprint_ , abs/1607.06450, 2016.  URL https://arxiv.org/abs/1607.06450.
  * Bansal et al. [2024] Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover.  Comparing bad apples to good oranges: Aligning large language models via joint preference optimization.  _arXiv preprint arXiv:2404.00530_ , 2024.
  * Barbaresi [2021] Adrien Barbaresi.  Trafilatura: A web scraping library and command-line tool for text discovery and extraction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pp. 122–131, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-demo.15.  URL https://aclanthology.org/2021.acl-demo.15.
  * bench authors [2023] BIG bench authors.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  In _Transactions on Machine Learning Research (TMLR)_ , 2023.  https://openreview.net/forum?id=uyTL5Bvosj.
  * Bender et al. [2021] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , 2021.
  * Bevendorff et al. [2018] Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast.  Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl.  In _European Conference on Information Retrieval Research (ECIR)_ , 2018.  https://github.com/chatnoir-eu/chatnoir-resiliparse.
  * Bevendorff et al. [2021] Janek Bevendorff, Martin Potthast, and Benno Stein.  FastWARC: Optimizing Large-Scale Web Archive Analytics.  In _International Symposium on Open Search Technology (OSSYM)_ , 2021.  https://github.com/chatnoir-eu/chatnoir-resiliparse.
  * Bi et al. [2024] DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Yu Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang, Yu mei You, Shuiping Yu, Xin yuan Yu, Bo Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_ , 2024.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi.  PIQA: reasoning about physical commonsense in natural language.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pp. 7432–7439. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/6239.
  * Black et al. [2022] Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pp. 95–136, virtual+Dublin, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.bigscience-1.9.  URL https://aclanthology.org/2022.bigscience-1.9.
  * Bloom [1970a] Burton H. Bloom.  Space/time trade-offs in hash coding with allowable errors.  _Communications of the ACM_ , 1970a.  https://doi.org/10.1145/362686.362692.
  * Bloom [1970b] Burton H Bloom.  Space/time trade-offs in hash coding with allowable errors.  _Communications of the ACM_ , 13(7):422–426, 1970b.
  * Brandfonbrener et al. [2024] David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham M Kakade.  Color-filter: Conditional loss reduction filtering for targeted language model pre-training.  _arXiv preprint_ , 2024.
  * Broder [1997a] Andrei Z. Broder.  On the resemblance and containment of documents.  In _Compression and Complexity of Sequences_ , 1997a.
  * Broder [1997b] A.Z. Broder.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171)_ , pp. 21–29, 1997b.  doi: 10.1109/SEQUEN.1997.666900.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
  * Bugliarello et al. [2022] Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic.  IGLUE: A benchmark for transfer learning across modalities, tasks, and languages.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 2370–2392. PMLR, 2022.  URL https://proceedings.mlr.press/v162/bugliarello22a.html.
  * Calandriello et al. [2024] Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al.  Human alignment of large language models through online preference optimisation.  _arXiv preprint arXiv:2403.08635_ , 2024.
  * Carlini et al. [2023] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models, 2023.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _ArXiv preprint_ , abs/2107.03374, 2021.  URL https://arxiv.org/abs/2107.03374.
  * Chen et al. [2023] Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Ré.  Skill-it! a data-driven skills framework for understanding and training language models.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), _Advances in Neural Information Processing Systems_ , volume 36, pp. 36000–36040. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/70b8505ac79e3e131756f793cd80eb8d-Paper-Conference.pdf.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  In _Journal of Machine Learning Research (JMLR)_ , 2022.  https://arxiv.org/abs/2204.02311.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _ArXiv preprint_ , abs/2210.11416, 2022.  URL https://arxiv.org/abs/2210.11416.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 2924–2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1300.  URL https://aclanthology.org/N19-1300.
  * Clark et al. [2021] Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith.  All that’s ‘human’ is not gold: Evaluating human evaluation of generated text.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 7282–7296, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.565.  URL https://aclanthology.org/2021.acl-long.565.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _ArXiv preprint_ , abs/1803.05457, 2018.  URL https://arxiv.org/abs/1803.05457.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _ArXiv preprint_ , abs/2110.14168, 2021.  URL https://arxiv.org/abs/2110.14168.
  * Common Crawl [2007] Common Crawl.  Common Crawl, 2007.  https://commoncrawl.org.
  * Computer [2023] Together Computer.  Redpajama: an open dataset for training large language models, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Conneau & Lample [2019] Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pp. 7057–7067, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.html.
  * Cui et al. [2023] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2023.
  * Cui et al. [2022] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh.  DC-BENCH: Dataset condensation benchmark.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=Bs8iFQ7AM6.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In _International Conference on Machine Learning (ICML)_ , 2023.  https://proceedings.mlr.press/v202/dehghani23a.html.
  * Dodge et al. [2021] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 1286–1305, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.98.  URL https://aclanthology.org/2021.emnlp-main.98.
  * Du et al. [2022] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  Glam: Efficient scaling of language models with mixture-of-experts.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 5547–5569. PMLR, 2022.  URL https://proceedings.mlr.press/v162/du22c.html.
  * Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  _arXiv preprint arXiv:2404.04475_ , 2024.
  * Elazar et al. [2023] Yanai Elazar, Akshita Bhagia, Ian H. Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge.  What’s in my big data?  _ArXiv preprint_ , abs/2310.20707, 2023.  URL https://arxiv.org/abs/2310.20707.
  * Ethayarajh et al. [2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization.  _ArXiv preprint_ , abs/2402.01306, 2024.  URL https://arxiv.org/abs/2402.01306.
  * Face [2023] Hugging Face.  What’s going on with the open llm leaderboard?  https://huggingface.co/blog/open-llm-leaderboard-mmlu, 2023.
  * Fan et al. [2023] Simin Fan, Matteo Pagliardini, and Martin Jaggi.  Doge: Domain reweighting with generalization estimation.  _ArXiv preprint_ , abs/2310.15393, 2023.  URL https://arxiv.org/abs/2310.15393.
  * Fang et al. [2023] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar.  Data filtering networks.  _ArXiv preprint_ , abs/2309.17425, 2023.  URL https://arxiv.org/abs/2309.17425.
  * Fourrier et al. [2023] Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall.  Lighteval: A lightweight framework for llm evaluation, 2023.  URL https://github.com/huggingface/lighteval.
  * Gadre et al. [2024a] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 36, 2024a.  https://arxiv.org/abs/2304.14108.
  * Gadre et al. [2024b] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt.  Language models scale reliably with over-training and on downstream tasks.  _ArXiv preprint_ , abs/2403.08540, 2024b.  URL https://arxiv.org/abs/2403.08540.
  * Gao et al. [2021] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800gb dataset of diverse text for language modeling.  _ArXiv preprint_ , abs/2101.00027, 2021.  URL https://arxiv.org/abs/2101.00027.
  * Ge et al. [2024] Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding.  Data mixing made efficient: A bivariate scaling law for language model pretraining.  _ArXiv preprint_ , abs/2405.14908, 2024.  URL https://arxiv.org/abs/2405.14908.
  * Geva et al. [2019] Mor Geva, Yoav Goldberg, and Jonathan Berant.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 1161–1166, Hong Kong, China, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1107.  URL https://aclanthology.org/D19-1107.
  * Geva et al. [2021] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361, 2021.  doi: 10.1162/tacl_a_00370.  URL https://aclanthology.org/2021.tacl-1.21.
  * Gillick & Liu [2010] Dan Gillick and Yang Liu.  Non-expert evaluation of summarization systems is risky.  In _Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , pp. 148–151, Los Angeles, 2010. Association for Computational Linguistics.  URL https://aclanthology.org/W10-0722.
  * Glorioso et al. [2024] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge.  Zamba: A compact 7b ssm hybrid model.  _ArXiv preprint_ , abs/2405.16712, 2024.  URL https://arxiv.org/abs/2405.16712.
  * Gokaslan et al. [2019] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex.  Openwebtext corpus, 2019.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Grave et al. [2018] Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov.  Learning word vectors for 157 languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, 2018. European Language Resources Association (ELRA).  URL https://aclanthology.org/L18-1550.
  * Groeneveld [2023] Dirk Groeneveld.  The big friendly filter.  https://github.com/allenai/bff, 2023.
  * Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al.  Olmo: Accelerating the science of language models.  _ArXiv preprint_ , abs/2402.00838, 2024.  URL https://arxiv.org/abs/2402.00838.
  * Gunasekar et al. [2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need.  _Preprint_ , 2023.  https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need.
  * Gururangan et al. [2023] Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt.  OpenLM: a minimal but performative language modeling (lm) repository, 2023.  https://github.com/mlfoundations/open_lm.
  * Hendrycks et al. [2021a] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021a.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Hendrycks et al. [2021b] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021b.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://arxiv.org/abs/2203.15556.
  * Hu et al. [2024] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.
  * Ibrahim et al. [2024] Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish.  Simple and scalable strategies to continually pre-train large language models.  _arXiv preprint arXiv:2403.08763_ , 2024.
  * Ivison et al. [2023] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.  _arXiv preprint arXiv:2311.10702_ , 2023.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Florian Bressand Diego de las Casas, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b.  _ArXiv preprint_ , abs/2310.06825, 2023.  URL https://arxiv.org/abs/2310.06825.
  * Jin et al. [2019] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu.  PubMedQA: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2567–2577, Hong Kong, China, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1259.  URL https://aclanthology.org/D19-1259.
  * Johnson et al. [2017] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _ArXiv preprint_ , abs/1702.08734, 2017.  URL https://arxiv.org/abs/1702.08734.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1601–1611, Vancouver, Canada, 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1147.  URL https://aclanthology.org/P17-1147.
  * Joulin et al. [2017] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pp. 427–431, Valencia, Spain, 2017. Association for Computational Linguistics.  URL https://aclanthology.org/E17-2068.
  * Kaddour [2023] Jean Kaddour.  The minipile challenge for data-efficient language models.  _ArXiv preprint_ , abs/2304.08442, 2023.  URL https://arxiv.org/abs/2304.08442.
  * kaggle200000Jeopardy [2019] kaggle200000Jeopardy.  200,000+ Jeopardy! Questions — kaggle.com.  https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions, 2019.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _ArXiv preprint_ , abs/2001.08361, 2020.  URL https://arxiv.org/abs/2001.08361.
  * Kingma & Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Laurençon et al. [2022] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite.  The bigscience roots corpus: A 1.6tb composite multilingual dataset.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 31809–31826. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ce9e92e3de2372a4b93353eb7f3dc0bd-Paper-Datasets_and_Benchmarks.pdf.
  * Le Scao et al. [2022] Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy.  What language model to train if you have one million GPU hours?  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 765–782, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.findings-emnlp.54.
  * Lee et al. [2022] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8424–8445, Dublin, Ireland, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.577.  URL https://aclanthology.org/2022.acl-long.577.
  * Levesque et al. [2012] Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In _International conference on the principles of knowledge representation and reasoning_ , 2012.  https://aaai.org/papers/59-4492-the-winograd-schema-challenge.
  * Li et al. [2023a] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  _ArXiv preprint_ , abs/2305.06161, 2023a.  URL https://arxiv.org/abs/2305.06161.
  * Li et al. [2023b] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report.  _ArXiv preprint_ , abs/2309.05463, 2023b.  URL https://arxiv.org/abs/2309.05463.
  * Lin et al. [2024] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen.  Rho-1: Not all tokens are what you need.  _ArXiv preprint_ , abs/2404.07965, 2024.  URL https://arxiv.org/abs/2404.07965.
  * Ling et al. [2017] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 158–167, Vancouver, Canada, 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1015.  URL https://aclanthology.org/P17-1015.
  * Liu et al. [2020] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  In Christian Bessiere (ed.), _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pp. 3622–3628. ijcai.org, 2020.  doi: 10.24963/ijcai.2020/501.  URL https://doi.org/10.24963/ijcai.2020/501.
  * Liu et al. [2024a] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.  Lost in the middle: How language models use long contexts.  _Transactions of the Association for Computational Linguistics_ , 12:157–173, 2024a.
  * Liu et al. [2024b] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al.  Best practices and lessons learned on synthetic data for language models.  _ArXiv preprint_ , abs/2404.07503, 2024b.  URL https://arxiv.org/abs/2404.07503.
  * Liu et al. [2023] Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing.  Llm360: Towards fully transparent open-source llms, 2023.
  * Longpre et al. [2023a] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al.  The data provenance initiative: A large scale audit of dataset licensing & attribution in ai.  _ArXiv preprint_ , abs/2310.16787, 2023a.  URL https://arxiv.org/abs/2310.16787.
  * Longpre et al. [2023b] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito.  A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity.  _ArXiv preprint_ , abs/2305.13169, 2023b.  URL https://arxiv.org/abs/2305.13169.
  * Lozhkov et al. [2024a] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf.  Fineweb-edu, 2024a.  URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.
  * Lozhkov et al. [2024b] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder 2 and the stack v2: The next generation.  _ArXiv preprint_ , abs/2402.19173, 2024b.  URL https://arxiv.org/abs/2402.19173.
  * Luccioni & Viviano [2021] Alexandra Sasha Luccioni and Joseph D. Viviano.  What’s in the box? an analysis of undesirable content in the common crawl corpus.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2021.  URL https://api.semanticscholar.org/CorpusID:233864521.
  * Luukkonen et al. [2023] Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo.  FinGPT: Large generative models for a small language.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 2710–2726, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.164.  URL https://aclanthology.org/2023.emnlp-main.164.
  * Maini et al. [2024] Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly.  Rephrasing the web: A recipe for compute and data-efficient language modeling.  _ArXiv preprint_ , abs/2401.16380, 2024.  URL https://arxiv.org/abs/2401.16380.
  * Manber & Myers [1993] Udi Manber and Gene Myers.  Suffix arrays: a new method for on-line string searches.  _siam Journal on Computing_ , 22(5):935–948, 1993.
  * Mazumder et al. [2022] Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi.  Dataperf: Benchmarks for data-centric ai development.  _ArXiv preprint_ , abs/2207.10062, 2022.  URL https://arxiv.org/abs/2207.10062.
  * Meng et al. [2024] Yu Meng, Mengzhou Xia, and Danqi Chen.  Simpo: Simple preference optimization with a reference-free reward.  _arXiv preprint arXiv:2405.14734_ , 2024.
  * Mihaylov et al. [2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 2381–2391, Brussels, Belgium, 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1260.  URL https://aclanthology.org/D18-1260.
  * MosaicML [2023a] MosaicML.  Llm evaluation scores, 2023a.  https://www.mosaicml.com/llm-evaluation.
  * MosaicML [2023b] MosaicML.  llm-foundry/scripts/eval/local_data/EVAL_GAUNTLET.md at main · mosaicml/llm-foundry — github.com.  https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md, 2023b.
  * Muennighoff et al. [2022] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al.  Crosslingual generalization through multitask finetuning.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2022.  https://aclanthology.org/2023.acl-long.891.
  * Muennighoff et al. [2023a] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models.  _ArXiv preprint_ , abs/2308.07124, 2023a.  URL https://arxiv.org/abs/2308.07124.
  * Muennighoff et al. [2023b] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.  Scaling data-constrained language models.  In _Advances in Neural Information Processing Systems (NeuIPS)_ , 2023b.  https://arxiv.org/abs/2305.16264.
  * Muennighoff et al. [2024] Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela.  Generative representational instruction tuning.  _ArXiv preprint_ , abs/2402.09906, 2024.  URL https://arxiv.org/abs/2402.09906.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv preprint_ , abs/2303.08774, 2023.  URL https://arxiv.org/abs/2303.08774.
  * Paperno et al. [2016] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1525–1534, Berlin, Germany, 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1144.  URL https://aclanthology.org/P16-1144.
  * Parrish et al. [2022] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman.  BBQ: A hand-built bias benchmark for question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 2086–2105, Dublin, Ireland, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.165.  URL https://aclanthology.org/2022.findings-acl.165.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pp. 8024–8035, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
  * Patel et al. [2021] Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are NLP models really able to solve simple math word problems?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2080–2094, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.168.  URL https://aclanthology.org/2021.naacl-main.168.
  * PatronusAI [2023] PatronusAI.  Patronus AI | Patronus AI launches EnterprisePII, the industry’s first LLM dataset for detecting business-sensitive information — patronus.ai.  https://www.patronus.ai/announcements/patronus-ai-launches-enterprisepii-the-industrys-first-llm-dataset-for-detecting-business-sensitive-information, 2023.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _ArXiv preprint_ , abs/2306.01116, 2023.  URL https://arxiv.org/abs/2306.01116.
  * Penedo et al. [2024] Guilherme Penedo, Hynek Kydlíček, Leandro von Werra, and Thomas Wolf.  Fineweb, 2024.  URL https://huggingface.co/datasets/HuggingFaceFW/fineweb.  Software.
  * Peng et al. [2023a] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu.  RWKV: Reinventing RNNs for the transformer era.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2023a.  https://aclanthology.org/2023.findings-emnlp.936.
  * Peng et al. [2024] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr. au2, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu.  Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence.  _ArXiv preprint_ , abs/2404.05892, 2024.  URL https://arxiv.org/abs/2404.05892.
  * Peng et al. [2023b] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models.  _arXiv preprint arXiv:2309.00071_ , 2023b.
  * Pouransari et al. [2024] Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, and Oncel Tuzel.  Dataset decomposition: Faster llm training with variable sequence length curriculum.  _arXiv preprint arXiv:2405.13226_ , 2024.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _Preprint_ , 2019.  https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
  * Rae et al. [2021] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher.  _ArXiv preprint_ , abs/2112.11446, 2021.  URL https://arxiv.org/abs/2112.11446.
  * Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Raffel et al. [2020a] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67, 2020a.  URL http://jmlr.org/papers/v21/20-074.html.
  * Raffel et al. [2020b] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67, 2020b.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rajani et al. [2023] Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf.  No robots.  https://huggingface.co/datasets/HuggingFaceH4/no_robots, 2023.
  * Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 2383–2392, Austin, Texas, 2016. Association for Computational Linguistics.  doi: 10.18653/v1/D16-1264.  URL https://aclanthology.org/D16-1264.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, 2019.  doi: 10.1162/tacl_a_00266.  URL https://aclanthology.org/Q19-1016.
  * Rein et al. [2023] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.  Gpqa: A graduate-level google-proof q&a benchmark.  _ArXiv preprint_ , abs/2311.12022, 2023.  URL https://arxiv.org/abs/2311.12022.
  * Roemmele et al. [2011] Melissa Roemmele, Cosmin Adrian Bejan, , and Andrew S. Gordon.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium_ , 2011.  https://people.ict.usc.edu/~gordon/copa.html.
  * Romac et al. [2021] Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer.  Teachmyagent: a benchmark for automatic curriculum learning in deep RL.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 9052–9063. PMLR, 2021.  URL http://proceedings.mlr.press/v139/romac21a.html.
  * Rudinger et al. [2018] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 8–14, New Orleans, Louisiana, 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2002.  URL https://aclanthology.org/N18-2002.
  * Sachdeva et al. [2024] Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian J. McAuley, and Derek Zhiyuan Cheng.  How to train data-efficient llms.  _ArXiv preprint_ , abs/2402.09668, 2024.  URL https://arxiv.org/abs/2402.09668.
  * Sakaguchi et al. [2020] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pp. 8732–8740. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/6399.
  * Sap et al. [2019] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 4463–4473, Hong Kong, China, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1454.  URL https://aclanthology.org/D19-1454.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shazeer [2020] Noam Shazeer.  Glu variants improve transformer.  _ArXiv preprint_ , abs/2002.05202, 2020.  URL https://arxiv.org/abs/2002.05202.
  * Shi et al. [2023] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  _ArXiv preprint_ , abs/2310.16789, 2023.  URL https://arxiv.org/abs/2310.16789.
  * Shilov et al. [2024] Igor Shilov, Matthieu Meeus, and Yves-Alexandre de Montjoye.  Mosaic memory: Fuzzy duplication in copyright traps for large language models.  _ArXiv preprint_ , abs/2405.15523, 2024.  URL https://arxiv.org/abs/2405.15523.
  * Singh et al. [2024a] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel.  Beyond human data: Scaling self-training for problem-solving with language models.  _Transactions on Machine Learning Research_ , 2024a.  ISSN 2835-8856.  URL https://openreview.net/forum?id=lNAyUngGFK.  Expert Certification.
  * Singh et al. [2024b] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al.  Aya dataset: An open-access collection for multilingual instruction tuning.  _ArXiv preprint_ , abs/2402.06619, 2024b.  URL https://arxiv.org/abs/2402.06619.
  * Soboleva et al. [2023] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023.  URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.
  * Soldaini & Lo [2023] Luca Soldaini and Kyle Lo.  peS2o (Pretraining Efficiently on S2ORC) Dataset.  Technical report, Allen Institute for AI, 2023.  ODC-By, https://github.com/allenai/pes2o.
  * Soldaini et al. [2024] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al.  Dolma: An open corpus of three trillion tokens for language model pretraining research.  _ArXiv preprint_ , abs/2402.00159, 2024.  URL https://arxiv.org/abs/2402.00159.
  * Sorscher et al. [2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  https://openreview.net/forum?id=UmvSlP-PyV.
  * Su et al. [2021] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  _ArXiv preprint_ , abs/2104.09864, 2021.  URL https://arxiv.org/abs/2104.09864.
  * Talmor et al. [2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4149–4158, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1421.  URL https://aclanthology.org/N19-1421.
  * Team et al. [2024] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open models based on gemini research and technology.  _ArXiv preprint_ , abs/2403.08295, 2024.  URL https://arxiv.org/abs/2403.08295.
  * Team [2024] K2 Development Team.  Llm360 k2-65b: Scaling up fully transparent open-source llms.  Technical report, LLM360, 2024.  https://www.llm360.ai/paper2.pdf.
  * Team [2023] MosaicML NLP Team.  Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.  www.mosaicml.com/blog/mpt-7b.
  * Teknium [2023] Teknium.  Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023.  https://huggingface.co/datasets/teknium/OpenHermes-2.5.
  * Thudi & Maddison [2024] Anvith Thudi and Chris J. Maddison.  Finding optimally robust data mixtures via concave maximization.  _ArXiv preprint_ , abs/2406.01477, 2024.  URL https://arxiv.org/abs/2406.01477.
  * Tirumala et al. [2024] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos.  D4: Improving llm pretraining via document de-duplication and diversification.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Together Computer [2023] Together Computer.  Redpajama: an open dataset for training large language models, 2023.  https://github.com/togethercomputer/RedPajama-Data.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and Efficient Foundation Language Models.  _ArXiv preprint_ , abs/2302.13971, 2023a.  URL https://arxiv.org/abs/2302.13971.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _ArXiv preprint_ , abs/2307.09288, 2023b.  URL https://arxiv.org/abs/2307.09288.
  * Tunstall et al. [2023] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_ , 2023.
  * Üstün et al. [2024] Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al.  Aya model: An instruction finetuned open-access multilingual language model.  _ArXiv preprint_ , abs/2402.07827, 2024.  URL https://arxiv.org/abs/2402.07827.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 5998–6008, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Wang et al. [2022] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.  _arXiv preprint arXiv:2204.07705_ , 2022.
  * Warstadt et al. [2023] Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell.  Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora.  In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), _Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning_ , pp. 1–34, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.conll-babylm.1.  URL https://aclanthology.org/2023.conll-babylm.1.
  * Wei et al. [2022] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Wei et al. [2024] Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang.  Starcoder2-instruct: Fully transparent and permissive self-alignment for code generation.  https://huggingface.co/blog/sc2-instruct, 2024.
  * Wenzek et al. [2020] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pp. 4003–4012, Marseille, France, 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.494.
  * Wettig et al. [2024] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen.  Qurating: Selecting high-quality data for training language models.  _ArXiv preprint_ , abs/2402.09739, 2024.  URL https://arxiv.org/abs/2402.09739.
  * Workshop et al. [2022] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al.  Bloom: A 176b-parameter open-access multilingual language model.  _ArXiv preprint_ , abs/2211.05100, 2022.  URL https://arxiv.org/abs/2211.05100.
  * Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 23965–23998. PMLR, 2022.  URL https://proceedings.mlr.press/v162/wortsman22a.html.
  * Wortsman et al. [2023] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al.  Small-scale proxies for large-scale transformer training instabilities.  _ArXiv preprint_ , abs/2309.14322, 2023.  URL https://arxiv.org/abs/2309.14322.
  * Xia et al. [2024] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen.  Sheared LLaMA: Accelerating language model pre-training via structured pruning.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=09iOdaeOzp.
  * Xiao et al. [2023] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff.  C-pack: Packaged resources to advance general chinese embedding, 2023.
  * Xie et al. [2023a] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu.  Doremi: Optimizing data mixtures speeds up language model pretraining.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.  URL https://openreview.net/forum?id=lXuByUeHhd.
  * Xie et al. [2023b] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang.  Data selection for language models via importance resampling.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=uPSQv0leAu.
  * Xiong et al. [2023] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al.  Effective long-context scaling of foundation models.  _arXiv preprint arXiv:2309.16039_ , 2023.
  * Xue et al. [2021] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.
  * Yang et al. [2023a] Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica.  Rethinking benchmark and contamination for language models with rephrased samples.  _ArXiv preprint_ , abs/2311.04850, 2023a.  URL https://arxiv.org/abs/2311.04850.
  * Yang et al. [2023b] Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica.  Rethinking benchmark and contamination for language models with rephrased samples.  _ArXiv preprint_ , abs/2311.04850, 2023b.  URL https://arxiv.org/abs/2311.04850.
  * Yin et al. [2023] Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang.  Dynosaur: A dynamic growth paradigm for instruction-tuning data curation.  _arXiv preprint arXiv:2305.14327_ , 2023.
  * Yue et al. [2024] Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen.  Mammoth2: Scaling instructions from the web.  _arXiv preprint arXiv:2405.03548_ , 2024.
  * Zelikman et al. [2022] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.  Star: Bootstrapping reasoning with reasoning.  _ArXiv preprint_ , abs/2203.14465, 2022.  URL https://arxiv.org/abs/2203.14465.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  HellaSwag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4791–4800, Florence, Italy, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1472.  URL https://aclanthology.org/P19-1472.
  * Zhang et al. [2019] Biao Zhang, Ivan Titov, and Rico Sennrich.  Improving deep transformer with depth-scaled initialization and merged attention.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 898–909, Hong Kong, China, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1083.  URL https://aclanthology.org/D19-1083.
  * Zhang et al. [2024a] Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang.  Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024a.
  * Zhang et al. [2024b] Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen.  Map-neo: Highly capable and transparent bilingual large language model series.  _arXiv preprint arXiv:2405.19327_ , 2024b.
  * Zhao et al. [2024] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.  Wildchat: 1m chatGPT interaction logs in the wild.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=Bl8u7ZRlbM.
  * Zhao et al. [2023] Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li.  Pytorch fsdp: Experiences on scaling fully sharded data parallel.  In _Very Large Data Bases Conference (VLDB)_ , 2023.  https://dl.acm.org/doi/10.14778/3611540.3611569.
  * Zheng et al. [2024] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement.  _arXiv preprint arXiv:2402.14658_ , 2024.
  * Zhong et al. [2024] Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao.  Achieving> 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners.  _arXiv preprint arXiv:2404.14963_ , 2024.
  * Zhong et al. [2023] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.  Agieval: A human-centric benchmark for evaluating foundation models.  _ArXiv preprint_ , abs/2304.06364, 2023.  URL https://arxiv.org/abs/2304.06364.
  * Zhu et al. [2023] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.

## Appendix

###  Appendix A Contributions

All authors are listed alphabetically by last name.

Data processing

DCLM-baseline
Khyathi Chandu, Alex Fang, Saurabh Garg, Thao Nguyen, Vaishaal Shankar

Decontamination
Achal Dave, Saurabh Garg, Jeffrey Li, Vaishaal Shankar, Georgios Smyrnis

Deduplication
Achal Dave, Alex Fang, Jeffrey Li, Matt Jordan, Vaishaal Shankar

Extra data sources
Yonatan Bitton, Mayee Chen, Giannis Daras, Achal Dave, Alex Fang, Joshua
Gardner, Maciej Kilian, Jeffrey Li, Niklas Muennighoff, Marianna Nezhurina,
Vaishaal Shankar, Hanlin Zhang

Baseline datasets

Pre-processing existing datasets
Achal Dave, Alex Fang, Samir Gadre, Reinhard Heckel, Sedrick Keh, Marianna
Nezhurina, Vaishaal Shankar, Georgios Smyrnis

Reproductions
Amro Abbas, Hritik Bansal, Yonatan Bitton, Yair Carmon, Khyathi Chandu, Alex
Fang, Dhruba Ghosh, Cheng-Yu Hsieh, Maor Ivgi, Matt Jordan, Sedrick Keh,
Jeffrey Li, Kyle Lo, Luca Soldaini, Hanlin Zhang, Jieyu Zhang

Data curation

Filtering
Hritik Bansal, Alex Fang, Saurabh Garg, Maor Ivgi, Matt Jordan, Jeffrey Li,
Marianna Nezhurina, Vaishaal Shankar

Rewriting
Saurabh Garg, Maor Ivgi, Niklas Muennighoff

Mixing
Achal Dave, Jeffrey Li, Georgios Smyrnis

Evaluations

Evaluation framework
Alon Albalak, Kushal Arora, Hritik Bansal, Achal Dave, Maor Ivgi, Sedrick Keh,
Vaishaal Shankar, Rulin Shao, Rui Xin

Evaluation runs
Kushal Arora, Achal Dave, Alex Fang, Jeffrey Li, Sedrick Keh, Vaishaal Shankar

Evaluation metrics
Achal Dave, Alex Fang, Jeffrey Li, Vaishaal Shankar

Training

Training code
Achal Dave, Samir Gadre, Suchin Gururangan, Kalyani Marathe, Jean Mercat, Hadi
Pouransari, Sunny Sanyal, Georgios Smyrnis, Igor Vasiljevic, Mitchell Wortsman

Data preprocessing
Alex Fang, Matt Jordan, Vaishaal Shankar, Georgios Smyrnis

Training runs
Kushal Arora, Achal Dave, Alex Fang, Samir Gadre, Jenia Jitsev, Sedrick Keh,
Jeffrey Li, Marianna Nezhurina, Vaishaal Shankar, Georgios Smyrnis

Trillion-token models
Alex Fang, Jeffrey Li, Hadi Pouransari, Vaishaal Shankar

Instruction tuning
Kushal Arora, Hritik Bansal, Aaron Gokaslan, Etash Guha, Niklas Muennighoff

Competition setup

Competition design
Yair Carmon, Achal Dave, Alex Fang, Samir Gadre, Reinhard Heckel, Jeffrey Li,
Ludwig Schmidt, Vaishaal Shankar

Competition tooling
Gabriel Ilharco, Maor Ivgi, Jeffrey Li, Sarah Pratt

Paper writing
Alon Albalak, Hritik Bansal, Yair Carmon, Achal Dave, Alexandros G. Dimakis,
Alex Fang, Samir Gadre, Etash Guha, Reinhard Heckel, Maor Ivgi, Sedrick Keh,
Jeffrey Li, Niklas Muennighoff, Sarah Pratt, Ludwig Schmidt, Vaishaal Shankar,
Georgios Smyrnis

Leadership and advising

Advising
Alaa El-Nouby, Fartash Faghri, Dirk Groeneveld, Reinhard Heckel, Jenia Jitsev,
Sham Kakade, Pang Wei Koh, Thomas Kollar, Kyle Lo, Niklas Muennighoff, Sewoong
Oh, Sujay Sanghavi, Luca Soldaini, Shuran Song, Alexander Toshev, Stephanie
Wang, Luke Zettlemoyer

Leadership
Yair Carmon, Achal Dave, Alexandros G. Dimakis, Ludwig Schmidt, Vaishaal
Shankar

Project coordination
Achal Dave, Ludwig Schmidt, Vaishaal Shankar

###  Appendix B Additional related work

Data curation methods have been proposed that can be grouped into two
categories: methods that aim to enhance performance, and those with non-
performance related goals. Performance-oriented methods include language
detection, heuristics-based filtering, quality filtering, data deduplication,
data mixing, and synthetic data. Non-performance-oriented filters include the
removal of copyrighted text, toxic content, personally identifiable
information (PII), opt-out, and evaluation data.

###### Language detection.

Language detection methods most often rely on a fastText classifier that has
been trained to identify 157 languages [66, 121, 150], but past methods have
also utilized other classifiers including a naive Bayes classifier [131]. When
collecting multilingual datasets, another curation option is to filter web
pages based on country domains or by selecting URLs that are correlated with
data from certain languages [103, 147, 124].

###### Heuristics.

It is widely known that web-scraped text contains high quantities of
boilerplate HTML, error messages, stock tickers, and other undesirable data
for training language models, much of which can be detected and removed by
heuristic-based systems. The exact heuristics used by each data curation
method vary but can be grouped into five categories: item count, repetition
count, existence, ratio, and statistics. For example, Rae et al. [128] remove
any lines that contain at least two of the following stop words: _the, be, to,
of, and, that, have, with_ , defining an item count heuristic. An example of a
statistic might be removing documents that have a mean line length greater
than 100 characters [34].

###### Quality filtering.

Filtering for “high-quality” data (data which was written by humans and has
likely gone through an editing process [8]) is a common step for data curation
pipelines. The most commonly used method for quality filtering is to train a
binary classifier on data from a perceived high-quality dataset (e.g.
Wikipedia) and a perceived low-quality dataset (e.g. unfiltered web text) and
filter out data where the classifier assigns sufficiently low scores [59, 30,
49]. A less commonly used method is train a language model on the high-quality
dataset and calculate perplexity on the data to be filtered, where high
perplexity scores suggest that the data is lower quality [170, 113].

Recently, works have proposed the use of pretrained language models to
identify and curate high-quality data through prompting for various dimensions
of perceived quality [171, 139, 100]. Ankner et al. [12] even find that it’s
possible to use a small pretrained model (125M parameters) to prune training
data for models as large as 3B. MiniPile [82] demonstrated that a 1M document
subset of the pile, selected by clustering and removing low-quality clusters,
can lead to small LMs that maintain performance on GLUE, while significantly
reducing the scale of training data. RHO-1 [92] has a similar goal to quality
filtering, but rather than filtering data out of the dataset, they propose
Selective Language Modeling, an objective function that selectively masks the
loss of tokens that are predicted to be low quality.

###### Deduplication.

Deduplication has proven to be a beneficial step in almost every data curation
pipeline. The methods used vary in complexity, including deduplication based
on URLs, hashing, string metrics, and using model-based representations. URL
deduplication has long been in use for deduplicating web snapshots [3].
Commonly used hash-based deduplication methods include Bloom filters [25,
150], suffix array-based methods [88, 121], and MinHash-based methods [29]
such as MinHashLSH [30]. Model-based methods include SemDeDup [1] which embeds
each point in a dataset, clusters data points together, and removes data
points within clusters that are too similar, and D4 [159] which further
applies the SSL prototypes method from Sorscher et al. [151] and removes the
most prototypical example from each cluster.

###### Data mixing.

When the training dataset is composed of data from multiple domains or sources
(e.g. web text, Wikipedia, and books), then an additional challenge for data
curation is to determine what percent of the final dataset comes from each
source, known as data mixing. Methods for data mixing include using heuristics
(such as human judgment) [59, 161], or empirically determining the best domain
weights according to some downstream evaluation [49, 128]. More principled
approaches have been proposed that are based on Group DRO [177], multi-armed
bandits [7], and information theory [6]. Further methods have been proposed
building off of these principled approaches, including DoGE [54], Skill-it
[35], and ShearedLlama [175], each bringing some improvements. Thudi &
Maddison [158] develop MixMax, a provably optimal method under a concave
objective, which improves upon Group DRO-based alternatives but has not been
proven at scales typical of language modeling. Ge et al. [60] propose BiMix, a
unified scaling law that simultaneously models the behaviors of data quantity
and mixing weights, using only small models as proxies to calculate the
scaling laws.

###### Synthetic data.

With the improvements in the ability of language models to accurately model
text distributions, the generation of synthetic data has become an additional
avenue for data curation. Notable methods for pretraining include the Phi
models [69, 2], which generate synthetic textbook data from the GPT series of
models, as well as WRAP [104] which uses a similar method to the Phi models,
but demonstrates that the synthetic data generation pipeline is feasible with
much smaller models (1.8B and 7B parameters). Beyond generating synthetic data
for pretraining, Singh et al. [146] propose ReSTEM, a method for generating
synthetic data for math and coding benchmarks, which uses binary feedback (eg.
whether the code gives the correct output) to repeatedly filter self-generated
data. Similarly, Zelikman et al. [185] propose STaR, which bootstraps a
dataset of rationales for commonsense question answering and mathematics
datasets.

Non-performance related methods have been designed for a variety of purposes,
including to remove copyrighted content [98, 144, 145], toxic speech [131,
164], private information [9, 101], opt-out data [101, 112] or to
decontaminate data to avoid benchmark leakage [182]. While these methods are
less relevant to this work, they are nonetheless important in real-world data
curation pipelines.

###  Appendix C Benchmark rules

This section provides detailed guidelines for submissions in the two DCLM
tracks.

####  General rules

The following applies to both the filtering and mixing tracks.

  1. 1.

Submissions should include documentation detailing their key components.

  2. 2.

The dataset underlying a submission to the leaderboard, or fully working code
to reproduce it, should be freely available to encourage reproducibility of
submissions. Submissions that do not satisfy this requirements may still be
accepted, but we will mark them as such in the leaderboard.

  3. 3.

Tokenization must be performed with our provided script that tokenizes the
data and performs a global shuffle.

  4. 4.

Submissions cannot make any changes to the training or evaluation code.

  5. 5.

Use of evaluation data (test data from our evaluation tasks) for purposes
other than evaluation and decontamination is forbidden.

####  Filtering track

The defining characteristic of entries in the filtering track is that they
form the dataset by applying a processing pipeline on the subset of DCLM-Pool
corresponding to the chosen compute scale (see Table 1) without including any
external data. The rationale behind this requirement is twofold. First, the
size and quality of initial data for filtering affects both the processing
cost and the quality of the processed dataset. By fixing the initial dataset
we level the playing field and allow comparison to focus on core curation
techniques. Second, we wish to encourage the development of methods
potentially relevant even at frontier-model scale. Using the 7B-2x pool
(containing roughly 16T tokens) for the 400M-1x compute scale (requiring
roughly 8B tokens for training) would allow filtering strategies that keep
less than 0.1% of the data and cannot scale to generating a trillion-token
dataset.

As we wish to encourage creative and performant submissions, our requirement
for using only DCLM-Pool comes with the following qualifications:

  1. 1.

Modifying HTML extraction. We create DCLM-Pool by extracting text from Common
Crawl archives using resiliparse, which eases the computational burden on
participants who may not have resources to extract text themselves. However,
we additionally specify the Common Crawl archives for each pool to allow
experimentation with text extraction. Participants may either start with our
parsed DCLM-Pool data or work directly with the relevant Common Crawl WARC
archives.

  2. 2.

Using models trained on external data. We allow the DCLM-Pool processing
pipeline to leverage models for quality filtering, paraphrasing, etc. These
models may be trained on external data with the exception of evaluation data
as per the general guidelines. We will not accept submissions abusing this
allowance to introduce external data via a backdoor, e.g., by “paraphrasing”
documents from DCLM-Pool into memorized data.

####  Mixing track

In the mixing track, participants are free to use any data source, provided it
meets the general guidelines by being freely available and not including
evaluation data. Submissions to the mixing track should clearly document their
data sources, the weight given to each source, and the ratio of tokens used
for training (fixed for every benchmark scale) to the overall custom pool
size.

###  Appendix D Tooling

###### Download.

For the construction of our pool, we download WARC files from Common Crawl,
and process them via resiliparse, we do this by streaming data directly from
S3 to EC2 using the Ray data processing framework. This is the starting point
for our data processing pipeline. For the dataset released to participants, we
release various sizes of DCLM-Pool, that we make available for download. For
details on the data, see Appendix E.

###### Processing.

Given raw pool of text, it is often useful to define a processing pipeline to
clean, modify and filter it. We provide a robust framework to do that at
scale, by sharding the pool and processing it in parallel. Namely, to process
a pool one needs to define a sequence of _Mappers_ , each taking a single