document with its associated metadata as input, and output a list of
documents. Our mappers include:

  1. 1.

Filters which either retain or discard the input document according to some
filtering criteria such as having a maximum or minimum length.

  2. 2.

Enrichers which always return a list of with the page as is, adding additional
information to the metadata, such as detected language or number of tokens.

  3. 3.

Modifiers change the content of the text itself, and can also split the
document to create several new documents. This is useful for example, as a
participant may design a function to remove padding white-space.

In particular, we implement all mappers used in RefinedWeb (which includes
those from Gopher as a subset) and C4 along with many new ones, and allow
users to integrate custom mappers into their pipeline. Additionally, while
mappers allow for document-level processing, in some cases it may also be
necessary to execute corpora-level operations. For instance, a user may wish
to deduplicate spans that appear in several documents. Our tooling also
supports global functions that depend on all documents.

###### Contamination Analysis.

We use the tools provided by Lee et al. [88] as a base and adapt them to
evaluate the contamination of our training set with the evaluation sets. As
done in Touvron et al. [162], we measure the number of tokens that appear in
the same consecutive sequence of at least 10 tokens, between a training sample
and an evaluation sample. With this number, we calculate how many tokens on
average per evaluation sample are “contaminated”, appearing both in the
training and the evaluation data.

###### Tokenization and shuffling.

Once documents have been mapped, filtered, or globally processed, we provide
standardized code to tokenize and shuffle data. The output of this code is a
trainable dataset artifact. For tokenization, our code uses the GPT-NeoX [24]
tokenizer. Our tokenization code adopts Ray 222https://github.com/ray-
project/ray for cluster management and scales from a single node setups for
small datasets to multiple nodes for larger ones. After tokenizing, we perform
a global shuffle of our dataset.

###### Training Setup.

We base our training code on OpenLM [70], and provide configuration files for
each of our scales. We also provide scripts that train models using each
configuration, and produce json files that describe a trained model in detail.
For further training details, see Appendix F.

###### Evaluation.

We base our evaluation pipeline on the evaluation tasks provided by LLM-
foundry [110]. Using one of the aforementioned model json files as input, our
tools evaluate the associated checkpoint on all of our tasks. A new json file
is then produced, including the evaluation results in each task, as well as
aggregate metrics. This json file can then be submitted via a pull request to
submit the results to our leaderboard.

###### Reproducibility.

All of our results, including data processing, model training, evaluations,
and plots included in this paper, are reproducible using our open-source
framework and the recipes in https://datacomp.ai/dclm. We list compute
requirements for our code in Appendix Q.

###  Appendix E DCLM-Pool

DCLM-Pool was collected by taking all 5.1M Common Crawl WARC dumps from 2008
to 2022 (inclusive) and extracting text from the html using the resiliparse
framework. We opted to omit 2023 and above to prevent large amounts of
language model generated text from polluting our datasets and to provide a
hold out for future use. We release DCLM-Pool on HuggingFace with `CC-BY-4`
license. We release DCLM-Pool as a set of `.jsonl` files similar to Dolma-V1
and RedPajama. We provide the fields that are in the `.jsonl` in Table 10. The
entire pool is 5.1M gzip compressed `.jsonl\` files, and 340TB compressed on
disk. The use of this dataset is also subject to CommonCrawl’s Terms of Use:
https://commoncrawl.org/terms-of-use.

Common Crawl respects `robots.txt`, and thus our pool does so as well, giving
content creators a mechanism to opt out of Common Crawl and DCLM-Pool. Since
DCLM-Pool is a large subset of Common Crawl it will contain some PII data,
however Common Crawl does honor deletion requests and periodically redacts
dumps. We designed DCLM-Pool to maintain a one-to-one mapping between raw
Common Crawl WARC files and DCLM-Pool `.jsonl` files, allowing us to update
DCLM-Pool based on redactions.

We note that Common Crawl includes raw data as collected from the web without
filtering. While some of our pools, such as DCLM-baseline, underwent some
filtering of malicious URLs, none have had any special treatment for PII and
sensitive content to preserve representativeness of the raw data. For a more
complete discussion on PII and consent regarding our pools, see Appendix S.

Table 10: Metadata provided in DCLM-Pool data. Label | Additional notes
---|---
metadata.Content-Length | Length of the content.
metadata.Content-Type | Type of the content.
metadata.WARC-Block-Digest | Digest for data integrity.
metadata.WARC-Concurrent-To | Related WARC record.
metadata.WARC-Date | Date of the WARC record.
metadata.WARC-IP-Address | IP address of the source.
metadata.WARC-Identified-Payload-Type | Identified payload type.
metadata.WARC-Payload-Digest | Payload digest for integrity.
metadata.WARC-Record-ID | Unique ID of the WARC record.
metadata.WARC-Target-URI | Target URI of the record.
metadata.WARC-Type | Type of WARC record.
metadata.WARC-Warcinfo-ID | Related warcinfo record ID.
text | Text content.
url | URL of the source.
warcinfo | Information about the WARC file.

###  Appendix F Training details

###### Overview.

Our training setup follows closely that of Wortsman et al. [174] and Gadre et
al. [58]. Specifically, we build our training infrastructure using the OpenLM
[70], which supports decoder-only, pre-normalization Transformers [165],
following an architecture inspired by GPT-2 [127] and Llama [161]. OpenLM is a
PyTorch [118, 13] code-base that targets FSDP modules for distributed training
[191].

###### Architecture details.

We utilize LayerNorm [15] without bias parameters for all normalization, qk-
LayerNorm [47] on queries and keys for training stability, SwiGLU [143]
multilayer perceptrons (MLPs), and a depth-scaled initialization scheme
following Zhang et al. [187]. Our sequence length, during pre-training is
2048. We pack multiple sequences into batches to fill the entire context, with
an EOS token to split documents. We allow causal attention to attend across
documents; we experimented with masking attention across documents but early
experiments indicated little impact on downstream performance.

###### Training sets and tokenization.

Since the focus of our paper is dataset development, we train on over 270 data
distributions, mostly filtered from Common Crawl. For the majority of our
experiments we use GPT-NeoX [24] for tokenization, which yields a vocabulary
size of 50k.

###### Optimization details.

As metioned in the main body, we train with a standard next-token prediction
objective. Following Chowdhery et al. [36], we employ z-loss to encourage
output logit magnitudes to remain in a numerically stable range.

###### Hyperparameters.

We detail the hyperparameters for our models in Table 11. For the 400M-1x and
1B-1x, we follow hyperparameters from [58], which were tuned to optimize
perplexity on a validation set containing tokens from recent arXiv papers, the
OpenLM codebase itself, and news articles. For the 1B-1x scale, we also
investigated alternative hyperparameters in Table 13, and find the
hyperparameters from [58] perform best. For the 7B-1x and 7B-2x, we used a
higher learning rate, and a lower weight decay, guided by the hyperparameter
sweep in Table 12. We use a cooldown of 3$e$-5 for all experiments. For Table
2, we trained with a lower learning rate following [58] as these experiments
were performed before our sweep. Specifically, we used a learning rate of
3$e$-4 and weight decay of 0.33.

Table 11: Main models and hyperparameters used in our investigation. For each scale, we list the number of layers $n_{\text{layers}}$, number of attention heads $n_{\text{heads}}$, model width $d_{\text{model}}$, and width per attention head $d_{\text{head}}$. Batch sizes are global and in units of sequences. Each sequence has 2,048 tokens.  Scale | $n_{\text{layers}}$ | $n_{\text{heads}}$ | $d_{\text{model}}$ | $d_{\text{head}}$ | Warmup |  Learning rate |  Weight decay | z-loss |  Batch size
---|---|---|---|---|---|---|---|---|---
400M-1x | 24 | 8 | 1,024 | 128 | 2,000 | 3$e$-3 | 0.033 | 1$e$-4 | 512
1B-1x | 24 | 16 | 2,048 | 128 | 5,000 | 3$e$-3 | 0.033 | 1$e$-4 | 256
7B-1x, 7B-2x | 32 | 32 | 4,096 | 128 | 5,000 | 2$e$-3 | 0.05 | 5e-6 | 2,048
Table 12: Learning rate and weight decay sweep (7B-1x scale). We evaluated the impact of learning rate and weight decay on an earlier iteration of DCLM-baseline. Based on this sweep, we specify the settings for Table 11 for the 7B-1x and 7B-2x scales. LR | WD | Core
---|---|---
1$e$-03 | 0.1 | 44.1
2$e$-03 | 0.05 | 44.8
3$e$-03 | 0.033 | 44.7
1$e$-02 | 0.01 | 43.8

###  Appendix G Evaluation details

Below we outline the tasks we used to evaluate our models in LLM Foundry. We
also examine the LightEval [56] evaluation pipeline used in the FineWeb-Edu
[100] evaluations.

####  Evaluation Tasks

We divide our evaluations into two high-level categories: Core (22 tasks) and
Extended (53 tasks). The set of Core tasks were selected due to their ability
to provide a low variance signal of learning, even at small scales. We include
a diverse range of tasks aimed at assessing a variety of model capabilities.

######  Core tasks.

  * •

The AGI Eval LSAT-AR dataset [194] (3-shot) tests for model knowledge in the
legal domain and evaluates analytical reasoning capabilities.

  * •

The ARC easy and ARC challenge datasets [40] (10-shot) contain four-way
multiple choice questions taken from grade 3-9 science exams, where questions
in the easy dataset require knowledge of basic science, and the challenge
questions require some procedural reasoning.

  * •

We use a series of 6 datasets from Big-Bench [18] (all 10-shot): (1) QA
Wikidata which requires models to complete factual statements with the correct
answer, (2) Dyck languages where the model needs to complete a partially
balanced expression consisting of parentheses and braces, (3) Operators where
the model is given some newly defined operators and asked to compute the
output from some expression using those operators, (4) Repeat Copy Logic which
requires the model to differentiate instructions from text-to-copy and to
perform a sequence of operations, (5) CS Algorithms which requires the model
to execute algorithms such as recursion and dynamic programming, and (6)
Language Identification where the model is expected to identify the language
of a sequence of natural language text.

  * •

BoolQ [38] (10-shot) is a binary question answering dataset where the model is
expected to answer questions about relevant passages.

  * •

CommonsenseQA [153] (10-shot) is a 5-way multiple choice question answering
dataset which evaluates the models ability to understand and apply commonsense
knowledge on everyday scenarios.

  * •

COPA [136] (0-shot) consists of causal reasoning questions where the model is
given two possible outcomes to a scenario and must use commonsense to select
the outcome that is more likely.

  * •

CoQA [134] (0-shot) is a conversational question answering dataset where the
model is given a passage and conversation between two participants and then
expected to extract an answer from the passage to a question from one of the
participants.

  * •

HellaSwag [186] (0-shot and 10-shot) is a 4-way multiple choice commonsense
reasoning dataset, where the model is required to understand implicit context
and common knowledge in order to correctly select the continuation to a
context.

  * •

Jeopardy [83] (10-shot) is a dataset of questions posed in the format of the
“Jeopardy!” quiz show, covering a wide variety of topics.

  * •

LAMBADA [116] (0-shot) is a collection of narratives where a human is able to
guess the final word of the narrative, but is not able to if they are only
given the final sentence. To perform well on this task requires the model to
attend to context from the full narrative and cannot simply rely on the local
context.

  * •

OpenBookQA [108] (0-shot) is a 4-way multiple choice question answering
dataset that requires the model to use multi-step reasoning and commonsense
knowledge.

  * •

PIQA [23] (10-shot) is a binary multiple choice question answering dataset
that requires the model to use physical commonsense reasoning to answer
correctly.

  * •

SQuAD [133] (10-shot) is a question answering dataset where the model is given
a question and a passage containing the answer to that question.

  * •

The Winograd Schema Challenge [89] (0-shot) is binary multiple choice pronoun
resolution task where the model is given a context and asked to determine
which entity a pronoun refers to, requiring the model to exhibit commonsense
knowledge and contextual understanding.

  * •

The Winogrande [140] (0-shot) dataset extends the Winograd Schema Challenge
dataset by expanding the dataset to a wider variety of domains.

######  Extended tasks.

  * •

We use a series of 4 additional tasks from the AGI Eval suite of datasets
[194] (all 3-shot): (1) LSAT-LR and (2) LSAT-RC test for model knowledge in
the legal domain and evaluate logical reasoning and reading comprehension,
respectively, (3) SAT-En evaluates the model’s capabilities in English, and
(4) SAT-Math evaluates the model’s capability in math using chain-of-thought
prompting.

  * •

AQuA [93] (3-shot) is a 4-way multiple choice question answering dataset that
evaluates the model on algebra questions using chain-of-thought prompting.

  * •

BBQ [117] (3-shot) is a multiple choice question answering dataset designed to
detect model’s biases along nine social dimensions.

  * •

We use a series of 9 additional datasets from Big-Bench [18] (all 10-shot):
(1) Conceptual Combinations which evaluates the model’s capability to parse
conceptual combinations by selecting sentences where these combinations are
used correctly, (2) Conlang Translation where the model is expected to deduce
a new translation from English to an obscure constructed language based on a
limited number of translation examples, (3) Elementary Math QA which is a
multiple choice question answering dataset of simple quantitative reasoning
problems, (4) Logical Deduction which requires a model to parse, understand,
and apply information about objects and relationships between objects to infer
new information, (5) Misconceptions evaluates whether a model can discern
popular misconceptions from truth, (6) Novel Concepts measures the models
ability to creatively construct a necessary abstraction that is unlikely to
have existed in training data, (7) Strange Stories measures a model’s capacity
for Theory of Mind, (8) Strategy QA is a test that requires a model to answer
questions requiring multi-step implicit reasoning, (9) Understanding Fables
which evaluates the model’s capability to understand the moral of a short
story.

  * •

Enterprise PII classification [120] (10-shot) is a binary classification task
that evaluates whether a model can detect PII (e.g. usernames, emails) within
text.

  * •

GPQA-main and GPQA-diamond [135] (5-shot) are 4-way multiple choice question
answering datasets written by domain experts in biology, physics, and
chemistry, which are intended to be very difficult for non-experts to answer
(even with access to the web). The diamond set is a high-quality subset
including only questions where two experts answer correctly, but most non-
experts answer incorrectly.

  * •

GSM8K [41] (3-shot) is a dataset of grade school math word problems that
requires between 2 to 8 steps to solve, where the model uses chain-of-thought
prompting.

  * •

LogiQA [94] (10-shot) is a 4-way multiple choice question answering dataset
that evaluates logical reasoning.

  * •

Math QA [11] (10-shot) is a 5-way multiple choice question answering dataset
that evaluates math word problem solving capabilities, built on top of AQuA.

  * •

MMLU [72] (0-shot and 5-shot) is a 4-way multiple choice question answering
dataset that covers 57 different domains and tasks, evaluating both world
knowledge and problem solving capabilities.

  * •

PubMedQA [78] (10-shot) is a 3-way multiple choice question answering dataset
which evaluates the model’s ability to answer biomedical research questions
given context from a relevant research article.

  * •

Simple arithmetic with spaces and without spaces [110] (10-shot) are datasets
consisting of simple arithmetic problems with up to 3 operations using numbers
with up to 3 digits, evaluating a model’s ability to follow the correct order
of operations and perform arithmetic.

  * •

Social Interaction QA [141] (10-shot) is a binary multiple choice question
answering dataset that evaluates a model’s social commonsense intelligence.

  * •

SVAMP [119] (3-shot) is a set of challenging elementary-level math word
problems that uses chain-of-thought prompting.

  * •

Trivia QA [80] (3-shot) is an open-ended question answering dataset that
evaluates the world knowledge of a model.

  * •

The Winogender male and Winogender female datasets [138] (10-shot) are
variants of the winograd schemas method that creates a minimal pair of
sentences that differ only by the gender of one pronoun, designed to evaluate
a model’s gender bias.

  * •

HumanEval [34] (0-shot) is a code completion dataset composed of hand-written
problems where the model is given a function signature, and docstring and
expected to correctly produce a function that will pass several unit tests.

####  LightEval

Given the multiple ways of evaluating accuracy [53], we conducted a miniature
study using the LightEval evaluation framework [56]. Notably, under this
framework, we are able to achieve scores above random (25%) for 0-shot MMLU
for 1B models by considering the log-probabilities of entire answer passages
as opposed to single letters. The 1B Hugging Face model trained on FineWeb-Edu
[100] has shown to work well on this, so we wanted to more closely examine how
LightEval evaluation scores correlate with evaluation scores from LLM Foundry.
We present our findings in Figure 5.

Figure 5: Comparisons Between LightEval MMLU scores (x-axis) and LLM Foundry
MMLU scores (y-axis). LightEval is able to provide signal (i.e. score above
random baseline) earlier for weaker models, but the LightEval scores at larger
scales appear to be capped at a much lower threshold and are more closely
clumped together.

The key difference between LightEval and LLM Foundry for multiple choice tasks
like MMLU is that LightEval considers the log probabilities of entire answer
sequences, whereas LLM Foundry only considers log probabilities of single
letters. Nonetheless, Figure 5 shows a positive correlation between the two
evaluation frameworks on MMLU 0-shot accuracy.

In Figure 5 we were able to reproduce the MMLU scores reported in the FineWeb-
Edu blog [100]. Notably, we found that LightEval indeed gave MMLU scores above
random for the 1B scales, whereas in LLM Foundry, all the 1B models have
accuracies around 0.25. At larger scales, however, the LightEval scores for
the models become quite cramped together, which may make it more difficult to
compare models and may make the comparisons more susceptible to noise. For
example, the models Gemma-7B, Llama3-8B, and Mistral-7B all have scores
between 0.43 and 0.44 in LightEval, while their scores range from 0.56 to 0.62
for LLM Foundry. We also see that FineWeb-Edu 7B-2x and DCLM 7B-2x perform
quite similarly in LightEval, but DCLM-7B is better by close to 10 points in
LLM Foundry. In conclusion, we believe that LightEval can be potentially a
good choice when evaluating smaller models, but other frameworks like LLM
Foundry could give clearer signals when comparing larger models.

One limitation of this study is that we took MMLU as a representative task,
and we did not evaluate on other tasks. In the future, it would be interesting
to compare with additional tasks, as well as additional frameworks like
Eleuther LLM Harness.

###  Appendix H Hyperparameter study

A potential concern is that the training recipe can change conclusions about
which dataset is optimal for training, due to interaction between training
hyperparameters and dataset distributions. To address this confounder, we show
in Table 13 that orderings between datasets are preserved for various
combinations of weight decay and learning rate. Moreover we find that
performance gains from optimal hyper-parameter choice and dataset design tend
to be orthogonal and complement each other. We illustrate this effect in Table
14.

Table 13: Rankings are stable across hyperparameters (1B-1x scale). We train
models on 3 datasets with 5 hyperparameter settings, varying learning rate and
weight decay settings. Across the hyperparameter settings, the dataset ranking
remains largely stable, with DCLM-baseline outperforming RedPajama, which in
turns outperforms C4. With improved hyperparameters, the gaps between the
datasets grows: e.g., at ‘Default’ (the best hyperparameter setting), DCLM-
baseline outperforms RedPajama by 4.5 points and RedPajama outperforms C4 by 2
points, while at ‘0.1x Learning Rate’ (the lowest performing setting), the
gaps reduce to 3.3 points and 0.9 points respectively. Note: When changing
learning rate, we also update weight decay so the product of the two remains
the same.

Hyperparameters | Dataset |  Learning Rate (LR) |  Weight Decay (WD) | Core | Extended
---|---|---|---|---|---
0.1x Learning Rate | C4 | 3.0e-04 | 3.3e-01 | 22.1 | 11.6
RedPajama | 3.0e-04 | 3.3e-01 | 23.0 | 11.8
DCLM-baseline | 3.0e-04 | 3.3e-01 | 26.3 | 14.4
0.1x Weight Decay | C4 | 3.0e-03 | 3.3e-03 | 21.8 | 11.6
RedPajama | 3.0e-03 | 3.3e-03 | 23.0 | 11.8
DCLM-baseline | 3.0e-03 | 3.3e-03 | 28.3 | 15.1
Default | C4 | 3.0e-03 | 3.3e-02 | 23.7 | 12.5
RedPajama | 3.0e-03 | 3.3e-02 | 25.7 | 13.5
DCLM-baseline | 3.0e-03 | 3.3e-02 | 30.2 | 15.4
10x Weight Decay | C4 | 3.0e-03 | 3.3e-01 | 21.8 | 12.0
RedPajama | 3.0e-03 | 3.3e-01 | 22.5 | 11.9
DCLM-baseline | 3.0e-03 | 3.3e-01 | 27.1 | 14.0
10x Learning Rate | C4 | 3.0e-02 | 3.3e-03 | 22.7 | 12.3
RedPajama | 3.0e-02 | 3.3e-03 | 26.0 | 13.2
DCLM-baseline | 3.0e-02 | 3.3e-03 | 29.0 | 15.0

Table 14: Improvements from better hyperparameters stack with better datasets. (7B-1x scale). We evaluate the impact of the most influential step in our dataset design, model based filtering (‘fastText filtering’), stacked with a better hyperparameter setting. We see that for both MMLU and Core benchmarks, the two inteventions (better dataset and better hyperparmaeters) seem to be orthogonal and stack on top of each other. Hyperparameters |  fastText filtering | LR | WD | MMLU | Core
---|---|---|---|---|---
Low LR | ✗ | 3.0e-04 | 0.33 | 25.4 | 38.3
✓ | 3.0e-04 | 0.33 | 29.2 | 41.0
High LR | ✗ | 1.0e-03 | 0.1 | 25.5 | 39.7
✓ | 1.0e-03 | 0.1 | 38.3 | 44.1

###  Appendix I Model-based quality filters

We presented results for several different model-based quality filters in
Section 4.4. In this section, we describe their implementations in further
detail, focusing especially on fastText classifiers which were our method of
choice for DCLM-baseline.

####  fastText classifiers

###### Training

We use the supervised fastText package from Joulin et al. [81] to train models
to classify between chosen “high-quality” reference data which are given
positive labels, and web-crawled data which are given negative labels. We then
apply these classifiers to score each document from the pool we wish to
filter, taking the predicted probability of the positive label as the score
and computing a percentile-based threshold. In terms of training
hyperparameters, we mostly used the default choices from the fastText package;
the only hyperparameter change that we tried was to expand the feature space
from unigrams only to both unigrams and bigrams (via setting the wordNgrams
argument to be 2 instead of the default 1). This helped improve the quality of
downstream filtered datasets, as shown in Table 15 (which extends Table 5 from
Section 4.4).

Table 15: fastText feature-space ablation (7B-1x scale). Adding bigrams to the feature space helps over the default setting of unigrams only.  Dataset | Threshold | Features | Core | MMLU | Extended
---|---|---|---|---|---
OH-2.5 \+ ELI5 | 10% | Unigrams + Bigrams | 41.0 | 29.2 | 21.4
OH-2.5 \+ ELI5 | 10% | Unigrams | 40.0 | 28.3 | 22.1

###### Data preparation.

The bulk of our experimentation for training fastText models focused on
constructing their underlying training sets, specifically the positively
labeled reference data. For each experiment, we fixed the size of the training
set to be 400K examples (i.e., 200K positive, 200K negative). The negatively
labeled examples were sampled randomly from a set of documents that came from
an earlier (smaller) version of our RefinedWeb reproduction. This version used
trafilatura as the extractor instead of resiliparse, which we hypothesize
might actually help for training the filtering model; as shown in Appendix J,
trafilatura more aggressively removes boilerplate content that may appear in
many pages (especiall from the same website). This type of content, if left
in, may lead to the fastText models over-relying on these “spurious” features
instead of the main contents of the page. For the positively labeled reference
data, we tried several different sources, some of which involved further pre-
processing:

  * •

Wikipedia. We use the processed version from RedPajama [160] and apply English
filtering by only keeping pages from the en.wikipedia.org domain. To encourage
the classifier to rely on the core content of each page, we remove occurrences
of the section titles "See Also" and "References", at least one of which
occurs in 90% of articles.

  * •

OpenWebText2. We use this dataset as is, taken from the version in The Pile
[59].

  * •

GPT-3 Approx. We mix together Wikipedia and OpenWebText2 along with the books
source from RedPajama [160]. Given the long length of individual books, we
instead define examples by extracting chunks of text that are at most 2048
tokens long.

  * •

OH-2.5 \+ ELI5. Our goal for this mix was to source instruction and question-
answer formatted data that is both high-quality and covers a wide range of
potential topics. We sample 100K examples from OH-2.5, which we do not further
pre-process. For ELI5, each raw page from the r/ExplainLikeImFive subreddit
contains a post asking a specific question and then some number of comments
aiming to answer said question. We curate examples for training fastText
models by taking a post and combining it with the top-scoring answer (using
the karma score derived from community up/down-votes). If there are ties, the
longest answer is chosen. We also filter these examples by keeping only those
where the post has score $\geq$ 3, the best comment has score $\geq$ 5, and
there are at least 3 comments total.

####  Other quality filtering baselines

We also examined other quality filters, though found none as effective as the
fastText methods described above, as shown in Table 4. We now provide further
details for some of these baselines.

###### PageRank.

An intuitively promising, but ultimately unfruitful approach was to consider
page centrality metrics such as PageRank and Harmonic centrality metrics, with
the idea that more "central" web text would yield higher quality data. We
collected PageRank metrics from Common Crawl’s host level webgraph
dataset333https://commoncrawl.org/web-graphs and omitted any hosts that did
not appear in the crawl. Next we partitioned our RefinedWeb reproduction into
quintiles based on their PageRank score and trained several models at the
1B-1x scale. These results are collated in Table 16, but unfortunately no
quintile performed better than a pool sampled from the union of all quintiles.

Table 16: PageRank-based filtering (1B-1x scale). Using PageRank score to select data is not helpful for improving upon our RefinedWeb reproduction. Using any quintile based on this score performs worse than a random sample from the same initial pool.  Quintile | All | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
Core | 27.8 | 26.1 | 27.3 | 26.6 | 26.3 | 27.1

###### AskLLM.

A recent line of work studies using instruction-tuned models as annotators to
determine the potential usefulness of a document. Sachdeva et al. [139]
proposed AskLLM, in which the authors prompted Flan-T5 models [37] to evaluate
whether the given document _“ $\dots$ contain[s] informative signal for pre-
training a large-language model? An informative data point should be well-
formatted, contain some usable knowledge of the world, and strictly NOT have
any harmful, racist, sexist, etc. content.”_. We implemented this method
ourselves, testing several models as annotators, different settings for
maximal sequence length, and several prompts on a small scale. We found that
the best configuration was using Mistral-7B-Instruct-v0.2 [77], clipping the
document at 1024 tokens, and taking the cumulative probabilities of _Yes_ and
_yes_ tokens as the model score. We used the following prompt template:

Evaluate the following paragraph for LLM pre-training suitability: \- Is it
well-structured or contains useful examples to natural texts? \- Does it offer
insights, useful facts or relevant information? \- Does it teach how to comply
with open-ended tasks such as writing letters, poems, emails etc.? \- Is it
free from harmful content? If most criteria are met based on the content
below, indicate ’yes’ for suitable. Otherwise, indicate ’no’ for unsuitable.
### <input> ### Is it suitable for LLM pre-training? OPTIONS: \- yes \- no

where <input> is replaced with the document tokens, clipped if too long, and
appended with ‘‘…[The rest of the paragraph is omitted]’’ in such cases. While
this method worked slightly better than random sampling from our pool (see
Table 4), it significantly underperformed compared to our fastText
experiments. Considering the high costs associated with applying it at scale,
we did not perform this experiment on a larger scale.

###### Semantic deduplication.

Following the success of the different deduplication methods we used (Appendix
K), we studied the effect of Semantic deduplication as proposed by Abbas et
al. [1]. In this approach, the authors propose embedding the documents using
pre-trained language models, clustering them using k-means, and removing all
but one document from each group of closely related documents to encourage
diversity in the dataset. We began by embedding each document in a pool of
approximately 100 million documents (following Abbas et al. [1]’s best
practices) with BGE-base [176]. We then used faiss-GPU [79] to perform
spherical k-means clustering, with 20 iterations and $K=11000$. We sampled
documents after discarding 25% of the data. As seen in Table 4, this
intervention only negatively impacted the trained model. We hypothesize that
the model used for embedding has a significant impact on the outcomes of this
method. However, due to the large computational overhead when scaled, making
it infeasible, we opted to rely on the deduplication methods outlined in
Appendix K and leave this line of research for future work.

###  Appendix J Text extraction comparison

Here, we share more detailed quantitative and qualitative comparisons between
our chosen extractor, resiliparse [20], and the two alternatives previously
used by other datasets: WET files, trafilatura [17].

####  Profiling

We compute basic summary statistics for each extractor based on a sample of 10
WARC files (corresponding to 900K individual pages), presenting the results in
Table 17. Notably, both resiliparse and trafilatura result at least 2x shorter
documents on average compared to WET files. As shown in the examples in
Section J.2, WET files indeed contain many additional lines with seemingly
little value for pre-training (e.g. navigation bars, boilerplate notices,
copyright statements). trafilatura and resiliparse trim most of these lines
out, with the former being more strict about doing so. Between the two,
resiliparse still keeps in about 10% more text; some of this additional text
may provide useful content such as section titles and dates for articles. In
terms of runtime, the two are much farther apart, with resiliparse being
roughly 8x faster.

Table 17: Text extractor profiling. Characters and tokens are averaged over the number of resulting output pages (note that this may differ for each extractor due to due to the possibility of extraction failures). Throughput is measured in GBs of input WARCs processed per second for each CPU core.  Extractor | Avg. Chars | Avg. Tokens | Throughput (GB / sec / core)
---|---|---|---
WET | 6,580 | 2,824 | –
resiliparse | 3,227 | 1,329 | $4.55\times 10^{-3}$
trafilatura | 2,901 | 1,179 | $0.56\times 10^{-3}$

####  Extraction examples

#####  J.2.1 Example set 1

[Trafilatura] HERE is a sampling of some of the better antiques and flea
markets around the United States. Two or Three Times a Year BRIMFIELD Route
20, Brimfield, Mass. 01010; 413-245-3436. Second weekend of May and July, and
the second weekend after Labor Day. RENNINGER’S OUTDOOR EXTRAVAGANZA Noble
Street, Kutztown, Pa.; 717-385-0104. Thursday, Friday and Saturday of the last
weekend of April, June, September. FARMINGTON ANTIQUES WEEKEND Farmington Polo
Grounds, Town Farm Road, Farmington, Conn. 06032; 508-839-9735. Starting
Wednesday before shows open; 203-677-7862. June 9-10 and Sept. 1-2. Monthly
ANN ARBOR ANTIQUES MARKET, P.O. Box 1512, Ann Arbor, Mich. 48106;
313-662-9453. May through October, third Sunday. Continue reading the main
storyKANE COUNTY FLEA MARKET, Kane County Fairgrounds, P.O. Box 549, St.
Charles, Ill. 60174; 708-377-2252. Year-round, first weekend. THE METROLINA
EXPO, 7100 Statesville Road, Charlotte, N.C. 28213; 704-596-4643. Year-round,
first weekend of every month. SPRINGFIELD ANTIQUE SHOW AND FLEA MARKET, Clark
County Fairgrounds, Route 41, Springfield, Ohio, 45501; 513-325-0053. Year-
round, third weekend. Weekly BAKERSFIELD SWAP-O-RAMA, 4501 Wible Road,
Bakersfield, Calif. 93313; 805-831-9342. Saturday and Sunday. LAMBERTVILLE
ANTIQUE MARKET, Route 29, Lambertville, N.J. 08530. Weekend number:
609-397-0456. Weekday: 215-752-4485, between 5 and 7 P.M. Market on Saturday
and Sunday. ATLANTA FLEA MARKET AND ANTIQUE CENTER, 5360 Peachtree Industrial
Boulevard, Chamblee, Ga. 30341; 404-458-0456. Friday, Saturday and Sunday.
Continue reading the main story

[Resiliparse] This is a digitized version of an article from The Times’s print
archive, before the start of online publication in 1996. To preserve these
articles as they originally appeared, The Times does not alter, edit or update
them. Occasionally the digitization process introduces transcription errors or
other problems. Please send reports of such problems to
<EMAIL_ADDRESS>May 10, 1990, Page 00006 The New York Times
Archives HERE is a sampling of some of the better antiques and flea markets
around the United States. Two or Three Times a Year BRIMFIELD Route 20,
Brimfield, Mass. 01010; 413-245-3436. Second weekend of May and July, and the
second weekend after Labor Day. RENNINGER’S OUTDOOR EXTRAVAGANZA Noble Street,
Kutztown, Pa.; 717-385-0104. Thursday, Friday and Saturday of the last weekend
of April, June, September. FARMINGTON ANTIQUES WEEKEND Farmington Polo
Grounds, Town Farm Road, Farmington, Conn. 06032; 508-839-9735. Starting
Wednesday before shows open; 203-677-7862. June 9-10 and Sept. 1-2. Monthly
ANN ARBOR ANTIQUES MARKET, P.O. Box 1512, Ann Arbor, Mich. 48106;
313-662-9453. May through October, third Sunday. Continue reading the main
story KANE COUNTY FLEA MARKET, Kane County Fairgrounds, P.O. Box 549, St.
Charles, Ill. 60174; 708-377-2252. Year-round, first weekend. THE METROLINA
EXPO, 7100 Statesville Road, Charlotte, N.C. 28213; 704-596-4643. Year-round,
first weekend of every month. SPRINGFIELD ANTIQUE SHOW AND FLEA MARKET, Clark
County Fairgrounds, Route 41, Springfield, Ohio, 45501; 513-325-0053. Year-
round, third weekend. Weekly BAKERSFIELD SWAP-O-RAMA, 4501 Wible Road,
Bakersfield, Calif. 93313; 805-831-9342. Saturday and Sunday. LAMBERTVILLE
ANTIQUE MARKET, Route 29, Lambertville, N.J. 08530. Weekend number:
609-397-0456. Weekday: 215-752-4485, between 5 and 7 P.M. Market on Saturday
and Sunday. ATLANTA FLEA MARKET AND ANTIQUE CENTER, 5360 Peachtree Industrial
Boulevard, Chamblee, Ga. 30341; 404-458-0456. Friday, Saturday and Sunday.
Continue reading the main story

[WET file] A Guide To Markets - The New York Times NYTimes.com no longer supports Internet Explorer 9 or earlier. Please upgrade your browser. LEARN MORE » Sections Home Search Skip to content Skip to navigation View mobile version The New York Times Archives|A Guide To Markets Search Subscribe Now Log In 0 Settings Close search Site Search Navigation Search NYTimes.com Clear this text input Go https://nyti.ms/29nVV3Q Loading... See next articles See previous articles Site Navigation Site Mobile Navigation Advertisement Archives | 1990 A Guide To Markets MAY 10, 1990 Continue reading the main story Share This Page Continue reading the main story About the Archive This is a digitized version of an article from The Times’s print archive, before the start of online publication in 1996. To preserve these articles as they originally appeared, The Times does not alter, edit or update them. Occasionally the digitization process introduces transcription errors or other problems. Please send reports of such problems to<EMAIL_ADDRESS>May 10, 1990, Page 00006 The New York Times Archives HERE is a sampling of some of the better antiques and flea markets around the United States. Two or Three Times a Year BRIMFIELD Route 20, Brimfield, Mass. 01010; 413-245-3436. Second weekend of May and July, and the second weekend after Labor Day. RENNINGER’S OUTDOOR EXTRAVAGANZA Noble Street, Kutztown, Pa.; 717-385-0104. Thursday, Friday and Saturday of the last weekend of April, June, September. FARMINGTON ANTIQUES WEEKEND Farmington Polo Grounds, Town Farm Road, Farmington, Conn. 06032; 508-839-9735. Starting Wednesday before shows open; 203-677-7862. June 9-10 and Sept. 1-2. Advertisement Continue reading the main story Monthly ANN ARBOR ANTIQUES MARKET, P.O. Box 1512, Ann Arbor, Mich. 48106; 313-662-9453. May through October, third Sunday. Continue reading the main story Advertisement Continue reading the main story KANE COUNTY FLEA MARKET, Kane County Fairgrounds, P.O. Box 549, St. Charles, Ill. 60174; 708-377-2252. Year-round, first weekend. THE METROLINA EXPO, 7100 Statesville Road, Charlotte, N.C. 28213; 704-596-4643. Year-round, first weekend of every month. SPRINGFIELD ANTIQUE SHOW AND FLEA MARKET, Clark County Fairgrounds, Route 41, Springfield, Ohio, 45501; 513-325-0053. Year-round, third weekend. Weekly BAKERSFIELD SWAP-O-RAMA, 4501 Wible Road, Bakersfield, Calif. 93313; 805-831-9342. Saturday and Sunday. LAMBERTVILLE ANTIQUE MARKET, Route 29, Lambertville, N.J. 08530. Weekend number: 609-397-0456. Weekday: 215-752-4485, between 5 and 7 P.M. Market on Saturday and Sunday. ATLANTA FLEA MARKET AND ANTIQUE CENTER, 5360 Peachtree Industrial Boulevard, Chamblee, Ga. 30341; 404-458-0456. Friday, Saturday and Sunday. A version of this list appears in print on May 10, 1990, on Page C00006 of the National edition with the headline: A Guide To Markets. Order Reprints| Today’s Paper|Subscribe Continue reading the main story What’s Next Loading... Go to Home Page » Site Index The New York Times Site Index Navigation News World U.S. Politics N.Y. Business Tech Science Health Sports Education Obituaries Today’s Paper Corrections Opinion Today’s Opinion Op-Ed Columnists Editorials Op-Ed Contributors Letters Sunday Review Video: Opinion Arts Today’s Arts Art & Design Books Dance Movies Music N.Y.C. Events Guide Television Theater Video: Arts Living Automobiles Crossword Food Education Fashion & Style Health Jobs Magazine N.Y.C. Events Guide Real Estate T Magazine Travel Weddings & Celebrations Listings & More Reader Center Classifieds Tools & Services N.Y.C. Events Guide Multimedia Photography Video NYT Store Times Journeys Subscribe Manage My Account NYTCo Subscribe Subscribe Home Delivery Digital Subscriptions Crossword Email Newsletters Gift Subscriptions Group Subscriptions Education Rate Mobile Applications Replica Edition Site Information Navigation © 2019 The New York Times Company Home Search Accessibility concerns? Email us at<EMAIL_ADDRESS>We would love to hear from you. Contact Us Work With Us Advertise Your Ad Choices Privacy Terms of Service Terms of Sale Site Information Navigation Site Map Help Site Feedback Subscriptions

#####  J.2.2 Example set 2

[Trafilatura] Possible Duplicate: When should I use an em-dash, an en-dash,
and a hyphen? When do I put a - in a sentence? Is it a more powerful comma?
With a bigger pause? Possible Duplicate: When should I use an em-dash, an en-
dash, and a hyphen? When do I put a - in a sentence? Is it a more powerful
comma? With a bigger pause? This question has been asked before and already
has an answer. If those answers do not fully address your question, please ask
a new question. The dashes you described are known respectively as the en-dash
and the em-dash. To describe the difference between their origins, Mental
Floss writes: An en dash (–) is bigger than a hyphen but shorter than an em
dash (—). Th e names come from an obscure typographical measurement system,
but the dashes have now taken on a life of their own in grammar. The em dash
is the spork of English grammar: It ain’t particularly pretty, but you can use
it for most anything. Em dashes can replace colons or sets of parentheses, or
represent a sudden change in thought or tone. So when do you use an en-dash?
Again from Mental Floss: To show numerical ranges, signifying “up to and
including”—of dates, ages, pages, etc. (Example: “I read pages 7–22 last
night.”) The storied “compound adjective hyphen,” an event so rare in the
English language that proofreaders shiver with excitement whenever they come
across it. Basically “pro-American” gets a regular hyphen because “American”
is only one word, whereas “pro–Falkland Islands” gets an en dash because
“Falkland Islands” is two words. So, too phrases like “Civil War–era.” What
about an em-dash? From here: Similar to an extended hyphen (-), an em dash is
used to show a break in thought or a shift of tone. If you’d like to read more
about the differences between a hyphen (-), en-dash (–), and em-dash (—), see
the blog post here which summarizes the above.

[Resiliparse] 1 Possible Duplicate: When should I use an em-dash, an en-dash,
and a hyphen? When do I put a - in a sentence? Is it a more powerful comma?
With a bigger pause? marked as duplicate by waiwai933, MrHen, user2683,
Robusto, Thursagen Jul 13 ’11 at 0:32 This question has been asked before and
already has an answer. If those answers do not fully address your question,
please ask a new question. 0 The dashes you described are known respectively
as the en-dash and the em-dash. To describe the difference between their
origins, Mental Floss writes: An en dash (–) is bigger than a hyphen but
shorter than an em dash (—). Th e names come from an obscure typographical
measurement system, but the dashes have now taken on a life of their own in
grammar. The em dash is the spork of English grammar: It ain’t particularly
pretty, but you can use it for most anything. Em dashes can replace colons or
sets of parentheses, or represent a sudden change in thought or tone. So when
do you use an en-dash? Again from Mental Floss: 1\. To show numerical ranges,
signifying “up to and including”—of dates, ages, pages, etc. (Example: “I read
pages 7–22 last night.”) 2\. The storied “compound adjective hyphen,” an event
so rare in the English language that proofreaders shiver with excitement
whenever they come across it. Basically “pro-American” gets a regular hyphen
because “American” is only one word, whereas “pro–Falkland Islands” gets an en
dash because “Falkland Islands” is two words. So, too phrases like “Civil
War–era.” What about an em-dash? From here: Similar to an extended hyphen (-),
an em dash is used to show a break in thought or a shift of tone. If you’d
like to read more about the differences between a hyphen (-), en-dash (–), and
em-dash (—), see the blog post here which summarizes the above. Not the answer
you’re looking for? Browse other questions tagged or ask your own question.

[WET file] syntax - What’s the difference between - and -- in a phrase? - English Language & Usage Stack Exchange Stack Exchange Network Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Visit Stack Exchange Log In Sign Up current community English Language & Usage help chat English Language & Usage Meta your communities Sign up or log in to customize your list. more stack exchange communities company blog Tour Start here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us By using our site, you acknowledge that you have read and understand our Cookie Policy, Privacy Policy, and our Terms of Service. English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. Join them; it only takes a minute: Sign up Here’s how it works: Anybody can ask a question Anybody can answer The best answers are voted up and rise to the top Home Questions Tags Users Unanswered What’s the difference between - and — in a phrase? [duplicate] Ask Question 1 Possible Duplicate: When should I use an em-dash, an en-dash, and a hyphen? When do I put a - in a sentence? Is it a more powerful comma? With a bigger pause? syntax dashes symbols share|improve this question edited Apr 13 ’17 at 12:38 Community 1 asked Jul 13 ’11 at 0:10 curiouscurious 123115 marked as duplicate by waiwai933, MrHen, user2683, Robusto, Thursagen Jul 13 ’11 at 0:32 This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question. add a comment | 1 Answer 1 active oldest votes 0 The dashes you described are known respectively as the en-dash and the em-dash. To describe the difference between their origins, Mental Floss writes: An en dash (–) is bigger than a hyphen but shorter than an em dash (—). Th e names come from an obscure typographical measurement system, but the dashes have now taken on a life of their own in grammar. The em dash is the spork of English grammar: It ain’t particularly pretty, but you can use it for most anything. Em dashes can replace colons or sets of parentheses, or represent a sudden change in thought or tone. So when do you use an en-dash? Again from Mental Floss: To show numerical ranges, signifying “up to and including”—of dates, ages, pages, etc. (Example: “I read pages 7–22 last night.”) The storied “compound adjective hyphen,” an event so rare in the English language that proofreaders shiver with excitement whenever they come across it. Basically “pro-American” gets a regular hyphen because “American” is only one word, whereas “pro–Falkland Islands” gets an en dash because “Falkland Islands” is two words. So, too phrases like “Civil War–era.” What about an em-dash? From here: Similar to an extended hyphen (-), an em dash is used to show a break in thought or a shift of tone. If you’d like to read more about the differences between a hyphen (-), en-dash (–), and em-dash (—), see the blog post here which summarizes the above. share|improve this answer edited Aug 4 ’15 at 17:00 zwol 2,51911424 answered Jul 13 ’11 at 0:16 simchonasimchona 30.9k5112139 add a comment | Not the answer you’re looking for? Browse other questions tagged syntax dashes symbols or ask your own question. asked 7 years, 9 months ago viewed 2,336 times active 3 years, 8 months ago Featured on Meta Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) Linked 279 When should I use an em-dash, an en-dash, and a hyphen? Related 279 When should I use an em-dash, an en-dash, and a hyphen? 4 When to use -, – and —? 6 What is the difference between `-` and `--` 5 Does “cost-benefit ratio” use a hyphen or an en-dash? 0 What kind of dash character should I use at the end of a famous saying to mark of the author? -1 dash non-restrictive element in the middle of a sentence 1 em dash followed by a comma 0 what’s the difference between a hyphen, a dash and a minus sign? 0 Using comma to delimit the name of a group and its constituents? 1 I tend to overuse the hyphen as a pause, and would appreciate some feedback on this Hot Network Questions How could we fake a moon landing now? Using audio cues to encourage good posture Chinese Seal on silk painting - what does it mean? How does the math work when buying airline miles? What causes the direction of lightning flashes? Maximum summed subsequences with non-adjacent items Most bit efficient text communication method? Significance of Cersei’s obsession with elephants? Is it a good idea to use CNN to classify 1D signal? What font is "z" in "z-score"? How can I use the Python library networkx from Mathematica? Fundamental Solution of the Pell Equation Dating a Former Employee Do I really need recursive chmod to restrict access to a folder? Generate an RGB colour grid How to Make a Beautiful Stacked 3D Plot How to find all the available tools in mac terminal? Has negative voting ever been officially implemented in elections, or seriously proposed, or even studied? What is the meaning of the simile “quick as silk”? Circuit to "zoom in" on mV fluctuations of a DC signal? What are the out-of-universe reasons for the references to Toby Maguire-era Spider-Man in Into the Spider-Verse? What does this Jacques Hadamard quote mean? Can a new player join a group only when a new campaign starts? Why wasn’t DOSKEY integrated with COMMAND.COM? more hot questions English Language & Usage Tour Help Chat Contact Feedback Mobile Company Stack Overflow Stack Overflow Business Developer Jobs About Press Legal Privacy Policy Stack Exchange Network Technology Life / Arts Culture / Recreation Science Other Stack Overflow Server Fault Super User Web Applications Ask Ubuntu Webmasters Game Development TeX - LaTeX Software Engineering Unix & Linux Ask Different (Apple) WordPress Development Geographic Information Systems Electrical Engineering Android Enthusiasts Information Security Database Administrators Drupal Answers SharePoint User Experience Mathematica Salesforce ExpressionEngine® Answers Stack Overflow em Português Blender Network Engineering Cryptography Code Review Magento Software Recommendations Signal Processing Emacs Raspberry Pi Stack Overflow Ha pycckom Programming Puzzles & Code Golf Stack Overflow en español Ethereum Data Science Arduino Bitcoin more (31) Photography Science Fiction & Fantasy Graphic Design Movies & TV Music: Practice & Theory Worldbuilding Seasoned Advice (cooking) Home Improvement Personal Finance & Money Academia Law more (15) English Language & Usage Skeptics Mi Yodeya (Judaism) Travel Christianity English Language Learners Japanese Language Arqade (gaming) Bicycles Role-playing Games Anime & Manga Puzzling Motor Vehicle Maintenance & Repair more (33) MathOverflow Mathematics Cross Validated (stats) Theoretical Computer Science Physics Chemistry Biology Computer Science Philosophy more (10) Meta Stack Exchange Stack Apps API Data Blog Facebook Twitter LinkedIn site design / logo © 2019 Stack Exchange Inc; user contributions licensed under cc by-sa 3.0 with attribution required. rev 2019.4.18.33353 English Language & Usage Stack Exchange works best with JavaScript enabled

###  Appendix K Deduplication

We perform extensive ablations and experimentation on various deduplication
pipelines. This section is organized by first describing the deduplication
methods considered and then outlining the ablations that lead us to the choice
of deduplication pipeline used in generating DCLM-baseline (and other DCLM
scales).

####  Deduplication methods

Prior work such as Lee et al. [88], Penedo et al. [121] use a two-stage
deduplication pipeline where near duplicates are first removed at a inter-
document level by identifying and removing near-duplicates using the MinHash
algorithm, and then at an intra-document level where any substring of a
predetermined length that occurs more than once in the entire corpus is
removed. Intuitively, this strategy makes sense as the notion of a "duplicate"
is poorly defined and can include documents such as: (i) exact copies of
entire documents (targeted at the document-level); (ii) documents where the
majority of the text is a duplicate, but there are unique differences in just
the header or footer (targeted at the document-level); or (iii) documents
where there are significant sections of unique text, but also massively
repeated boilerplate text (targeted at the intra-document level). Performing
multiple resolutions of deduplication can target all such cases, and further,
a deduplication pipeline that can target near-duplicates, often referred to as
"fuzzy deduplication" can identify documents that humans would intuitively
refer to as duplicates.

While we ultimately rely on a Bloom filter based method of deduplication for
our datasets, we describe the other pipelines considered:

###### MinHash.

MinHash is a locality-sensitive hashing technique used to group sets into
collections based on their Jaccard similarity [28]. In the context of
deduplicating text datasets, MinHash was first employed in Lee et al. [88] and
then used in numerous other projects [121, 43]. We point readers to the main
text of Lee et al. [88] and Appendix G.3.1 of Penedo et al. [121] for more
details. The primary hyperparameters of note are the n-gram-size, and the
number of permutations used. Following Lee et al. [88], Penedo et al. [121],
we use an n-gram-size of 5 tokens and target a Jaccard similarity of 0.8.
Departing from prior work, however, we modify the number of MinHash
permutations used. Both Lee et al. [88] and Penedo et al. [121] use a total of
9,000 permutations, split into 450 buckets of 20 hashes each. We found this to
be overly expensive and notice that similar Jaccard similarity plots can be
attained with a much smaller number of permutations. For all of our ablations,
we instead use a total of 1,395 permutations, split into 93 buckets of size
15. These hyperparameters were chose programmatically to mimic the Jaccard
similarity plots as closely as possible, in an $\ell_{2}$ sense, with a fixed
hash budget. See Figure 6 for more details.

Figure 6: Probability of two documents with Jaccard similarity (x-axis) being
marked as duplicates (y-axis) with varying (number of buckets, bucket size)
parameters. (450, 20) corresponds to Lee et al. [88], Penedo et al. [121], our
experiments used (93, 15), chosen to be a cheaper alternative emulating the
same performance as (450, 20). The parameters (14,9) were used by Penedo et
al. [122].

###### Suffix arrays.

Suffix arrays, first introduced in Manber & Myers [105], enable efficient
identification and removal of substrings of a large corpus of text. This is
done by first concatenating all text in the corpus together and then sorting
each suffix. By scanning this sorted list, substrings with a common prefix can
by identified by scanning the prefices of neighboring elements in the sorted
list. This latter step can be done in an embarassingly parallel fashion, but
the implementation we employed, borrowed from the codebase provided in Lee et
al. [88] is not done in a multi-node fashion and requires loading the entire
corpus into RAM. We directly employ the hyperparameters used in Lee et al.
[88] and remove all repeated substrings that are at least 50 tokens long.

###### Bloom filters.

Bloom filters are a data structure that enable space-efficient set membership
queries [26]. Explicitly, in sublinear space, a Bloom filter maintains a
sketch of a set, that supports an insert operation, and a probabilistic
membership_query operation, where the latter will never return any false
negatives (i.e., return False for an element in the set), but will
occasionally return a false positive (i.e., return True for an element not in
the set). These were first used in the context of exact-duplicate removal in
Soldaini et al. [150], but have since been extended to perform near-duplicate
document and paragraph removal in a tool known as BFF (Big Friendly Filter)
[67], and we further modify BFF to perform deduplication at the document and
paragraph level simultaneously. We found that this technique is vastly more
efficient than a MinhHash and SuffixArray pipeline. However there is one
important caveat in that MinHash performs document-level deduplication at a
document vs. document level, whereas BFF performs document-level deduplication
at a document vs. corpus level.

###### Paragraph + document BFF.

Here we outline our modified Bloom filter based deduplication algorithm. Upon
initialization, we require an estimate of the number of tokens in our entire
corpus as well as a desired false-positive rate, to initialize a Bloom filter
with a fixed size and number of hashers. The optimal number of hashers, $k$,
is given by the formula

$k=-\frac{\ln{\epsilon}}{\ln{2}},$

where $\epsilon$ is the desired false positive rate. The optimal size $m$ for
$k$ hashers and $n$ tokens can then be computed by solving for $m$ in the
following formula:

$\epsilon=\Big{(}1-e^{\frac{-kn}{m}}\Big{)}^{k}.$

While this does not admit an easy analytical solution, it is trivial to solve
for $m$ by using a binary search algorithm.

Once we have established a Bloom filter, we proceed through each document in
our corpus and perform the following steps. First we tokenize the document
using the UniSeg tokenizer444https://www.unicode.org/reports/tr29/, and then
further break the document into paragraphs by splitting on the newline
character \n. For each document, we maintain counters total_ngrams, and
contained_ngrams. Each paragraph is then handled in turn, according to
hyperparameters denoting min_ngram_size, max_ngram_size, and threshold:

  * •

If the paragraph is fewer than min_ngram_size tokens long, it is left as is.

  * •

If the paragraph is in between min_ngram_size and max_ngram_size (inclusive)
then total_ngrams is incremented and this n-gram’s membership is checked in
the Bloom filter. If it is present in the Bloom filter, it is removed from the
paragraph and contained_ngrams is incremented. Otherwise, it is added to the
Bloom filter.

  * •

If the paragraph is at longer than max_ngram_size tokens, then each n-gram of
size max_ngram_size increments the total counter and is checked against the
Bloom filter. If present, the contained_ngrams counter is incremented. If
greater than threshold fraction of the n-grams in this paragraph are contained
in the Bloom filter, then the entire paragraph is removed from the document.
Otherwise, every non-contained n-gram is added to the Bloom filter.

Once all paragraphs have been processed, if the ratio between the counters
contained_ngrams and total_ngrams is greater than threshold, then the entire
document is removed from the corpus.

To finalize our discussion on the Bloom filter based deduplication, we offer
brief explanations on the hyperparameter choices made.

  * •

False Positive Rate: The two parameters that dictate the memory footprint
required by BFF are the number of tokens and the false positive rate. However
we only can control the false positive rate, and we notice that the Bloom
filter size scales linearly with the negative log of the false positive rate.
In particular, for a corpus of 1T tokens, occupying roughly 2TB of disk space,
ensuring no false positives, i.e. setting the false positive rate to $1/1T$,
would require 6.5TB of RAM. Here we argue analytically that a false positive
rate of even as low as 0.01 suffices, which we support with experimentation in
the next section.

In choosing a false positive rate for the n-gram-based Bloom filter, it’s
important to recognize that removal of a paragraph or document is dictated by
having greater than a threshold fraction of the n-grams contained in the set.
As an example, suppose we are given a paragraph of $N$ n-grams, where $S$ of
them are already contained in the Bloom filter and we set threshold to $T$.
Because Bloom filters do not allow false negatives, every one of the $S$
n-grams are marked (correctly) as contained, and $N-S$ of them could
potentially be marked as a false positive. Indeed, of the $N-S$ of these
n-grams, at least $TN-S$ of them would need to be marked as a false positive,
each of which occurs independently with probability $\epsilon$. This is
equivalent to $N-S$ Bernoulli random variables with parameter $\epsilon$, and
can be bounded by a crude Hoeffding bound. In this particular case, the
probability that a document or paragraph is falsely marked as a duplicate is
bounded by:

$\exp\Big{(}\frac{-2\cdot\big{(}TN-S-\epsilon\cdot(N-S)\big{)}^{2}}{N-S}\Big{)}$

To put things concretely, in a document with 100 n-grams and a threshold of
0.8 and a false positive rate of 0.01, if 60 of the n-grams have been seen
before, the probability of the document being marked as a duplicate is less
than $10^{-8}$. Unless otherwise specified, we always use a false positive
rate of 0.01.

  * •

min_ngram_size: In choosing a size for minimum n-grams, we recognize that many
documents contain paragraphs that are itemized lists and are quite short; for
example, recipes often include bullet-pointed ingredients lists, and MMLU
multiple choice questions may often be quite short. While we originally
noticed improved Core scores by setting a minimum size to 5 tokens, we noticed
that this caused a worse performance on MMLU. After manual inspection, we
settled on a min and max n-gram size of 13 tokens.

  * •

threshold: Ablations did not show a noticable difference in deduplication
performance.

####  Deduplication experiments

#####  K.2.1 Deduplication ablations: pipeline at 1B-1x scale

We first perform ablations regarding the full pipeline choice for
deduplication at the 1B-1x scale. We start with a pool of 76B tokens
subsampled from Common Crawl with the preprocessing steps from Penedo et al.
[121] applied. Then we apply a combination of deduplication steps, and
subsample the pool further to the 28B tokens required for the 1B-1x scale.
Finally we train and evaluate the Core score and the percentage of tokens that
were removed by deduplication. The main questions we seek to answer from this
round of ablations are:

  * •

For multi-step deduplication pipelines, how much of a contribution does each
step provide?

  * •

Which deduplication pipeline is worth scaling up to larger pool sizes?

Results are contained in Table 18. The main conclusions we can arrive at from
this table are as follows: i) Suffix Array deduplication seems to help more
than MinHash deduplication, thereby giving some signal to the source of the
gains procured by a MinHash+SuffixArray pipeline; ii) BFF provides comparable
performance to a full Exact+MinHash+SuffixArray pipeline, giving strong
evidence that the multiresolution BFF could be an easily scalable alternative
to the relatively more expensive MinHash+SuffixArray pipeline of prior works.
Interestingly, it appears that a SuffixArray pipeline seems to outperform
MinHash alone, though this falls within the range of variance for the Core
score due to the nondeterminism in subsampling the dataset and training a
model.

Table 18: Deduplication ablations (1B-1x scale). Starting from a pool of 76B
tokens acquried from Common Crawl with the RefinedWeb Penedo et al. [121]
pipeline applied, we evaluate the removal rate and Core score on different
combinations of deduplication methods. Our Bloom filter method performs just
as well as a combination of exact deduplication, MinHash and Suffix Array
based techniques.

Exact Dedup | MinHash | Suffix Array | Bloom Filter | Tokens | Removal Rate | Core |  $\Delta$ from Baseline
---|---|---|---|---|---|---|---
✗ | ✗ | ✗ | ✗ | 76B | 00% | 40.1 | +0.0
✓ | ✗ | ✗ | ✗ | 66B | 13% | 41.0 | +0.9
✗ | ✓ | ✗ | ✗ | 62B | 18% | 40.9 | +0.8
✗ | ✗ | ✓ | ✗ | 51B | 33% | 41.4 | +1.3
✗ | ✗ | ✗ | ✓ | 56B | 26% | 41.7 | +1.6
✓ | ✓ | ✗ | ✗ | 58B | 24% | 40.2 | +0.1
✓ | ✗ | ✓ | ✗ | 49B | 36% | 41.3 | +1.3
✗ | ✓ | ✓ | ✗ | 48B | 37% | 41.2 | +1.2
✓ | ✓ | ✓ | ✗ | 45B | 41% | 41.7 | +1.6

#####  K.2.2 Deduplication ablations: pipeline at 7B-1x / 7B-2x scale

To further check the effects of BFF versus the more classical
MinHash+SuffixArray we ran several experiments at the 7B-1x scale. Here we
also introduce another hyperparameter, which we refer to as shards. By
"sharding," we mean we break a dataset into chunks of roughly equal size and
run the deduplication pipeline on each one of them independently. This is
primarily done for engineering purposes, in that sharding is an easy way to
further parallelize deduplication and convert single-node algorithms to multi-
node algorithms. However, there are the side benefits of sharding for
deduplication in that more shards yields a larger token pool: there are fewer
documents to compare against and many documents which are repeated only a
small number of times can survive such a process. Additionally there is some
recent evidence that sharding seems to improve evaluation performance [122].
We also note that RefinedWeb [121] performs their deduplications on a 100-way
sharding of the Common Crawl pool.

For this round of ablations, we start with a pool sourced from one tenth of
Common Crawl and run the preprocessing steps from Penedo et al. [121] and
apply various deduplication pipelines. Then we subsample down to 138B tokens
and train and evaluate models at the 7B-1x scale. The main questions we seek
to answer from this round of ablations are:

  * •

Is BFF still competitive with a MinHash+SuffixArray pipeline at larger scales?

  * •

Which BFF hyperparameters yield the highest Core and MMLU performance at this
scale?

Results are contained in Table 19. The first point to note is that BFF with a
min_ngram_size at 13 and 20 yields Core scores and MMLU scores that are
comparable to the scores attained by a MinHash+SuffixArray deduplicated pool
at the same scale. The second point to note regards the BFF min_ngram_size and
sharding: interestingly a lower min_ngram_size yields higher Core scores, but
lower MMLU scores. We also see that fewer shards decreases the token yield,
but has variable effect on the Core score. We examine the hyperparameters for
BFF more fully in the next subsection.

Table 19: Deduplication Ablations (7B-1x scale). Starting with a pool from Common Crawl and the RW-Filter pipeline processing applied, we compared several BFF hyperparameters against the MinHash and Suffix Array pipeline of [88, 121]. Our best BFF run and the prior works are bolded.  Method | min-ngram | max-ngram | Shards | MMLU | Core | Token Yield
---|---|---|---|---|---|---
Bloom Filter | 5 | 13 | 32 | 25.0 | 52.0 | 4T
Bloom Filter | 5 | 13 | 1 | 27.1 | 52.7 | 1.3T
Bloom Filter | 13 | 13 | 32 | 28.7 | 51.7 | 3.8T
Bloom Filter | 20 | 20 | 32 | 27.7 | 51.5 | 4T
Bloom Filter | 20 | 20 | 10 | 28.4 | 51.2 | 3T
MinHash+SA | N/A | N/A | 16 | 29.1 | 51.9 | 3.2T

Encouraged by these results, next we examine the top candidates for a scalable
deduplication pipeline at the 7B-2x scale. Again we start with a pool obtained
from one tenth of Common Crawl and generate several deduplicated pools. The
questions of interest are the same as above and we summarize the results in
Table 20. The key takeaways from this round of ablations is that at the 7B-2x
scale, BFF with a min_ngram_size of 13 and 10 shards attains nearly identical
performance to a MinHash+SuffixArray pipeline, whereas BFF with a
min_ngram_size of 20 and 32 shards starts to lag behind, and that a
min_ngram_size of 5 yields competitive Core scores, but falters in MMLU
evaluations. While these experiments also vary the sharding choice, we view
sharding primarily as a choice made to trade-off scalability with token yield.
Larger shards are more expensive and less parallelizable and can decrease the
token yield. For this round of ablations, the primary interest is to gain
signal about how BFF compares to MinHash and Suffix Arrays at scale, and which
are the correct hyperparameters for BFF. On this latter point, we chose to
move forward with a min_ngram_size of 13 for generating DCLM-baseline.

Table 20: Deduplication Ablations (7B-2x scale). From the same pools as in Table 19, we trained and evaluated models at the 7B-2x scale. Notice that a min_ngram_size of 5 yields competitive Core results but drastically reduces MMLU scores. Method | min_ngram | max_ngram | Shards | MMLU | Core | Token Yield
---|---|---|---|---|---|---
BFF | 20 | 20 | 10 | 43.6 | 55.7 | 3T
BFF | 13 | 13 | 10 | 44.3 | 55.5 | 3T
BFF | 5 | 13 | 32 | 32.0 | 54.2 | 4T
MinHash+SA | N/A | N/A | 16 | 44.4 | 55.4 | 3.2T

#####  K.2.3 BFF hyperparameter ablations

While the above ablations largely focused on the Core score and MMLU as
performance metrics, these are expensive and not suited for large swaths of
ablations. Here we instead explore statistics of datasets deduplicated by BFF
as we toggle the ngram_size hyperparameters, false positive rate, and input
dataset size. We run separate experiments for each hyperparameter and finish
each paragraph with the choice of hyperparameter we use for all larger scale
runs.

###### False positive rate

.

Table 21: False positive rate ablations. Starting with a pool of 75B tokens from the RW-Filter pipeline, we ran BFF with default hyperparameters, varying the false-positive rate to indicate that this does not have a large bearing on output pool size.  False Positive Rate |  Removal Rate (Bytes)
---|---
0.1 | 20.47%
0.01 | 20.47%
0.001 | 20.47%

Here we start with the 75B token data pool as in Section K.2.1 and focus on a
paragraph-only level BFF. In other words, we run BFF as described above,
except omit the full-document removal step. We use the default hyperparameters
for n-gram sizes as in Groeneveld [67], of 5 and 13 for min_ngram_size and
max_ngram_size and a threshold of 0.8. We specifically look at the effect of
changing the false positive rate and compute the removal rate (in bytes) of
the output. From Table 21, we can see that a false positive rate of 0.1
suffices for a reasonably small pool such as this one. For larger pools, to be
safe, we always set the false positive rate to 0.01.

###### Min n-gram size.

From Table 19 and Table 20, we saw that altering the ngram_size
hyperparameters can affect both token yield and evaluation metrics. In
particular, we seek to examine how surviving documents are altered by
deduplication. As a proxy for this, we focus on the document lengths and
removal rates. Results for this paragraph and the two following paragraphs are
collated in Table 22. One key observation is that as the min_ngram_size
parameter is reduced, the mean and median document lengths become shorter.
This indicates that too-low a min_ngram_size parameter can dramatically affect
the language statistics of the dataset and should be avoided. This tracks with
intuitive sense where many documents include linebreak separated lists where
each list element is short and possibly repeated: e.g., many webpages include
recipes that might call for "1 stick of butter", which would get removed with
a min_ngram_size of 5 but would injuriously damage the source document.

Table 22: BFF hyperparameter ablations. Starting with a pool of 341B tokens taken from Common Crawl with the RW-Filter pipeline applied, we run our Bloom filter deduplication with various hyperparameters noting how the document length and pool size change after deduplication. The input pool statistics are noted in the first row. Min | Max | Threshold | Avg Tokens/Doc | Median Tokens/Doc | Total Documents | Total Tokens
---|---|---|---|---|---|---
N/A | N/A | N/A | 883 | 451 | 386M | 341.0B
1 | 13 | 0.8 | 778 | 403 | 246M | 191B
5 | 13 | 0.8 | 802 | 426 | 250M | 201B
13 | 13 | 0.8 | 836 | 456 | 246M | 205B
13 | 25 | 0.8 | 839 | 458 | 248M | 208B
13 | 50 | 0.8 | 833 | 453 | 253M | 211B
13 | 13 | 0.75 | 842 | 460 | 241M | 203B
13 | 13 | 0.8 | 836 | 456 | 246M | 205B
13 | 13 | 0.9 | 822 | 446 | 256M | 211B
13 | 13 | 0.99 | 797 | 427 | 275M | 218B

###### Max n-gram size.

Next we examine increasing the max_ngram_size hyperparameter. Starting with
the chosen min_ngram_size parameter of 13, decided in the previous paragraph,
we consider max_ngram_size parameters of 13, 25, and 50. Contrary to the
min_ngram_size, we do not see a dramatic alteration of language statistics as
this parameter becomes increased. For simplicity, we choose to use a
max_ngram_size of 13 for large-scale pools.

###### Threshold.

The threshold hyperparameter dictates how close a document must be to
previously seen n-grams before it is considered a duplicate. We ablate this
choice from 0.75 to 0.99, examining how this affects document length
statistics and removal rates. Interestingly, as the threshold increases,
documents get shorter, mirroring the statistics seen for reducing the
min_ngram_size. As expected, higher thresholds yield lower removal rates.
Following the Jaccard similarity choice used in MinHash deduplication and
noting that 0.8 yields median tokens/doc closest to the baseline, we use a
threshold of 0.8 going forward.

###### Shards.

Finally we simulate how shards affect the statistics of the deduplicated
datasets. As above, the key statistics we focus on here are the removal rate
and the average and medium document lengths. This is mostly to get a sense for
how these features change as the dataset scales, with the prevailing thought
that dramatically altering document statistics might adversely effect
downstream evaluations. For larger pools, we can always shard them as heavily
as desired, so we treat sharding as a hyperparameter that controls removal
rate and document statistics. Results are collated in Table 23 and Figure 7.
The key takeaways here are that removal rates increase monotonically with
dataset size as expected, but do so in a concave fashion. This provides some
signal for how heavily to shard an input pool if a desired token yield is
specified. The next point of interest is to consider the document lengths as
the dataset scales. These decrease monotonically as the pool increases in
size.

Table 23: Deduplication shard size. We run a single-shard BFF with the ngram_size set to 13, false positive rate 0.01, threshold of 0.80 on pools of varying size. As the pool size scales the deduplication rate increases, documents get shorter and the removal rate increases. Input Tokens | Input Documents | Avg Tokens/Doc | Median Tokens/Doc | Token Removal Rate
---|---|---|---|---
114B | 129M | 466 | 866 | 29%
227B | 257M | 460 | 848 | 35%
341B | 386M | 456 | 836 | 40%
455B | 516M | 453 | 826 | 43%
569B | 643M | 450 | 918 | 46%

In building DCLM-baseline, at the point of deduplication, the dataset is
approximately 70TB in size. Since Table 20 shows that BFF had the best
performance at a 10-way shard with a roughly 7TB input size, we adhere to a
100-way sharding for DCLM-baseline, where each shard is roughly 700GB in size.

Figure 7:  Deduplication shard size. We run a single-shard BFF with the
ngram_size set to 13, false positive rate 0.01, threshold of 0.80 on pools of
varying size. Larger pools have a larger removal rate, but this scales in a
concave fashion. The removal rates for tokens and documents begin to diverge
at larger scales.

#####  K.2.4 Global MinHash on Open Datasets

Finally, to get a sense for the duplicates remaining in a dataset after a full
processing pipeline has been applied, we run a global (i.e., one shard)
MinHash on several open datasets. These results are collated in table table
24. We evaluate our DCLM-baseline, the official RefinedWeb dataset from
HuggingFace, our emulation of the RefinedWeb pipeline, and Dolma V1. MinHash
is performed using 14 buckets and a bucket size of 9, corresponding to the
green curve in fig. 6.

We note several observations here. First, we note that pools deduplicated with
a Bloom Filter still have large numbers of "fuzzy duplicates" in the
MinHash/Jaccard Similarity sense. This indicates that what the Bloom Filter
considers a duplicate and what MinHash considers duplicates are not identical
concepts. Second, we see that while MinHash is a roughly idempotent procedure,
deduplication over shards fails to remove a large portion of the duplicates.
Third, we see that our 100-shard Bloom filter deduplication applied to DCLM-
baseline still leaves many duplicates in the dataset, yet does not seem to
adversely effect downstream performance. This calls into question the general
prevailing thought that the presence of any duplicates hinders downstream
performance: we instead conjecture that either i) only large amounts of
duplicates are detrimental to downstream performance, or ii) aggressive
single-sharded deduplication eliminates many high quality documents. We leave
such experimentation for future work.

Table 24: Global MinHash on Open Datasets We perform a global MinHash on several open datasets and evaluate the number of duplicates that would be removed. We denote the deduplication applied to generate each pool and the number of shards used (* implies inferred sharding). DolmaV1 contained approximately 600M documents containing only the empty string, so we report numbers with and without the empty strings in the dataset. Dataset | Num Documents | Deduplication Applied | Shards | MinHash Removal Rate
---|---|---|---|---
DCLM-baseline | 3.2B | (Fuzzy) Bloom Filter | 100 | 85%
RefinedWeb (official) | 968M | MinHash+SA | 1* | 0%
RefinedWeb (ours) | 2.0B | MinHash+SA | 16 | 45%
Dolma V1 (w/ empty) | 5.2B | (Exact) Bloom Filter | 1 | 43%
Dolma V1 (w/o empty) | 4.6B | (Exact) Bloom Filter | 1 | 36%

###  Appendix L Mixing sources

In Section 4.5, we showed that mixing our dataset with the usual sources did
not improve its general performance, and hypothesized that this is due to the
more stringent filtering performed for our Common Crawl portion. One could
argue that improved filtering in the other sources could lead to similar
improvements in performance. As such, we perform an experiment where we apply
the same fastText classifier for filtering the other sources, as we do for our
DCLM-baseline.

We take several source from RedPajama [160], and filter them with the fastText
classifier applied in our DCLM-baseline, while keeping only the highest scored
ones. We then add the resulting data to our pretraining dataset, and train
models at the 1B-1x scale. The results of this can be seen in Table 25. We see
that despite the more uniform handling of mixing across various sources, the
additional sources still decrease performance.

We leave further analysis on potential mixtures of our datasets with other
sources for future work. We also leverage the mixing track as a way for
participants to explore such directions.

Table 25: Mixing with filtered data. We evaluate our models on mixtures of data, where we combine our DCLM-baseline with filtered data from other sources of RedPajama [160]. We find that the case where we use only DCLM-baseline performs the best in our experiments. Evaluation is done at the 1B-1x scale. Dataset Mixture | Core | MMLU | Extended
---|---|---|---
DCLM-baseline only | 31.7 | 26.5 | 16.6
DCLM-baseline \+ Filtered Wiki | 31.0 | 24.9 | 16.3
DCLM-baseline \+ Filtered Books | 30.6 | 25.8 | 15.5
DCLM-baseline \+ Filtered Arxiv | 31.5 | 26.0 | 15.6
DCLM-baseline \+ Filtered Github | 30.0 | 24.4 | 15.2

###  Appendix M Human judgment

Figure 8: Accuracy measurements against ROC-AUC of different quality filters
on subsets of our human annotated samples. Top: Majority, bottom: Agreement.
Left: Core score, middle: StrategyQA, and right: SQuAD. All models share the
same scale (1B-1x) and training hyperparameters and are based on the same pre-
filtered pool, using similar filtering-ratios for different classifiers
(keeping top $\sim 15\%$ of the pool). The horizontal line marks the baseline
score of a model trained on random subset of the unfiltered pool. While it may
seem there is some positive correlation for StrategyQA, the opposite is true
for SQuAD and in both cases the $R^{2}<0.3$. Similar to what seen in Core
score, for almost all other tasks, there is no apparent relationship.

Prior work suggests that using human annotators may introduce undesired bias
or noise into the data due to under-training of the annotators for the task,
lack of skill or motivation, or unintended leakage of subjective bias [63, 61,
39]. However, human annotators are still widely considered the gold standard
for annotating data with a clear task at hand. A natural hypothesis is that if
human annotators could manually filter the large pool of raw data, we would
end up with a particularly high-quality dataset. To test this, we ask 16
English-speaking AI graduate students and professors to annotate approximately
500 randomly selected documents from a pool of data without a quality filter.
We obtain three annotations per document and use the majority vote in each as
the gold label. The average inter-annotator agreement is 71%. We further
extract the subset of 281 samples where all three annotators are in agreement,
naming the full data Majority and the subset Agreement.

We then evaluate various quality filters from Section 4.4 on this data to
search for correlation between dataset quality (as measured by Core accuracy)
and filter agreement with human labels. Figure 8 (a) depicts the Core scores
of models trained on datasets filtered with the respective quality filter
against ROC-AUC555ROC-AUC measures a classifier’s ability to distinguish
between classes by summarizing the trade-off between true positive and false
positive rates across all thresholds, making it a robust and common metric for
model performance. See
https://en.wikipedia.org/wiki/Receiver_operating_characteristic for further
details. of our quality-filters on the Majority data. Notably, both the best
and worst fastText-based filters score about the same on the Majority data
($\sim 73\%$ ROC-AUC), while the AskLLM filter that is highly correlated with
human annotations ($\leavevmode\nobreak\ \sim 82\%$ ROC-AUC) performs much
worse as a quality filter ($\sim 28.5\%$ Core compared to $>31\%$ in several
fastText classifiers).

We continue this study by inspecting correlations to specific downstream tasks
and comparing them to the ROC-AUC on the Agreement data, where all three
annotators agreed on the label. Figure 8 depicts the scores on a few tasks
against the ROC-AUC on the annotated data of the representative set of quality
filters. While some positive correlation may be observed for StrategyQA [62],
the opposite is true for other QA datasets such as SQuAD [133], and in both
cases the $R^{2}<0.3$. In most other downstream tasks, the results are similar
to Figure 8 (a), where no correlation can be observed. This suggests that
human intuition may not reliably identify the most useful documents for
language model training purposes. We hypothesize that human curators may
create datasets that lack sufficient diversity and leave further investigation
of these hypotheses to future research.

Figure 9: Histogram of length in words for samples in our human-annotated data
(capped at 2,000).

###### Collecting the data.

We sampled 499 random documents from our pool, after going through the rule-
based quality filters and deduplication. Figure 9 shows a histogram of the
length of the documents in words. We asked 16 English speak AI graduate and
professors to annotate each example as a good candidate to be included in an
LM pretraining corpus (see instructions and some examples given to annotators
below).666All annotation efforts were done voluntarily and were not paid for.
Out of the 499 samples, in 281 samples there was full agreement between all
three annotators. We release both datasets in https://datacomp.ai/dclm.

####  Instructions given to annotators

Your task is to assess the quality of the documents (0-bad, 1-good), sampled
randomly from RefinedWebV2, in terms of their usefulness for LLM training. How
do you judge the usefulness of a document? While this is a subjective task,
there are a few things to keep in mind: 1\. We encourage you to review the
details of the evaluation tasks that the LLMs are expected to excel in [link
to spreadsheet with the tasks was provided]. Broadly, these tasks include
reading comprehension, language understanding, world knowledge, and
commonsense reasoning. 2\. Check whether a particular document will be useful
for answering any open-ended imperative tasks or questions. For example:
writing a letter to a friend, writing a poem, replying to an email, cooking a
dish, etc. 3\. Check whether the data contains harmful content that you would
not want an LLM to generate. This can include personal information, vulgar
language, etc. 4\. See the examples below to build some intuition.

####  Examples

Doc1 (Bad)

Welcome to the Southern California Connection Prayer Line Ministry Sponsored
by the Central Filipino Church - Women’s Ministry Department Gwen Shorter -
Women’s Ministry Director Joanne Williams - Prayer Leader Dial: (712) 432-0075
Enter Participant Code: 624255 To hear the most recent call, please dial (712)
432-1085 Enter Participant Code: 624255 Sunday through Friday - at 7 am and 7
pm Psalm 91 (King James Version)Prayer Line Moderators |Sunday, Monday and
Friday |Joanne Williams, Marsha Harold |Tuesday, Wed., and Thursday |Gwen
Shorter, Gloria Duckett Note: Moderators schedule may vary from time to time.
Webmasters: Michael Wong, Will Fults Hard Copies - For CD’s of programs ($3.00
each) Please call Gloria @ 541-476-0038 Click on the links below for some
detailed information: Prayer Line Speakers - Biographys and Photographs Prayer
Line Recordings by Date View Prayer Line Testimonials - Submit Prayer Line
Testimonial Download, Print and Share: |free web hit counter As of December 9,
2010.

Doc2 (Bad)

Host: Zander Program Category: Music Frequency: Weekly Length: 2 Hours Terms:
Barter Delivery Method: Internet |“Zander’s knowledge of music and his
straight-forward approach has struck a huge interest among our listeners. The
Rockin’ 80’s is EXACTLY what we’ve been looking for!"" - Terry West, WQLA The
Rockin’ 80’s is the only 80’s show with a mix of the best rock from the decade
of excess plus “oh wow” tracks that add spice to the weekly line up. The two
hour version of the show features rarities from the 80’s “Lost and Found”, an
80’s “Two-Fer,” spotlighting two contrasting songs from one band played back
to back and much more. Featured core musical artists include Guns N’ Roses,
Motley Crue, Van Halen, Rush and AC/DC. |3733 Park East Drive • Room 222 •
Cleveland, Ohio 44122 P: 216-831-3761 • F: 216-514-4699 • Email us |©2014
Envision Networks. All rights reserved. Site design by Single Source Marketing

Doc3 (Good)

|Chinese Five-Spice Noodles with Shitake Mushrooms |Recipes - Chinese Five
Spice Chinese Five-Spice Noodles with Shitake Mushrooms Cook the noodles until
tender according to the package directions. Drain and rinse under cold running
water, and drain well again. Bring 2 1/2 cups water to a boil in a small
saucepan. Add the dried mushrooms, cover, and simmer for 3 minutes. Strain the
liquid through a paper coffee filter into a bowl to remove any grit, then
squeeze the mushrooms over the bowl. Roughly chop the mushrooms, then set
aside with the liquid. Heat the butter and oil in a wok or large saucepan over
medium heat. Add the Chinese five-spice powder and shallots and saute,
stirring frequently, until tender, about 2 minutes. Add the garlic and saute 2
minutes more. Increase the heat to medium-high and add the fresh and
reconstituted-dried shitakes with 1/2 cup of the reserved mushroom water.
Cook, stirring frequently, until the mushrooms are tender, about 3 minutes.
Add the soy sauce and rice wine. Increase the heat to high and cook, stirring,
for 1 minute. Add the remaining mushroom-soaking water. Add the noodles to the
wok and stir until heated through and coated with the sauce, about 1 minute.
Garnish with the green onions and sesame seeds, if using, and serve at once.
Makes 4 servings

Doc4 (Good)

"Is it necessary to purchase a travel book or is it realistic that we can get
similar information from other resources? Usually, most individuals have a
major question on buying a travel book. So here are the pros and cons of
purchasing one such book. Advantages of a Travel Book A travel book, which may
be a paperback or e-book, comes in handy while traveling. Glancing through a
travel book enables you to understand the custom and culture of a particular
place in the world. So you can adapt yourself to that particular environment
and stay there comfortably for longer periods. \- They Come In Handy — The
travel guide comes in various forms such as, e-books, paperbacks and the file
formats. You can have easy access to these books, which would assist you with
all details compatible to the region you are traveling to. \- They Provide
Enormous Information — Electronic or traditional travel guides provide you
with answers to all types of questions such as how to learn some sayings that
can be used in the place where you are traveling to? How to get data on where
to reside, what to see and where to eat? How to get a clear knowledge about
the history of a specific region or the atmosphere that it has? \- They Suit
To Your Requirements — To access full information about a specific country or
a region, both types of general and specific travel books are made available.
The e-book may easily fit into your e-book reader whereas the paperback can
fit into your backpack. Disadvantages of Travel Book \- The Price — The e-book
and paperback travel guides are very expensive compared to the information
obtained from travel websites or from those who have moved or traveled to that
region. \- Qualitative Images In Travel Books — Most travel books are in black
and white. Only a few e-books consist of colored photos. Hence make a thorough
revision before purchasing a travel guide or an e-book. \- Travel Books Make
The Trip Less Natural — Traveling can be made more spontaneous by acquiring
suggestions from locals than from travel books."

###  Appendix N Decontamination

In Section 4.6, we examined the results of the contamination analysis with
respect to MMLU. Here, we present some analysis with respect to the _other_
validation sets. Instead of the MMLU-specific decontamination that we
performed in Section 4.6, here we follow a more general approach based on
token overlaps.

Overall, a generally applicable decontamination rule is difficult to specify,
given the potentially subjective nature of what constitutes as contamination
in text data as well as the diversity in formats across tasks. Following
Touvron et al. [162], we search for contaminated tokens that exist in
overlapping 10-grams (or longer) between DCLM-baseline and our downstream
tasks. We measure the percentage of samples in each evaluation set where more
than 80% of the tokens are contaminated (such samples are considered “dirty”
per Touvron et al. [162]), as well as the percentage where less than 20% of
the tokens are contaminated (considered “clean” by the same criterion).

We examine the difference of performance of the same 7B-2x model trained on
DCLM-baseline, between the full evaluation set and the evaluation samples that
are marked as “not dirty” per the criterion in Touvron et al. [162] (less than
80 % of the tokens in the sample are marked as contaminated), and between the
full evaluation set and samples marked as “clean” using the same criterion
(less than 20 % of the tokens are marked). Results can be seen in Figure 10,
where we see that the difference in performance over the full dataset and the
“not dirty” samples is minimal. In fact, for BoolQ and SQuAD, which are marked
as highly contaminated, our model performs slightly better on the “not dirty”
subset. Moreover, in Figure 11 we see that the difference in performance
between the full evaluation set and the “clean” subset is similarly small for
most datasets. We note here that it’s difficult to identify a correct
threshold for what counts as a contaminated sample (as 20 % token overlap
might lead to many false positives, but at the same time 80 % might be too
high to detect all contaminated samples).

###  Appendix O Instruction tuning

Figure 10: Analysis of performance on the “not dirty” subset. The x-axis is
the percentage of samples from each evaluation task where more than 80% of the
tokens are contaminated (such samples are considered “dirty” per Touvron et
al. [162]). The y-axis is the performance of our 7B-2x model trained on DCLM-
baseline over the full training set, minus the performance on the “not dirty”.
Each point is an evaluation task in our Core subset, as well as MMLU. There is
no clear correlation with changes in performance over the full and the “not
dirty” evaluation subsets and contamination. Figure 11: Analysis of
performance on the “clean” subset. The x-axis is the percentage of samples
from each evaluation task where more than 20% of the tokens are contaminated
(such samples are considered “not clean” per Touvron et al. [162]). The y-axis
is the performance of our 7B-2x model trained on DCLM-baseline over the full
training set, minus the performance on the “clean” subset (less than 20% of
the tokens contaminated). Each point is an evaluation task in our Core subset,
as well as MMLU. Most evaluation tasks (including MMLU) have similar
performance in the full eval and in the “clean” subset.

Instruction tuning has emerged as a critical step to allow users to interact
with pretrained language models [168, 166, 111, 114, 183]. To investigate
whether models trained on DCLM-baseline can have strong instruction-following
capabilities, we instruction-tune the DCLM-baseline (7B) with the
OpenHermes-2.5 (OH-2.5) dataset [157]. Specifically, we train our model to
predict the response given the instruction from the instruction-tuning
dataset. We train DCLM-baseline using the Adam [85] optimizer for $10$ epochs
with a $10\%$ warmup ratio and a cosine learning rate schedule. We perform a
hyperparameter search over the three learning rates
$\\{1\text{e-}7,5\text{e-}6,2\text{e-}5\\}$ and report the best-performing
numbers on the evaluation metrics i.e., AlpacaEval 2.0 length-controlled win-
rate [50]. Post-training, we generate the responses for the instructions from
AlpacaEval using a sampling temperature of $0.3$ and maximum response length
of $500$. We benchmark our model with relevant baselines (e.g., LlaMA-2-Chat
[162], Zephyr-7B [163]) taken directly from the AlpacaEval leaderboard as well
as Mistral-7B [77] finetuned in the same manner as DCLM-baseline.

Table 26: Instruction tuning results on AlpacaEval2.0. We see that DCLM-baseline w/ OH-2.5 performs similarly to Mistral-7B finetuned also on OH-2.5, indicating similar behavior during instruction tuning. Also, with better data, we see DCLM-IT can be even better and can beat many existing models of similar scales.  Model | AlpacaEval2.0 LC Win-rate (%)
---|---
Our runs |
DCLM-IT | 16.6
Mistral-7B w/ OH-2.5 | 15.4
DCLM-baseline w/ OH-2.5 | 13.8
Reported from the leaderboard |
LLaMA-3-Instruct-8B | 22.9
Mistral-v0.2-7B | 17.1
Mistral-7B w/ OH-2.5 | 16.2
Zephyr-Beta-7B | 13.2
Vicuna-v1.3-13B | 10.8
Gemma-Instruct-7B | 10.4
Nous-Hermes-13B | 9.7
DaVinci001 | 9.0
LLaMA-2-Chat-13B | 8.4
Alpaca-7B | 5.9

We present the results in Table 26. We find that DCLM-baseline finetuned with
OH-2.5 outperforms various instruct models such as Zephyr-Beta-7B and Gemma-
Instruct-7B. This indicates that we can elicit high-quality responses from the
pretrained DCLM-baseline model with instruction-tuning. In addition, we
observe that the DCLM-baseline slightly lags behind Mistral-7B-OH-2.5 meaning
DCLM-baseline is competitive with other existing models of the same scale for
finetuning. The small difference in performance might be attributed to the
DCLM-baseline w/ OH-2.5 having longer generations on average than Mistral-7B
w/ OH-2.5 or the lesser number of tokens seen during DCLM-baseline pretraining
in comparison to Mistral-7B.

A follow-up question is whether DCLM-baseline can be finetuned to be even more
competitive with models of similar scale. To further improve the instruction-
following capabilities of DCLM-baseline, we curate a custom dataset, DCLM-IT,
by combining some of the best instruction-tuning datasets including
UltraFeedback [45], Tulu-v2 SFT [76], CodeFeedback [192], OH-2.5, Nectar
[195], NoRobots [132], WildChat [190], WebInstruct [184], and StarCoder2-Self-
OSS-Instruct [169]. There are roughly $4$ million instances and $8$ billion
tokens in this dataset. Subsequently, we perform instruction-tuning and
response generation of DCLM-baseline on this dataset with the training recipe
mentioned above. We present the results in Table 26. We find that DCLM-IT
outperforms DCLM-baseline w/ OH-2.5 by $2.8$ percentage points. Our results
highlight that there is room to enhance the instruction-following capabilities
of DCLM-baseline with better datasets such as DCLM-IT. We further clarify that
the current instruction-tuned models do not undergo any alignment procedures
such as PPO [142], DPO [129] or others [52, 32, 16, 107]. We leave the
development of aligned versions of DCLM-baseline for future research.

###  Appendix P Scaling up

The final run 2.5T shown in Figure 1 and Table 9 uses a two stage training
procedure as followed in Groeneveld et al. [68], Hu et al. [74] and Team et
al. [154]. For stage 1 we use the hyper-parameters from Table 27.

Table 27: Hyper-parameters for large scale run. Note the LR schedule uses a training length of 4.4T, but we do not train for the full length as we stop early and cooldown. Hyper-parameter/Config | Value
---|---
Training Tokens | 4,409,222,758,400
Warmup Steps | 10,000
Initial Learning Rate | $2\times 10^{-3}$
Weight Decay | 0.05
Final Learning Rate | $3\times 10^{-5}$
Global Batch Size | 2048
Accumulation Steps | 2
Query-Key Normalization | True
Z Loss | $5\times 10^{-6}$
Table 28: Results for 2.5T run, first row was run for 2T + 200B (cooldown), second row was run for 2T + 270B (cooldown), third is evaluation of average of weights of first two rows (0.2*CoolDown #1 + 0.8*CoolDown #2) Model | MMLU | Core | Extended
---|---|---|---
CoolDown #1 (200B Tokens) | 62.7 | 55.9 | 43.8
CoolDown #2 (270B Tokens) | 63.4 | 55.9 | 44.3
Final Model Soup | 63.9 | 56.0 | 43.7

After 2T tokens, we cooldown on a re-weighted pre-training distribution. For
the cooldown distribution we use a mix of 70% DCLM-baseline with a tighter
fastText threshold (top 7% rather than top 10%) and 30% ProofPile. We keep all
the hyperparameters the same as Table 27, so we cooldown to the same final
learning rate, just over a smaller number of tokens. Before the cool-down MMLU
performance was approximately 52%, and the LR was approximately $1\times
10^{-3}$.

We performed 2 independent cooldowns, one for 270B tokens and another for 200B
tokens, and created a “model soup" [173] with a weight of 0.8 on the 270B
cooldown and a weight of 0.2 on the 200B cooldown. Thus we the total number of
tokens seen by this model is 2.5T. We present results of each individual
cooldown and the model soup in Table 28. The model in Figure 1 and Table 9
uses the final model soup after long-context training for 100B tokens as
described in Section P.2.

####  Instruction tuning the scaled up model

In Appendix O we show how instruction tuning the above “model soup" for 80B
additional tokens leads to strong performance on instruction tuning benchmarks
and out-performs instruction tuned variants of similar 7B models such as
Gemma-7B. In addition to the IT benchmarks covered in Appendix O, In Table 29
we show that a small amount of instruction tuning provides large improvements
in “Extended" evals at the cost of a small degradation in “Core" and “MMLU"
evals. Notably we note that our GSM8k performance goes 2.5% to 52.5% which is
comparable to other similar language models that mixed IT data into
pretraining such as Gemma-7B.

Table 29: Effect of Instruction Tuning. We compare our final model with its instruction-tuned variant, both trained on a 4k context length. Including instruction tuning maintains performance on language tasks such as MMLU and results in considerable gains on 5-shot GSM8K with chain-of-thought, demonstrating the effectiveness of this training in performing complex reasoning. Model | Params | Tokens | Core | Extended | MMLU | GSM8K
---|---|---|---|---|---|---
DCLM-baseline | 7B | 2.5T | 56.0 | 43.7 | 63.9 | 2.1
DCLM-baseline-IT | 7B | 2.58T | 55.0 | 46.5 | 62.9 | 52.5

####  Continual learning to extend context length

In this section, we present continual learning results for adapting the above
DCLM-baseline 7B model (with an original context length of 2048) to a context
length of 8192, similar to [179]. We follow the continual learning recipe
described in [75], loading the DCLM-baseline 7B checkpoint and warming up to a
maximum learning rate of $10^{-4}$ over 2000 steps, then annealing with a
cosine schedule to $10^{-5}$. All other hyper-parameters remain the same as
original pretraining. The global batch size remains $2^{22}$ tokens per
optimization step. We employ a variable sequence length curriculum as in
[126], including batches of sequences ranging from 64 to 8192 in length. For
this continual learning stage, we train with a total of $\sim 120$B tokens
randomly sampled from the main dataset and distributed as follows among
different sequence lengths:
$64:2^{33},128:2^{33},256:2^{33},512:2^{33},1024:2^{33},2048:2^{33},4096:2^{35},8192:2^{35}$.
We use the Grow-Linear curriculum (from short to long sequences) with 4 cycles
as described in [126]. As proposed by [125] and similar to [179] for long-
context continual learning, we increase the RoPE [152] base frequency from
10,000 to 100,000 during the continual learning stage for long context
adaptation. The average context length for 20-Docs and 30-Docs is $\sim 4k$
and $\sim 6k$, respectively. Hence, the original DCLM with context length of
2048 model has poor performance for these benchmarks.

We show that the above strategy results in similar performance on regular
evaluations as the starting checkpoint and significantly improves on the
multi-document question-answering evaluation. We use the evaluation setup
described in [95]: the context is filled with $k$ documents followed by a
question. We ensure that one of the $k$ documents includes the answer to the
question (a.k.a., golden document). We use $k=1,10,20,30$, and for each case,
we run the evaluation multiple times by changing the position of the golden
document in the context and report the average. Results are reported in table
30. We demonstrate that long context adaptation results in a checkpoint
(DCLM-8k) that matches the original model on regular evaluations and
significantly improves multi-document QA showing its long-context
capabilities.

Table 30: Regular and long-context evaluations for DCLM-Baseline 7B model, and
DCLM-8k 7B model that is adapted to 8192 context length through continual
learning for additional $\sim 120$B tokens.

Model | Params | Tokens | Context length | Regular Evaluations | Multi-Document Evaluations
---|---|---|---|---|---
Core | MMLU | Extended | 1-Doc | 10-Docs | 20-Docs | 30-Docs
Llama-2 | 7B | 2T | 4096 | 49.2 | 45.8 | 34.1 | 48.5 | 27.3 | 25.6 | NA
DCLM | 7B | 2.5T | 2048 | 56.0 | 63.9 | 43.7 | 72.0 | 43.4 | NA | NA
DCLM-8k | 7B | 2.6T | 8192 | 57.1 | 63.7 | 45.4 | 76.9 | 49.8 | 46.1 | 38.8

###  Appendix Q Account of compute costs

We note the compute cost of training runs for each competition scale in Table
1. In total, we estimate that our runs for DCLM sum up to approximately 1M
H100 hours. Our precise estimate from our experimental test bed is 772K H100
hours for training, but this is likely an underestimate due to additional
compute that was not tracked, such as due to training failures.

###  Appendix R Existing assets used

In this section, we describe the assets we use in our benchmark and their
associated licenses.

####  Evaluation data

Appendix G discusses all downstream tasks we use for our evaluation. Below we
mention them again, and specify their licenses.

  * •

The AGI Eval LSAT-AR dataset [194] is distributed under the MIT license as
indicated in https://github.com/zhongwanjun/AR-LSAT.

  * •

The ARC easy and ARC challenge datasets [40] are distributed under the
Creative Commons Attribution-Sharealike 4.0 International license as indicated
in https://allenai.org/data/arc.

  * •

We use a series of 6 datasets from Big-Bench [18] (1) QA Wikidata, (2) Dyck
languages, (3) Operators, (4) Repeat Copy Logic, (5) CS Algorithms, and (6)
Language Identification. They are distributed under the Apache 2.0 license as
indicated in https://github.com/google/BIG-bench/blob/main/LICENSE.

  * •

BoolQ [38] is distributed under the Creative Commons Share-Alike 3.0 license
as indicated in https://huggingface.co/datasets/google/boolq.

  * •

CommonsenseQA [153] is available through the official website https://www.tau-
nlp.org/commonsenseqa with no specific license attached.

  * •

COPA [136] is distributed under the BSD-2 clause license as indicated in
https://shorturl.at/t7I4k, though we note the original distribution website is
no longer available.

  * •

CoQA [134] contains several parts, each of which is distributed under its own
license, indicated here https://stanfordnlp.github.io/coqa/. Namely, the
authors mention that CoQA contains passages from seven domains and make five
of these public under the following licenses:

    * –

Literature and Wikipedia passages are shared under CC BY-SA 4.0 license.

    * –

Children’s stories are collected from MCTest which comes with MSR-LA license.

    * –

Middle/High school exam passages are collected from RACE which comes with its
own license.

    * –

News passages are collected from the DeepMind CNN dataset which comes with
Apache license.

  * •

HellaSwag [186] is distributed under the MIT license as indicated in
https://github.com/rowanz/hellaswag/blob/master/LICENSE.

  * •

Jeopardy [83] is available through
https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions, with no
specific license attached.

  * •

LAMBADA [116] is distributed under the Creative Commons Attribution 4.0
International license as indicated in https://zenodo.org/records/2630551.

  * •

OpenBookQA [108] is distributed under the Apache 2.0 license as indicated in
https://github.com/allenai/OpenBookQA/blob/main/LICENSE.

  * •

PIQA [23] is distributed under the (Academic Free License v. 3.0 as indicated
in https://github.com/ybisk/ybisk.github.io/tree/master/piqa.

  * •

SQuAD [133] is distributed under the CC-BY-SA-4.0 license as indicated in
https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-
by-sa-4.0.md.

  * •

The Winograd Schema Challenge [89] is distributed under the Creative Commons
Attribution 4.0 International License license as indicated in
https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html.

  * •

The Winogrande [140] is distributed under the Apache 2.0 license as indicated
in https://github.com/allenai/winogrande/blob/master/LICENSE.

  * •

We use a series of 4 additional tasks from the AGI Eval suite of datasets
[194] (1) LSAT-LR, (2) LSAT-RC, (3) SAT-En, and (4) SAT-Math. These suite is
distributed under the MIT license as indicated in
https://github.com/ruixiangcui/AGIEval/blob/main/LICENSE.

  * •

AQuA [93] is distributed under the Apache 2.0 license as indicated in
https://github.com/google-deepmind/AQuA/blob/master/LICENSE.

  * •

BBQ [117] is distributed under the CC-By-4 license as indicated in
https://github.com/nyu-mll/BBQ/blob/main/LICENSE.

  * •

We use a series of 9 additional datasets from Big-Bench [18]: (1) Conceptual
Combinations, (2) Conlang Translation, (3) Elementary Math QA, (4) Logical
Deduction, (5) Misconceptions, (6) Novel Concepts, (7) Strange Stories, (8)
Strategy QA, and (9) Understanding Fables. They are distributed under the
Apache 2.0 license as indicated in https://github.com/google/BIG-
bench/blob/main/LICENSE.

  * •

Enterprise PII classification [120] is distributed via
https://github.com/mosaicml/llm-foundry as indicated in
https://www.patronus.ai/announcements/patronus-ai-launches-enterprisepii-the-
industrys-first-llm-dataset-for-detecting-business-sensitive-information. LLM-
foundry itself is released under the Apache-2.0 license.

  * •

GPQA-main and GPQA-diamond [135] are distributed under the MIT license as
indicated in https://github.com/idavidrein/gpqa/blob/main/LICENSE.

  * •

GSM8K [41] is distributed under the MIT license as indicated in
https://github.com/openai/grade-school-math/blob/master/LICENSE.

  * •

LogiQA [94] is distributed in through the official public repository at
https://github.com/lgw863/LogiQA-dataset with no specific license attached.

  * •

Math QA [11] is distributed under the Apache 2.0 license as indicated in
https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md.

  * •

MMLU [72] is distributed under the MIT license as indicated in
https://github.com/hendrycks/test/blob/master/LICENSE.

  * •

PubMedQA [78] is distributed under the MIT license as indicated in
https://github.com/pubmedqa/pubmedqa/blob/master/LICENSE.

  * •

Simple arithmetic with spaces and without spaces [110] is distributed under
the Apache-2.0 through https://github.com/mosaicml/llm-foundry.

  * •

Social Interaction QA [141] is distributed by AllenAI under the CC-BY-4.0
license as indicated in https://allenai.org/data/socialiqa.

  * •

SVAMP [119] is distributed under the MIT license as indicated in
https://github.com/arkilpatel/SVAMP/blob/main/LICENSE.

  * •

Trivia QA [80] is distributed under the Apache 2.0 license as indicated in
https://github.com/mandarjoshi90/triviaqa/blob/master/LICENSE.

  * •

The Winogender male and Winogender female datasets [138] are distributed under
the MIT license as indicated in https://github.com/rudinger/winogender-
schemas/blob/master/LICENSE.

  * •

HumanEval [34] is distributed (both code and data) under the MIT license as
indicated in
https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md.

####  Raw sources

Our main external asset used in constructing DCLM-Pool and its filtered
version DCLM-baseline is Common Crawl [42]. In their Terms of Use
(https://commoncrawl.org/terms-of-use), they grant a limited, non-transferable
license to access and use their service, primarily for innovation, education,
and research, with several restrictions on usage. While being relatively
permissive, it does not conform to any specific common licenses and emphasize
that the usage must comply with local and international laws, and users must
respect third-party copyrights. We urge the user’s discretion in verifying
their use abide by these terms-of-use.

In addition to the above, as described in Sections 5, 4.5 and 4.4 and
Appendices L and O we make use of the following datasets:

  1. 1.

OpenHermes2.5 [157] for instruction finetuning and to train some of our
quality filters. While the authors do not provide a specific license and refer
users to determine the license by following the links for the subsets they
use777https://huggingface.co/datasets/teknium/OpenHermes-2.5/discussions/9#65f2a0254ab77537428cc000,
we note that the dataset is based in part on outputs from OpenAI models, and
thus cannot be used for training new models for commercial purposes.

  2. 2.

StarCoder [90] and StarCoder2 [101] are used for some of our ablations
(Section 4). While constructed from permissive data by extracting datasets
that mention permissive licenses (e.g. MIT, Apache 2.0), they involve various
licenses, and as described in the Terms of
Use888https://huggingface.co/datasets/bigcode/the-stack#terms-of-use-for-the-
stack, require the user to follow all terms-of-use and licenses of the
different datasets it comprises of.

  3. 3.

ProofPile2 [14] is used to scale up the dataset to the trillion tokens scale
(Section 5). The authors do not alter the licenses of underlying datasets and
ask users to follow guidelines and licenses as described in these datasets.

  4. 4.

GSM8k [41] was used in some of the ablations in Section 4 and follows the MIT
license.

  5. 5.

RedPajama [160] is used for ablations in Section 4.5 and Appendix L. Note that
we do not release models or datasets that include this data. RedPajama filters
the datasets it uses keeping only permissive licenses, and refers the user to
adhere to underlying licenses where appropriate, as described in
https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T.

  6. 6.

UltraFeedback [45] is used for instruction tuning and is under the MIT
License.

  7. 7.

Tulu V2 SFT mixture [76] is used for instruction tuning and is under the Open
Data Commons License Attribution family.

  8. 8.

CodeFeedback [192] is used for instruction tuning and is under the Apache 2.0
License.

  9. 9.

Nectar [195] is used for instruction tuning and is under the Apache 2.0
License.

  10. 10.

NoRobots [132] is used for instruction tuning and is under the Creative
Commons Attribution Non-Commercial 4.0.

  11. 11.

WildChat [190] is used for instruction tuning and is under the AI2 ImpACT
License - Low Risk Artifacts ("LR Agreement").

  12. 12.

WebInstruct [184] is used for instruction tuning and is under the Apache 2.0
License.

  13. 13.

StarCoder2-Self-OSS-Instruct [169] is used for instruction tuning and is under
the Open Data Commons License Attribution family.

####  Libraries

The main libraries used in our benchmark pipeline are:

  1. 1.

transformers uses the Apache 2.0 License.999
https://github.com/huggingface/transformers/blob/main/LICENSE

  2. 2.

PyTorch uses a similar license to the 3-caluse BSD, and is defined in
https://github.com/pytorch/pytorch/blob/main/LICENSE.

  3. 3.

OpenLM [70] which is provided with MIT
license.101010https://github.com/mlfoundations/open_lm/blob/main/LICENSE

  4. 4.

llm-foundry uses the Apache 2.0 License.111111https://github.com/mosaicml/llm-
foundry/blob/main/LICENSE

  5. 5.

ChatNoir Resiliparse uses the Apache 2.0
License.121212https://github.com/chatnoir-eu/chatnoir-
resiliparse/blob/develop/LICENSE

  6. 6.

BFF uses the Apache 2.0
License.131313https://github.com/allenai/bff/blob/main/LICENSE

  7. 7.

Ray uses the Apache 2.0 License.141414https://github.com/ray-
project/ray/blob/master/LICENSE

  8. 8.

slurm is accessible under the GPL
license.151515https://github.com/SchedMD/slurm/tree/master?tab=License-1-ov-
file

  9. 9.

fastText [81] uses the MIT
License.161616https://github.com/facebookresearch/fastText/blob/main/LICENSE

  10. 10.

nltk uses the Apache 2.0
License.171717https://github.com/nltk/nltk/blob/develop/LICENSE.txt

  11. 11.

langdetect uses the Apache 2.0
License.181818https://github.com/Mimino666/langdetect/blob/master/LICENSE

In addition, the installation may include common ML and web development
packages, and we urge commercial users to verify their endowment to refrain
from license violations.

###  Appendix S Datasheet

####  Motivation

  1. Q1

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.

     * •

The purpose of DCLM and the associated DCLM-Pool and DCLM-baseline datasets is
to enable the study of what makes a strong pretraining dataset for large
language models. These models are transformative to society and act as the
foundation of numerous applications, but they are often associated with steep
costs. While prior work explores many curation techniques, it is often coupled
with various architectural and training design choices and evaluated in
different settings, making controlled comparison nearly impossible. This slows
down progress and forces a lot of duplicate work between research teams. Prior
work mainly focuses on data curation in the context of supervised datasets and
smaller scales (see Section 2 and Appendix B). In our initial release of DCLM,
we focus on 53 downstream language understanding tasks that also include
reasoning abilities, math, code, and more. For details see Section 3.5 and
Appendix G.

  2. Q2

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

     * •

DCLM-Pool and DCLM-baseline were created by a group of researchers with the
following affiliations, listed in alphabetical order: Allen Institute for
Artificial Intelligence, Apple, Carnegie Mellon University, Columbia
University, Contextual AI, Cornell University, DatologyAI, Harvard University,
Hebrew University, Juelich Supercomputing Center, Research Center Juelich,
SambaNova Systems, Stanford University, SynthLabs, Tel Aviv University, Toyota
Research Institute, TU Munich, University of California, Los Angeles,
University of California, Santa Barbara, University of Southern California,
The University of Texas at Austin, University of Washington.

  3. Q3

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

     * •

Funding for this research was generously provided by the University of
Washington, the University of Texas (Austin), the Institute for Foundations of
Machine Learning (IFML), and Open Philanthropy.

  4. Q4

Any other comments?

     * •

We anticipate that DCLM benchmark, tooling and pools will drive data-centric
research in ML and AI, fostering the development of the next generation of
web-scale datasets, enhancing model abilities, lowering training costs and
develop knowledge sharing across research teams.

####  Composition

  1. Q5

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

     * •

Each instance represented a web-crawled page (document). It contains the URL
and the corresponding HTML content. Each sample is also tagged with metadata
about its crawl time and additional information such as the detected language,
for processed instances such as those in DCLM-baseline. Additional information
can be found in Appendix E.

  2. Q6

How many instances are there in total (of each type, if appropriate)?

     * •

DCLM-Pool contains $\sim$200B documents, all of which are of the same
instance, and comes from hundreds of millions of different sources. The subset
DCLM-baseline contains approximately 3B documents.

  3. Q7

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

     * •

DCLM-Pool is an unfiltered web-text corpus comprised of all Common Crawl data
prior to 2023. As such, it represent the full breadth of possible instances
from this source. However, we note that Common Crawl does not cover the entire
web data, due to reach and compute limitations for instance. For our DCLM-
baseline, we use various filtering and deduplication strategies as described
in Section 4 in the explicit attempt to improve its quality for preatining,
thus removing low-quality instances, and in doing so, becoming non-
representative of the full set of instances. For a complete treatment and
visualization of our data processing funnel, see Sections 4, 4.3 and 4.2 and
Appendix E.

  4. Q8

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.

     * •

Each sample contains a web-page url for and the extracted HTML content
associated with. Additionally, each sample contains metadata fields shown in
Table 10 (e.g., WARC-Type, WARC-date, Content-Type etc.).

  5. Q9

Is there a label or target associated with each instance? If so, please
provide a description.

     * •

We do not provide any labels associated with the samples, as they are used to
pretrain language models by performing self-supervised next-token prediction.

  6. Q10

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

     * •

No, each sample is the full text as extracted from the HTML content, and the
respective metadata.

  7. Q11

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

     * •

No, the dataset is released as it is with no explicit attempt to establish
relationships between instances. Some links may be drawn based on metadata
information such the as the source URL, but we do not deliberately form any
such connections.

  8. Q12

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

     * •

No. The evaluation procedure is made of tasks as described in Section 3.5. We
also attempt to prevent test set contamination in as described in Section 4.6
and Appendix N.

  9. Q13

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

     * •

DCLM-Pool is based on Common Crawl, which can be thought of as a snapshot of
the internet at a given time. Hence, there can be considerable noise (e.g.,
placeholder text, broken links, failed extraction of HTML content, duplicate
data, etc.)

  10. Q14

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.

     * •

Each sample is associated with a URL that links other external resources on
the internet with no guarantee that the resources will exist in perpetuity or
that that the resources will not change. However, the dataset itself contains
already extracted HTML content and is thus self-contained for the purposes of
this benchmark as described in Appendix C.

  11. Q15

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.

     * •

The dataset consists of data that was publicly accessible on the internet at
the time of collection. However, it is possible that some of the data may
include confidential information, such as private data that is unintentionally
or maliciously made public.

  12. Q16

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

     * •

Given the diverse backgrounds of individuals worldwide, it is highly plausible
that DCLM-Pool contains content that could be upsetting. Since our dataset
consists of text scraped from the internet, it may include hateful, racist,
sexist, and other offensive or toxic material. We consider the dataset a
research artifact and hope future work will critically examine DCLM-Pool to
develop improved safety filters. Our processed dataset, DCLM-baseline does
apply a reproduction of the content-filtering from RefinedWeb. This involves
url-based filtering using a domain banlist curated from Blacklists
UT1191919https://dsi.ut-capitole.fr/blacklists/index_en.php and a set of
banned url-substrings curated from the LDNOOBW
202020https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-
Bad-Words/blob/master/en list. While these banlists are extensive, they may
still let in content that is harmful.

  13. Q17

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

     * •

As a snapshot of the Internet, the dataset may include information about
people which they shared intentionally or that was shared about them without
permission.

  14. Q18

Does the dataset identify any subpopulations (e.g., by age, gender)?

     * •

Our DCLM-Pool does not explicitly identify subpopulations in its metadata, as
it is unclear how one can define such division over raw text data from the
web.

  15. Q19

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

     * •

As names and other identifiers are frequent in web data, it is likely that
some content can be linked back to specific individuals. However, in most
public sites which Common Crawl scrape people publish such information
willingly, knowing it will be visible and public.

  16. Q20

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,