#  SlimPajama-DC: Understanding Data Combinations for LLM Training

Zhiqiang Shen† Tianhua Tao†,‡ Liqun Ma† Willie Neiswanger§
Zhengzhong Liu† Hongyi Wang♮ Bowen Tan♮ Joel Hestness♯
Natalia Vassilieva♯ Daria Soboleva♯ Eric Xing†
†MBZUAI ‡UIUC §Stanford University ♮CMU ♯Cerebras Systems

( )

###### Abstract

This paper aims to understand the impacts of various data combinations (e.g.,
web text, wikipedia, github, books) on the training of large language models
using SlimPajama. SlimPajama [33] is a rigorously deduplicated, multi-source
dataset, which has been refined and further deduplicated to 627B tokens from
the extensive 1.2T tokens RedPajama dataset [7] contributed by Together. We’ve
termed our research as SlimPajama-DC, an empirical analysis designed to
uncover fundamental characteristics and best practices associated with
employing SlimPajama in the training of large language models. During our
research with SlimPajama, two pivotal observations emerged: (1) Global
deduplication vs. local deduplication. We analyze and discuss how global
(across different sources of datasets) and local (within the single source of
dataset) deduplications affect the performance of trained models. (2)
Proportions of high-quality/highly-deduplicated multi-source datasets in the
combination. To study this, we construct six configurations of SlimPajama
dataset and train individual ones using 1.3B Cerebras-GPT [11] model with
Alibi [28] and SwiGLU [32]. Our best configuration outperforms the 1.3B model
trained on RedPajama using the same number of training tokens by a significant
margin. All our 1.3B models are trained on Cerebras 16$\times$ CS-2 cluster
with a total of 80 PFLOP/s in bf16 mixed precision. We further extend our
discoveries (such as increasing data diversity is crucial after global
deduplication) on a 7B model with large batch-size training. Our models and
the separate SlimPajama-DC datasets are available at: link1 and original
SlimPajama is at: link2.

###### Contents

  1. 1 Introduction
  2. 2 Dataset Overview
    1. 2.1 Number of Tokens
    2. 2.2 Dataset Token Frequency Statistics
    3. 2.3 Dataset Processing Procedure
      1. 2.3.1 Low-length Document Filtering
      2. 2.3.2 Global Deduplication
  3. 3 Dataset Combination Configurations
    1. 3.1 SlimPajama
    2. 3.2 RefinedWeb
  4. 4 Network Architecture and Training Details
    1. 4.1 Network Architecture
    2. 4.2 Training Details
  5. 5 Results and Analysis
    1. 5.1 Huggingface Leaderboard Evaluation with Harness
    2. 5.2 More Evaluations
    3. 5.3 Training Loss
  6. 6 Application: Large Batch-size Training on 7B
    1. 6.1 7B Training Data Combination
    2. 6.2 7B Model Training Configurations
    3. 6.3 Fast Training with Large Batch-size
    4. 6.4 Progressive Training on Weight Decay
    5. 6.5 Results of Pre-training and Instruction Tuning
  7. 7 Related Work
    1. 7.1 RedPajama, SlimPajama and Others.
    2. 7.2 Data Processing and Optimization Approaches
    3. 7.3 Data Combination for Training Large Language Models
    4. 7.4 Large Batch Training for Large Language Models
  8. 8 Conclusion
  9. A Data Proportion Details
  10. B MMLU

##  1 Introduction

The success of modern large-scale models is deeply rooted in their training
data. For large language models, the emphasis is not merely on generic text
but on “diverse text”. To guarantee the model’s linguistic expertise and its
comprehensive understanding of the world, this text must span a broad spectrum
of domains, genres, languages, and more. Consequently, the composition of the
pretraining data domains, such as Github, Wikipedia, books, and web text like
CommonCrawl, plays a critical role in the performance of large language
models. In our research, we delve into the domain/source weightings of
training data. Leveraging SlimPajama-DC, we investigate two primary areas: (1)
global-level and local-level deduplication, and (2) the efficacy of various
combinations of thoroughly deduplicated datasets. The first emphasis basically
encourages the model to be trained on all sources as no cross-domain overlaps
inside, and the second helps us understand how to manage the integration and
proportions of diverse domains, especially as datasets for LLM training
continue to expand in variety.

Generic Deduplication. Multi-source datasets often combine data from various
origins, each with its unique distribution of information. When training large
language models, handling data redundancy is critical to ensure that the model
generalizes well and does not exhibit undue biases, making training faster and
more efficient. Highly deduplicated datasets ensure that the model isn’t
repeatedly exposed to the same or very similar data points, making the
training more efficient. Redundant data can slow down convergence and might
make the model overfit to frequently seen patterns. Deduplication helps in
efficient utilization of the model’s capacity. In general, deduplication is
the process of removing duplicate data to address this redundancy.

Global Deduplication vs. Local Deduplication. The global deduplication process
removes duplicates from the entire combined datasets. When we’re using data
from multiple sources, there might be overlaps across sources. Global
deduplication identifies and removes these overlapping instances irrespective
of their source. In local deduplication, duplicates are removed within each
individual source dataset before merging them. However, if two source datasets
have overlapping data, those duplicates will still be present in the final
combined dataset since deduplication was only done locally within each
dataset. In most current open-source LLM training data [7, 36, 38], only local
deduplication is performed within each data source, which neglects the
redundancy across the different sources. Given the effects, global
deduplication performed in SlimPajama is generally preferable for training
large language models, especially when using multi-source datasets. It ensures
a balanced representation of information and prevents the pitfalls associated
with data redundancy. However, more hardware memory is naturally required by
this strategy.

Different Combinations of Highly-deduplicated Datasets. A model trained on
diverse data is more likely to generalize well across various tasks. It’s
exposed to a wider range of vocabulary, syntax, and semantics, enabling it to
handle a broad scope of queries. If diverse sources are chosen such that they
represent different cultures, beliefs, and demographics, the model might be
more balanced and less prone to biases. However, if many sources share common
biases, the final dataset might amplify them. Different sources can provide
both a breadth and depth of knowledge on various topics. Combining a technical
dataset with a general news dataset, for example, would allow the model to
understand both in-depth technical details and broad general knowledge. It’s
crucial to note that data quality often outweighs the quantity. In this work,
we aim to shed light on this fascinating perspective of comprehensive data
combination on SlimPajama.

Specialization vs. Generalization Trade-off. In general, combining many
specialized datasets can lead to a jack-of-all-trades model, which might not
be as adept at specific tasks as a model trained on a specialized dataset.
While the model can tackle a wide range of tasks, it might not have the depth
of understanding that a specialized model might have for a particular domain.
In this study, we also explore specialization and generalization ability using
both individual and combined data sources.

The remainder of this paper is organized as follows. In Section 2, we
elaborate the details of dataset statistics, token distributions, and data
processing procedure. Section 3 describes dataset combination configurations
for this SlimPajama-DC study. Our model architecture and training details are
provided in Section 4, followed by the results and analysis in Section 5 on
the range of various tasks in the zero- and few-shot settings. Section 6
presents an application of efficient Large Batch-size (LBS) training on a 7B
model. Section 7 reviews related work and Section 8 concludes this study.

##  2 Dataset Overview

###  2.1 Number of Tokens

SlimPajama has a total of 627B tokens across different domains, as shown in
Table 1. It includes validation and test sets with 500M tokens each, and these
have been cleaned to ensure no overlap with the training data. For the
SlimPajama-DC study, our entire training dataset for each configuration
contains 330B tokens after tokenization which is carefully selected from the
original SlimPajama dataset. We tested different sampling strategies for
different domains of our training data: (1) each token is trained only once
during training, such as Commoncrawl, and (2) we perform more than one epoch
for training on particular sources, such as the Wikipedia and Github domains.
The detailed domain source proportions of various combinations are shown in
Table 3.

Dataset | SlimPaj. | RedPaj. | LLaMA-1 | RefinedWeb | GPT3 | MassiveText
---|---|---|---|---|---|---
Commoncrawl | 52.2% | 72.6% | 67.0% | 100% | 60.0% | 0.0%
C4 | 26.7% | 14.4% | 15.0% | 0.0% | 0.0% | 10.0%
GitHub | 5.2% | 4.9% | 4.5% | 0.0% | 0.0% | 3.0%
Books | 4.2% | 2.1% | 4.5% | 0.0% | 16.0% | 27.0%
ArXiv | 4.6% | 2.3% | 2.5% | 0.0% | 0.0% | 0.0%
Wikipedia | 3.8% | 2.0% | 4.5% | 0.0% | 3.0% | 2.0%
StackExchange | 3.3% | 1.7% | 2.0% | 0.0% | 0.0% | 0.0%
WebText2 | 0.0% | 0.0% | 0.0% | 0.0% | 22.0% | 0.0%
MassiveWeb | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 48.0%
News | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 10.0%
Total tokens | 637B | 1.2T | 1.0/1.4T | 600B | 300B | 300B

Table 1: Data source proportions for various datasets.

###  2.2 Dataset Token Frequency Statistics

(a) All Tokens

(b) Tokens Composed of Letters

(c) Top 1000 Tokens

(d) Numbers and Commonly Used Operators

(e) Whitespace Tokens

(f) Non-Alphanumeric Tokens

Figure 1: Confusion matrix using KL divergence between the distributions of
token statistics for different datasets.

To examine the similarity between various datasets in SlimPajama, we calculate
the KL divergence between two domain distributions of token counts from
different datasets, as shown in Fig. 1a. Given that distinct datasets may
emphasize dissimilar token types, we subsequently delve into the differences
in the distribution of these datasets across token subsets exhibiting distinct
characteristics: (1) Tokens exclusively comprising letters (Fig. 1b); (2) The
union set of tokens with the top 1000 frequencies on each dataset (Fig. 1c);
(3) Numbers and commonly used operators, like ‘30’, ‘+’ and ‘=’ (Fig. 1d); (4)
Whitespace Tokens, like ‘$\backslash$n$\backslash$n’ and ‘$\backslash$t’ (Fig.
1e); (5) Non-alphanumeric tokens, like ‘#’ and ‘====’ (Fig. 1f).

There exists a degree of similarity in the distribution of different token
subsets among RefinedWeb, Book, C4, and CommonCrawl, as well as between Github
and StackExchange. Notably, when it comes to the distribution of non-
alphanumeric tokens, Arxiv differs significantly from most datasets. While on
the distribution of whitespace tokens, Refinedweb shows notable distinctions
in comparison to Github and StackExchange. Among numbers and commonly used
operators, the distribution of all datasets is relatively consistent.

###  2.3 Dataset Processing Procedure

Figure 2: SlimPajama preprocessing pipeline.

SlimPajama was created by filtering low-length documents and applying
MinHashLSH deduplication to the 1.2T token RedPajama dataset to reduce it to
627B tokens. RefinedWeb [27] shows that training on deduplicated data improves
training compute efficiency and decreases the chance of LLMs generating
memorized text from the dataset. By removing duplicate and low-length
examples, it ultimately improves the training compute efficiency and model
performance. The overview of SlimPajama preprocessing pipeline is shown in
Fig. 2 and the preprocessing code is under
https://github.com/Cerebras/modelzoo.

Data source | Document filter rate | Byte duplication rate
---|---|---
Commoncrawl | 0.02% | 63.76%
C4 | 4.7% | 6.85%
GitHub | 0.0% | 46.16%
Books | 0.0% | 2.01%
ArXiv | 0.62% | 0.06%
Wikipedia | 0.0% | 2.24%
StackExchange | 0.32% | 0.20%
Total | 1.86% | 49.60%
Table 2: Document low-length filter rates and data source byte duplication
rates.

####  2.3.1 Low-length Document Filtering

Additional global filtering is performed to remove short, low-quality
documents. After removing punctuation, consecutive spaces, newlines, tabs, and
leading or trailing escape characters, documents with less than 200 characters
were further filtered out. These documents typically contain only metadata and
no useful information. A low-length filter was applied to every corpora other
than Books and GitHub where it was found useful for short documents. The
percentage of documents filtered out from each corpus within the SlimPajama
dataset is detailed in Table 2. In total, this additional step removed 1.86%
of the documents.

####  2.3.2 Global Deduplication

When building SlimPajama, it is observed that every corpus included in it
contained duplicates with the most significant duplication found in
CommonCrawl and GitHub. RefinedWeb [27] also found similar rates of
deduplication in the CommonCrawl data. It is most common to perform
deduplication within each dataset source separately [36, 7, 42, 13] to reduce
implementation complexity and meet resource constraints. This local
deduplication approach does not have the ability to remove overlap between
data sources which can be significant for web-scraped data. Instead, global
deduplication removes duplication within and between each data source.
Following [4, 27, 1, 31], global-level deduplication is performed using
MinHashLSH algorithm. To facilitate global deduplication efforts and
reproducibility for other researchers, a tool designed for scalable
performance is offered under the above link.

Specifically, global MinHashLSH deduplication is performed using a Jaccard
similarity threshold of 0.8, document signatures constructed with preprocessed
lowercase 13-grams, and schema following [22]. To unify a representation of
the same content, punctuation, consecutive spaces, newlines, tabs, and leading
or trailing escape characters are removed. The level of deduplication
performed per data source is presented in Table 2. The initial implementation
of MinHashLSH did not scale to trillion token datasets like RedPajama without
running out of memory. This is overcome by optimizing the memory usage and
parallelization to perform deduplication on 64 CPU cores with 1.4TB GB peak
memory usage, which can be easily decreased by creating multiple MinHashLSH
objects to query.

##  3 Dataset Combination Configurations

###  3.1 SlimPajama

Combination Strategies. As shown in Table 3, the adjusted domain weights
establish a new training distribution. Using this distribution, we adopt a
standard training approach to learn a consistent model architecture. This
architecture remains unchanged across various domain weights and is trained
using data from diverse combination distributions. Across different setups, we
maintain the total training tokens to be the same. Our examination of domain
weights in large language model training focuses on three main areas: 1)
Incrementally increasing the diversity of source combinations, as seen in
configurations 1, 2, and 3. 2) With consistent data sources, we explore
varying domain proportions as presented in configurations 2, 4, and 5. 3) We
assess the significance of individual domain sources concerning the final
model’s performance. Note that given the minimal impact of ArXiv and
StackExchange, we have opted to omit them from the ablations in configuration
3 to conserve training resources and keep relatively sufficient training
tokens for CommonCrawl. The detailed configurations are as follows:

  * •

Configuration-1: 330B CommonCrawl

  * •

Configuration-2: 300B CommonCrawl + 30B Github

  * •

Configuration-3: 250B CommonCrawl + 30B Github + 26B Books + 24B Wikipedia

  * •

Configuration-4: 250B CommonCrawl + 80B Github (adjust sampling proportion)

  * •

Configuration-5: 250B CommonCrawl + 80B Wikipedia (adjust sampling proportion)

  * •

Configuration-6: 330B RefinedWeb CommonCrawl

###  3.2 RefinedWeb

RefinedWeb [27] is a massive English web dataset that is constructed using
rigorous filtering and extensive deduplication of CommonCrawl. We use it as
the comparison to our SlimPajama-DC CommonCrawl-only training.

| sub dataset | DC-1 | DC-2 | DC-3 | DC-4 | DC-5 | DC-6
---|---|---|---|---|---|---|---
SlimPajama | Commoncrawl | 100.0% | 90.9% | 75.8% | 75.8% | 75.8% | 0.0%
C4 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0%
GitHub | 0.0% | 9.1% | 9.1% | 24.2% | 0.0% | 0.0%
Books | 0.0% | 0.0% | 7.9% | 0.0% | 0.0% | 0.0%
ArXiv | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0%
Wikipedia | 0.0% | 0.0% | 7.3% | 0.0% | 24.2% | 0.0%
StackExchange | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0%
RefinedWeb | Commoncrawl | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 100.0%
Total (Tokens) |  | 330B | 330B | 330B | 330B | 330B | 330B

Table 3: Six configurations of sub-dataset combinations in SlimPajama.

##  4 Network Architecture and Training Details

###  4.1 Network Architecture

Cerebras-GPT Architecture [11]. Cerebras-GPT architecture shares similarities
with those built on GPT-3 [4], particularly in the use of an autoregressive
transformer decoder. However, a key difference lies in the attention mechanism
employed. While GPT-3 utilizes a mix of dense and sparse-banded attention,
Cerebras-GPT consistently uses dense attention across all decoder blocks. In
terms of model dimensions, we either adhere to an aspect ratio of
approximately 80 ($\text{d}_{\text{model}}$/$\text{n}_{\text{layers}}$) or
maintain dimensions that are congruent with GPT-3 models. Additionally, all of
our models are trained to handle a maximum sequence length of 2,048 tokens.
The detailed architecture is shown in Table 4.

Alibi [28]. Alibi introduces a more streamlined and efficient positional
approach called Attention with Linear Biases. Rather than adding positional
embeddings to word embeddings, ALiBi applies a bias to query-key attention
scores, penalizing them based on their distance.

SwiGLU [32]. SwiGLU is an activation function which is a variant of GLU [9].
The formulation is as follows:

$\operatorname{SwiGLU}(x,W,V,b,c,\beta)=\operatorname{Swish}_{\beta}(xW+b)\otimes(xV+c)$
(1)

where $x$ is a vector of the hidden representation at a particular position in
the sequence. $W,V,b,c$ are the matrices and bias vectors, respectively.

Model | n_params | n_layers | d_model | n_heads | d_heads | batch size | learning rate
---|---|---|---|---|---|---|---
GPT-3 XL | 1.3B | 24 | 2,048 | 24 | 128 | 1M | 2.0$\times$10-4
Our DC | 1.3B | 24 | 2,048 | 24 | 128 | 2M | 1.2$\times$10-2
GPT-3 | 6.7B | 32 | 4,096 | 32 | 128 | 2M | 1.2$\times$10-4
LLaMA | 6.7B | 32 | 4,096 | 32 | 128 | 4M | 3.0$\times$10-4
Our LBS | 6.7B | 32 | 4,096 | 32 | 128 | 14.3M | 1.8$\times$10-4

Table 4: Detailed model sizes, architectures, and optimization hyper-
parameters. Our LBS model details are presented in Sec. 6.

###  4.2 Training Details

Tokenizer. We use an adapted GPT-NeoX [2] BPE-based tokenizer similar to that
used in GPT-2 for all of our experiments, which has a vocabulary size of
50,277. Our entire training dataset for each configuration contains 330B
tokens after tokenization, and each model takes about 2.5 days on Cerebras
16$\times$ CS-2S cluster.

Optimizer. We employ the AdamW optimizer [26] to train our models, adopting
these specific hyper-parameters: $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.95, and
eps = 1.0e-08. Our chosen learning rate follows a linear scheduler,
culminating in a final learning rate that’s 10% of its peak value.
Additionally, we apply a weight decay of 0.1, limit the gradient using a clip
value of 1.0, and implement a 150-step warmup.

Other Hyperparameters. In our model, the filter size is 5,461, hidden size is
2,048 and attention dropout rate is 0. SwiGLU is used as the nonlinearity and
alibi is used for position embedding. Mixed precision and bfloat16 are
employed during model training. More hyperparameters are shown in Table 4.

##  5 Results and Analysis

This section presents the analytical experiments and results on different
combinations of SlimPajama. We first discuss the results following Huggingface
Leaderboard Evaluation. Then, we demonstrate the importance of global
deduplication and a diverse range of data sources in enhancing LLM’s
performance by conducting additional comprehensive evaluations across various
topics. Finally, we visualize the training loss curves of different data
domain combinations and provide insights on how they connect to the models’
performance.

###  5.1 Huggingface Leaderboard Evaluation with Harness

Following the Huggingface Leaderboard Evaluation [12], we also assess our
models on four key benchmarks using the Eleuther AI Language Model Evaluation
Harness [14]. This unified framework facilitates the evaluation of generative
language models across a broad scope of tasks. Specifically, our tests
comprised:

1) AI2 Reasoning Challenge (25-shot) [6]: This entails a series of grade-
school level science questions.

2) HellaSwag (10-shot) [41]: This benchmark gauges commonsense inference.
While straightforward for humans, with an average accuracy of 95%, it poses
challenges for state-of-the-art models.

3) MMLU (5-shot) [16]: Designed to assess a text model’s multitask
proficiency, this test spans 57 diverse tasks, including elementary
mathematics, US history, computer science, law, among others.

4) TruthfulQA (0-shot) [23]: This evaluates a model’s inclination to echo
inaccurate information frequently encountered online. However, it’s pertinent
to note that within the Harness, TruthfulQA is essentially a 6-shot task, as
it consistently commences with six examples, even when initialized with zero
for the number of few-shot examples.

As shown in Table 5, with the exception of DC-5, our average results are all
better than RedPajama-1.3B which is also trained on 330B tokens. Among our
combinations, the DC-1 (which relies solely on SlimPajama Commoncrawl)
achieves the highest scores for ARC and MMLU among all tested configurations.
Yet, its performance on TruthfulQA ranks at the bottom. On the other hand,
DC-3 obtains the top average accuracy across all SlimPajama data combinations,
while DC-6 stands out with the best results on HellaSwag and superior average
performance across the board. A potential strategy to harness the strengths of
each configuration might involve a sequential training process on DC-1, DC-3,
and DC-6.

Furthermore, SlimPajama is built using global deduplication across all
sources. This suggests that merging all domains typically yields better
results than selective combinations, given the absence of overlaps among
different domain datasets. This also highlights the importance of global
deduplication and a diverse range of data sources in enhancing LLM overall
performance.

Model | Average | ARC | HellaSwag | MMLU | TruthfulQA
---|---|---|---|---|---
Cerebras-GPT-1.3B [11] | 33.5 | 26.3 | 38.5 | 26.6 | 42.7
GPT-neo-1.3B [3] | 36.0 | 31.2 | 48.5 | 24.8 | 39.6
RedPajama-1.3B [7] | 38.0 | 37.2 | 55.8 | 24.9 | 34.3
DC-1-1.3B | 38.5 | 36.3 | 56.0 | 27.0 | 34.8
DC-2-1.3B | 38.4 | 33.9 | 55.5 | 25.7 | 38.6
DC-3-1.3B | 38.6 | 34.7 | 56.0 | 25.6 | 38.0
DC-4-1.3B | 38.5 | 35.2 | 54.7 | 25.7 | 38.3
DC-5-1.3B | 37.6 | 33.4 | 53.3 | 26.0 | 37.6
DC-6-1.3B | 41.0 | 35.1 | 64.7 | 26.2 | 37.9

Table 5: Results of six dataset combination configurations following
Huggingface Leaderboard Evaluation [12] with Harness [14].

###  5.2 More Evaluations

As shown in Table 6, we present additional evaluations across various domains
to investigate the fine-grained capabilities offered by different data
combinations. Except for DC-6 (model trained on RefinedWeb data),
incorporating more sources, such as DC-3, typically leads to improved average
performance.

Upon analysis, we find that specific mixtures excel in particular evaluation
benchmarks. For example, DC-1 obtains the highest accuracy in the arc
challenge and race. Meanwhile, DC-3 outperforms others in the wsc273, swag,
and pawsx, and DC-5 emerges as the top performance in the xstory cloze
evaluation. Moreover, all of our configurations are superior in the average
performance over the comparisons of GPT-neo-1.3B [3] and RedPajama-1.3B [7].

Eval |  Neo [3] |  RedPaj. [7] | DC-1 | DC-2 | DC-3 | DC-4 | DC-5 | DC-6 | LBS
---|---|---|---|---|---|---|---|---|---
| 1.3B | 1.3B | 7B
humaneval (p@1) | - | - | - | - | - | - | - | - | 9.5
bigbench* | 32.4 | 33.1 | 33.8 | 32.0 | 34.0 | 34.5 | 33.0 | 33.8 | 35.0
arc_easy | 61.1 | 66.7 | 66.1 | 66.9 | 66.5 | 66.4 | 65.5 | 66.8 | 74.7
arc_challenge | 25.9 | 33.5 | 36.3 | 33.9 | 34.7 | 35.2 | 33.4 | 35.1 | 44.3
boolq | 62.0 | 55.6 | 63.4 | 65.6 | 62.5 | 64.2 | 50.6 | 61.7 | 66.9
PIQA | 71.1 | 72.4 | 70.8 | 69.2 | 70.7 | 68.6 | 67.8 | 75.7 | 77.4
race | 34.1 | 34.4 | 37.3 | 36.7 | 37.3 | 36.5 | 34.6 | 36.6 | 38.2
winogrande | 54.9 | 60.5 | 60.3 | 59.7 | 59.8 | 60.1 | 60.5 | 61.2 | 64.4
openbookqa | 33.6 | 33.0 | 35.6 | 34.8 | 34.0 | 34.0 | 34.4 | 37.4 | 39.8
copa | 69.0 | 77.0 | 70.0 | 73.0 | 75.0 | 74.0 | 70.0 | 81.0 | 86.0
wsc273 | 75.1 | 78.0 | 76.2 | 78.0 | 81.0 | 76.9 | 76.6 | 79.5 | 85.0
swag | 67.8 | 68.8 | 69.2 | 68.5 | 70.1 | 67.8 | 68.3 | 70.0 | 73.8
pawsx* | 50.6 | 51.5 | 51.4 | 52.3 | 53.1 | 52.2 | 50.5 | 50.8 | 54.7
xstory_cloze* | 51.1 | 51.5 | 51.0 | 51.3 | 52.0 | 51.5 | 52.2 | 51.6 | 55.3
Average | 53.0 | 55.1 | 55.5 | 55.5 | 56.2 | 55.5 | 53.6 | 57.0 | 61.2

Table 6: Results of six dataset combination configurations of 1.3B models and
our LBS-7B model details are presented in Sec. 6. Bigbench is evaluated under
3-shot using the average of multiple choice grade. Arc_easy and arc_challenge
are evaluated using 5-shot, 25-shot, and 25-shot, respectively. All other
evaluation benchmarks are tested on 0-shot. * represents the results are
averaged across multiple sub-items inside each benchmark dataset.

Risk of random guessing score on 1.3B models. It is widely recognized that
small models, such as the 1.3B variant, may struggle to achieve satisfactory
predictions on specific benchmarks like MMLU. Their results could resemble
random choices, not truly capturing the model’s actual capabilities. To more
accurately showcase a model’s true potential and reflect the ability of
different data combinations, we introduce a novel metric RRGS (risk of random
guessing score) to evaluate the degree of random guessing. Since 25% in MMLU
represents the baseline score for a guess, this metric evaluates the variance
using average $\ell_{1}$ distance around this base value across all sub-items.
A larger variance would suggest a reduced likelihood of predictions resulting
from mere chance. Given a MMLU score vector $X$ of length $N$ with sub-item
scores $s_{1},s_{2},\dots,s_{n}$, RRGS can be formulated as:

$\text{RRGS}=1-\frac{1}{N}\sum_{i=1}^{N}(|s_{i}-0.25|)$ (2)

where $i$ is the index of sub-item in MMLU and $N$ is the number of items of
MMLU. This metric utilizes the probabilities of variance to baseline 25%,
aiming to assess the extent to which a model’s prediction resembles random
guessing on the MMLU benchmark. The metric has three variations: (1) Consider
only items with scores exceeding 25%, i.e., $i\in{\text{\\{positive item
set\\}}}$. (2) Focus solely on items with scores less than 25%, i.e.,
$i\in{\text{\\{negative item set\\}}}$. (3) Include all items and sum them up.
The results are shown in Table 7. Generally, a model with a higher MMLU
average score will have a low risk of random guessing probability.

It is also crucial to employ a broader and more diverse set of benchmarks,
such as in Table 6. Additionally, for a detailed understanding, we have
cataloged the complete MMLU results for every sub-item in Table 12. This
offers a lens into the knowledge assimilated by the pretrained models within
each sub-domain on this comprehensive benchmark.

| DC-1 | DC-2 | DC-3 | DC-4 | DC-5 | DC-6
---|---|---|---|---|---|---
MMLU | 0.27 | 0.257 | 0.256 | 0.257 | 0.260 | 0.262
$\text{RRGS}_{\text{pos}}$ | 0.964 | 0.964 | 0.968 | 0.965 | 0.970 | 0.963
$\text{RRGS}_{\text{neg}}$ | 0.974 | 0.973 | 0.975 | 0.974 | 0.969 | 0.973
$\text{RRGS}_{\text{all}}$ | 0.968 | 0.968 | 0.971 | 0.969 | 0.970 | 0.967

Table 7: Evlauation of random guessing probability on sub-items of MMLU.

###  5.3 Training Loss

Figure 3: Illustration of training loss curves. DC-2’s curve closely resembles
those of DC-3 and 5, so it has been excluded from the figure for clarity.

Fig. 3 presents the training loss curves for various data combinations, from
which several insights can be observed: 1) While DC-6 demonstrated the highest
average accuracy in our quantitative evaluations, its training loss was also
the most substantial. This suggests that a lower training loss doesn’t
necessarily correlate directly with superior model performance. 2) DC-4, with
a considerable portion of its data coming from code domain, exhibited the
lowest training loss. This implies that as the amount of code in training
increases, the training loss diminishes. 3) The training loss values for other
combinations appeared to be relatively consistent with one another.

##  6 Application: Large Batch-size Training on 7B

###  6.1 7B Training Data Combination

Our 7B large batch size (LBS) training dataset is primarily based on
Slimpajama, however, to obtain a sufficient proportion of web text, we have
incorporated additional web data from the Commoncrawl corpus in RedPajama. We
have also adjusted the proportions of various data sources in line with our
1.3B model training. For instance, we elevate the sampling frequency of Github
and Wikipedia and increase the diversity of data sources by adding S2orc [25]
and Stack-Markdown [21] following [38], as detailed in Table 8. It’s crucial
to understand that our primary focus is not solely on achieving the best
performance. Instead, we place a higher emphasis on optimizing data
combinations and ensuring the convergence of training large language models
with large batch sizes. Consequently, we continue to utilize the
SlimPajama/RedPajama Commoncrawl instead of higher-quality RefinedWeb.

dataset | proportion
---|---
Slimpj.Arxiv | 4% (54B)
Slimpj.StackExchanges | 3.2% (43B)
Slimpj.Github | 4.9% (66B)
Slimpj.Wikipedia | 7.5% (101B)
Slimpj.Books | 4.3% (57B)
Slimpj.C4 | 17.6% (236B)
S2orc | 3% (40B)
Markdown | 3% (40B)
Slimpj.CC | 34.5% (462B)
Redpaj.CC (ext.) | 18% (241B)
Total | 1.34T

Table 8: Data combination of 7B model training in large batch size style.

###  6.2 7B Model Training Configurations

Architecture. For the 7B model training, we adopt MPT architecture [38], the
max sequence length is 2,048. We use Triton [35] with Flash Attention [8] as
the self-attention implementation. Alibi is enabled to make model more
flexible for input length extrapolation. The model’s total number of
parameters is 6.7B.

Tokenizer. The tokenizer used for 7B training is adapted GPT-NeoX-20b.
Following [38], the model’s vocabulary size is adjusted to 50,432 for improved
mfu and leaving a few tokens available that can be used in subsequent
training.

Optimizer. We employ the AdamW optimizer to train our models, adopting these
specific hyper-parameters: $\beta_{1}$ set at 0.9 and $\beta_{2}$ at 0.95. We
adopt a learning rate schedule that traces a cosine pattern, concluding with a
learning rate that is 10% of its maximum value. Along with this, we use a
multi-stage weight decay scheduler as described in Sec. 6.4, cap the gradient
with a clipping value of 1.0, and use a warmup spanning 2,000 steps.

System and platform. For our 7B model training with a large batch size, we use
232 NVIDIA A100 GPUs (80G). We employ llm-foundry [37] as the training
platform. We use FSDP with activation checkpointing enabled to save memory
consumption. We also use the automatic mixed precision of bf16 in training.

###  6.3 Fast Training with Large Batch-size

Large batch training allows a larger learning rate, leading to a faster
convergence of large models. Also, utilizing a larger batch size can optimize
hardware resource usage to make training procedures more efficient.
Additionally, fewer batches are required, which further accelerates the
training process. As shown in Table 9, our large batch training scheme
achieves much higher throughput and mfu than LLaMA [36] and MPT [38] with
fewer total training GPU hours. Overall, in a convex optimization framework,
leveraging a larger portion of the dataset typically leads to enhanced
results. However, for most large deep models that involve non-convex
optimizations, the precise nature of the loss landscape remains elusive,
making the scenario more intricate. Many prior works [17, 19] have noticed
that training with larger batches often results in overfitting compared to
those using smaller batch sizes for the same network. When utilizing large
batch training, there is a propensity for the model to become stuck or even
gravitate towards potential saddle points within the loss landscape. While
large batch training methods often focus on the nearest relative minima they
encounter, networks trained with smaller batches usually navigate the loss
landscape more thoroughly before committing to an optimal minimum. The minima
reached through large batch training can be distinctly different from those
achieved with smaller batch training methods. In the following, we introduce
an approach to mitigate overfitting when training large language models in a
large batch-size scheme.

model | batch size | # GPUs (A100-80G) | throughput | mfu | GPU-hours
---|---|---|---|---|---
LLaMA-7B | 4M | – | – | – | 82,432
MPT-7B | 4M | 232 | 3,310 | 0.4575 | 84.351
LBS-7B (ours) | 14M | 232 | 3,626 | 0.5011 | 76,999

Table 9: Training speed of throughput (tokens per sec on each GPU), model
FLOPs utilization (mfu) [5] and total GPU-hours (per trillion training
tokens).

###  6.4 Progressive Training on Weight Decay

Figure 4: Loss curve of our LBS-7B training.

Prior work [24] observed that dropout operation is utilized only in the early
stages of training and is deactivated in subsequent phases. Models that
incorporate this early dropout strategy tend to exhibit reduced final training
loss compared to models that do not use dropout. In contrast to this, our
approach emphasizes the role of weight decay during large model training. We
introduce a novel training strategy for large language models, wherein the
training process is segmented into various stages. Within each stage, a
distinct weight decay is applied to the model to serve specific objectives.
We’ve termed this approach Progressive Training on Weight Decay (PTWD). Owing
to this methodology, our model, even when trained with a large batch size and
extremely small iterations, achieves smooth convergence. As illustrated in
Fig. 4, our training strategy consists of three distinct phases. Initially, we
negate weight decay by setting it to zero and allow the model to train until
full convergence is achieved. It usually can reach a lower loss level within
this stage compared to using weight decay, even if it slightly overfits.
Following this, in the second phase, we introduce a substantial weight decay,
with a value of 0.5 in our experiments, to suppress the overfitting. Once the
loss values stabilize, we transition to the third phase, wherein a standard
weight decay of 0.1 is implemented, a value consistent with many other LLMs
training. Intriguing, each phase spontaneously converges to roughly 1/3 of the
total training budget, ensuring effective allocation of training budget
throughout the process.

###  6.5 Results of Pre-training and Instruction Tuning

The results from our pretraining and subsequent instruction tuning on ShareGPT
dataset are presented in Table 10. Notably, after instruction tuning, there is
a significant enhancement in MMLU and TruthfulQA metrics. In contrast, the
performance on ARC and HellaSwag has a slight decrease. On the whole, the
average accuracy witnessed a substantial boost following instruction tuning.
More evaluation results on the pretrained LBS model are provided in Table 6.

Model | Average | ARC | HellaSwag | MMLU | TruthfulQA
---|---|---|---|---|---
Ours-LBS-7B-Base | 44.1 | 44.3 | 69.8 | 26.1 | 36.1
Ours-LBS-7B-Instruct | 46.4 | 43.5 | 68.0 | 32.1 | 42.1

Table 10: Results of our large batch-size (LBS) trained 7B models following
Huggingface Leaderboard Evaluation [12] using Harness [14].

##  7 Related Work

###  7.1 RedPajama, SlimPajama and Others.

RedPajama [7] aims to develop open-source large language models and begins by
replicating the LLaMA training dataset [36], which boasts over 1.2 trillion
tokens. This collaborative effort involves entities such as Together,
Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research, and the MILA Québec AI
Institute. SlimPajama [33] stands as the highly deduplicated, multi-source,
open-source dataset tailored for training large language models. This dataset
emerged by refining and eliminating duplicates from the whole 1.2T token
RedPajama dataset. Through meticulous filtering of subpar data and repetitive
content, it reduced the dataset size by 49.6%, scaling it down from 1.2T to
627B tokens. SlimPajama provides superior quality and computational efficiency
for training tasks than the original RedPajama dataset. Other efforts also
have been made in this direction to construct diverse datasets, such as Pile
[13]. It is an English text corpus of 825 GiB, which is designed for the
training of large-scale language models with increased training dataset
diversity to improve general cross-domain knowledge and downstream
generalization capability. It contains a combination of 22 distinct, high-
quality subsets. These subsets incorporate both pre-existing and freshly
curated data, with a significant portion sourced from scholarly or
professional domains.

###  7.2 Data Processing and Optimization Approaches

There have been several advancements in data processing and optimization. The
seminal method of importance sampling [20] stands out as a Monte Carlo
approach designed to evaluate attributes of a particular distribution, even
when the samples are drawn from a distribution that differs from the one under
exploration. SlimPajama’s deduplication mechanism is an adaptation of
importance sampling, incorporating a heuristic that values unique data points.
Recently, several data selection frameworks [18, 15, 34, 40] have been
introduced, inspired by the concept of importance sampling. Among them, DSIR
[40] presents a framework for the data selection challenge by aiming to choose
a subset from a large, unlabeled raw dataset that aligns with a specific
target distribution, given a set of unlabeled target examples. It builds upon
the traditional importance resampling method, adapting it for data selection
in large-scale models. DSIR operates as a scalable algorithm, determining
importance weights within a reduced feature space and then selecting data
based on these importance resampling weights. In [34], the authors delve into
the relationship between error scaling and dataset size. Their theoretical
exploration suggests that by using a robust data pruning metric, which
prioritizes which training examples to remove, the proposed method can
suppress traditional power law scaling, potentially reaching exponential
scaling for pruned dataset sizes.

###  7.3 Data Combination for Training Large Language Models

The training of large language models, such as GPT [29, 30, 4] and BERT [10],
requires significant amounts of data to capture and generalize over the vast
intricacies of human language. As a result, researchers often combine data
from various sources, such as web text, Github, Books, ArXiv, Wikipedia, etc.
There are some related work and difficulties that have been explored in the
context of data combination for training large language models. (1)
Concatenation of diverse datasets: One of the simplest methods for combining
data is to concatenate various corpora, covering diverse topics, styles, and
sources. This ensures that the model gets a broad view of the language. (2)
WebText and similar corpora: For OpenAI’s GPT-2, a dataset called WebText [30]
was curated by scraping content from the internet. This kind of data provides
a rich mix of formal, informal, factual, and opinionated text, thus offering
diverse training material. (3) Balancing and weighting: Simply combining data
may lead to issues if one source is overrepresented. Prior studies have
applied weights to different data portions or ensure that the combined dataset
is balanced in terms of sources, styles, and other criteria. For instance,
DoReMi [39] first trains a small proxy model using group distributionally
robust optimization across domains, generating domain weights (or mixture
proportions) without relying on information from subsequent tasks. Following
this, they utilize these domain weights to resample a dataset, on which then
train a full-size model. (4) Multimodal Training: Combining text with other
data forms, like images or sounds, can also enhance language model training,
especially for tasks that require understanding across modalities.

###  7.4 Large Batch Training for Large Language Models

Large language models inherently possess a structure that supports
parallelization, especially when optimized using techniques that allow for
batch training. When computational resources permit, large batch sizes are
favored to expedite the training of large models containing potentially
millions or billions of parameters. At a fundamental level, larger batch sizes
enhance the quality of each gradient update since they consider a more
considerable chunk of the dataset. Conversely, a smaller batch size means that
model parameter updates are based on gradients derived from a limited dataset
portion. This smaller dataset slice might not comprehensively capture the
intricate relationships between features and labels. Therefore, it might seem
that larger batch sizes consistently offer advantages in training. However,
[19] pointed out that this perspective does not factor in the model’s capacity
to generalize to new, unseen data, nor the intricate, non-convex optimization
landscape of contemporary large models. In practice, multiple studies [17, 19]
have demonstrated that while larger batch sizes might hasten convergence, they
can impair a model’s generalization to new datasets, irrespective of the deep
network type. This observed disparity has been named as the Generalization
Gap. A method [17] to address this gap involves starting from a smaller batch
size and gradually enlarging it as training advances. In our study, we explore
this problem through a new and unique angle of progressive weight decay
training.

##  8 Conclusion

We have presented SlimPajama-DC, a comprehensive study on understanding the
data domain weights and combinations for training large language models.
Notably, SlimPajama-DC can operate on compact models, and its advantages can
be seamlessly transferred to models that are several times larger. This leads
to a remarkable acceleration in training on the SlimPajama with the optimal
sampling probabilities across domains for larger models. Through this, we aim
to spark further exploration into data-centric methods to enhance the
efficiency of large language model training.

## References

  * [1] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
  * [2] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al.  Gpt-neox-20b: An open-source autoregressive language model.  arXiv preprint arXiv:2204.06745, 2022.
  * [3] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021.  If you use this software, please cite it using these metadata.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [7] Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset, 2023\.
  * [8] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In Advances in Neural Information Processing Systems, 2022.
  * [9] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In International conference on machine learning, pages 933–941. PMLR, 2017.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [11] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al.  Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.  arXiv preprint arXiv:2304.03208, 2023.
  * [12] Nathan Habib Sheon Han Nathan Lambert Nazneen Rajani Omar Sanseviero Lewis Tunstall Thomas Wolf Edward Beeching, Clémentine Fourrier.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * [13] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2020.
  * [14] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, Sept. 2021.
  * [15] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith.  Don’t stop pretraining: Adapt language models to domains and tasks.  arXiv preprint arXiv:2004.10964, 2020.
  * [16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In International Conference on Learning Representations, 2021.
  * [17] Elad Hoffer, Itay Hubara, and Daniel Soudry.  Train longer, generalize better: closing the generalization gap in large batch training of neural networks.  Advances in neural information processing systems, 30, 2017.
  * [18] Angelos Katharopoulos and François Fleuret.  Not all samples are created equal: Deep learning with importance sampling.  In International conference on machine learning, pages 2525–2534. PMLR, 2018.
  * [19] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836, 2016.
  * [20] Teun Kloek and Herman K Van Dijk.  Bayesian estimates of equation system parameters: an application of integration by monte carlo.  Econometrica: Journal of the Econometric Society, pages 1–19, 1978\.
  * [21] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries.  The stack: 3 tb of permissively licensed source code.  Preprint, 2022.
  * [22] Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman.  Mining of massive data sets.  Cambridge university press, 2020.
  * [23] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022.
  * [24] Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell.  Dropout reduces underfitting.  In ICML, 2023.
  * [25] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld.  S2orc: The semantic scholar open research corpus.  arXiv preprint arXiv:1911.02782, 2019.
  * [26] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [27] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  arXiv preprint arXiv:2306.01116, 2023.
  * [28] Ofir Press, Noah A Smith, and Mike Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation.  arXiv preprint arXiv:2108.12409, 2021.
  * [29] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [31] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  arXiv preprint arXiv:2112.11446, 2021.
  * [32] Noam Shazeer.  Glu variants improve transformer.  arXiv preprint arXiv:2002.05202, 2020.
  * [33] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023\.
  * [34] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  Advances in Neural Information Processing Systems, 35:19523–19536, 2022.
  * [35] Philippe Tillet, Hsiang-Tsung Kung, and David Cox.  Triton: an intermediate language and compiler for tiled neural network computations.  In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019.
  * [36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [37] https://github.com/mosaicml/llm-foundry.  Llm foundry.  Mosaicml, 2023.
  * [38] https://www.mosaicml.com/blog/mpt-7b.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  Mosaicml blog, 2023.
  * [39] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu.  Doremi: Optimizing data mixtures speeds up language model pretraining.  arXiv preprint arXiv:2305.10429, 2023.
  * [40] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang.  Data selection for language models via importance resampling.  arXiv preprint arXiv:2302.03169, 2023.
  * [41] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  arXiv preprint arXiv:1905.07830, 2019.
  * [42] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.

## Appendix

##  Appendix A Data Proportion Details

Dataset | Slimpajama | Redpajama | LLaMA 1
---|---|---|---
Commoncrawl | 52.2% | 333B | 72.6% | 878B | 67.0% | 670/938B
C4 | 26.7% | 170B | 14.4% | 175B | 15.0% | 150/210B
GitHub | 5.2% | 33B | 4.9% | 59B | 4.5% | 45/63B
Books | 4.2% | 27B | 2.1% | 26B | 4.5% | 45/63B
ArXiv | 4.6% | 29B | 2.3% | 28B | 2.5% | 25/35B
Wikipedia | 3.8% | 24B | 2.0% | 24B | 4.5% | 45/63B
StackExchange | 3.3% | 21B | 1.7% | 20B | 2.0% | 20/28B
Total | 100.0% | 637B | 100.0% | 1.2T | 100% | 1.0/1.4T
| RefinedWeb | GPT3 | MassiveText
Commoncrawl | 100% | 600B | 60.0% | 180B | 0.0% | 0
C4 | 0.0% | 0B | 0.0% | 0 | 10.0% | 30B
GitHub | 0.0% | 0B | 0.0% | 0 | 3.0% | 9B
Books | 0.0% | 0B | 16.0% | 48B | 27.0% | 81B
Wikipedia | 0.0% | 0B | 3.0% | 9B | 2.0% | 6B
WebText2 | 0.0% | 0B | 22.0% | 66B | 0.0% | 0
MassiveWeb | 0.0% | 0B | 0.0% | 0 | 48.0% | 144B
News | 0.0% | 0B | 0.0% | 0 | 10.0% | 30B
Total | 100.0% | 600B | 100.0% | 300B | 100.0% | 300B

Table 11: Detailed data source proportions for various datasets.

##  Appendix B MMLU

In this section, we provide the detailed item-by-item results in MMLU, as
shown in Table 12, it is interesting to notice that on some sub-domains in
MMLU, the results from our configured 1.3B models are even better than GPT-3
175B and LLaMA2 7B models.

|  | GPT-3 | Llama2 | SlimPajama-DC 1.3B
---|---|---|---|---
|  | 175B | 7B | DC-1 | DC-2 | DC-3 | DC-4 | DC-5 | DC-6
Abstract Algebra | STEM | 30.0 | 29.0 | 27.0 | 26.0 | 28.0 | 25.0 | 27.0 | 21.0
Anatomy | STEM | 48.0 | 37.0 | 23.0 | 23.0 | 25.9 | 27.4 | 34.1 | 19.3
Astronomy | STEM | 49.0 | 33.6 | 25.0 | 19.7 | 21.7 | 23.0 | 27.0 | 20.4
Business Ethics | Other | 46.0 | 40.0 | 24.0 | 22.0 | 30.0 | 26.0 | 24.0 | 28.0
Clinical Knowledge | Other | 48.0 | 35.1 | 30.2 | 26.8 | 25.7 | 24.9 | 18.9 | 26.0
College Biology | STEM | 45.0 | 37.5 | 23.6 | 24.3 | 23.6 | 27.1 | 25.7 | 31.9
College Chemistry | STEM | 26.0 | 32.0 | 26.0 | 19.0 | 21.0 | 29.0 | 19.0 | 25.0
College Computer Science | STEM | 46.0 | 29.0 | 37.0 | 36.0 | 33.0 | 32.0 | 36.0 | 33.0
College Mathematics | STEM | 34.5 | 33.0 | 35.0 | 29.0 | 21.0 | 31.0 | 25.0 | 36.0
College Medicine | Other | 48.0 | 30.6 | 26.0 | 23.1 | 26.6 | 26.0 | 27.8 | 24.9
College Physics | STEM | 28.0 | 26.5 | 24.5 | 24.5 | 24.5 | 21.6 | 22.6 | 21.6
Computer Security | STEM | 57.0 | 45.0 | 24.0 | 30.0 | 28.0 | 19.0 | 27.0 | 27.0
Conceptual Physics | STEM | 36.5 | 36.6 | 27.7 | 30.2 | 23.8 | 22.1 | 28.5 | 24.3
Econometrics | Social Science | 33.0 | 23.7 | 24.6 | 25.4 | 24.6 | 30.7 | 23.7 | 29.8
Electrical Engineering | STEM | 50.0 | 26.9 | 29.0 | 24.1 | 23.5 | 26.2 | 29.0 | 28.3
Elementary Mathematics | STEM | 30.0 | 24.3 | 26.2 | 25.9 | 25.9 | 27.5 | 25.1 | 23.5
Formal Logic | Humanities | 29.0 | 27.0 | 35.7 | 24.6 | 15.9 | 20.6 | 16.7 | 29.4
Global Facts | Other | 37.0 | 29.0 | 30.0 | 31.0 | 33.0 | 30.0 | 37.0 | 17.0
High School Biology | STEM | 48.0 | 34.5 | 25.8 | 26.5 | 24.8 | 25.5 | 24.8 | 21.9
High School Chemistry | STEM | 33.0 | 28.1 | 27.6 | 19.7 | 24.1 | 27.1 | 27.1 | 25.1
High School Computer Science | STEM | 39.0 | 31.0 | 29.0 | 26.0 | 25.0 | 26.0 | 27.0 | 27.0
High School European History | Humanities | 54.0 | 44.2 | 23.6 | 28.5 | 25.5 | 24.9 | 26.7 | 20.6
High School Geography | Social Science | 58.0 | 34.3 | 34.3 | 20.7 | 22.2 | 19.2 | 17.7 | 18.2
High School Government And Politics | Social Science | 58.0 | 44.6 | 35.2 | 16.6 | 21.8 | 25.9 | 21.8 | 21.8
High School Macroeconomics | Social Science | 40.5 | 35.4 | 34.4 | 25.9 | 23.8 | 22.8 | 24.6 | 32.8
High School Mathematics | STEM | 28.0 | 24.8 | 26.7 | 25.2 | 25.2 | 28.5 | 26.7 | 23.7
High School Microeconomics | Social Science | 42.0 | 31.9 | 23.5 | 23.1 | 25.2 | 25.2 | 21.4 | 23.5
High School Physics | STEM | 28.0 | 26.5 | 27.8 | 26.5 | 21.9 | 27.2 | 29.8 | 26.5
High School Psychology | Social Science | 61.0 | 47.3 | 32.3 | 23.1 | 23.8 | 22.9 | 23.7 | 22.2
High School Statistics | STEM | 30.5 | 35.2 | 21.3 | 21.3 | 19.9 | 22.2 | 23.2 | 40.7
High School Us History | Humanities | 53.0 | 39.7 | 24.5 | 21.6 | 24.5 | 24.5 | 27.5 | 27.9
High School World History | Humanities | 56.0 | 40.9 | 29.1 | 25.7 | 24.5 | 27.4 | 25.7 | 24.9
Human Aging | Other | 50.0 | 40.8 | 14.8 | 30.5 | 37.2 | 30.5 | 27.4 | 35.9
Human Sexuality | Social Science | 54.0 | 36.6 | 28.2 | 22.1 | 22.9 | 22.1 | 25.2 | 32.1
International Law | Humanities | 55.5 | 51.2 | 26.5 | 30.6 | 39.7 | 32.2 | 30.6 | 24.0
Jurisprudence | Humanities | 55.0 | 38.9 | 26.9 | 22.2 | 26.9 | 27.8 | 25.0 | 25.9
Logical Fallacies | Humanities | 48.0 | 39.3 | 19.6 | 27.0 | 29.5 | 23.9 | 27.6 | 30.7
Machine Learning | STEM | 31.0 | 23.2 | 17.9 | 33.0 | 23.2 | 28.6 | 30.4 | 30.4
Management | Other | 56.0 | 35.0 | 26.2 | 29.1 | 27.2 | 21.4 | 23.3 | 25.2
Marketing | Other | 60.0 | 46.6 | 22.2 | 24.4 | 23.9 | 25.2 | 28.2 | 26.9
Medical Genetics | Other | 40.0 | 43.0 | 27.0 | 24.0 | 24.0 | 22.0 | 23.0 | 31.0
Miscellaneous | Other | 60.0 | 42.4 | 22.5 | 27.5 | 27.6 | 29.3 | 26.2 | 24.1
Moral Disputes | Humanities | 44.5 | 40.2 | 29.5 | 25.7 | 24.9 | 24.9 | 24.0 | 26.6
Moral Scenarios | Humanities | 26.0 | 24.3 | 27.3 | 24.6 | 24.3 | 23.8 | 24.6 | 23.9
Nutrition | Other | 47.0 | 37.6 | 28.1 | 23.2 | 25.2 | 25.8 | 25.8 | 21.6
Philosophy | Humanities | 51.0 | 39.9 | 28.0 | 28.9 | 26.7 | 29.3 | 28.3 | 26.1
Prehistory | Humanities | 53.0 | 36.1 | 26.5 | 25.9 | 29.3 | 26.9 | 27.5 | 26.5
Professional Accounting | Other | 33.0 | 25.9 | 27.0 | 29.1 | 27.0 | 27.3 | 27.0 | 23.4
Professional Law | Humanities | 34.5 | 30.2 | 27.1 | 25.0 | 25.8 | 24.6 | 26.9 | 25.4
Professional Medicine | Other | 36.0 | 44.5 | 19.9 | 31.6 | 22.8 | 21.0 | 27.9 | 21.7
Professional Psychology | Social Science | 44.5 | 35.1 | 26.3 | 27.3 | 25.5 | 25.2 | 27.5 | 26.8
Public Relations | Social Science | 48.0 | 40.9 | 33.6 | 30.9 | 28.2 | 29.1 | 26.4 | 27.3
Security Studies | Social Science | 52.0 | 31.8 | 39.2 | 17.5 | 18.8 | 21.2 | 16.3 | 21.2
Sociology | Social Science | 53.0 | 46.8 | 25.4 | 24.4 | 22.9 | 24.9 | 23.9 | 24.4
Us Foreign Policy | Social Science | 69.0 | 46.0 | 27.0 | 31.0 | 24.0 | 25.0 | 28.0 | 33.0
Virology | Other | 46.0 | 30.1 | 21.7 | 30.1 | 31.3 | 27.1 | 28.3 | 30.7
World Religions | Humanities | 55.0 | 50.9 | 27.5 | 25.2 | 32.8 | 29.8 | 32.2 | 26.3
Humanities |  | 40.6 | 34.0 | 27.1 | 25.8 | 26.9 | 26.2 | 26.4 | 26.0
STEM |  | 36.7 | 30.5 | 26.5 | 25.8 | 24.4 | 26.1 | 27.1 | 26.7
Social Science |  | 50.5 | 38.3 | 30.3 | 24.0 | 23.6 | 24.5 | 23.3 | 26.1
Other |  | 49.0 | 38.1 | 24.6 | 27.1 | 27.8 | 25.9 | 26.5 | 25.9
All |  | 43.9 | 35.1 | 27.0 | 25.7 | 25.6 | 25.7 | 26.0 | 26.2

Table 12:  MMLU. 5-shot results per domain on the test sets.