0.8pt

#  DsDm: Model-Aware Dataset Selection with Datamodels

Logan Engstrom
<EMAIL_ADDRESS>
MIT  Axel Feldmann
<EMAIL_ADDRESS>
MIT  Aleksander Mądry
<EMAIL_ADDRESS>
MIT

###### Abstract

When selecting data for training large-scale models, standard practice is to
filter for examples that match human notions of data quality. Such filtering
yields qualitatively clean datapoints that intuitively should improve model
behavior. However, in practice the opposite can often happen: we find that
selecting according to similarity with “high quality” data sources may not
increase (and can even hurt) performance compared to randomly selecting data.

To develop better methods for selecting data, we start by framing dataset
selection as an optimization problem that we can directly solve for: given
target tasks, a learning algorithm, and candidate data, select the subset that
maximizes model performance. This framework thus avoids handpicked notions of
data quality, and instead models explicitly how the learning process uses
train datapoints to predict on the target tasks. Our resulting method greatly
improves language model (LM) performance on both pre-specified tasks and
previously unseen tasks. Specifically, choosing target tasks representative of
standard LM problems and evaluating on diverse held-out benchmarks, our
selected datasets provide a 2× compute multiplier over baseline methods.

##  1 Introduction

Suppose we want to train a large-scale machine learning model. What data
should we train on? The simple answer is: as much data as possible. For
example, we train language and vision models on vast quantities of text
[Radford et al., 2019] and image-caption [Ramesh et al., 2021] data from
sources like internet crawls. This seemingly straightforward recipe yields
models that generalize remarkably well to a broad range of tasks.

A closer look, however, reveals that choosing training data is not actually so
straightforward. Indeed, not all data is equally useful; for example, internet
data sources frequently contain “low quality” data like spam, poor writing, or
nonsense text. Therefore, in practice, we tend to filter training data
according to intuitive notions of quality, e.g., choosing documents similar to
a “high quality” data source like Wikipedia or discarding documents with fewer
than five sentences. These steps choose (qualitatively) “clean” samples that
should _intuitively_ improve performance. However, do such samples improve
performance in practice too?

Contributions. We find that the opposite can happen: selecting data according
to similarity with “high quality” data sources may not improve (and, in fact,
can even hurt) model performance. Specifically, we train language models with
standard, similarity-based selection methods previously used to select data
for models like PaLM and GPT-3 [Brown et al., 2020, Xie et al., 2023b], and
find these methods do not outperform (and can even underperform) selecting
data at random (cf. Section 4).

To develop better methods for selecting training data, we start from first
principles. That is, we avoid intuitive notions of data quality, and instead
frame dataset selection as an optimization problem where the goal is to—given
target tasks, a learning algorithm, and a candidate data pool—select the data
that maximizes model performance. However, actually finding the optimal
solution to this problem is difficult. While we can calculate the performance
of a _specific_ training set by training a model on that set (and then
evaluating), it is (generally) unclear how to calculate the best possible
training subset without examining every possible subset one by one, a
computationally infeasible procedure.

We instead approximate the optimal subset by (approximately) modeling how the
learning algorithm actually uses training data to predict. Specifically, in
Section 2, we model target task performance as a function of training subset
using datamodels (which efficiently approximate the mapping between training
subset and model performance [Ilyas et al., 2022]), and select the subset that
maximizes our estimate. Then, in Section 3, we demonstrate that our resulting
method, dataset selection with datamodels (DsDm), consistently improves
language model performance on diverse target tasks (e.g., SQuAD [Rajpurkar et
al., 2016] and LAMBADA [Paperno et al., 2016]), even when existing selection
methods do not.

DsDm-selected data can improve performance on pre-specified tasks. However, in
practice we train large-scale models to generalize to yet unseen tasks. Our
framework suggests a principled approach to selecting data in this scenario
too: choose target tasks similar to those we expect at deployment time, then
select the optimal dataset subset for these target tasks. Following this
strategy, in Section 4, we choose target tasks that cover a range of natural
language problem categories (SQuAD, Jeopardy [MosaicML, 2023], and LAMBADA),
and select data from C4, a canonical web crawl [Raffel et al., 2020]. Our
selections deliver a 2× compute multiplier on a diverse set of test
benchmarks: DsDm-selected datasets yield LMs that perform as well as those
trained with 2× the compute budget on randomly selected data (we train up to
1.8B parameter models). In contrast, no baseline method outperforms randomly
selecting data—even at the same compute budget.

##  2 Estimating the optimal dataset selection with DsDm

To select better data for training large-scale models, we start by defining
the optimal dataset selection as an optimization problem. We then select data
by finding a train subset that is _approximately_ the best solution to that
problem. Specifically, we use datamodels [Ilyas et al., 2022] to approximate
how the learning algorithm uses data to predict on the tasks of interest. We
describe the resulting framework in more detail below.

###  2.1 Task-optimal dataset selection

We frame dataset selection as an optimization problem where the goal is to
minimize trained model loss on a set of target tasks with respect to training
data choice. Given a learning algorithm $\mathcal{A}$ (e.g., SGD on a neural
network) that maps train set to trained model, and a target distribution
$\mathcal{D}_{\mathrm{targ}}{}$ (e.g., a language modeling task), the size-$k$
task-optimal dataset selection over the set $\mathcal{S}$ of available data
(e.g., documents from an internet scrape) is the subset

$\displaystyle S^{*}$
$\displaystyle\vcentcolon={}\operatorname*{arg\,min}\limits_{S\subset\mathcal{S},|S|=k}\mathcal{L}_{\mathcal{D}_{\mathrm{targ}}{}}(S),$
(1) $\displaystyle\mbox{where }\mathcal{L}_{\mathcal{D}}(S)$
$\displaystyle\vcentcolon={}\mathbb{E}_{x\sim\mathcal{D}}\left[\ell(x;\mathcal{A}(S))\right],$

that minimizes the trained model population loss
$\mathcal{L}_{\mathcal{D}_{\mathrm{targ}}{}}(S)$, where $\ell(x;g)$ denotes
the loss (e.g., cross-entropy loss) for model $g$ on example $x$. Note the
expectation in the population loss is over both target dataset and learning
algorithm randomness (as, e.g., SGD is a non-deterministic algorithm).

In our setting, minimizing (1) is difficult. Indeed, we do not have an easy-
to-optimize, closed-form expression for trained model loss in terms of
training set choice $S$ for large-scale model learning algorithms.
111Depending on the setup, we may have such a form for other classes of
learning algorithms, like linear regression (with influence functions [Cook,
1977, Giordano et al., 2019]) or kernel regression [Bierens, 1988]. While we
can directly calculate the trained model loss for a given $S$ by actually
training on $S$ with $\mathcal{A}$ (and then evaluating loss), using this
method to find the best subset is generally computationally infeasible: we
would need to train (and evaluate) a model for each of the
${|\mathcal{S}|\choose k}$ possible size-$k$ train subsets.

###  2.2 Estimating model loss efficiently with datamodels

To circumvent this computational challenge, we trade optimality for
feasibility, and instead estimate the best train subset. Specifically, we
approximate the trained model loss in place of calculating it directly, then
select the subset that minimizes our approximation.

The core primitive we use to approximate the trained model loss is
datamodeling [Ilyas et al., 2022], a framework originally designed to predict
how choice of training set changes model predictions. More precisely, a
datamodel for a fixed sample $x$ approximates the mapping from train subset
choice $S$ (out of the available dataset $\mathcal{S}$) to resulting trained
model loss on a sample $x$, i.e., the function:

$\mathcal{L}_{x}(S)\vcentcolon=\mathbb{E}\left[\ell(x;\mathcal{A}(S))\right].$

Previous work used datamodels primarily for reliability purposes, e.g., to
detect data poisoning [Khaddaj et al., 2022] or train-test leakage [Ilyas et
al., 2022]. In contrast, we leverage datamodels to cheaply approximate the
trained model loss $\mathcal{L}_{x}$. Formally, given a candidate data subset
$S\subset\mathcal{S}$, datamodels take as input the corresponding
characteristic vector

$\text{1}_{S}\in\\{0,1\\}^{|\mathcal{S}|}\hskip 10.0pt\hbox{ such that}\hskip
10.0pt\left(\text{1}_{S}\right)_{i}=\begin{cases}1&\text{if
}\mathcal{S}_{i}\in S\\\ 0&\text{otherwise}\end{cases},$ (2)

instead of the subset $S$ directly. Then, the datamodel $\tau_{\theta_{x}}$
for $x$ is the parameterized function that optimally predicts
$\mathcal{L}_{x}$ over a (chosen) distribution of train subsets
$\mathcal{D}_{\mathcal{S}}$, i.e.,

$\displaystyle\tau_{\theta_{x}}:\\{0,1\\}^{|\mathcal{S}|}\to\mathbb{R},\qquad\text{
where }\qquad\theta_{x}=\arg\min_{\theta}\
\widehat{\mathbb{E}}^{(m)}_{S_{i}\sim\mathcal{D}_{\mathcal{S}}}\left[L_{{\text{reg}}}\left(\tau_{\theta}(\text{1}_{S_{i}}),\
\mathcal{L}_{x}(S_{i})\right)\right],$ (3)

where $L_{{\text{reg}}}(\cdot,\cdot)$ is a regression loss function (e.g.,
mean squared error), and $\widehat{\mathbb{E}}^{(m)}$ is an $m$-sample
empirical expectation. Note that in practice, we estimate the datamodel
parameters that minimize (3) (i.e., we estimate the parameters of the function
we use to approximate model loss).

##### Linear datamodels.

So far we have only defined the datamodeling framework; we have not actually
defined the parameterized function $\tau_{\theta}$ or described how to
estimate the parameters $\theta$. In this work, we instantiate datamodels as a
linear function of the characteristic vector $\text{1}_{S}$ (a standard choice
[Ilyas et al., 2022, Saunshi et al., 2023]), such that

$\tau_{\theta_{x}}(\text{1}_{S})\vcentcolon=\theta_{x}^{\top}\text{1}_{S}.$

Note that, being a linear model, $\tau_{\theta_{x}}$ treats the inclusion of
an example $\mathcal{S}_{i}$ in the train set as having a fixed effect on
$\mathcal{L}_{x}(S)$ irrespective of the other examples in $S$ (this fixed
effect is exactly the value of index $i$ of $\theta_{x}$).

In this work, to estimate linear datamodel parameters $\theta_{x}$ we largely
follow the procedures of previous work [Park et al., 2023, Ilyas et al.,
2022]—in particular, we use the TRAK estimator—but make changes needed for the
language modeling domain (see Appendix B for full details).

###  2.3 DsDm: Dataset Selection with Datamodels

Recall that our goal is to estimate the candidate data subset that minimizes
trained model loss on the target task (cf. (1)). To do so, we approximate the
mapping between training subset $S$ and target distribution loss
$\left(\textrm{i.e., }\mathcal{L}_{\mathcal{D}_{\mathrm{targ}}}(S)\right)$
with datamodels as a primitive, then select the candidate data subset that
minimizes our approximation of the target loss.

Specifically, given a train subset $S$, we estimate the corresponding target
distribution loss with an $n$-sample empirical expectation of datamodel loss
estimates over $\mathcal{D}_{\mathrm{targ}}{}$ samples:

$\widehat{\mathcal{L}}_{\mathcal{D}_{\mathrm{targ}}{}}{}(S)=\widehat{\mathbb{E}{}}^{(n)}_{x_{i}\sim\mathcal{D}_{\mathrm{targ}}{}}\left[\tau_{\theta_{x_{i}}}(\text{1}_{S})\right]=\frac{1}{n}\sum_{i=1}^{n}\theta_{x_{i}}^{\top}\text{1}_{S}=\text{1}_{S}^{\top}\left(\frac{1}{n}\sum_{i=1}^{n}\theta_{x_{i}}\right).$

Then, our size-$k$ dataset selection with datamodels (DsDm) estimate of the
optimal dataset selection is the subset that minimizes the approximated target
loss $\widehat{\mathcal{L}}_{\mathcal{D}_{\mathrm{targ}}{}}{}(S)$ with respect
to training set choice:

$\widehat{S}_{\mathrm{DM}}\vcentcolon=\operatorname*{arg\,min}_{S\subset\mathcal{S},|S|=k}\widehat{\mathcal{L}}_{\mathcal{D}_{\mathrm{targ}}{}}{}(S)=\operatorname*{arg\,min}_{S\subset\mathcal{S},|S|=k}\text{1}_{S}^{\top}\left(\frac{1}{n}\sum_{i=1}^{n}\theta_{x_{i}}\right)=\mathop{\mathrm{arg\,bot}\mbox{\hskip
0.6pt-}k}{}\left(\frac{1}{n}\sum_{i=1}^{n}\theta_{x_{i}}\right).$

In our instantiation, the considered datamodels are linear, so DsDm selects
the examples corresponding to the smallest $k$ indices of
$\frac{1}{n}\sum_{i=1}^{n}\theta_{x_{i}}$. (Note that linear datamodels are a
design choice: DsDm can use any datamodel parameterization that can be
optimized over.)

##  3 Evaluating DsDm

To what extent does DsDm actually minimize trained model target task loss? In
this section, we demonstrate that DsDm consistently reduces LM target task
loss in practice. In contrast, baseline targeted dataset selection methods—all
of which ignore the model training process and instead select data according
to textual similarity with target task samples—often do not outperform
randomly selecting training data. Below, we describe our experimental setup,
then discuss results.

###  3.1 Setup

To capture the effectiveness of a given data selection method, we measure the
extent to which it reduces the optimal dataset selection objective of (1),

$\mathcal{L}_{\mathcal{D}_{\mathrm{targ}}{}}(S)\vcentcolon=\mathbb{E}_{x\sim\mathcal{D}}\left[\ell(x;\mathcal{A}(S))\right],$

across varying target tasks. For each considered target task, we split samples
into a target set and a separate test set, and only use the target set to
select training subsets. We then train an LM on the resulting dataset, and
inspect target task performance (using the test set). Below, we describe the
experimental setup as well as the baselines we use (see Appendix C for more
setup details).

##### Target tasks, candidate dataset, and model training.

We consider four separate LM target tasks: LAMBADA [Paperno et al., 2016], CS-
Algorithms [Srivastava et al., 2022], SQuAD [Rajpurkar et al., 2016], and
Jeopardy [Tunguz, 2019]; see Appendix C.1 for more details on each task. Our
candidate dataset $\mathcal{S}$ is the English subset of the Colossal Cleaned
Common Crawl (C4), a standard web scrape [Raffel et al., 2020].222Each
candidate example $\mathcal{S}_{i}$ is a sequence-length (1024 token) corpus
slice; $|\mathcal{S}|\approx 217{,}000{,}000$ (cf. Appendix A.1). On each
selected train dataset, we train a 125M parameter GPT-2 style model on 6
billion tokens.

##### Baselines.

We compare DsDm with two standard targeted dataset selection methods, both of
which select according to textual similarity between candidate training
samples and $\mathcal{D}_{\mathrm{targ}}{}$ samples: Classifier (selects the
top examples in $\mathcal{S}$ given by a logistic model trained to classify,
on FastText features, between $\mathcal{S}$ and
$\mathcal{D}_{\mathrm{targ}}{}$ samples; used by GPT-3/PaLM/The Pile
[Chowdhery et al., 2022, Gao et al., 2020]) and DSIR (Data Selection with
Importance Resampling chooses train samples with n-grams that distributionally
match those of $\mathcal{D}_{\mathrm{targ}}{}$ [Xie et al., 2023b]). We also
compare with randomly selecting data (Random).

Figure 1: Target task performance by selection method, varying dataset
selection size. We train a 125M models on a fixed number of tokens for each
selection, adjusting epochs accordingly. DsDm consistently improves
performance, even when baselines do not outperform randomly selecting data
(e.g., on SQuAD and CS-Algorithms). DsDm models also consistently match a
larger model trained with 10× the compute budget on random data (a Chinchilla-
optimal 1.3B model). DsDm performance decreases with larger selection
fraction, indicating that higher ranked DsDm samples (i.e., data in the
smallest selections) tend to improve performance more than less highly ranked
samples (i.e., data only present in larger selections). We measure the average
log-probability of the label across samples. The “random” shaded area is the
range of values achieved by 10 Random models trained on one epoch of data
(Random performance is not x-axis dependent). Measuring accuracy in place of
log-probability yields similar conclusions (cf. Figure 9).

###  3.2 Results

In Figure 1 we display the mean log-probability (of the label given the
context, across task samples; larger is better) achieved on each target task
by training a model with each selection method (varying dataset selection
size). Each model was trained on the same number of total tokens, with models
trained on smaller fractions of C4 traversing more epochs. We find that DsDm
most improves target task performance on all tasks. Models trained with DsDm
even outperform a larger model trained with 10× the compute on randomly
selected data. Additionally, DsDm performance decreases with larger selection
fraction, indicating that the samples predicted by DsDm to most improve
performance actually do so in practice. After all, smaller selections will
contain more useful data (as predicted by DsDm) on average compared to larger
selections (e.g., all methods select the same subset for selection fraction
$1$).

In contrast, baselines that select according to textual similarity with the
target task, Classifier and DSIR, do not consistently outperform randomly
selecting data (e.g., on SQuAD and CS-Algorithms). These results suggest that
similarity with the target task does not suffice to find useful train samples.
Note that the only task that baselines match DsDm on, LAMBADA (a passage
completion task), is also the only one without contextual instructions. We
hypothesize that n-gram similarity may not capture how instructions define
tasks.

  1. (1)

s, forms, and modification alternative can be overwhelming. So save the time,
chance, money, budget, energy, also effort and implement these tips to acquire
a obvious concept of what you would like and things you need before you start
the quest and think about the right variations and pick right decoration, here
are some recommendations and photos on deciding on the best leather sectional
sofas toronto.\nThe design need to create impact to your sofa. Could it be
modern, luxury, minimalist, or traditional? Co

  2. (2)

ises; soldier of fortune.\n3\. a person who undertakes great commercial risk;
speculator.\n4\. a person who seeks power, wealth, or social rank by
unscrupulous or questionable means: They thought John was an adventurer and
after their daughter’s money.\n"There can be adventurer souls."\n"There can be
adventurer sirs."\n"There can be adventurer reflexes."\n"There can be
adventurer realises."\n"There can be adventurer profiles."\n"There can be
adventurer problems."\n"There can be adventurer paths."\n"There can be

(a) DsDm samples

  1. (1)

in Alexandria, where it was begun; and the Greek Bible of the Hellenistic Jews
and the Catholic Church may rightly be styled the Alexandrian Greek version of
the Old Testament.\nIn the early days of the Church the Septuagint was widely
used among the Jews; as a rule, though there are exceptions, when the Old
Testament is quoted in the New Testament it is from the Greek, not the Hebrew
Bible that the quotation is made. The early Jewish-Christians and the great
majority of the Jews had the same Bible, and Gent

  2. (2)

the Central Committee of the Party, that is, by the Politburo, the Orgburo
(Organizational Bureau), and the Secretariat. The decisions made were
implemented through the Presidium of the Supreme Soviet of the USSR, the
Council of People’s Commissars of the USSR, the GKO, and the General
Headquarters of the Supreme Command, which had been established on August 8.
Strategic direction of the armed forces was carried out by the General
Headquarters through its working body, the General Staff. Major questions as

(b) DSIR samples

  1. (1)

ris and St Gleb, dating from the mid-12th century, was much rebuilt in
succeeding periods, before being restored to its original shape in the 20th
century. The crowning achievement of Chernigov masters was the exquisite
Church of St Paraskeba (Pyatnitskaya), constructed at the turn of the 12th and
13th centuries. This graceful building was seriously damaged in the Second
World War; its original medieval outlook was reconstructed. The earliest
residential buildings in the downtown date from the late 17th cen

  2. (2)

their professional careers.\nDr Simpson’s first line is classic.\nlatest date
in the year it’s been that cold in 50 years of record keeping.\nBack in March,
2007, Al Gore told Congress that "the science is settled."\nscience is
settled. The Sun revolves around the Earth, not vice versa.\nscience," spent
the rest of his life under house arrest.\n& Tax Bill (its actual name) through
the House? Hopefully, some "cooler"\nseem, may have nothing to do with global
warming.\nPaul, let me give you a little advice.\nYou migh

(c) Classifier samples

Figure 2: Train examples selected by each method for SQuAD. Selected
Classifier and DSIR train examples are intuitively “high quality,” and more
similar to SQuAD samples (which are Wikipedia excerpts with questions) than
DsDm examples are. DsDm samples do not match SQuAD, but do contain QA-style
text—like (1) left (an advertisement posing a question) or (2) left (a
dictionary definition). Samples randomly chosen from the top 0.01% that each
method is most likely to select (cf. Appendix C.3). “\n” is a newline.

To better understand how dataset choice relates to performance, we inspect the
datapoints each method is most and least likely to select (for SQuAD: in
Figure 2, for all other targets: in Appendix C.3). We find that:

##### Useful data is not necessarily similar to the target task (or
intuitively helpful at all).

Looking at selected data for SQuAD in Figure 2, DSIR and Classifier select
data that is more qualitatively similar to SQuAD samples (which are Wikipedia
excerpts with questions, cf. Appendix Figure 5) than DsDm. Instead, DsDm
samples often contain question answering-related text that does not match the
SQuAD format; DsDm performance shows that qualitatively similar data is not
necessarily the best data. However, helpful data is not always intuitively
useful. Indeed, the DsDm examples for CS-Algorithms and Jeopardy (cf. Appendix
Figures 21 and 15) often contain seemingly nonsense text. Yet, DsDm yields the
best models for these tasks.

#####  DsDm discards “mislabeled” data.

Samples that DSIR and Classifier are least likely to select are qualitatively
different from those of DsDm. Inspecting Appendix Figure 11 for data selected
for SQuAD: least likely samples for all methods are incoherent/malformed, but
those of DsDm also often contain QA text. Despite this, such DsDm samples
examples hurt model performance: training on them is worse than selecting
randomly (cf. Appendix Figure 10). We liken these samples to “mislabeled”
examples from supervised learning, and conjecture excluding such data could
(in part) explain DsDm performance.

##  4 Selecting data for broad model capabilities

So far, we have shown that DsDm consistently reduces loss on pre-specified
target tasks. However, when we train large-scale models in practice our hope
is that they will perform well on yet unseen tasks too. Our framework suggests
a straightforward approach to improving this kind of performance: choose
target tasks that match those we expect to see at model deployment time, then
estimate the optimal dataset selection for these “proxy” target tasks.

In this section, we demonstrate that this approach to selecting data can
greatly improve held-out task performance compared to baselines. Specifically,
we consider three target tasks that cover a broad range of language modeling
problem categories—LAMBADA (language understanding problems), SQuAD (reading
comprehension problems), and Jeopardy (world knowledge problems)—and estimate
the optimal training dataset selection for these tasks (all together) via
DsDm. We then compare models trained on this data with models trained via
existing dataset selection baselines. Overall, evaluating on a diverse set of
held-out benchmarks (meant to model “yet unseen tasks”), we find that: (a)
randomly selecting data is a surprisingly strong baseline—no baseline
selection method outperforms selecting data at random—and (b) our approach
yields models that match those trained with 2× the training compute on
randomly selected data. In particular, models trained with our approach
reliably improve performance on benchmarks that are qualitatively related to
the target tasks. We describe our setup below, and defer additional details to
Appendix D.

##### Model training, scaling DsDm, selection baselines, and evaluation.

We train GPT-2 style LMs with varying compute budgets. To train the best
possible model for a given compute budget, we use Chinchilla-optimal
parameter-to-train-tokens ratios [Hoffmann et al., 2022] and train up to 1.8B
parameter models. To select with DsDm, we use 125M proxy models: we calculate
DsDm subsets for 125M models, then train on these selections at each compute
budget (instead of computing DsDm separately for each model class). DsDm cost
scales linearly with model size, so this procedure greatly reduces overhead
(cf. Appendix B.5). For baselines, we compare with two methods that select via
textual similarity with a specified “high quality” data source (DSIR and
Classifier, the baselines of Section 3), a data deduplication method (SemDeDup
[Abbas et al., 2023]), and selecting data randomly. We evaluate on 15 standard
benchmarks (cf. Table 1).

##### Target tasks.

We execute each targeted dataset selection method using its originally
proposed target task. For DsDm, we apply the framework described above: we
select three target tasks that cover a broad range of LM problem
categories—LAMBADA, SQuAD, and Jeopardy—then estimate the optimal dataset
selection for these tasks together (i.e., $\mathcal{D}_{\mathrm{targ}}{}$ as
an equal mix of these tasks). For Classifier and DSIR, we target a replication
of the “high quality” target distribution proposed by these methods (a mix of
Wikipedia [Foundation, 2022], Books1 [Presser, 2021], and OpenWebText
[Gokaslan et al., 2019], cf. Appendix D.1).

Figure 3: Left: mean benchmark performance, varying training compute budget.
Right: mean performance for 1.3B models. Randomly selecting data is a strong
baseline: no active selection baseline outperforms random selection. In
contrast, DsDm outperforms all baselines across compute budgets (left panel),
and even matches training with 2× the compute on randomly selected data (when
training 1.3B models, right panel). Our train budgets correspond to training
125M, 356M, 760M, and 1.3B parameter Chinchilla-optimal LMs. To contextualize
1.3B results, we show the performance of a model trained on randomly selected
data with 2× the 1.3B compute budget (i.e., a 1.8B Chinchilla-optimal model).
For more details, see Appendix D.1.

###  4.1 Results

In Figure 3, we display the mean benchmark performance of models trained with
each selection method, varying training compute budget. Randomly selecting
data is a strong baseline: all baseline methods generally match or perform
worse than random selection across training compute budgets (Figure 3 left).
In the case of Classifier and DSIR, we hypothesize that data selected via
similarity with a fixed source hurts model performance by trading off data
diversity for (qualitatively) “cleaner” data.

In contrast, DsDm is a 2× compute multiplier: DsDm yields 1.3B models that
match models trained with 2× the compute budget on randomly selected data
(Figure 3, right). Furthermore, across compute budgets, DsDm consistently
outperforms all selection baselines (Figure 3, left).

Table 1: Accuracies on the considered benchmarks for 1.3B models trained with each selection method, along with a model trained with 2× the 1.3B compute budget on randomly selected data (a 1.8B model; Chinchilla-optimal models with larger parameter counts train with more tokens as well). In parentheses, we contextualize accuracy with the difference compared to a 1.3B model trained on randomly selected data. |  | Accuracy (%)
---|---|---
| Model Parameters | 1.3B | 1.8B
| Method | DsDm | Random | Classifier | DSIR | SeDeDup | Random
Category | Benchmark |  |  |  |  |  |
Commonsense Reasoning | copa |  63.0 (+1) |  62.0 (+0) |  66.0 (+4) |  67.0 (+5) |  68.0 (+6) |  64.0 (+2)
openbook_qa |  31.2 (–2) |  33.4 (+0) |  32.0 (–1) |  32.0 (–1) |  32.2 (–1) |  33.6 (+0)
piqa |  69.0 (+0) |  68.9 (+0) |  69.4 (+1) |  65.7 (–3) |  69.7 (+1) |  71.5 (+3)
Language Understanding | cbt |  88.2 (+2) |  86.4 (+0) |  85.1 (–1) |  92.4 (+6) |  86.2 (+0) |  88.4 (+2)
hellaswag |  42.3 (–3) |  44.9 (+0) |  42.7 (–2) |  40.4 (–5) |  44.9 (+0) |  50.1 (+5)
winogrande |  51.1 (–1) |  52.2 (+0) |  50.5 (–2) |  55.3 (+3) |  50.3 (–2) |  50.9 (–1)
Reading Comprehension | boolq |  58.0 (+3) |  54.9 (+0) |  60.9 (+6) |  61.0 (+6) |  49.9 (–5) |  53.4 (–2)
coqa |  25.5 (+7) |  18.8 (+0) |  16.7 (–2) |  16.5 (–2) |  22.9 (+4) |  24.9 (+6)
news_qa |  15.6 (+8) |  7.5 (+0) |  5.1 (–2) |  5.5 (–2) |  8.6 (+1) |  9.5 (+2)
Symbolic Problem Solving | bb_copy_logic |  3.1 (+0) |  3.1 (+0) |  0.0 (–3) |  0.0 (–3) |  3.1 (+0) |  3.1 (+0)
bb_dyck_lang |  11.9 (–2) |  13.5 (+0) |  3.4 (–10) |  1.0 (–13) |  7.3 (–6) |  8.9 (–5)
bb_operators |  13.3 (+3) |  10.5 (+0) |  6.7 (–4) |  10.5 (+0) |  11.4 (+1) |  9.5 (–1)
World Knowledge | arc_easy |  47.6 (+3) |  44.8 (+0) |  44.7 (+0) |  39.6 (–5) |  43.5 (–1) |  48.5 (+4)
bb_qa_wikidata |  48.1 (+8) |  40.6 (+0) |  48.3 (+8) |  37.7 (–3) |  45.5 (+5) |  53.6 (+13)
trivia_qa |  7.1 (+3) |  3.7 (+0) |  2.5 (–1) |  3.5 (+0) |  2.4 (–1) |  4.1 (+0)

Going beyond aggregate performance, we find that DsDm greatly improves on
benchmarks related to the target tasks, while simultaneously not reducing
performance on unrelated categories (on average). More precisely, inspecting
individual benchmark performance in Table 1, DsDm most improves reading
comprehension and world knowledge benchmarks compared to selecting randomly.
We hypothesize that our choice of target tasks leads to improved performance
on these benchmarks (which are qualitatively similar to SQuAD, a reading
comprehension task, and Jeopardy, a world knowledge task). Furthermore, in
these categories DsDm consistently matches or outperforms training with 2× the
compute budget on randomly selected data (i.e., the 1.8B model in Table 1).
Crucially, DsDm improves on these categories while also not reducing
performance in other categories. As a comparison, DSIR—which targets mostly
formal text—performs well on language understanding tasks but poorly on other
categories (e.g., world knowledge and symbolic problem solving).

Figure 4: Per-category performance for 760M models trained with DsDm-selected
data, varying target task. DsDm target tasks generally improve performance on
(qualitatively) related benchmark task categories. Specifically, models
targeted towards SQuAD/Jeopardy/LAMBADA improve accuracy on reading
comprehension/world knowledge/language understanding, respectively. Targeting
all three tasks at once improves overall accuracy. However, target tasks can
also reduce performance on (qualitatively) unrelated tasks. For example,
targeting with LAMBADA (a language understanding task) reduces performance on
world knowledge tasks compared to randomly selecting. See each category’s
constituent benchmarks in Table 1. We plot accuracy improvement compared to
randomly guessing on benchmarks (e.g., some benchmarks are multiple-choice).

##### Target tasks improve performance on qualitatively similar benchmarks.

So far, we have only targeted DsDm with a mix of LAMBADA, Jeopardy and SQuAD.
How does target task choice change model behavior? We find that targeting a
specific task generally improves performance on qualitatively related tasks.
To demonstrate, in Figure 4 we display accuracy by benchmark category while
varying target task across LAMBADA, Jeopardy, SQuAD, and all at once. Here,
targeting a task generally improves accuracy on related tasks, e.g., SQuAD
most improves reading comprehension, and Jeopardy most improves world
knowledge. Furthermore, targeting all tasks at once improves overall accuracy
the most. However, targeting can also decrease accuracy on unrelated tasks.
For example, targeting LAMBADA, a language understanding task, reduces world
knowledge accuracy compared to randomly selecting data. Our results suggest
that we can tailor target tasks to improve deployment-time performance, but
also that we need to be careful to choose targets that are diverse enough to
capture a range of downstream problems.

#####  DsDm is necessary to improve performance (with the targeted tasks).

DsDm selections yield much better models than Classifier and DSIR selections.
However, we have not yet compared these selection methods head-to-head with
the same target task. Classifier and DSIR target a mix of “high quality”
sources, while DsDm targets three LM tasks (Jeopardy, SQuAD, and LAMBADA). To
what extent does selecting with DsDm drive performance compared to the
difference in target tasks? We demonstrate that selecting with DsDm is
necessary to improve performance on the considered target tasks. Specifically,
we train models on data selected with DSIR and Classifier targeting LAMBADA,
Jeopardy and SQuAD, and find that (just as when targeting “high quality text”)
neither outperforms randomly selecting data (cf. Appendix Figure 25).

##  5 Discussion

Ostensibly, the sole goal of our dataset selection framework is improve model
performance by better selecting training data. However, one can view our
framework more broadly. That is, one can also use our framework to select data
that boosts any chosen downstream property of our trained models—not just
performance on a given benchmark. In this sense, our framework (and
accompanying method) unlocks data curation as another stage of the model
training pipeline that we can intervene on to control the downstream model
behavior in a fine-grained manner. Below, we discuss in more detail the
broader opportunities this view opens up as well as the other aspects of the
framework, such as proxy modeling and computational efficiency.

##### Applications and broader opportunities.

DsDm can optimize for any specified downstream model behavior. Indeed, by an
appropriate choice of the target tasks, we can use our framework to improve a
wide range of model behaviors, including: “aligning” models at pretraining
time (in addition to or in place of existing methods, which typically operate
post model training [Bai et al., 2022, Ziegler et al., 2019, Taori et al.,
2023]); optimizing for notions of fairness; and improving performance on
specific domains of interest (such as low-resource languages or programming).

##### Training stronger models with weaker proxy models.

We select data for large models by using smaller models to proxy large model
behavior (recall that we use DsDm to select data for smaller proxy models,
then train large models on these selections). Despite that these proxy models
are much worse than larger models on benchmarks (cf. Appendix Table 2), the
corresponding selections nonetheless greatly improve performance. Furthermore,
training on proxy models’ selections is the simplest possible approach to
scaling. Therefore, we suspect that scaling to larger models less naïvely
could yield even better results. More broadly, our findings are in line with
previous work showing that smaller models can still be leveraged to determine
better training hyperparameters for larger models [Kaplan et al., 2020,
Coleman et al., 2020, Hoffmann et al., 2022, Yang et al., 2022, Xie et al.,
2023a].

##### Computational cost.

DsDm is relatively inexpensive to compute in practical model training
scenarios. At a high level, the most expensive part of estimating DsDm is
computing the gradient for each training example on a handful of small proxy
models (in our case, four 125M parameter LMs—see Appendix B.5 for a full cost
breakdown). To contextualize DsDm cost with model training: computing
gradients also dominates the cost of training LMs. Since the cost of computing
a 125M model gradient is orders of magnitude lower than the cost of computing
gradients for standard model sizes,333For reference: models trained today
generally range from 3B to 175B parameters. The cost of a gradient is
(roughly) linear in model size, so it is $24\times$ to $1400\times$ more
expensive to compute gradients for these models vs. 125M models. even a small
compute multiplier (let alone the 2× improvement DsDm seems to offer) quickly
makes the overhead of computing DsDm worthwhile. Additionally, after computing
DsDm on a set of datapoints once, the cost of computing DsDm on those
datapoints again is essentially negligible (as the required computations are
easy to cache). Therefore, we can amortize DsDm’s computational cost over the
entire “lifetime” of training on the given dataset.

##  6 Related Work

Current methods for selecting LM pretraining datasets tend to follow a two-
step framework: (a) choose an intuitively “high quality” reference corpus,
like Wikipedia [Foundation, 2022], then (b) select data that matches it. There
are two standard methods that adhere to this framework: DSIR (Dataset
Selection with Importance Reweighting [Xie et al., 2023b]) and Classifier
(originally introduced in Brown et al. [2020] and used by other work [Gao et
al., 2020, Chowdhery et al., 2022, Du et al., 2022]). Other work on selecting
data for LM pretraining has included deduplicating examples in LM activation
space [Abbas et al., 2023], and selecting examples with the largest difference
in loss between LMs trained on the candidate and reference sets [Moore and
Lewis, 2010, Axelrod, 2017, Feng et al., 2022]. Simpler methods for selecting
data are also commonplace. These include removing documents that are too short
or contain too many special characters [Raffel et al., 2020, Computer, 2023,
Xie et al., 2023b]. In the LM domain, a related (but different) task to
dataset selection is choosing weights for sampling from mixtures of data
sources [Chen et al., 2023b, Xie et al., 2023a, Albalak et al., 2023].

Beyond LM pre-training, previous work also selects data in other domains.
These works aim to: improve the performance on a given task [Wei et al., 2015,
Kaushal et al., 2019, Wang et al., 2020, Killamsetty et al., 2021a, Chitta et
al., 2021, Mindermann et al., 2022], identify core-sets of large training
datasets [Sener and Savarese, 2017, Phillips, 2017, Coleman et al., 2020,
Mirzasoleiman et al., 2020, Paul et al., 2021, Killamsetty et al., 2021b,
Okanovic et al., 2023], and instruction fine-tune LMs [Chen et al., 2023a, Cao
et al., 2023]. Broadly, such methods select by prompting pretrained models,
discriminating on proxies for model uncertainty like loss or gradient norm,
matching on gradients, or deduplicating in model output space.

##  7 Conclusion

In this work, we cast dataset selection as an optimization problem: given
target tasks, a learning algorithm, and a candidate training dataset, choose
the training maximizes performance. We then propose a method for approximating
the solution to this optimization problem, DsDm, that selects by modeling how
the learning algorithm uses training data to predict on the target tasks. We
show that our method reliably improves target task performance in the LM
setting, and furthermore use our framework to improve broader model
generalization. By choosing target tasks similar to those we expect to see at
deployment time, we can greatly improve model performance on yet unseen tasks.

Our findings prompt us to take on a much broader view of the role of dataset
selection stage in model training. In particular, our framework demonstrates
that dataset selection can be an effective tool for fine-grain control of
model behavior. Indeed, we hypothesize that carefully choosing data can not
only improve downstream task performance, but also other downstream properties
of trained models, such as notions of predictor fairness, alignment with human
preferences, or capabilities in specific domains like low-resource languages
or programming. We also suspect that current methods for datamodeling only
scratch the surface of understanding how models learn from data—and that we
can greatly improve our ability to manipulate model behavior through training
data by developing better datamodeling techniques.

#### Acknowledgments

LE funded by the Google PhD Fellowship. Work supported in part by the NSF
grants CNS-1815221 and DMS-2134108, and Open Philanthropy. This material is
based upon work supported by the Defense Advanced Research Projects Agency
(DARPA) under Contract No. HR001120C0015.

## References

  * Abbas et al. [2023] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication.  _arXiv preprint arXiv:2303.09540_ , 2023.
  * Albalak et al. [2023] Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang.  Efficient online data mixing for language model pre-training.  _Training_ , 20000(40000):60000, 2023.
  * Axelrod [2017] Amittai Axelrod.  Cynical selection of language model training data.  _arXiv preprint arXiv:1709.02279_ , 2017.
  * Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR, 2023.
  * Bierens [1988] Hermanus Josephus Bierens.  The nadaraya-watson kernel regression function estimator.  1988\.
  * Bisk et al. [2019] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language, 2019.
  * Black et al. [2022] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach.  Gpt-neox-20b: An open-source autoregressive language model, 2022.  URL https://arxiv.org/abs/2204.06745.
  * Brown et al. [2020] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Cao et al. [2023] Yihan Cao, Yanbin Kang, and Lichao Sun.  Instruction mining: High-quality instruction data selection for large language models.  _arXiv preprint arXiv:2307.06290_ , 2023.
  * Chen et al. [2023a] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al.  Alpagasus: Training a better alpaca with fewer data.  _arXiv preprint arXiv:2307.08701_ , 2023a.
  * Chen et al. [2023b] Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré.  Skill-it! a data-driven skills framework for understanding and training language models.  _arXiv preprint arXiv:2307.14430_ , 2023b.
  * Chitta et al. [2021] Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet.  Training data subset search with ensemble active learning.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(9):14741–14752, 2021.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv:1803.05457v1_ , 2018.
  * Coleman et al. [2020] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia.  Selection via proxy: Efficient data selection for deep learning.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Computer [2023] Together Computer.  Redpajama: an open dataset for training large language models.  https://github.com/togethercomputer/RedPajama-Data, October 2023.
  * Cook [1977] R Dennis Cook.  Detection of influential observation in linear regression.  _Technometrics_ , 19(1):15–18, 1977.
  * Du et al. [2022] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al.  Glam: Efficient scaling of language models with mixture-of-experts.  In _International Conference on Machine Learning_ , pages 5547–5569. PMLR, 2022.
  * Feng et al. [2022] Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc.  Automatic document selection for efficient encoder pretraining.  _arXiv preprint arXiv:2210.10951_ , 2022.
  * Foundation [2022] Wikimedia Foundation.  English wikipedia.  https://huggingface.co/datasets/wikipedia, 2022.
  * Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Giordano et al. [2019] Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick.  A swiss army infinitesimal jackknife.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 1139–1147. PMLR, 2019.
  * Gokaslan et al. [2019] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * Hill et al. [2015] Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston.  The goldilocks principle: Reading children’s books with explicit memory representations.  _arXiv preprint arXiv:1511.02301_ , 2015.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  In _arXiv preprint arXiv:2203.15556_ , 2022.
  * Ilyas et al. [2022] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Predicting predictions from training data.  In _International Conference on Machine Learning (ICML)_ , 2022.
  * Johnson and Lindenstrauss [1984] William B Johnson and Joram Lindenstrauss.  Extensions of lipschitz mappings into a hilbert space.  In _Contemporary mathematics_ , 1984.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In Regina Barzilay and Min-Yen Kan, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1147.  URL https://aclanthology.org/P17-1147.
  * Joulin et al. [2016] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_ , 2016.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kaushal et al. [2019] Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan.  Learning from less data: A unified data subset selection and active learning framework for computer vision.  In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1289–1299. IEEE, 2019.
  * Khaddaj et al. [2022] Alaa Khaddaj, Guillaume Leclerc, Aleksandar Makelov, Kristian Georgiev, Andrew Ilyas, Hadi Salman, and Aleksander Madry.  Backdoor or feature? a new perspective on data poisoning.  2022\.
  * Killamsetty et al. [2021a] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer.  Glister: Generalization based data subset selection for efficient and robust learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 8110–8118, 2021a.
  * Killamsetty et al. [2021b] Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer.  Retrieve: Coreset selection for efficient and robust semi-supervised learning.  _Advances in Neural Information Processing Systems_ , 34:14488–14501, 2021b.
  * Koh and Liang [2017] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _International Conference on Machine Learning_ , 2017.
  * Liu et al. [2018] Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.  Generating wikipedia by summarizing long sequences.  _arXiv preprint arXiv:1801.10198_ , 2018.
  * Mihaylov et al. [2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_ , 2018.
  * Mindermann et al. [2022] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al.  Prioritized training on points that are learnable, worth learning, and not yet learnt.  In _International Conference on Machine Learning_ , pages 15630–15649. PMLR, 2022.
  * Mirzasoleiman et al. [2020] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec.  Coresets for data-efficient training of machine learning models.  In _International Conference on Machine Learning_ , pages 6950–6960. PMLR, 2020.
  * Moore and Lewis [2010] Robert C Moore and William Lewis.  Intelligent selection of language model training data.  In _Proceedings of the ACL 2010 conference short papers_ , pages 220–224, 2010.
  * MosaicML [2021] MosaicML.  Composer, 2021.  URL https://www.github.com/mosaicml/composer.
  * MosaicML [2023] MosaicML.  LLM Foundry, 2023.  URL https://www.github.com/mosaicml/llm-foundry.
  * Okanovic et al. [2023] Patrik Okanovic, Roger Waleffe, Vasilis Mageirakos, Konstantinos E Nikolakakis, Amin Karbasi, Dionysis Kalogerias, Nezihe Merve Gürel, and Theodoros Rekatsinas.  Repeated random sampling for minimizing the time-to-accuracy of learning.  _arXiv preprint arXiv:2305.18424_ , 2023.
  * Paperno et al. [2016] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The lambada dataset: Word prediction requiring a broad discourse context.  _arXiv preprint arXiv:1606.06031_ , 2016.
  * Park et al. [2023] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry.  Trak: Attributing model behavior at scale.  In _Arxiv preprint arXiv:2303.14186_ , 2023.
  * Paul et al. [2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  _Advances in Neural Information Processing Systems_ , 34:20596–20607, 2021.
  * Phillips [2017] Jeff M Phillips.  Coresets and sketches.  In _Handbook of discrete and computational geometry_ , pages 1269–1288. Chapman and Hall/CRC, 2017.
  * Pregibon [1981] Daryl Pregibon.  Logistic regression diagnostics.  In _The Annals of Statistics_ , 1981.
  * Presser [2021] Shawn Presser.  Bookcorpusopen.  https://huggingface.co/datasets/bookcorpusopen, 2021.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_ , 2020.
  * Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_ , 2016.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  Coqa: A conversational question answering challenge, 2019.
  * Roemmele et al. [2011] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_ , 2011.
  * Sakaguchi et al. [2021] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106, 2021.
  * Saunshi et al. [2023] Nikunj Saunshi, Arushi Gupta, Mark Braverman, and Sanjeev Arora.  Understanding influence functions and datamodels via harmonic analysis.  In _ICLR_ , 2023.
  * Sener and Savarese [2017] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_ , 2017.
  * Srivastava et al. [2022] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_ , 2022.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Trischler et al. [2016] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman.  Newsqa: A machine comprehension dataset.  _arXiv preprint arXiv:1611.09830_ , 2016.
  * Tunguz [2019] Bojan Tunguz.  Jeopardy! questions, 2019.  URL https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions.
  * Wang et al. [2020] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig.  Optimizing data usage via differentiable rewards.  In _International Conference on Machine Learning_ , pages 9983–9995. PMLR, 2020.
  * Wei et al. [2015] Kai Wei, Rishabh Iyer, and Jeff Bilmes.  Submodularity in data subset selection and active learning.  In _International conference on machine learning_ , pages 1954–1963. PMLR, 2015.
  * Xie et al. [2023a] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu.  Doremi: Optimizing data mixtures speeds up language model pretraining.  _arXiv preprint arXiv:2305.10429_ , 2023a.
  * Xie et al. [2023b] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang.  Data selection for language models via importance resampling.  _arXiv preprint arXiv:2302.03169_ , 2023b.
  * Yang et al. [2022] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.  Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.  _arXiv preprint arXiv:2203.03466_ , 2022.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_ , 2019.
  * Zhang et al. [2010] Yin Zhang, Rong Jin, and Zhi-Hua Zhou.  Understanding bag-of-words model: a statistical framework.  _International journal of machine learning and cybernetics_ , 2010.
  * Zhu and Hastie [2005] Ji Zhu and Trevor Hastie.  Kernel logistic regression and the import vector machine.  _Journal of Computational and Graphical Statistics_ , pages 185–205, 2005.
  * Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_ , December 2015.
  * Ziegler et al. [2019] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

###### Contents

  1. 1 Introduction
  2. 2 Estimating the optimal dataset selection with DsDm
    1. 2.1 Task-optimal dataset selection
    2. 2.2 Estimating model loss efficiently with datamodels
    3. 2.3 DsDm: Dataset Selection with Datamodels
  3. 3 Evaluating DsDm
    1. 3.1 Setup
    2. 3.2 Results
  4. 4 Selecting data for broad model capabilities
    1. 4.1 Results
  5. 5 Discussion
  6. 6 Related Work
  7. 7 Conclusion
  8. A Experimental Setup
    1. A.1 Candidate dataset
    2. A.2 Target tasks
      1. A.2.1 Mitigating train-test leakage
    3. A.3 Data selection baselines
      1. A.3.1 Targeted baselines
      2. A.3.2 Untargeted baselines
    4. A.4 LM training details
    5. A.5 Evaluation metrics
      1. A.5.1 Log-probability
      2. A.5.2 Accuracy
  9. B Datamodel estimation
    1. B.1 Datamodels refresher
      1. B.1.1 Estimating datamodels with data regression
    2. B.2 Estimating datamodels with TRAK
      1. B.2.1 Datamodels for logistic regression
      2. B.2.2 Transforming learning algorithms to linear regression
      3. B.2.3 TRAK estimator
    3. B.3 Datamodels for language modeling
    4. B.4 TRAK setup
    5. B.5 Computational cost
  10. C Evaluating task-optimal dataset selection
    1. C.1 Experimental setup
    2. C.2 Omitted figures
    3. C.3 Sample dataset selections
  11. D Evaluating data selections for broad model performance
    1. D.1 Experimental setup
    2. D.2 Omitted figures

##  Appendix A Experimental Setup

In this section we discuss general experimental setup, including candidate
data pool, considered target tasks, baselines, evaluation metrics, and model
training choices.

###  A.1 Candidate dataset

Our candidate dataset is the full English subset of C4 [Raffel et al., 2020].
We use the train split of the en.noblocklist subset of the C4 version prepared
by AllenAI at https://huggingface.co/datasets/c4. The subset name noblocklist
signifies that curse words were not filtered in the subset.

To split the text from the documents into examples, we tokenize all the
documents, concatenate them together (separated by end-of-text tokens), and
then slice the result into 1024 token chunks. These 1024 token examples
generally contain between 3,000 and 6,000 characters (roughly a thousand
words). The final candidate dataset has 216,948,746 examples. We tokenize with
the Pythia tokenizer [Black et al., 2022, Biderman et al., 2023].

As a public internet crawl, C4 contains diverse text. To contextualize the
dataset, we show (excerpts) of random C4 samples in Figure 24.

###  A.2 Target tasks

We describe each of the considered target tasks below. We both describe the
tasks, and how we split samples into distinct sets of “target samples” (to
select datasets for a target task) and “holdout samples” (to evaluate models
on the target task):

  * •

SQuAD. The Stanford Question-Answering Dataset (SQuAD [Rajpurkar et al.,
2016]) is an open book, reading comprehension dataset of questions about
Wikipedia articles. The goal is to answer questions using the corresponding
article as context. Our target set is 25% of the SQuAD train set (23107
examples), our holdout set is the SQuAD validation set (10557 examples).

  * •

Jeopardy. Jeopardy [Tunguz, 2019] is a set of trivia questions taken directly
from the show “Jeopardy!” We use the version of Jeopardy published by MosaicML
[MosaicML, 2023].444Located at: https://github.com/mosaicml/llm-
foundry/blob/v0.2.0/scripts/eval/local_data/world_knowledge/jeopardy_all.jsonl
We include all the samples save for the “Word Origins” subset.555We originally
intended this subset as a hold-out set for our broader evaluation, but decided
not to use the subset as we deemed it unfairly close to the original task to
serve as a true hold-out set. We randomly partition the remaining samples into
876 target samples and 876 holdout samples.

  * •

LAMBADA. LAnguage Modeling Broadened to Account for Discourse Aspects (LAMBADA
[Paperno et al., 2016]) is an open-ended cloze task measuring broad context
text understanding. The goal is to predict the last word of curated passages
from BooksCorpus [Zhu et al., 2015] given the rest of the passage as context.
The task is meant to be challenging: Paperno et al. [2016] only select
passages such that crowdworkers could not guess the final word given the final
sentence alone (up until the final word), but could guess the final word given
the entire passage. We use the LAMBADA version curated by
EleutherAI.666Located at
https://huggingface.co/datasets/EleutherAI/lambada_openai/viewer/en. Finally,
we split the LAMBADA test set into separate target and holdout sets, then
remove 6 samples from the LAMBADA holdout set due to overlap with samples in
our candidate train dataset (cf. Subsection A.2.1 for details on this
procedure). We conclude with 2570 holdout samples and 2577 target samples.

  * •

CS-Algorithms. BIG-bench CS Algorithms [Srivastava et al., 2022] measures the
ability of models to solve basic algorithmic problems. In particular, this
benchmark contains two kinds of problems: testing for balanced parentheses,
and finding the longest common subsequence of multiple strings. For each
considered example, the goal is to directly output the answer to the posed
algorithmic question. We randomly split the test set into 660 target samples
and 660 holdout samples.

We include samples of each benchmark in Figure 5 (SQuAD), Figure 6 (Jeopardy),
Figure 7 (LAMBADA), and Figure 8 (CS-Algorithms). We evaluate in the 0-shot
(for LAMBADA and CS-Algorithms) and 3-shot (for SQuAD and Jeopardy) regimes.
In the 3-shot setting, we separate each example with a single newline. We use
standard prompts for each task (see the samples for details).

  1. 1.

Context: The chloroplasts of some hornworts and algae contain structures
called pyrenoids. They are not found in higher plants. Pyrenoids are roughly
spherical and highly refractive bodies which are a site of starch accumulation
in plants that contain them. They consist of a matrix opaque to electrons,
surrounded by two hemispherical starch plates. The starch is accumulated as
the pyrenoids mature. In algae with carbon concentrating mechanisms, the
enzyme rubisco is found in the pyrenoids. Starch can also accumulate around
the pyrenoids when CO2 is scarce. Pyrenoids can divide to form new pyrenoids,
or be produced “de novo”.
Question: What shape are pyrenoids?
Answer: roughly spherical

  2. 2.

Context: In this dioxygen, the two oxygen atoms are chemically bonded to each
other. The bond can be variously described based on level of theory, but is
reasonably and simply described as a covalent double bond that results from
the filling of molecular orbitals formed from the atomic orbitals of the
individual oxygen atoms, the filling of which results in a bond order of two.
More specifically, the double bond is the result of sequential, low-to-high
energy, or Aufbau, filling of orbitals, and the resulting cancellation of
contributions from the 2s electrons, after sequential filling of the low
$\sigma$ and $\sigma^{*}$ orbitals; $\sigma$ overlap of the two atomic $2p$
orbitals that lie along the O-O molecular axis and $\pi$ overlap of two pairs
of atomic $2p$ orbitals perpendicular to the O-O molecular axis, and then
cancellation of contributions from the remaining two of the six 2p electrons
after their partial filling of the lowest $\pi$ and $\pi^{*}$ orbitals.
Question: What is a descriptive term for a low-to-high energy bond?
Answer: Aufbau

Figure 5: Random SQuAD samples. Context is normal text, and the continuation
label is hightlighted.

  1. 1.

WORLD HISTORY: In 1191 this Lion-Hearted king of England captured Cyprus &
Acre during the Crusades
Answer: Richard I

  2. 2.

LITERATURE: 1719 novel about a mariner who lived 8 & 20 years all alone in an
uninhabited island
Answer: Robinson Crusoe

Figure 6: Random Jeopardy samples. Context is normal text, and the
continuation label is hightlighted.

  1. 1.

The Simplification Movement wasn’t really an organized movement. It was more
of an ideological shift by a large number of believers. There were quite a few
Simpletons among the Mother Assembly denomination, but the High Sire had never
recognized their movement as an order or organization. However, some other
denominations were founded on the principles of the Simplification Movement

  2. 2.

“Here,” said Jacob, handing them what was a rope attached to the ground next
to them, the other end at the bottom of the well. “You first.”
Will stood there. Why am I doing this? he thought.
“Come on, let’s go!” ordered Jacob.
Will took the rope and began to climb down the well.
“Thatta boy, you’ve got this,” said Jacob

Figure 7: Random LAMBADA samples. We show the context as normal text, and the
continuation label as hightlighted.

  1. 1.

Given two strings, determine the length of the longest common subsequence.

Strings: REFVJLZIV PJIQB
Length of longest common subsequence: 2

  2. 2.

Determine whether the given sequence of parentheses is properly matched.

Sequence: [ ] ( ) ( ( ( ) ) )
Valid/Invalid? Valid

Figure 8: Random CS-Algorithms samples. We show the context as normal text,
and the continuation label as hightlighted.

####  A.2.1 Mitigating train-test leakage

We mitigate train-test leakage by filtering out test examples that overlap
with our candidate data samples. Specifically, we define a test example as
“leaked” if both its context and continuation are present in a single C4
example. To upper-bound train-test leakage, we test for the context and
continuation separately (i.e., for a given test sample, the context and
continuation do not have to be contiguous in a train sample to count as
leaked) We investigate train-test leakage for all the test examples in each of
the test sets (i.e., LAMBADA, SQuAD, Jeopardy, and CS-Algorithms) across the
entire candidate train set (i.e., the C4 English subset). Note that we match
strings after lowercasing and removing whitespace.

We find 6 LAMBADA test examples with overlap in C4, and remove them from our
LAMBADA test split. We do not find any train-test leakage for SQuAD, Jeopardy,
or CS-Algorithms.

###  A.3 Data selection baselines

We consider four baselines for selecting language modeling data. These fall
into two categories: targeted data selection methods (which select data
according to a target distribution), and untargeted data selection methods
(which do not take in a target distribution).

####  A.3.1 Targeted baselines

The two targeted dataset selection methods we consider, Classifier (originally
used to select the GPT-3 dataset [Brown et al., 2020]) and DSIR, both select
according to textual similarity with a target distribution. We describe the
details of these methods below:

#####  Classifier.

The dataset selection method originally developed to select data for GPT-3,
and additionally used to select data for PaLM [Chowdhery et al., 2022] and The
Pile [Gao et al., 2020]. The method trains a logistic regression model on
FastText [Joulin et al., 2016] features to classify between (held-out) samples
of the candidate dataset (in our case, C4) and the target distribution, then
chooses training data according to how likely the model predicts the data as
being sampled from the target distribution. To more specifically describe
Classifier: the method keeps a given document if the scored document
satisfies:

$\epsilon>1-\mathtt{document\\_score},\epsilon\sim\mathrm{Lomax}(\alpha),$

where a Lomax sample is drawn for each considered document, and where
$\mathtt{document\\_score}$ is the classifier-given probability that the given
sample is in the target distribution. Sampling a threshold according to the
Lomax distribution is meant to improve diversity of the selected data. In this
work, we learn the classifier on the C4 en.noblocklist validation set, and
choose $\alpha=12$ via the parameter selection procedure described in Brown et
al. [2020] (score each document in C4 with the classifier, then fit the
parameters of a Lomax distribution via maximum likelihood estimation according
to these scores).

#####  DSIR.

Dataset Selection with Importance Resampling [Xie et al., 2023b] aims to
select a data subset with a similar distribution as the target task in terms
of n-gram counts. DSIR comprises two steps: (a) find the (hashed) n-gram
counts for each train set example (each example is represented as a vector of
counts, with n-grams hashed into buckets to reduce dimensionality), then (b)
importance sample to select candidate train set examples that are distributed
similarly to target distribution samples in terms of n-gram counts. DSIR
calculates importance weights by modeling the distribution of examples (in
feature space) under the target distribution and under the candidate data
distribution separately, using bag-of-words style models. In greater detail,
DSIR consists of the following steps:

  1. 1.

Fit $\hat{p}_{\mathrm{feat}}$ and $\hat{q}_{\mathrm{feat}}$, estimates of the
distributions of target examples and candidate training examples in hashed
n-gram space (respectively). DSIR parameterizes $\hat{p}_{\mathrm{feat}}$ and
$q_{\mathrm{feat}}$ through the following general procedure for estimating the
distribution of hashed n-grams777For example, if we wanted to make a $d$
dimensional hashed n-gram feature vector for a document, we would find all the
n-grams in the document, hash the n-grams into integers up to size $d$, then
go through each integer and increment the corresponding feature vector index.
for a given set of documents. First, calculate the hashed n-gram counts (with
$d$ hash buckets) across the documents as the vector
$\gamma\in\mathbb{R}^{d}$, where $\gamma_{k}$ corresponds to the number of
n-grams that hash to $k$ in the documents. Then, normalize $\gamma$ so that
its values sum to $1$, forming a probability distribution over buckets.
Finally, parameterize the distribution of hashed n-grams for this set of
documents as a bag-of-words style model [Zhang et al., 2010] such that the
probability of a document with hashed n-gram counts $c$ is
$\prod_{i=1}^{d}\gamma_{d}^{c_{i}}$ (here, the bag-of-words model is over
hashed n-grams instead of words).

  2. 2.

Calculate importance weights for each example in the candidate training set,
such that example $i$ with counts $c$ has weight
$w_{i}=\frac{\hat{p}_{\mathrm{feat}}(c)}{\hat{q}_{\mathrm{feat}}(c)}$.

  3. 3.

Sample examples without replacement according to the categorical distribution
with (unscaled) weights $w_{i}$.

For more details on DSIR, see Section 4 of Xie et al. [2023b]. We adapt
implementations of both DSIR and Classifier from
https://github.com/p-lambda/dsir.

##### Considered target distributions.

We apply targeted dataset selection methods with different target
distributions depending on the context. In Section 3, we measure the extent to
which different selection methods can reduce loss on individual target tasks,
so we select data for individual tasks (i.e., Jeopardy, SQuAD, CS-Algorithms,
and LAMBADA). In Section 4 we use these targeted baselines to select data for
general purpose language modeling, so we use the recommended target task from
each work (intuitively high-quality data sources; see Appendix D.1 for more
details).

####  A.3.2 Untargeted baselines

The two untargeted dataset selection methods we consider are: Random (select
data randomly) and SemDeDup (Semantic Deduplication [Abbas et al., 2023]).
SemDeDup selects by clustering data according to the last layer activations
for the last token in the given document, then choosing only the examples in
each cluster that have the lowest cosine similarity with the cluster centroid.
We follow the hyperparameters from the original work (11,000 clusters,
deduplicating down to 20% of the dataset for optimal model performance). We
use the implementation from https://github.com/facebookresearch/SemDeDup/.

###  A.4 LM training details

We train GPT-2 family decoder-only transformer models [Radford et al., 2019,
Liu et al., 2018] using LLM-Foundry [MosaicML, 2023]. To train models, we use
ADAM ($\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=10^{-8}$), sequence length 1024,
batch size 1024, a cosine learning rate schedule (with 200 warm up batches and
$\alpha=0.1$), and $\ell_{2}$ gradient clipping with threshold $1$. We train
on A100s (with BF16 precision) and H100s (with FP8 precision), and tokenize
text with the BPE tokenizer used by Pythia [Biderman et al., 2023].

We summarize the remaining hyperparameter choices used to train the models in
this work in Table 2 (including weight decay, learning rate, model
architecture, and training token count). We select all hyperparameters to
minimize 125M held-out perplexity on C4. The only exception: we increase the
weight decay for the Section 4 models to ensure that larger parameter model
training runs converge (with smaller weight decay, larger models diverge in
loss). Model parameterization choices (i.e., number of heads or layers),
optimizer hyperparameters, and learning rate schedule generally chosen
according to the default LM training configurations in LLM-Foundry.

##### Chinchilla-optimal compute ratios.

To train the best possible LM for a given compute budget, one must trade off
two hyperparameters that control used compute: model size and number of
training tokens. We use Chinchilla-optimal parameter-to-training-token ratios
to trade these parameters off [Hoffmann et al., 2022]. In our compute regime,
this (roughly) amounts to training on a number of tokens equal to 20× the
number of parameters.

Table 2: Training configurations for models trained across this work. Accuracy measured as the mean accuracy across the benchmarks considered in Section 4 for a model trained with randomly selected data with the corresponding configuration. The Chinchilla-optimal (760M, 1.3B, 1.8B) models [Hoffmann et al., 2022] of Section 4 are much more accurate than the 125M models used to calculate datamodels. Following previous work, we approximate FLOPs (Floating Point OPerations) via $\texttt{parameters}\times\texttt{tokens}\times 6$ [Kaplan et al., 2020, Hoffmann et al., 2022]; FLOPs proxy the computational cost of training a given model. LR is learning rate, WD is weight decay. Each batch contains 1024 samples of 1024 tokens each. | Hyperparameters |  |  |
---|---|---|---|---
Parameters | LR | WD | $d_{\textrm{model}}$ | Heads | Layers | Tokens | Batches | Train FLOPs |  Accuracy
Estimating datamodels |
125M | $6\times 10^{-4}$ | $2\times 10^{-4}$ | $768$ | $12$ | $12$ | $8.4\times 10^{10}$ | $80000$ | $6.3\times 10^{19}$ | $31.8\%$
Section 3: Evaluating optimal dataset selection estimators |
125M | $6\times 10^{-4}$ | $2\times 10^{-4}$ | $768$ | $12$ | $12$ | $2.6\times 10^{10}$ | $25000$ | $2.0\times 10^{19}$ | $-$
Section 4: Evaluating unseen-task generalization (chosen as ~Chinchilla-optimal) |
125M | $6\times 10^{-4}$ | $4\times 10^{-4}$ | $768$ | $12$ | $12$ | $2.5\times 10^{9}$ | $2400$ | $1.9\times 10^{18}$ | $26.6\%$
356M | $6\times 10^{-4}$ | $4\times 10^{-4}$ | $1024$ | $16$ | $24$ | $7.0\times 10^{9}$ | $6700$ | $1.5\times 10^{19}$ | $29.2\%$
760M | $6\times 10^{-4}$ | $4\times 10^{-4}$ | $1536$ | $12$ | $24$ | $1.5\times 10^{10}$ | $14400$ | $6.9\times 10^{19}$ | $32.9\%$
1.3B | $6\times 10^{-4}$ | $4\times 10^{-4}$ | $2048$ | $16$ | $24$ | $2.6\times 10^{10}$ | $24700$ | $2.0\times 10^{20}$ | $36.3\%$
1.8B | $6\times 10^{-4}$ | $4\times 10^{-4}$ | $2432$ | $19$ | $24$ | $3.7\times 10^{10}$ | $34931$ | $4.0\times 10^{20}$ | $38.3\%$

###  A.5 Evaluation metrics

In this work, we measure model performance using two different metrics: log-
probability (in Section 3, to compare model performance on target tasks) and
accuracy (in Section 4, to compare model performance on a broad set of yet
unseen tasks). Below, we describe how we measure both metrics.

####  A.5.1 Log-probability

To calculate mean log-probability, we compute the log-probability of the model
generating the correct label, then aggregate the mean across benchmark
samples. More specifically, all the tasks we evaluate with log-probability are
open-ended LM tasks (e.g., LAMBADA), where the goal is to generate a desired
continuation from the context (e.g., for LAMBADA, generate the last word of a
paragraph, given the rest of the paragraph as context). Therefore, the log-
probability of the model answering correctly is the log-probability that the
model generates the label, given the context. This is, for a sample $x$ with
$k$ continuation tokens starting at index $C$,

$\mathrm{Log\\_Probability}(x;g_{w}{})=\sum_{i=C}^{C+k}\log(p_{i}),\textrm{
where $p_{i}$ is the correct-label probability given by model $g_{w}$ at index
$i$}.$ (4)

####  A.5.2 Accuracy

To evaluate accuracy, we use one of three separate accuracy procedures
depending on the considered benchmark: (a) multiple choice accuracy, (b) exact
text match, or (c) fuzzy text match. These are:

  * •

Multiple choice accuracy: For multiple choice question benchmarks, we choose
the answer with the maximal predicted probability out of the possible choices,
then measure the accuracy as the fraction of correct answers.

  * •

Exact match: We mark an example as correct if the generated tokens for the
context exactly match the label tokens, then measure the accuracy as the
fraction of correct answers.

  * •

Fuzzy match: For open-ended benchmarks like TriviaQA whose questions have
multiple textually different but correct answers, we measure whether our model
is correct on a given example through the following procedure. We generate
text for the example context, then normalize this text with the standard
TriviaQA text normalizer888Default choice for this procedure accuracy
measurement in the MosaicML Composer [MosaicML, 2021], see
https://github.com/mandarjoshi90/triviaqa/blob/master/evaluation/triviaqa_evaluation.py
(which removes articles/extraneous white space/punctuation and normalizes
underscores/casing), and finally count the example as correct if the resulting
normalized text exactly matches any of the (normalized) labels. We then
measure accuracy as the fraction of correct answers.

Table 3 lists the exact accuracy procedure used for each considered benchmark.

##  Appendix B Datamodel estimation

In this section, we describe how we estimate datamodels for GPT-2 style LMs.
We start by briefly giving an overview of datamodels (cf. Appendix B.1), then
describe the datamodel estimator we use, TRAK (cf. Appendix B.2). Finally, we
conclude by instantiating datamodels for language modeling (cf. Appendix B.3),
and analyzing the computational cost of our procedure (cf. Appendix B.5). For
the impatient reader, we include a standalone section on how to mechanically
compute datamodel estimates with TRAK (without background) in Appendix B.4.

###  B.1 Datamodels refresher

The goal of datamodeling is to approximate the mapping from choice of training
subset to trained model loss on a given, fixed sample. Datamodels frame this
problem as a supervised learning problem: datamodels learn an approximation
from the former to the latter. Recall from Section 2.2 that the datamodel
$\tau_{\theta}$ for an example $x$ is a parameterized function that, given a
candidate training dataset $\mathcal{S}$, learning algorithm $\mathcal{A}$
(mapping train set to trained model), and model output function $f$ (in the
main text, we simplify by refering to this quantity as the loss $\ell$; but in
reality $f$ can capture any function of the trained model) that maps test
example and model to resulting loss, optimally predicts the model output on
$x$ over a (chosen) distribution of train subsets $\mathcal{D}_{\mathcal{S}}$,
i.e.,

$\displaystyle\tau_{\theta_{x}}:\\{0,1\\}^{|\mathcal{S}|}\to\mathbb{R},\qquad\text{
where }\qquad\theta_{x}=\arg\min_{\theta}\
\widehat{\mathbb{E}}^{(m)}_{S_{i}\sim\mathcal{D}_{\mathcal{S}}}\left[L_{{\text{reg}}}\left(\tau_{\theta}(\text{1}_{S_{i}}),\
f(x;\mathcal{A}(S))\right)\right],$ (5)

where $L_{{\text{reg}}}(\cdot,\cdot)$ is a regression loss function (e.g.,
mean squared error), and $\widehat{\mathbb{E}}^{(m)}$ is an $m$-sample
empirical expectation. Note that datamodels operate on the characteristic
vector $\text{1}_{S}$ of each subset (cf. Equation 2), not the subset
directly.

In this work, we parameterize $\tau_{\theta_{x}}$ as linear in the choice of
training data, i.e., such that

$\tau_{\theta_{x}}(\text{1}_{S})=\text{1}_{S}^{\top}{\theta_{x}}.$

Intuitively, such linear datamodels model each datapoint $\mathcal{S}_{i}$ as
having a constant effect on the loss when included in the training set (this
effect is exactly the value of $\theta_{x}$ in index $i$).

####  B.1.1 Estimating datamodels with data regression

So far we have only defined linear datamodels. How do we actually estimate the
linear parameters $\theta_{x}$? When introducing datamodels, Ilyas et al.
[2022] originally did so with a linear regression predicting loss from
training subset—i.e., directly minimizing Equation 5 by collecting a large
amount of “training data”—pairs of (randomly chosen training data subset,
corresponding trained model output on $x$)—then learning the mapping from
train subset to output on the collected training data.

This estimator, which we refer to as data regression, proceeds in two steps.
The first step is to collect regression data. Here, we repeatedly: sample a
random train subset $S_{i}$ (from a chosen distribution
$S_{i}\sim\mathcal{D}_{S}$999A standard choice is uniformly random subsets of
a fixed size.), train a model $\mathcal{A}(S_{i})$ on the subset, then
evaluate the model output on $x$ (and record the train subset, model output
pairs). This step yields “training data” for the regression in the form of $m$
train subset, loss pairs:
$\\{\left(\text{1}_{S_{i}},\ell(x;\mathcal{A}\left(S_{i}\right))\right)\\}_{i=1}^{m}$
(recall that our datamodel takes as input the characteristic vector of subsets
rather than subsets directly). Then, the second step is to actually estimate
the linear datamodel parameters with linear regression. Here, the regression
minimizes the (empirical) squared error over datamodel parameters:

$\displaystyle\theta_{x}$
$\displaystyle=\operatorname*{arg\,min}_{\theta}\widehat{\mathbb{E}}^{(m)}_{S_{i}\sim\mathcal{D}_{\mathcal{S}}}\left[L_{{\text{reg}}}\left(\tau_{\theta}(\text{1}_{S_{i}}),\
\ell(x;\mathcal{A}(S))\right)\right]$
$\displaystyle=\operatorname*{arg\,min}_{\theta}\widehat{\mathbb{E}}^{(m)}_{S_{i}\sim\mathcal{D}_{\mathcal{S}}}\left[\left(\text{1}_{S_{i}}^{\top}\theta-\ell(x;\mathcal{A}(S))\right)^{2}\right].$

Linear regression estimates the datamodel parameters directly, and
asymptotically yields the true datamodel parameters (with enough “training
data,” or pairs of training subset, corresponding trained model output).

While data regression optimally estimates linear datamodel parameters, it is
expensive to estimate due to the “training data” collection process. Obtaining
a single training datapoint for the regression—i.e., a single train set,
corresponding loss on $x$ pair—is expensive because training even a single
model can be expensive (particularly for the large-scale model setting), and
in practice, previous work has found that we need to train at (at least)
thousands of models to collect enough regression datapoints [Ilyas et al.,
2022].

###  B.2 Estimating datamodels with TRAK

Rather than estimating with data regression, we estimate linear datamodel
parameters with a more computationally efficient linear datamodel estimator:
TRAK [Park et al., 2023]. TRAK estimates datamodels more efficiently by
exploiting the fact that datamodels are efficient to calculate for convex
learning problems: TRAK (approximately) transforms the original learning
algorithm into a convex learning problem, computes datamodels in this new
regime, then returns these datamodels as an estimate of the datamodels for the
originally considered learning algorithm. TRAK trades off approximation error
(i.e., the transformation is inexact) for computational efficiency.

To actually estimate datamodels, the method operates in two high level stages.
Given a held out sample $x$, learning algorithm $\mathcal{A}$ and training
dataset $\mathcal{S}$, TRAK first constructs a new algorithm
$\mathcal{A}^{\prime}$ that approximates the corresponding trained model
output on $x$ as if the model output were obtained by solving a convex problem
over the train set datapoints, such that $f(x;\mathcal{A}(S))\approx
f(x;\mathcal{A}^{\prime}(S))$. Then, TRAK estimates the datamodel parameters
for the original learning problem by estimating the datamodel parameters for
executing $\mathcal{A}^{\prime}$ on $S$ (datamodels are inexpensive to compute
for convex problems like $\mathcal{A}^{\prime}$). We break these stages into
two steps below, and start with a primer on calculating datamodels for the
logistic regression setting.

####  B.2.1 Datamodels for logistic regression

We first describe how to efficiently estimate datamodels for models with a
convex objective. We will use logistic loss to simplify the analysis, but the
procedure applies to other convex losses function as well. Consider a
(generalized, including biases) binary classification task learning from
$n=|\mathcal{S}|$ candidate training samples:

$\mathcal{S}=\\{z_{1},...,z_{n}:z_{i}=(x_{i},b_{i},y_{i})\\},$

where each sample $z_{i}$ is a triplet containing an input $x_{i}$, a bias
$b_{i}$, and a binary label $y_{i}\in\\{-1,1\\}$. In this setup, training a
logistic regression model on a training subset $S\subset\mathcal{S}$ yields
the corresponding parameters $\mathcal{A}_{\mathrm{Log}}{}(S)$:

$\mathcal{A}_{\mathrm{Log}}{}(S)\vcentcolon=\arg\min_{\theta}\sum_{z_{i}\in
S}\log\left(1+\exp\left(-y_{i}\cdot\left(x_{i}^{\top}\theta+b_{i}\right)\right)\right).$
(6)

Note that including biases $b_{i}$ makes this version of logistic regression
more general; setting $b_{i}=0$ yields standard logistic regression.

How do we estimate datamodels for logistic regression? We start by defining
the output function that we want to approximate using datamodels in the first
place: we approximate the logistic linear model output

$f(z;\theta)\vcentcolon=x^{\top}\theta+b,\mbox{ where }z=(x,b,y).$

That is, we aim to construct datamodels that approximate the map from train
subset $S$ to linear model output $f(z;\mathcal{A}_{\mathrm{Log}}{}(S))$.

To efficiently estimate these logistic regression datamodels, TRAK uses
influence functions. Influence functions are a standard method for efficiently
approximating the effect of excluding a single training point (hence, “leave-
one-out”) on linear regression outputs compared to training on the entire set
[Pregibon, 1981] (and apply to other classes of models as well [Giordano et
al., 2019]). Specifically, the leave-one-out influence for training example
$i$ on example $z$, $\mathrm{IF}(z)_{i}$, approximates this effect as:

$\mathrm{IF}(z)_{i}\vcentcolon=\frac{x^{\top}(X^{\top}RX)^{-1}x_{i}}{1-x_{i}^{\top}(X^{\top}RX)^{-1}\cdot
p_{i}^{*}(1-p_{i}^{*})}(1-p_{i}^{*})\approx
f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}\setminus
z_{i})),$ (7)

where $X\in\mathbb{R}^{n\times k}$ is the matrix of stacked train example
inputs ($k$ the input dimension of each $x_{i}$),
$p^{*}_{i}=\left(1+\exp\left(-y_{i}\cdot
f(z_{i};\theta^{*})\right)\right)^{-1}$, and $R$ is an $n\times n$ matrix with
$R_{ii}=p_{i}^{*}(1-p_{i}^{*})$; this estimate arises from performing a Newton
step from logistic model parameters for $\mathcal{S}$ to minimize loss on
$\mathcal{S}\setminus z_{i}$. In practice, influence functions closely
approximate the effect of removing a single train example on logistic model
predictions [Koh and Liang, 2017]. Furthermore, influences are efficient to
estimate: computing the influence of $i$ on example $z$ requires only a few
inner products and scalar multiplications (the most expensive term to compute,
the inverse $\left(X^{\top}RX\right)^{-1}$, does not depend on $z$ or $i$ and
therefore can be computed just once).

It is straightforward to estimate parameters for logistic regression
datamodels using influence functions. We consider leave-one-out datamodels,
i.e., referring back to the datamodel definition of (5), datamodels for a
distribution of training sets $\mathcal{D}_{S}$ that is supported on train
subsets missing a single train example. In this setting, we can estimate a
leave-one-out linear datamodel $\tau_{\theta}$ with $\theta=\mathrm{IF}(z)$
and including a bias
$f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-\sum_{k=1}^{n}\mathrm{IF}(z)_{k}$,
i.e., in full:

$\tau_{\theta}(S)=\mathrm{IF}(z)^{\top}\text{1}_{S}+f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-\sum_{k=1}^{n}\mathrm{IF}(z)_{k}$
(8)

Then, on a data subset with a single removed example $S\setminus x_{i}$, the
datamodel approximation of $f(z;\mathcal{A}_{\mathrm{Log}}{}(S\setminus
x_{i}))$ is:

$\displaystyle\tau_{\theta}(S\setminus z_{i})$
$\displaystyle=\mathrm{IF}(z)^{\top}\text{1}_{S\setminus
x_{i}}+f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-\sum_{k=1}^{n}\mathrm{IF}(z)_{k}$
$\displaystyle=f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-\mathrm{IF}(z)_{i}$
$\displaystyle\approx
f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-\left(f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}))-f(z;\mathcal{A}_{\mathrm{Log}}{}(\mathcal{S}\setminus
z_{i}))\right)$ $\displaystyle=f(z;\mathcal{A}_{\mathrm{Log}}{}(S\setminus
z_{i})),$

which is the approximation of the effect of removing $z_{i}$ on $z$ given by
the influence function. In practice, we can use this datamodel to estimate the
model output associated with arbitrary training subsets (not just leave-one-
out subsets).

####  B.2.2 Transforming learning algorithms to linear regression

We now discuss how TRAK uses these logistic datamodels to estimate datamodels
for non-linear models. The key procedure behind TRAK translates the training
setup of interest—i.e., that defined by the learning algorithm $\mathcal{A}$
and candidate dataset $\mathcal{S}$—into a new setup with a carefully
constructed convex (in our case, logistic regression) learning algorithm
$\mathcal{A}^{\prime}$ (on the same candidate dataset $\mathcal{S}$). Here,
TRAK approximates the model output $f(z;\mathcal{A}(S))$ for a given subset
with the logistic output $f(z;\mathcal{A^{\prime}}(S))$, then estimates
datamodels for $\mathcal{A}^{\prime}$, which can be efficiently computed.

To set up this transformation, consider a (binary classification101010We use
binary classification for simplicity, but the analysis follows for other
standard losses as well.) machine learning model with learned parameters
$\theta^{*}=\mathcal{A}(\mathcal{S})$ (trained on the full candidate set) that
outputs a (binary) logit model output $f(z;\theta^{*})$ for the given example
$z$. TRAK starts by linearizing $f$ with a Taylor expansion at the model
weights $\theta^{*}$:

$\hat{f}(z;\theta)=f(z;\theta^{*})+\nabla_{\theta}f(z;\theta^{*})^{\top}(\theta-\theta^{*}).$
(9)

Here, the approximation $\hat{f}$ of $f$ is linear in the gradient of the
considered example $z$. $\hat{f}$ is a linear function that approximates the
model output $f$ for arbitrary parameters. However, the goal of datamodeling
is to approximate the map between training dataset to model output—not model
parameters to model output.

To model how training dataset choice changes model output, TRAK approximates
the original learning algorithm, $\mathcal{A}$, as minimizing the logistic
loss for the (linear) predictor $\hat{f}(z;\theta)$, over parameters $\theta$.
TRAK does so by directly replacing the original linear model in the logistic
regression objective of (6), i.e., $\theta$, with the linearization
$\hat{f}(z;\theta)$ (which is also linear in $\theta$). This yields the
logistic regression algorithm $\mathcal{A}^{\prime}$:

$\mathcal{A}^{\prime}(S)=\operatorname*{arg\,min}_{\theta}\sum_{z_{i}\in
S}\log\left(1+\exp\left(-y_{i}\cdot\left(\theta^{\top}\nabla_{\theta}f(z_{i};\theta^{*})+f(z_{i};\theta^{*})-\nabla_{\theta}f(z_{i};\theta^{*})^{\top}\theta^{*}\right)\right)\right).$

Rearranging the terms with new linear regression inputs
$x_{i}^{\prime}=\nabla_{\theta}f(z_{i};\theta^{*})$ and biases
$b_{i}^{\prime}=f(z_{i};\theta^{*})-\nabla_{\theta}f(z_{i};\theta^{*})^{\top}\theta^{*}$,
$\mathcal{A}^{\prime}$ is exactly logistic regression over dataset triplets
$(x^{\prime}_{i},b^{\prime}_{i},y_{i})$:

$\mathcal{A}^{\prime}(S)=\operatorname*{arg\,min}_{\theta}\sum_{z_{i}\in
S}\log\left[1+\exp\left(-y_{i}\cdot\left(\theta^{\top}x^{\prime}_{i}+b^{\prime}_{i}\right)\right)\right].$
(10)

Finally, TRAK estimates datamodel parameters for training $\mathcal{A}$ on
$\mathcal{S}$, the original problem of interest, by estimating datamodels for
the logistic regression algorithm $\mathcal{A}^{\prime}$ on $\mathcal{S}$.

####  B.2.3 TRAK estimator

In this section, we detail the exact form TRAK uses to estimate datamodels.
TRAK does not exactly estimate using the influence function estimate of (7)
with the input triplets $(x^{\prime}_{i},b^{\prime}_{i},y_{i})$ of (10), but
instead uses a similar form found by ablating over the relevant terms and
performing dimensionality reduction.

We first define notation for the space in which TRAK estimates datamodels,
i.e., the linear regression setting of (10). Suppose that
$\theta^{*}=\mathcal{A}(\mathcal{S})$ is the final model parameters obtained
after training on the entire candidate dataset. Recall that the logistic
regression problem of $\mathcal{A}^{\prime}$ “trains” on inputs
$x^{\prime}_{i}=\nabla_{\theta}f(x^{\prime}_{i},\theta^{*})$; we therefore
define the “feature map” $\phi$ that translates examples into this input space
as:

$\phi(z)\vcentcolon=\nabla_{\theta}f(z;\theta^{*})\in\mathbb{R}^{n}.$

We additionally define
$\Phi=\left[\phi(z_{1}),\ldots,\phi(z_{n})\right]^{\top}\in\mathbb{R}^{|\mathcal{S}|\times|\theta^{*}|}$
as the matrix of stacked candidate train set examples in this space. Finally,
we define

$Q\vcentcolon=\mathrm{diag}\left(\left\\{\frac{\partial{}L(y_{i},f(z_{i};\theta^{*}))}{\partial{}f(z_{i};\theta^{*})}\right\\}\right)\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{S}|},$

where $L$ is the convex loss we consider (in our case above, logistic loss).
$Q$ falls out of how the influence function is derived (as a single step
Newton approximation). As an example, in the logistic regression case above,
$Q$ is:

$Q=\mathrm{diag}\left(\\{1-p_{i}^{*}\\}\right)=\mathrm{diag}\left(\left\\{\left(1+\exp\left(y_{i}\cdot
f(z_{i};\theta^{*})\right)\right)^{-1}\right\\}\right),$

the $|\mathcal{S}|\times|\mathcal{S}|$ sparse matrix with correct prediction
probabilities on the diagonal.

With our notation in hand, we describe the TRAK estimator in two stages. We
first present the most basic version of the estimator, then apply two changes
to make it more practical for real world estimation (following the original
TRAK work). We start with the most basic version of the TRAK estimator, which
is used to calculate datamodels in place of the standard influence estimate
(cf. (8)),

$\mathrm{TRAK}(z)=\phi(z)^{\top}\left(\Phi^{\top}\Phi\right)^{-1}\Phi^{\top}Q\in\mathbb{R}^{|\mathcal{S}|}.$
(11)

To give intuition for this form: Park et al. [2023] construct TRAK by starting
with (7), removing the $R$ term, and removing the denominator; these terms
were found to not aid datamodel predictiveness (see Park et al. [2023] for
more details). The $Q$ term is a vectorized (over candidate train set) version
of the $1-p^{*}$ term in (7), and
$\phi(z)^{\top}\left(\Phi^{\top}\Phi\right)^{-1}\Phi$ is a vectorized (over
candidate train set) version of the numerator in (7).

Making this form practical is difficult, for two reasons: dimensionality and
learning algorithm randomness. For the former problem: calculating TRAK
requires inverting (and storing) the term $\left(\Phi^{\top}\Phi\right)^{-1}$,
a square matrix with side length equal to the number of model parameters. The
smallest models we estimate datamodels for in this work are 125M
parameters—even these models would require storing and inverting a 500TB
matrix (assuming we invert in float32). To circumvent this issue, TRAK reduces
the dimensionality of the input space using Johnson-Lindenstrauss (JL) random
projection matrices [Johnson and Lindenstrauss, 1984]; JL projections preserve
the inner-products between projected vectors (and the logistic regression
objective can be factored in terms of inner products between inputs [Zhu and
Hastie, 2005]).

For the latter problem, in practice $\theta^{*}=\mathcal{A}(\mathcal{S})$ is
generally not unique. For example, for large scale models, the final trained
model when training on the entirety of $\mathcal{S}$ changes based on
initialization or minibatch randomness. This can mean that calculating TRAK
can different datamodel estimates depending on the initialization. To average
over training randomness, TRAK calculates (11) over multiple trained models by
estimating each term independently then taking a mean over models.

To both (a) add random projections to reduce input dimensionality to
$d<<|\theta^{*}|$ and (b) average training randomness over $m$ models, we
start by defining a collection of model parameters $\\{\theta^{*}_{k}\\}_{k}$
in place of $\theta^{*}$, where each $\theta^{*}_{k}$ is a vector of model
parameters corresponding to training a model on $\mathcal{S}$ with
$\mathcal{A}$. We then define our new, dimensionality-reduced mapping to
trained model $k$ (with parameters $\theta_{k}^{*}$) gradient space as

$\phi_{k}(z)\vcentcolon=P_{k}^{\top}\nabla_{\theta}f(z;\theta_{k}^{*})\in\mathbb{R}^{d},\mbox{
where }P_{k}\sim\mathcal{N}(0,1)^{|\theta^{*}|\times d},$

replacing $\phi$ from (11), and the corresponding stacked, projected candidate
train vectors for model $k$ as
$\Phi_{k}=\left[\phi_{k}(z_{1}),\ldots,\phi_{k}(z_{n})\right]^{\top}\in\mathbb{R}^{|\mathcal{S}|\times
d}$, replacing $\phi$ from (11). We additionally $Q_{k}$ as:

$Q_{k}\vcentcolon=\mathrm{diag}\left(\left\\{\frac{\partial{}L(y_{i},f(z_{i};\theta^{*}_{k}))}{\partial{}f(z_{i};\theta^{*}_{k})}\right\\}\right)\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{S}|},$

replacing $Q$ from (11), and finally define the final TRAK estimator by
starting from the basic TRAK estimator (11) and averaging each term across $m$
models:

$\mathrm{TRAK}(z)=\left(\frac{1}{m}\sum_{k\in\left[m\right]}\left(\phi_{k}(z)^{\top}\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}\right)\right)\cdot\left(\frac{1}{m}\sum_{k\in\left[m\right]}Q_{k}\right)\in\mathbb{R}^{|\mathcal{S}|}$
(12)

###  B.3 Datamodels for language modeling

In this section, we discuss how to formulate datamodels for LMs. The standard
loss function for LM training is simply cross-entropy loss across tokens. The
main question is: what output function do we use? Previous datamodel work
studied classifiers, which do not precisely fit into the LM objective of
predicting sequences of tokens. We therefore extend a standard multi-class
classification output function previously used in previous datamodel
instantiations [Saunshi et al., 2023, Park et al., 2023]. These methods use
the “multi-class margin” output function:

$f(x;\theta)\vcentcolon=\log\left(\frac{p(x;\theta)}{1-p(x;\theta)}\right),$

where $p(x;\theta)$ is the probability of the correct class given by the model
$\theta$. Since each LM training example consists of many classification
problems, we employ what we call the “mean multi-class margin” output
function:

$f(x;\theta)\vcentcolon=\sum_{j=2}^{T}\log\left(\frac{p(x_{j}|x_{<j};\theta)}{1-p(x_{j}|x_{<j};\theta)}\right),$

where $T$ is the model context length, $x$ is a length $T$ token sequence, and
$p(x_{j}|x_{<j};\theta)$ is the probability that model $\theta$ correctly
predicts token $j$ given the previous tokens as context. To use this output
function with TRAK, the corresponding $Q$ is:

$Q=\mathrm{diag}\left(\left\\{1-\bar{p}_{i}\right\\}\right),$

where $\bar{p}_{i}$ is the mean probability that the model correctly predicts
the next token in example $i$ (across all $T-1$ continuation tokens in the
example)

###  B.4 TRAK setup

We design this section to be standalone, and repeat (12) along with
definitions of each of the terms. To understand the full background for this
form, read from the start of Appendix B.

Given an algorithm $\mathcal{A}$, candidate training set $\mathcal{S}$, and
model output function $f(z;\theta)$, TRAK estimates the datamodel parameters
for a given test input of interest $z$ as:

$\mathrm{TRAK}(z)=\left(\frac{1}{m}\sum_{k\in\left[m\right]}\left(\phi_{k}(z)^{\top}\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}\right)\right)\cdot\left(\frac{1}{m}\sum_{k\in\left[m\right]}Q_{k}\right)\in\mathbb{R}^{|\mathcal{S}|}.$

Before defining all these terms, we start with preliminary notation. Let $m$
be the number of trained reference models that we calculate TRAK with, with
$\\{\theta^{*}_{k}\\}_{k=1}^{m}$ as a set of $m$ parameter vectors for models
trained with $\mathcal{A}$. Let $N$ be the dimensionality of each
$\theta^{*}_{k}$, and let $d$ be a projection dimension such that $d<<N$. We
then define one projection matrix per model, with $\\{P_{k}\\}_{k=1}^{m}$ a
set of $m$ Johnson-Lindenstrauss projection matrices, such that each
$P_{k}\sim\mathcal{N}(0,1)^{N\times d}$ is drawn from a multivariate Gaussian.

We now define the constituent terms of the TRAK estimator as follows.
$\phi_{k}$ is the function mapping an example $z$ to its projected gradient
for model $k$:

$\phi_{k}(z)\vcentcolon=P_{k}^{\top}\nabla_{\theta}f(z;\theta_{k}^{*}),$

and $\Phi_{k}$ is the matrix of stacked training example gradients, with

$\Phi_{k}=\left[\phi_{k}(z_{1}),\ldots,\phi_{k}(z_{n})\right]^{\top}\in\mathbb{R}^{|\mathcal{S}|\times
d}.$

Finally, $Q_{k}$ is the diagonal matrix:

$Q_{k}\vcentcolon=\mathrm{diag}\left(\left\\{\frac{\partial{}L(y_{i},f(z_{i};\theta^{*}_{k}))}{\partial{}f(z_{i};\theta^{*}_{k})}\right\\}\right)\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{S}|}.$

In the LM setting we consider, we define $f$ and $Q_{k}$ as discussed in
Appendix B.3.

##### Computing TRAK.

We compute with $\mathcal{S}$ as C4 (cf. Appendix A.1), $\mathcal{A}$ as
training a 125M LM for 80000 batches, or a (random) 38% of C4, $m=4$
independently trained models, and $d=16384$ projection dimension.
Mechanically, to compute the final TRAK estimate, we proceed in three steps:

  1. 1.

Model training. First, train $m$ reference models on C4 (i.e.,
$\\{\theta^{*}_{k}\\}_{k=1}^{m}$). We train 4 LMs, each on roughly 82 million
samples of C4 (~38% of C4; 80,000 batches).

  2. 2.

Collect projected gradients. For each of the $m$ reference models, calculate
$\Phi_{k}$ by iterating over C4 and taking the gradient of each train sample
with respect to the output function. Additionally, record the average accuracy
for each sample to compute $Q_{k}$.

  3. 3.

Collect terms. Calculate each per-model term in (12), then average together
the terms and calculate the final TRAK estimate (i.e., calculate the
corresponding
$\phi_{k}(z)^{\top}\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}$
and $Q_{k}$ for each model $k$, then average the terms across models, then
matrix multiply the two aggregate terms together to obtain
$\mathrm{TRAK}(z)$). We calculate the final TRAK scores in a batched manner by
stacking the $\phi_{k}(z)$ for each sample $z$ that we calculate datamodels
for before multiplying by
$\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}$.

###  B.5 Computational cost

In Appendix B.4 we detailed the mechanics of estimating datamodels with TRAK.
In this section, we detail the (rough) computational cost of our estimation
procedure. We detail the cost of each of the steps in Appendix B.4 in terms of
per-example “forward and backward pass” (FBP) count for the given model class
(note that in this work, we only directly estimate datamodels for 125M LMs).
Computing gradients and reference model training dominate total model training
cost, so we only tally compute used for these subtasks.111111The costs we
ignore are projecting the gradient—which is a constant, essentially free
factor on top of taking the gradient—and computing TRAK from the projected
gradients, which is simply two inner products per projected gradient and a
projection-dimension sized

In this section we use the following notation: $m$ to denote the number of
models, $n_{\mathrm{model}}$ to denote the number of samples used to train
models used to compute TRAK, and $n_{\mathrm{dm}}$ to denote the number of
examples we compute datamodels for.

  1. 1.

Model training. Training $m$ models on $n_{\mathrm{model}}$ samples requires
$m\cdot n_{\mathrm{model}}$ FBPs.

  2. 2.

Collect projected gradients. Taking the (projected) gradient of the
$|\mathcal{S}|$ train samples for each of the $m$ models requires
$m\cdot|\mathcal{S}|$ FBPs. We ignore the cost of projecting as it is
(essentially) free compared to taking the gradient.121212Projecting accounts
for ¡1% of the time taken to compute the projected gradient. Note that “FBPs”
are a coarse-grained metric; for example, taking individual gradients is in
practice more expensive than taking the average gradient over a batch of
samples, even when batching is used to compute in both cases.

  3. 3.

Calculate $\mathrm{TRAK}$. Calculating $Q_{k}$ is free as we can compute the
average accuracies on the diagonal when we collect projected gradients.
Calculating
$\phi_{k}(z)^{\top}\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}$
for each sample $z$ we compute a datamodel for requires two stages: first,
compute and save $\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}\Phi_{k}^{\top}$,
then, second, compute each datamodel of interest by matrix multiplying with
$\phi_{k}(z)$. The first stage is essentially free,131313Computing
$\Phi_{k}^{\top}\Phi_{k}$ requires only $|\mathcal{S}|$ inner products;
inverting a square matrix at the projection dimension we use, 16384, is very
cheap; with these two quantities calculated, multiplying
$\left(\Phi_{k}^{\top}\Phi_{k}\right)^{-1}$ with $\Phi_{k}^{\top}$ takes only
$|\mathcal{S}|$ inner products. the second stage requires $n_{\mathrm{dm}}$
FBPs.

Our total cost is: $\left(m\cdot
n_{\mathrm{model}}\right)+\left(m\cdot|\mathcal{S}|\right)+\left(n_{\mathrm{dm}}\right)$
FBPs. Our constants in this work are $m=4$, $n_{\mathrm{model}}=82\times
10^{6}$, $|\mathcal{S}|\approx 217\times 10^{6}$, and $n_{\mathrm{dm}}\approx
30000$. These constants yield a total cost of $1.2\times 10^{9}$ FBPs. This
cost is dominated by taking projected gradients across the four models (~73%
of computation), along with actually training the 4 models (~7% each). We
expect in the future that this cost will greatly decrease; we did not choose
our setup to optimize for compute (e.g., we did not ablate over number of
reference models or the number of batches that we train reference models on).

##  Appendix C Evaluating task-optimal dataset selection

This section provides additional context for Section 3, in which we measure
DsDm at optimal dataset selection on varying target tasks. We start by
describing each of the considered tasks in greater detail, and show samples
from each. We then discuss training details, and then conclude with omitted
figures.

###  C.1 Experimental setup

We consider four separate LM target tasks: standard language modeling tasks:
LAMBADA (an open-ended cloze task measuring language understanding [Paperno et
al., 2016]), CS-Algorithms (an algorithmic problem solving benchmark
containing tasks like longest common subsequence identification [Srivastava et
al., 2022]), SQuAD (the Stanford Question-Answering Dataset, a reading
comprehension dataset of questions about Wikipedia articles [Rajpurkar et al.,
2016]), and Jeopardy (trivia questions from the “Jeopardy!” game show
[MosaicML, 2023]). We consider 0-shot (LAMBADA and CS-Algorithms) and 3-shot
(SQuAD and Jeopardy) settings. For more details on these target tasks,
including samples and target/holdout splits for evaluating, see Appendix A.2.

We train models according to the training procedure described in Section A.4,
with the hyperparameters described in the “Section 3” rows of Table 2. See
Appendix A.3 for baseline details, and Appendix A.1 for candidate dataset
details.

###  C.2 Omitted figures

##### Measuring performance with accuracy in place of log-probability.

We repeat the experiment of Figure 1—measuring model performance while varying
fraction of selected data and selection method—but measure accuracy in place
of log-probability. Figure 9 shows our results. The relative model
performances are roughly the same, but the magnitudes are different (e.g., the
best Jeopardy model attains roughly 0.03% accuracy) and “noisier” across model
retraining (i.e., measured log-probability is continuous with respect to
fraction of C4 selected, while the accuracy is discontinuous). We describe our
procedure for measuring accuracy in Section A.5.2. (We measure accuracy
according to exact match for LAMBADA and CS-Algorithms, and according to fuzzy
matching for SQuAD and Jeopardy.)

##### Counterfactual verification: training on the “worst” samples

We train a model on the “worst” (i.e., least likely to be chosen) samples
according to DsDm in Figure 10. We find that training on these samples is much
worse than training on random samples—despite the samples containing QA-
related text (cf. Figure 10).

###  C.3 Sample dataset selections

WARNING:
SAMPLES MAY INCLUDE OFFENSIVE TEXT

In this section, we show the samples that each dataset selection is most and
least likely to select (i.e., the “top” and “bottom” samples). We both (a)
describe how we choose the top and bottom examples for each dataset selection
method to visualize and (b) display examples of the data selected both
randomly and by DsDm, DSIR, Classifier across the tasks we investigate. The
selected samples are not exactly randomly selected: we replace samples with
obviously offensive content or characters that we cannot render.

We visualize the top and bottom $0.01\%$ of samples for each combination of
selection method, target dataset. For each selection method, we use the
following procedure to select the top/bottom candidate train examples from C4:

  * •

Random: We choose random examples; see Figure 24.

  * •

DsDm: We sort examples by corresponding linear datamodel weight (cf. Section
2.2). See Figure 12 (SQuAD), Figure 15 (Jeopardy), Figure 18 (LAMBADA) and
Figure 21 (CS-Algorithms).

  * •

DSIR: We sort examples by log-probability of inclusion. See Figure 14 (SQuAD),
Figure 16 (Jeopardy), Figure 19 (LAMBADA) and Figure 22 (CS-Algorithms).

  * •

Classifier: We sort examples by margin towards the “target dataset class” for
the trained classifier. See Figure 13 (SQuAD), Figure 17 (Jeopardy), Figure 20
(LAMBADA) and Figure 23 (CS-Algorithms).

We show only random 500 character excerpts; each full example would take up
pages of text.

Additionally, we fix SQuAD as the target dataset and visualize the top
examples side-by-side in Figure 2 (in the main text). We compare the bottom
examples side by side in Figure 11.

Figure 9: Target task performance by method, training 125M models and varying
dataset selection size. Performance measured in accuracy. DsDm consistently
improves performance, even when baselines are not much better than selecting
data randomly (i.e., on SQuAD and CS-Algorithms). DsDm models also
consistently match a 1.3B Random model (trained with more than 10x the compute
budget). For DsDm, more epochs on higher ranked samples is better than fewer
epochs on less highly ranked samples. Each model trains on 25.6 million
samples (equivalent to 12% of C4). The “random” shaded area is the range of
values achieved by 10 Random models each trained for one epoch (i.e., the
Random model training does not depend on the x-axis). Accuracy measured
according to Section A.5.2. Figure 10: Model performance when training on
worst data according to DsDm. Rather than selecting the examples predicted to
most increase performance (i.e., the standard DsDm selection mechanism), we
select for the examples predicted to most decrease performance. We find that
despite these examples containing QA-related text (cf. Figure 11), they yield
models that perform much worse than models trained on randomly selected data.

  1. (1)

ligent machines and the brain. I’m not really a brain expert.\n01:29:44 I’m
more a machine learning person, but I talk to neuroscientists and so
on.\n01:29:48 And I try, I really care about the big question of how is the
brain doing the really complex things that it does.\n01:30:10 Speaker 2: On
your path to the Promised Land?\n01:30:12 Yoshua: Yes, exactly, that’s
right.\n01:30:14 And I’ve been making those small steps on this particular
topic for about a year and a half.\n01:30:21 So it’s not like just somethin

  2. (2)

whom thy father, Prince of Wales, was first.\n2.1.177822Than was that young
and princely gentleman.\n2.1.183828Which his triumphant father’s hand had
won.\n2.1.185830But bloody with the enemies of his kin.\n2.1.187832Or else he
never would compare between.\n2.1.190836 Not to be pardoned, am content
withal.\n2.1.192838The royalties and rights of banished
Hereford?\n2.1.193839Is not Gaunt dead? And doth not Hereford
live?\n2.1.194840Was not Gaunt just? And is not Harry true?\n2.1.195841Did not
the one deserve to have

(a) DsDmwhiteDsDm samples

  1. (1)

n( 6)Michael Brown( 1)Michael Collins( 1)Michael Glessner( 2)Michael Graber(
150)Michael Greenstone( 1)Michael Ohler( 1)Michael Ohler and Phil Samuel(
1)Michael Raynor( 1)Michael Soerensen( 1)Michael Thompson( 1)Michael Whitaker(
7)Michel van Hove( 3)Michele Nemschoff( 1)Michele Westergaard( 1)Michelle
Tabart( 2)Mick Simonelli( 4)Mike Brown( 88)Mike Cassettari( 1)Mike Dalton(
4)Mike Lippitz( 5)Mike Myatt( 102)Mike Shipulski( 134)Mike Waite( 1)Miriam
Clifford( 1)Mitch Ditkoff( 81)Moises Norena( 5)Monique Vin

  2. (2)

.1 3000 76 12 5.3 2250 2000 2000 ı X ı X ı X 76 10 4. Shaded cells are
acceptable for motor codes.2 2500 75 22 9.12 3000 76 15 6.15 3000 76 17 7.)
3.I.25 2500 2500 No Porting ı ı X X NPT Porting 3/4” 3/4” 1” 3/4” 1” 1” 1” 1”
1 1/4” 1” 1 1/4” 1 1/4” IC ID IJ YC YD YF YJ YL ID IC IG YD YC YF YG YL ID ID
3/4” 1”* 3/4” 1 1/4”* 3/4” 1”* 1” 1” 1” 1 1/4”* 1” 1 1/2”* 1” 1 1/4” 1 1/4” 1
1/2”* 1 1/4” 1 1/2” 1 1/2” EC EJ EK AC AD AF AJ AK AL AP AR ED EG EH AD AC AF
AG AH AL AM AR ED X* 3/4” 3/4” 1” 3/4” 1” 1” 1” 1” 1

(b) DSIRwhiteDSIR samples

  1. (1)

Martin was one of my best friends growing up, and I am in shock to learn about
this. So many prayers and lots of love being sent your way.\noh my god. i’m so
shocked.!! it’s hard to find words..\nwhy must there be so shit-things like
cancer.!!\na wonderful life and I know that you are strong.!!!\nall the very
best - I’m thinking of you..\nThere are just no words. I’m so sorry. My heart
aches for you. I’ve long admired the two of you. You are such a beautiful
match, inside and out. I’ll keep you in my prayers, an

  2. (2)

will be using your checklist on my future SEO projects. Thank you
Bruce.\nExcellent article. Thanks so much for sharing this checklist. This is
very useful.\nWe regularly miss out on a number of these checkpoints. Thanks
for sharing and enabling us do the right job with our seo tasks.\nSEO is most
good technical way to promote your website in any search engine. Here you have
shared excellent article and information about SEO checklist. This techniques
should helpful for us to get rank first. Thanks for sharing

(c) ClassifierwhiteClassifier samples

Figure 11:  LeastwhiteLeast helpfulwhitehelpful training examples for SQuAD,
as ranked by each method. We choose samples randomly from the bottom $0.01\%$
of samples given by each method (see Appendix C.3 for methodology and more
samples across target tasks). “\n” denotes a newline.

  1. (1)

OI on the IBM C2060-350 exam papers is tremendous, with an absolute guarantee
to pass Application Developer C2060-350 tests on the first attempt.\nStill
searching for IBM C2060-350 exam dumps? Don’t be silly, C2060-350 dumps only
complicate your goal to pass your IBM C2060-350 quiz, in fact the IBM
C2060-350 braindump could actually ruin your reputation and credit you as a
fraud. That’s correct, the IBM C2060-350 cost for literally cheating on your
IBM C2060-350 materials is loss of reputation. Which is why

  2. (2)

ighly thought of by potential consumers. stockholder responsibility b. a.
Anheuser-Busch is exhibiting which of the following? a. cause marketing e. d.
social responsibility Answer: e Page(s): 88-89 LO: 3 AACSB: Ethics QD: Medium
Rationale: Social responsibility is the view that organizations are part of a
larger society and are accountable to that society for their actions. profit
responsibility c. Answer: a Page(s): 88-89 LO: 3 AACSB: Ethics QD: Hard -4-.
the larger Anheuser-Busch’s profits. the higher co

  3. (3)

erances, so not only are the two sticks in each pair matched, they are also
the same weight as the pairs you bought last year & the pairs that you will
buy next year.\nWhat we say: The Shaw C+ Wood Tip Drum Stick is a classic
British model drum stick that has been a part of the shaw brand for years.
Shaw Sticks are manufactured from the finest grades of selected American
hickory. They are matched to within precise tolerances, so not only are the
two sticks in each pair matched, they are also the same weight

(a) BestwhiteBest train samples for SQuAD (DsDm)

  1. (1)

ligent machines and the brain. I’m not really a brain expert.\n01:29:44 I’m
more a machine learning person, but I talk to neuroscientists and so
on.\n01:29:48 And I try, I really care about the big question of how is the
brain doing the really complex things that it does.\n01:30:10 Speaker 2: On
your path to the Promised Land?\n01:30:12 Yoshua: Yes, exactly, that’s
right.\n01:30:14 And I’ve been making those small steps on this particular
topic for about a year and a half.\n01:30:21 So it’s not like just somethin

  2. (2)

whom thy father, Prince of Wales, was first.\n2.1.177822Than was that young
and princely gentleman.\n2.1.183828Which his triumphant father’s hand had
won.\n2.1.185830But bloody with the enemies of his kin.\n2.1.187832Or else he
never would compare between.\n2.1.190836 Not to be pardoned, am content
withal.\n2.1.192838The royalties and rights of banished
Hereford?\n2.1.193839Is not Gaunt dead? And doth not Hereford
live?\n2.1.194840Was not Gaunt just? And is not Harry true?\n2.1.195841Did not
the one deserve to have

  3. (3)

te 2 1 Torrent Magnet Casino Royale (2006) Extended BRrip 720p x264 Dual Audio
Eng.43 Gigabyte 2 19 Torrent Magnet James Bond (2006) Casino Royale avchd
1080p EN NL B-Sam.93 Gigabyte. Telesync.XViD-pukka.36 Gigabyte 0 0 Torrent
Magnet Casino 802.45 MB 0 1 Torrent Magnet.3CD-WAF.05 Gigabyte 0 0 Torrent
Magnet yale. Magnet, james Bond: Casino Royale (2006) 1080p BrRip x264 -
yify.1 Gigabyte 282 87, torrent. X265-WAR 829.21 MB 5 20 Torrent Magnet.1
Gigabyte 5 5 Torrent Magnet Casino Royale (2006) DVDrip multis

(b) WorstwhiteWorst train samples for SQuAD (DsDm)

Figure 12: According to DsDm: the best and worst training examples for
improving SQuAD performance. Samples randomly chosen from the top/bottom
(respectively) $0.01\%$ of train samples as determined by DsDm (cf. Appendix
C.3 for details); we display (random) 512 character slices of samples. \n
denotes a newline.

  1. (1)

bout Tony Blair. And I guess I’m saying if we’re willing to go without a
second resolution, can Tony Blair go without a second resolution?\nMR.
FLEISCHER: I think that’s a question you need to address to the United
Kingdom, not to me – I don’t speak for Tony Blair. The President has been
abundantly plain on this issue and he has said the United States does not need
a second resolution. But because it’s important to our allies, that makes it
important to him.\nQUESTION: Thank you. I have two questions, if I m

  2. (2)

ris and St Gleb, dating from the mid-12th century, was much rebuilt in
succeeding periods, before being restored to its original shape in the 20th
century. The crowning achievement of Chernigov masters was the exquisite
Church of St Paraskeba (Pyatnitskaya), constructed at the turn of the 12th and
13th centuries. This graceful building was seriously damaged in the Second
World War; its original medieval outlook was reconstructed. The earliest
residential buildings in the downtown date from the late 17th cen

  3. (3)

ed in the sell order was not consistent with a bona fide intention to sell
within a reasonable time.\nQuestion: Rule 144(h) provides that the Form 144
shall be transmitted for filing “concurrently” with either the placing of a
sale order with a broker or the execution of the sale directly with a market
maker. Does “concurrently” mean that the Form 144 should be transmitted for
filing on the same day as the placing of a sale order or the execution of the
sale?\nAnswer: Yes. For example, if a person is filing a

(a) BestwhiteBest train samples for SQuAD (Classifier)

  1. (1)

ownload 3.6.1 and that the announcement will be removed when that changes. I
also see a lot of threads about blank pages, etc.\nI see within Wordpress that
the theme was updated Jan 11th.\nWhat should I do? Download what version? Wait
until 3.6.1 is fixed? Unfortunately I can’t wait to download the new eCommerce
plugin so I hope all goes well with changing themes if you suggest that I wait
to switch to Atahualpa.\nPlease let me know your thoughts and THANKS!!!\nBy
the way, I’m going to be using the eCommerce pl

  2. (2)

Martin was one of my best friends growing up, and I am in shock to learn about
this. So many prayers and lots of love being sent your way.\noh my god. i’m so
shocked.!! it’s hard to find words..\nwhy must there be so shit-things like
cancer.!!\na wonderful life and I know that you are strong.!!!\nall the very
best - I’m thinking of you..\nThere are just no words. I’m so sorry. My heart
aches for you. I’ve long admired the two of you. You are such a beautiful
match, inside and out. I’ll keep you in my prayers, an

  3. (3)

will be using your checklist on my future SEO projects. Thank you
Bruce.\nExcellent article. Thanks so much for sharing this checklist. This is
very useful.\nWe regularly miss out on a number of these checkpoints. Thanks
for sharing and enabling us do the right job with our seo tasks.\nSEO is most
good technical way to promote your website in any search engine. Here you have
shared excellent article and information about SEO checklist. This techniques
should helpful for us to get rank first. Thanks for sharing

(b) WorstwhiteWorst train samples for SQuAD (Classifier)

Figure 13: According to Classifier: the best and worst training examples for
improving SQuAD performance. Samples randomly chosen from the top/bottom
(respectively) $0.01\%$ of train samples as determined by Classifier (cf.
Appendix C.3 for details); we display (random) 512 character slices of
samples. \n denotes a newline.

  1. (1)

in Alexandria, where it was begun; and the Greek Bible of the Hellenistic Jews
and the Catholic Church may rightly be styled the Alexandrian Greek version of
the Old Testament.\nIn the early days of the Church the Septuagint was widely
used among the Jews; as a rule, though there are exceptions, when the Old
Testament is quoted in the New Testament it is from the Greek, not the Hebrew
Bible that the quotation is made. The early Jewish-Christians and the great
majority of the Jews had the same Bible, and Gent

  2. (2)

the Central Committee of the Party, that is, by the Politburo, the Orgburo
(Organizational Bureau), and the Secretariat. The decisions made were
implemented through the Presidium of the Supreme Soviet of the USSR, the
Council of People’s Commissars of the USSR, the GKO, and the General
Headquarters of the Supreme Command, which had been established on August 8.
Strategic direction of the armed forces was carried out by the General
Headquarters through its working body, the General Staff. Major questions as

  3. (3)

pled to the third terminal 912d of the third transistor MRP2. The third second
terminal 910b of the second transistor MRPN1, and the third terminal 914d of
the fourth transistor MRN2 is coupled to the second terminal 914b of the
fourth transistor MRN2.\nThe voltage source 906 has a first terminal 932a and
a second terminal 932b. The first terminal 932a of the voltage source 906 is
coupled to the first terminal 928a of the first dynamic body bias unit 902,
the first terminal 908a of the first transistor MRP1,

(a) BestwhiteBest train samples for SQuAD (DSIR)

  1. (1)

Mayon sa Naga (1) Mayon Volcano (2) mayor (1) Mayor Antonio Halili (1) MAYOR
INDAY (3) MAYOR INDAY DUTERTE (1) Mayor Inday Sara (1) Mayor Jed Patrick
Mabilog (1) Mayor Parojinog Sr. (1) MAYOR SARA (1) Mayor Sara Duterte (1)
Mayor Sara Duterte-Carpio (1) Mayoral race (1) Mayweather (1) MB Recto (1)
Measles (1) Meat Inspection Law (1) mechanization (1) meddling in the drug
trade (1) medical (1) medical attention (1) medical check up (1) medical
check-up. (1) Medical Intern (1) medical treatment (1) mediocre

  2. (2)