# Chain-of-Thought Reasoning is a Policy Improvement Operator

Hugh Zhang
Harvard University
<EMAIL_ADDRESS>
&David C. Parkes
Harvard University
<EMAIL_ADDRESS>
###### Abstract

Large language models have astounded the world with fascinating new
capabilities. However, they currently lack the ability to teach themselves new
skills, relying instead on being trained on large amounts of human-generated
data. We introduce _SECToR_ (Self-Education via Chain-of-Thought Reasoning), a
proof-of-concept demonstration that language models can successfully teach
themselves new skills using chain-of-thought reasoning. Inspired by previous
work in both reinforcement learning (Silver et al., 2017) and human cognition
(Kahneman, 2011), SECToR first uses chain-of-thought reasoning to slowly think
its way through problems. SECToR then fine-tunes the model to generate those
same answers, this time without using chain-of-thought reasoning. Language
models trained via SECToR autonomously learn to add up to 29-digit numbers
without any access to any ground truth examples beyond an initial supervised
fine-tuning phase consisting only of numbers with $6$ or fewer digits. Our
central hypothesis is that chain-of-thought reasoning can act as a policy
improvement operator, analogously to how Monte-Carlo Tree Search is used in
AlphaZero. We hope that this research can lead to new directions whereupon
language models can learn to teach themselves without the need for human
demonstrations.

> “Self-education is, I firmly believe, the only kind of education there is.”
>
> Isaac Asimov

Modelt$167+708=[$ CoT $\ldots\,]$A: 875$714+263=[$ CoT $\ldots\,]$A: 977Solve
problems usingCoT reasoningModelt+1Train model to generatethese solutions
withoutusing CoT reasoning.$2632+8647=\,?$$9401+5804=\,?$Repeat on even harder
problems$167+708=\,?$$714+263=\,?$Initial Problems$+$CoT Figure 1: To perform
self-learning, SECToR asks models to solve addition problems using chain-of-
thought reasoning by decomposing the problem step-by-step. It then trains the
next version of the model to solve those same problems directly _without using
chain-of-thought reasoning_. This process often results in an improved model
which can often solve even harder problems than original model, allowing the
self-learning loop to continue.

##  1 Introduction

Large language models are currently trained on vast corpora of human-generated
data (Vaswani et al., 2023; Devlin et al., 2019; Radford et al., 2019; Brown
et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023). While large
language models have demonstrated many surprising capabilities, the
possibility of reaching superhuman performance is a challenging proposition
when training solely on existing data. In this paper, we ask whether large
language models can autonomously teach themselves new skills rather than
solely depending on the availability of suitable data.

A positive answer to this question would open the door to a tantalizing
possibility. Although the discovery of scaling laws (Kaplan et al., 2020;
Hoffmann et al., 2022) for language models has created much excitement for
training increasingly larger language models, these would consume a
significant fraction of the high-quality (textual) data on the internet. The
issue of data exhaustion is an active area of research (Villalobos et al.,
2022), especially in light of results showing that repeated training on the
same data quickly leads to degeneration (Shumailov et al., 2023). If large
language models can effectively learn from data they themselves generate, this
could usher in a new era of scaling laws that are solely compute-driven,
independent of how much data is available.

Self-training is not a novel concept in AI. For instance, AlphaZero achieved
superhuman capabilities in Go, Chess, and Shogi via self-play (Silver et al.,
2017). Nevertheless, success with such a process has not yet been demonstrated
with language models. Recently, Wei et al. (2022) highlighted the intriguing
observation that language models often produce better results when prompted to
use chain-of-thought reasoning to solve the problem. This approach contrasts
with the conventional method where models are prompted to instantly generate
an answer, without producing any intermediate steps.

In this paper, we introduce _SECToR_ (Self-Education via Chain-of-Thought
Reasoning), which gives a proof-of-concept that large language models can
successfully teach themselves new abilities via chain-of-thought reasoning,
using addition as a benchmark task. Despite well-known difficulties for
language models to perform addition (Liu & Low, 2023; Nye et al., 2021; Lee et
al., 2023), language models trained via SECToR teach themselves to add numbers
with up to 29-digits _without access to any ground-truth examples_ for numbers
of length longer than 6 digits.

The observation that lies at the heart of SECToR is that chain-of-thought
reasoning can be considered a policy improvement operator: _regardless of the
quality_ of the underlying model (assuming it satisfies some minimum size and
training requirements), models that are prompted to use chain-of-thought
reasoning outperform directly sampling an answer from the model. SECToR uses
this observation to perform self-learning. During self-learning, SECToR
prompts the model to use chain-of-thought reasoning to generate solutions to
problems that it could not otherwise solve without such reasoning. Then, this
same model is fine-tuned to generate these exact solutions, this time _without
using chain-of-thought reasoning_. This leads to an improved version of the
model which can now directly solve problems that the previous version of the
model required using chain-of-thought reasoning to solve. If this improved
model, when again equipped with chain-of-thought reasoning, can solve a larger
set of problems than the original model could (even when equipped with chain-
of-thought reasoning), the self-learning process can continue. This procedure
is highly similar to the procedure that AlphaZero (Silver et al., 2017) used
to reach superhuman performance in Chess, Go, and Shogi, except using chain-
of-thought reasoning in place of Monte-Carlo Tree Search as the policy
improvement operator.

In the specific task of addition, SECToR begins with a pre-trained language
model and then performs an initial supervised fine-tuning period whereupon it
learns to perform addition both with and without using chain of thought
reasoning but only for addition problems with a small number of digits. After
the initial supervised training phase, it then undergoes a self-training phase
whereupon all training data is model-generated, _with zero access to ground-
truth data_. During each learning period, we train the language model digit-
by-digit using curriculum learning by requiring the model to perform
satisfactorily at $1$ to $N$ digit addition before introducing $N+1$ digit
problems to the dataset. Examples of each of these kinds of tasks are shown in
Figure 2. This setup is analogous to prior work such as AlphaGo (Silver et
al., 2016), where the model was initially trained on human-generated data
before undergoing self-training.

Task Type |  Example
---|---
Addition w/o CoT (fast) |  Q: 141 + 123 = ? A: 95.
Addition w/ CoT (slow) |  Q: 141 + 123 = ? A: The first number’s last digit is 1. The second number’s last digit is 3. 1 + 3 = 4. The last digit of this sum is 4 so our initial partial answer is 4. We can now recursively solve a smaller problem. Removing the last digit from each number we have 14 and 12. The next subproblem is 14 + 12.
Figure 2: Examples of the two types of addition tasks. One can view these two
types as the analogues of Type 1 (fast) and Type 2 (slow) reasoning (Kahneman,
2011).

A major challenge that has prevented past efforts of self-learning in language
models from succeeding, especially in arithmetic, is a phenomenon that we call
_error avalanching_. During self-training, when all training data is generated
by the model itself, there is no guarantee that the data is correct. _Error
avalanching_ occurs when small errors or inaccuracies in a model’s output
compounds rapidly over time, because each iteration of the model learns from
the outputs of the previous model amplifies the existing mistakes. We
demonstrate that, if left unchecked, error avalanching leads to severe
degradation of performance within only a few iterations of self-training. This
is consistent with past attempts to get language models to self-learn (for
addition as well as other tasks) in which improvement stagnates in only a few
steps (Zelikman et al., 2022; Lewkowycz et al., 2022; Bai et al., 2022). While
error avalanching is a fundamental issue in any bootstrapped process, SECToR
manages to largely mitigate error avalanching via several forms of self-
consistency checks (Figures 5 and 6) that minimize the number of mistakes
introduced to the dataset. Nevertheless, SECToR does not continue ad
infinitum, and training eventually terminates due to accumulated errors. We
return to this issue in the discussion.

Results. Our results indicate that the pre-trained 582M parameter ByT5 model
(Xue et al., 2022), after a supervised fine-tuning period that only provides
examples for $1$ to $6$ digits, can teach itself to perform up to 29-digit
addition with $98$+% accuracy (Table 2). Additionally, a smaller training run
with the 300M parameter version of ByT5 showed similar positive results,
successfully teaching itself to perform up to 24-digit addition after a
supervised learning phase where it was shown examples of up to 8 digits.

Figure 3: SECToR begins with a supervised fine-tuning phase in which the model
is trained with curriculum learning in which harder addition problems after
the model has mastered solving easier ones. SECToR begins self-training when
the models (when equipped with chain-of-thought reasoning) generalize
perfectly to $N+1$ digit addition despite only have been trained on $1$
through $N$ digit addition thus far. Empirically, this occurred at 7 digits
for the 582M parameter model. During self-training, training is largely
identical to the supervised training phase, except that all training data is
generated by the model itself using chain-of-thought reasoning and self-
consistency checks (see Figure 1).

##  2 Approach

###  2.1 AlphaZero and Policy Improvement Operators

The concept of a policy improvement operator plays a central role in
reinforcement learning. In reinforcement learning, a policy is the strategy by
which an agent chooses its actions, given the current state of the world. A
policy improvement operator is a function that takes in an arbitrary policy
and returns an improved policy, which is closer to the optimal policy, with
respect to a given reward function. Many reinforcement learning algorithms,
such as Q-learning or policy gradients, are built around a policy improvement
operator. A crucial property of a policy improvement operator is that it can
be used repeatedly, including for the improved policy that the operator has
just created. As such, if one continually applies a policy improvement
operator to a policy, it will eventually converge to the environment’s optimal
policy.

In AlphaZero, Monte-Carlo Tree Search (MCTS) served as the policy improvement
operator. Coulom (2006) proved that, regardless of the quality of the original
policy, running MCTS would provide a policy that was closer to the Nash
equilibrium of the game. However, AlphaZero required access to a perfect
simulator of the environment in order to perform MCTS. AlphaZero’s followup
work, MuZero (Schrittwieser et al., 2019), removed the requirement of direct
access to a simulator by learning a model of the world, but MuZero still
required continual query access to the true simulator to ensure that this
world model remained accurate during training.

Nevertheless, two-player zero-sum games are relatively simple, well-defined
environments. While a simulator is easily accessible for such small,
constrained environments, it is often not possible for general domains in the
real world, where environments may be ill-defined or otherwise difficult to
accurately simulate (e.g. real-world robotics, writing a Pulitzer Prize
winning novel). It remains an open question whether an analogue policy
improvement operator exists for a broader class of environments.

###  2.2 Reasoning as A Policy Improvement Operator

In this paper, we explore the hypothesis that _reasoning can serve as a policy
improvement operator_ , analogous to how MCTS functions in AlphaZero, but for
a broader range of environments. To complete the analogy, we consider the
language model’s conditional distribution over next token as the “policy,” the
previous context as the “environment,” and reasoning as the “policy
improvement” operator. The goal is for each version of the model to generate
the data upon which the next iteration of the model is trained using the
policy improvement operator. This form of self-training is highly similar to
the popular methods based on reinforcement learning from human feedback (RLHF)
(Christiano et al., 2023; Ouyang et al., 2022; Ziegler et al., 2020), with one
major exception: after an initial fine-tuning stage, all reward signal is
provided by the model itself, without any external or human input.

While large language models have not yet been shown to possess a complete
control of general reasoning skills, Wei et al. (2022) showed that asking
models to think through a problem step-by-step can often result in substantial
improvements in the model’s overall capabilities. This was dubbed chain-of-
thought reasoning. In particular, Wei et al. (2022) noticed that often a model
_unable to solve a problem_ without using reasoning was able to come up with
the correct answer if allowed to reason through the problem step-by step. One
way to view chain-of-thought reasoning is the model spending computational
resources at inference time to augment its own abilities. This is similar to
how AlphaZero spends compute to perform MCTS and augment its own game playing
abilities. Given that chain-of-thought reasoning can allow models to solve
problems that they otherwise could not, it is natural to ask whether repeated
application of this method might allow models to self-improve far beyond their
original capabilities.

At a high level, SECToR operates by using chain-of-thought reasoning as a
policy improvement operator to assist the model in solving problems that it
could not do without using the additional computation. It then constructs a
dataset of these new problem, solution pairs, which is then use to train the
model to solve these same problems _without using chain-of-thought reasoning._
The hope is that, having learned to directly solve problems that the previous
version of the model could only tackle with the additional computational power
of chain-of-thought reasoning, this improved model will be able to solve an
even larger set of problems when again augmented with chain-of-thought
reasoning. This process is designed to mimic how AlphaZero uses Monte-Carlo
Tree Search to teach itself to play Go.

###  2.3 Error Avalanching

The approach of repeatedly training the model on its own utterances usually
quickly results in a phenomenon we call error avalanching. Consider a scenario
where the model makes a small error in one iteration. When this output is fed
back into the model for the next round of training, the error becomes part of
the training data. As the model learns from this flawed data, the error may be
reinforced rather than corrected. In subsequent iterations, the error can get
compounded, leading to increasingly inaccurate outputs. This is akin to an
avalanche, where a small disturbance can trigger a massive slide.

Because we ask the model to learn without giving it any access to the ground
truth, it is important to prevent the compounding of errors, eventually
derailing the training process. In past attempts to use similar self-learning
approaches, this phenomenon caused the training process to go awry within just
a few steps (Zelikman et al., 2022; Jung et al., 2023). In order to mitigate
the effects of error avalanching, SECToR utilizes several consistency checks
largely based on the following idea: we ask the model to generate a large
number of “equivalent” variations of any given question and then ask the model
to (independently) solve each variation. If the answers are largely
consistent, we accept these answers into the training data. Otherwise, SECToR
discards the answers. These consistency checks are essential for SECToR to
minimize the introduction of incorrect data into the model’s self-training
loop.

##  3 Experiments

Figure 4: This figure describes the generalization accuracy of models to $N+1$
digit addition when they have only been trained on $1$ through $N$ digit
addition. Consistent with past work, we find that models typically show poor
length generalization when asked to perform standard (fast) addition. However,
if we allow the models to use chain-of-thought, we find that the
generalization accuracy quickly reaches almost perfect accuracy. The primary
insight of SECToR is that we can teach ourselves to add _without using chain-
of-thought_ (fast thinking) using our ability to add _with chain-of-thought_
(slow thinking).

###  3.1 Setup

Addition is a fundamental task in mathematics, but one on which language
models have historically struggled to perform. Nevertheless, addition has some
appealing properties as a toy dataset for evaluating self-learning. Firstly,
addition is a simple form of problem-solving. If a language model can teach
itself to solve addition problems, this may indicate potential for applying
general logic and reasoning to solve more complex problems. Additionally, it
is currently actively debated whether language models possess the ability to
reason symbolically (Arkoudas, 2023; Bender et al., 2021; Marcus, 2018). If a
model can teach itself to do addition, perhaps this indicates a potential for
a general ability to reason symbolically.

We ask whether models can teach themselves how to add very large numbers after
having been shown only addition examples with substantially fewer digits. For
simplicity, our setup includes only problems of the form $a+b$ or $a+b+1$,
where $a$ and $b$ are guaranteed to have the same number of digits. $a$ and
$b$ are uniformly sampled among all integers with the proper number of digits.
We train all models on a cluster of $8$ V100 GPUs with $32$ GB of memory each.
Prior work has indicated the importance of proper tokenization for performing
arithmetic (Liu & Low, 2023; Zhou et al., 2022). As such, we use the ByT5
family, which operates on the raw character/byte value and thus avoids any
potential issues with tokenization. All models are fine-tuned using the Adam
optimizer (Kingma & Ba, 2017) and the DeepSpeed library (Aminabadi et al.,
2022) with a constant learning rate of $10^{-4}$. We used a batch size of
$2048$ for our experiments with the 300M parameter model and $1024$ for the
582M parameter model, which were the maximum possible sizes that fit in memory
given our computational resources. Furthermore, we used 16-bit training via
bfloat16 to conserve memory.

###  3.2 Supervised Training

Without fine tuning, we find that the ByT5 models are quite poor at addition,
even when augmented with chain-of-thought reasoning, which is consistent with
prior work on benchmarking addition. SECToR thus begins by performing an
initial supervised fine-tuning phase consisting of addition problems with only
a small number of digits before beginning the self-training loop. This process
is described in Figure 3.

During this supervised training period, models are trained to perform addition
both with and without chain-of-thought reasoning. When performing “fast”
addition, models are asked to immediately output the solution to the addition
problem without using any intermediate tokens to reason through the answer. In
contrast, when performing “slow” addition, models are asked to perform a
single step of simplifying the $N$ digit problem into an $N-1$ digit problem
and a partial solution, similar to the method of teaching addition to
schoolchildren. Note that in our setup, “slow” addition does not fully solve
the problem: it merely performs the first step of the solution. Figure 2
depicts examples of both “fast” and “slow” addition. Training examples for
each type are prefixed with a special token depicting their type, so the model
can differentiate between the two possible answers. During the supervised
learning phase of training, all training examples are generated
programmatically by an external script. Learning both how to add numbers
instantly as well as via chain-of-thought addition can be viewed as a similar
idea to that of _process supervision_ (Lightman et al., 2023), which recently
achieved state-of-the-art performance on the MATH dataset (Hendrycks et al.,
2021).

The supervised training phase also follows a curriculum learning schedule
where a model must first achieve satisfactory performance on 1 through $N$
digit addition before $N+1$ digit examples are added to the dataset, starting
with $N=3$. Satisfactory performance is defined as achieving at least 75%
accuracy length $N$ addition problems without using chain-of-thought reasoning
and 100% accuracy (on length $N$ addition problems) when using chain-of-
thought reasoning on a held out test set of size $128$ of each type of
problem. Accuracy is measured by computing an exact token match between the
gold reference text and the model output while sampling at temperature $0$. We
do not require perfect accuracy on fast addition problems because SECToR’s
built in self-consistency checks are more robust to errors of this kind than
with errors in the chain-of-thought reasoning process. To prevent catastrophic
forgetting McCloskey & Cohen (1989), we mix examples from $1$ through $N$
digit addition while training the model on $N+1$ digit addition. The
supervised training period follows a curriculum learning setup largely to
minimize differences with the self-training phase of learning.

###  3.3 Self-Training

The second period of learning consists of self training. Here, the goal is to
ask whether the model can continue to teach itself skills even in the absence
of human demonstrations, as is often the case in reinforcement learning
setups.However, unlike standard reinforcement learning, SECToR does not give
the model the privilege of querying the environment: models must learn
entirely based on their own conceptions of the world. This means that all the
training occurs “in the model’s head” without any sort of grounding in the
real world. Thus, any errors incurred by the model cannot be corrected by
external feedback. Nevertheless, we demonstrate using addition as a proof of
concept, that models are able to successfully learn regardless.

The self-training phase of learning is largely identical to the supervised
training period, with one major exception. In the supervised training period,
data was generated with an external reference program that was guaranteed to
produce both the correct chain-of-thought decomposition and the correct answer
to the fast addition problems. In the self-training period, _all_ new training
data is generated by the model itself without any external verification of
correctness.

####  3.3.1 Generating New Addition Examples

The self-training phase requires the model being able to generate both the
“fast” and “slow” types of addition datapoints for numbers larger than it has
seen in training. While prior work has found that models trained only on $1$
to $N$ digit addition do not generalize very well to digit lengths greater
than $N$, we find that the reasoning capabilities of models generalize
exceptionally well to longer addition lengths. Figure 4 shows that the 560M
parameter model begins to achieve high generalization accuracy to $N+1$ digit
addition after training only on $1$ through $4$ digit addition. The 300M
parameter model shows similar results, but requires training on up to $5$
digit addition problems. We use this observation to generate new datapoints
during self-training as follows.

Self-training is begun only when models achieve near-perfect accuracy on slow,
chain-of-thought augmented $N+1$ digit addition. As such, to generate
candidate training examples for slow $N+1$ digit addition, we simply sample
from the model at temperature $0$. Generating fast addition problems is a more
complex process. Figure 4 shows that longer after models generalize using
chain-of-thought, generalization without such reasoning remains poor.

To generate solutions to $N+1$ digit addition problems, SECToR uses a novel
decoding method called _simplify-then-guess_ which utilizes a model’s
abilities to perform both fast and slow addition for $1$ through $N$ digit
addition. _Simplify-then-guess_ is inspired by the approaches of least-to-most
prompting Zhou et al. (2023) and self-consistency Wang et al. (2023b). In
least-to-most prompting, models are prompted to decompose problems into
simpler sub-problems before solving each one in sequence. simplify-then-guess
(Figure 6) follows in the footsteps of least-to-most prompting, but adds in a
built in self-consistency check inspired by Wang et al. (2023b). After each
simplification of the problem, simplify-then-guess asks the model to directly
guess the final solution without using any further reasoning steps. The final
answer is a majority vote over all intermediate guesses. For example, if a
model is tasked with solving an $8$-digit addition problem, it will first
simplify the problem into a $7$ digit addition problem before taking its first
guess of a solution. It then repeats this process with the 7-digit problem it
just generated, and so on. This process is described in Figure 6.

The primary advantage of simplify-then-guess over least-to-most prompting, is
that with least-to-most prompting, a single error at any point in the process
corrupts the entire solution. In contrast, simplify-then-guess has a built-in
error check in that, because the accuracy of each “guess” is unaffected by any
reasoning errors that occur after the guess is made. This error reduction is
critical for mitigating error avalanching.

Simplify-then-guess also has spiritual similarities to how AlphaZero often
does not perform MCTS to the end of the game, instead terminating search after
a certain depth and returning the output of the value network. The analogy for
simplify-then-guess is that SECToR runs $K$ steps of simplification (i.e.
search) before taking a guess at the answer (i.e. querying the value network)
instead of simplifying all the way to $1$-digit addition (i.e. running MCTS to
the end of the game). This takes direct advantage of the inductive curriculum-
style training in which a model learns to add $1$ to $N$ digit numbers before
being asked to generate training data for $N+1$ addition numbers.

Figure 5:  Simplify-then-guess operates by having the model simplify the
problem one step for $K$ iterations. After each simplification, the model
directly guesses the final solution without any further reasoning steps. These
guesses are then aggregated via majority vote to produce the final answer.

In SECToR, simplify-then-guess generates $K$ separate guesses for an addition
problem by applying between $1$ and $K$ simplification steps before fast
adding the remaining addition problem. It then takes a majority vote between
the generated guesses to construct a final guess for the answer. In this
paper, we use $K=5$, which was found to be a good balance between
computational speed and accuracy.

###  3.4 Commutativity Checks

Figure 6: This figure shows the impact of both chain-of-thought reasoning (via
simplify-then-guess) and the commutativity self-consistency check (note the
log scale y-axis). Directly generating training data for $N+1$ digit addition
via directly sampling from the model leads to extremely high error rates.
Generating training data with simplify-then-guess reduces this error, which is
again further reduced by the commutativity consistency check. When all methods
are put together, the error rate averages only two mistakes for every ten
thousand examples generated over the course of training.

Nevertheless, simplify-then-guess alone is not enough to mitigate error
avalanching. We perform an additional self-consistency check based on
commutativity. Specifically, we ask the model to solve independently (via
simplify-then-guess) both a problem ($a+b$) and its twin ($b+a$). If the
generated solutions are not equal, we discard these problems from the dataset.

For fast type addition problems, this check is simple to perform since the
output is just the answer. For slow addition problems, this is slightly
trickier since the chain-of-thought reasoning utterance for $A+B$ is not the
same as the answer for $B+A$. We solve this issue by checking that the answer
generated by simplifying for one step, followed by guessing an addition answer
(see previous section) answer is identical for a problem and its commutative
pair.

In the context of performing consistency checks on the auxiliary
simplification task, this consistency check procedure can lead to a large
number of false positive errors, which we define as discarding correct data
points because the model fails to add the subproblem correctly. Nevertheless,
accidentally discarding correct data is not the issue. Rather, the issue is
keeping wrong answers in the training set, and poisoning the future training.
Figure 6 describes the error rates over the course of training after using
both simplify-then-guess and the self-consistency check based on
commutativity.

##  4 Results

We report the results for a single training run for both the 300M parameter
ByT5 model as well as a larger 582M parameter model.

###  4.1 300M Parameter Model

The 300M model began self training after 6 digits. The training run is
described in Figure 9. Its last saved checkpoint was after successfully
learning up to 24 digit addition problems, having failed to successfully learn
continue the training loop with 25 digit addition. Figures 7 and 8 describe
the generalization accuracies of the 300M parameter model over the course of
training. Table 1 reports the addition accuracies achieved by the final
version of the model. Note that even though the model was unable to continue
training on the 25-digit iteration of learning, its previous checkpoint is
still able to do addition problems larger than $24$ digits with reasonable
accuracy.

Length | 1-19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27+
---|---|---|---|---|---|---|---|---|---
Accuracy (%) | 98-100 | 95 | 98 | 98 | 99 | 98 | 91 | 33 | 0
Table 1: Accuracy (out of 100 examples) of the final checkpoint of the 300M
model after training. For example, this table shows that the post-training
300M model can add 24 digit numbers with 98% accuracy without any chain-of-
thought reasoning. Figure 7: This figure describes the generalization accuracy
of the model’s addition capabilities over the course of training over the
training run of the 300M model. Blue lines indicate the supervised training
phase, while red lines indicate the self-training phase. We can see that even
at the end of training, models do not show much generalization in their
addition capabilities without using chain of thought.  Figure 8: This figure
describes the generalization accuracy of the model’s addition capabilities
over the course of training over the training run of the 300M model. Blue
lines indicate the supervised training phase, while red lines indicate the
self-training phase. While initially, models show little generalization to
lengths greater than than what they have seen in training, models quickly
learn to generalize well beyond their training distribution using chain-of-
thought, allowing them to continue teaching themselves addition problems
beyond what they have seen before.  Figure 9: This figure describes how much
training data was generated/consumed over the course of the training run of
the 300M model.

###  4.2 582M Parameter Model

The 582M model began self training after 6 digits (Figure 12). Its last saved
checkpoint was after successfully learning up to 28 digit addition problems,
having failed to successfully learn continue the training loop with 29 digit
addition. Figures 10 and 11 describe the generalization accuracies of the 582M
parameter model over the course of training. Table 2 reports the addition
accuracies achieved by the final version of the model.

Length | 1-29 | 30 | 31 | 32 | 33 | 34 | 35+
---|---|---|---|---|---|---|---
Accuracy (%) | 98-100 | 88 | 79 | 52 | 26 | 2 | 0
Table 2: Accuracy (out of 100 examples) of the final checkpoint of the 582M
model after training. For example, this table shows that the post-training
582M model can add 30 digit numbers with 88% accuracy without using any chain-
of-thought reasoning. Figure 10: This figure describes the generalization
accuracy of the model’s addition capabilities (without using chain-of-thought
reasoning) over the course of training over the training run of the 582M
model. Blue lines indicate the supervised training phase while red lines
indicate the self-training phase. Even at the end of the training run, no
model achieves perfect generalization to even two digits beyond what was
included in training.  Figure 11: This figure describes the generalization
accuracy of the model’s addition capabilities over the course of training over
the training run of the 582M model if the model is allowed to use chain-of-
thought reasoning. Blue lines indicate the supervised training phase, while
red lines indicate the self-training phase. While initially, models show
little generalization to lengths greater than than what they have seen in
training, models quickly learn to generalize well beyond their training
distribution using chain-of-thought, allowing them to continue teaching
themselves addition problems beyond what they have seen before. This result
stands in stark contrast to the analogous results without using chain-of-
thought reasoning (Figure 10).  Figure 12: This figure describes how much
training data was generated/consumed over the course of the training run of
the 582M model.

While the 300M and 582M parameter ByT5 models required an initial supervised
learning phase before being able to begin self-learning, we note that there
seems to be an inverse correlation between the size of the model and the
length of the supervised training period, both in terms of the maximum length
of addition problems seen as well as the total training examples. For
examples, Figure 9 shows that the 300M model required almost 4 million
training examples of up to length 8 before generalizing sufficiently well to
begin training. In contrast, Figure 12 shows that the 582M model required only
0.5 million training examples of up to length 6 before generalizing. We
hypothesize that a sufficiently large model might be able to forgo the
supervised training phase entirely and begin self-training immediately out of
the box, possibly with the assistance of in-context learning.

##  5 Related Work

###  5.1 Teaching Transformers Arithmetic

While many attempts have been made to teach large language models to perform
addition, to our knowledge, none have demonstrated that language models do so
by teaching themselves. Liu & Low (2023) fine-tuned the LLAMA family of
models, successfully teaching the models 8-digit addition using a supervised
fine-tuning, while methods such as Zhou et al. (2022) try to teach addition
via in-context learning instead of fine-tuning. Nye et al. (2021) propose
augmenting a language model by allowing it to emit intermediate computations
to a temporary “scratchpad,” which allowed the model to perform up to 10-digit
addition.

Lee et al. (2023) found significant challenges with length generation with
transformers, a crucial part of SECToR. A closer inspection, however, finds
that their results are entirely consistent with our findings. The main
difference is that Lee et al. (2023) only tested generalization in the chain-
of-thought regime with three-digit numbers. Figure 4 describes our results
with length generalization based on the size of the model and the number of
digits trained. We do not observe the emergence of substantial generalization
to $N+1$ digits until around training both models on $1$ through $5$ digit
addition.

###  5.2 Chain Of Thought Reasoning

Chain of thought reasoning was introduced by Wei et al. (2022) as a method of
improving performance in solving reasoning problems. Since then, this method
has been expanded upon and found to improve performance across many domains
(Wu et al., 2023; Zhang et al., 2023a; Feng et al., 2023; Yao et al., 2023;
Lightman et al., 2023).

###  5.3 Self-Training

AlphaZero was one of the original works demonstrating the concept of self-
learning, whereupon the training process employed Monte-Carlo Tree Search
(MCTS) as a policy improvement operator. Specifically, AlphaZero used MCTS to
improve upon the original policy of the game, before distilling this improved
policy into the original policy. MuZero, its follow-up, improved upon the
original work by removing the requirement of having access to a perfect
simulator of the game by learning a world model of the underlying board game
and performing a search using that world model. However, MuZero still required
occasional queries to the underlying world model to keep its world model
intact during training. Additionally, while MCTS is a powerful tool for
learning in board games, it is not a general policy improvement operator.
SECToR builds upon this prior work and can be used to train models to perform
up to 30-digit addition without _any_ queries to the ground truth world model
after the initial self-training period.

Prior approaches have been tried to perform self-learning in language models.
For example, StAR Zelikman et al. (2022) generates additional data for
training by asking models to give rationales for a correct question-answer
pair where none exist. It then trains again on those self-generated rationales
and demonstrates improved performance on several reasoning-based benchmarks.
Impossible distillation Jung et al. (2023) takes an off-the-shelf LLM and then
distills a high-quality dataset by filtering its generations. It then
undergoes a self-distillation phase where it does the same process on its own
generations, before undergoing one further phase of self-improvement. However,
the above methods fail to mitigate _error avalanching_ in the training
process, resulting in only a few steps of self-improvement before the process
terminates.

Other methods rely on self-improvement via methods other than updating the
weights. For example, methods such as Voyager (Wang et al., 2023a) exploit the
powerful in-context learning abilities of large language models to self-learn
Minecraft abilities. Reflexion (Shinn et al., 2023) uses a similar approach to
improve accuracy on the HumEval coding benchmark. SwiftSage (Lin et al., 2023)
uses a similar idea of “fast-and-slow” thinking, but does not perform the
self-learning loop of fine-tuning a model on its own generations.

###  5.4 Snowballing

Much work has also been done on the difficulties of self-training, especially
in language models. Shumailov et al. (2023) give both theoretical and
empirical evidence that models repeatedly trained on their own outputs,
without any filtering mechanisms, quickly degenerate into nonsense because of
error avalanching. SECToR mitigates this by using chain-of-thought reasoning
and self-consistency checks minimize the number of errors that accrue in the
training process. Zhang et al. (2023b) coins the term _hallucination
snowballing_ , which can be viewed as a specific form of _error avalanching_
in the context of generating factual content. Dziri et al. (2023) give some
theoretical justification that errors may snowball exponentially. Wang et al.
(2023b) proposed using self-consistency as a method of improving reasoning
compared to standard greedy decoding by sampling various methods of solving
the problem, and then using a majority-vote system among the generated
answers.

##  6 Discussion and Future Directions

We demonstrate that chain-of-thought reasoning can serve as a policy
improvement operator and show a proof-of-concept demonstration that language
models can teach themselves addition. In contrast to prior efforts in which
self-improvement fizzles out after only a few steps, models trained with
SECToR manage to stay on pace for over twenty steps. Nevertheless, there
remain a large number of directions to continue to explore regarding self-
learning with language models.

Limitations. While SECToR demonstrates the possibility of self-learning in
addition with language models, it is far from showing that models can self-
learn in general. A natural question is to ask whether SECToR or a similar
process generalizes to more complex tasks, such as multiplication or perhaps
even general mathematics or programming. Secondly, models trained with SECToR
do not improve forever. We speculate that a larger model, or a stronger
consistency check, might allow for the models to continue improving beyond 29
digits.

Self-learning vs. Learning to Self-Learn. While SECToR is a process by which
models teach themselves new concepts, they arguably do not _learn_ to teach
themselves new concepts. In our experiments, SECToR provides scaffolding
around the model which, while never performing any aspect of addition, assists
the language model in learning. For example, SECToR stops fine tuning once
accuracy hits a certain threshold and this threshold was not chosen by the
model itself. One can imagine using ideas from recent work, such as
Toolformer, to give a language model access to tools or API commands that
allow it to fine tune itself and add datapoints to the dataset. If successful,
one could imagine a process analogous to SECToR, except that the process would
consist solely of sampling from the model and executing the generated commands
with no additional assistance. If so, one could imagine a model learning to
teach itself a wide variety of concepts, including those in which it has to
self-discover the learning process, as well as the new concepts itself.

Grounding. Grounding is an area of active discussion in the research
community, with many criticizing language models for their perceived lack of
grounding in the real world. Some have suggested that future models may need
to be embodied to effectively learn in the real world (Marcus, 2018; Bender &
Koller, 2020; Tamari et al., 2020; Bisk et al., 2020). We believe our results
with SECToR raise the possibility that large language models can succeed
without grounding in domains that benefit from the existence of very strong
self-consistency checking (e.g., mathematics, programming, etc.). While the
model in the present paper is arguably grounded in true arithmetic during the
supervised training phase, during the self-training phase, of which the
majority of training occurs, models trained with SECToR receive no information
from the external world and are, in this sense, ungrounded. Nevertheless, they
manage to teach themselves addition problems that are orders of magnitude
larger than they have ever seen during supervised fine-tuning. Might it be
possible for models to bootstrap their learning in other domains without
access to an incremental source of external signal or grounding?

Safety. While SECToR is merely a proof-of-concept demonstration of the
possibility of self-learning in language models, this line of research brings
both tremendous opportunities as well as potential risks. One risk is that
self-learning may amplify preexisting biased or erroneous information in the
model during the self-training loop. This is not a concern when considering
purely objective domains such as addition, but may be an issue if self-
learning is more broadly applied to other domains with less objectivity.
Additionally, as models gain proficiency in autonomous learning, the
boundaries of their capabilities may become less and less predictable, raising
questions of how such models can be controlled and used in a safe manner. We
believe that alleviating these concerns is an important direction for future
research.

##  7 Acknowledgements

We thank Kenneth Li, Sonja Johnson-Yu, Daniel Bashir, Zhou Fan, and Safwan
Hossain for their feedback and discussions about this paper. We also thank
Microsoft Azure and the Harvard Data Science Initiative for access to compute.
The first author is supported by an NSF Graduate Research Fellowship and a
Kempner Institute Fellowship.

## References

  * Aminabadi et al. (2022) Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He.  DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale, June 2022.  URL http://arxiv.org/abs/2207.00032.  arXiv:2207.00032 [cs].
  * Arkoudas (2023) Konstantine Arkoudas.  GPT-4 Can’t Reason, August 2023.  URL http://arxiv.org/abs/2308.03762.  arXiv:2308.03762 [cs].
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional AI: Harmlessness from AI Feedback, December 2022\.  URL http://arxiv.org/abs/2212.08073.  arXiv:2212.08073 [cs].
  * Bender & Koller (2020) Emily M. Bender and Alexander Koller.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5185–5198, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.463.  URL https://aclanthology.org/2020.acl-main.463.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, pp. 610–623, New York, NY, USA, March 2021. Association for Computing Machinery.  ISBN 978-1-4503-8309-7.  doi: 10.1145/3442188.3445922.  URL https://dl.acm.org/doi/10.1145/3442188.3445922.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian.  Experience Grounds Language, November 2020.  URL http://arxiv.org/abs/2004.10151.  arXiv:2004.10151 [cs].
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), _Advances in neural information processing systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling Language Modeling with Pathways, October 2022\.  URL http://arxiv.org/abs/2204.02311.  arXiv:2204.02311 [cs].
  * Christiano et al. (2023) Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences, February 2023.  URL http://arxiv.org/abs/1706.03741.  arXiv:1706.03741 [cs, stat].
  * Coulom (2006) Rémi Coulom.  Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search.  volume 4630, May 2006.  ISBN 978-3-540-75537-1.  doi: 10.1007/978-3-540-75538-8˙7.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019.  URL http://arxiv.org/abs/1810.04805.  arXiv:1810.04805 [cs].
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi.  Faith and Fate: Limits of Transformers on Compositionality, June 2023.  URL http://arxiv.org/abs/2305.18654.  arXiv:2305.18654 [cs].
  * Feng et al. (2023) Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang.  Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective, June 2023.  URL http://arxiv.org/abs/2305.15408.  arXiv:2305.15408 [cs, stat].
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring Mathematical Problem Solving With the MATH Dataset.  _arXiv:2103.03874 [cs]_ , March 2021.  URL http://arxiv.org/abs/2103.03874.  arXiv: 2103.03874.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training Compute-Optimal Large Language Models, March 2022.  URL http://arxiv.org/abs/2203.15556.  arXiv:2203.15556 [cs].
  * Jung et al. (2023) Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi.  Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing, May 2023.  URL http://arxiv.org/abs/2305.16635.  arXiv:2305.16635 [cs].
  * Kahneman (2011) Daniel Kahneman.  _Thinking, fast and slow_.  Thinking, fast and slow. Farrar, Straus and Giroux, New York, NY, US, 2011\.  ISBN 978-0-374-27563-1 978-1-4299-6935-2.  Pages: 499.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling Laws for Neural Language Models, January 2020.  URL http://arxiv.org/abs/2001.08361.  arXiv:2001.08361 [cs, stat].
  * Kingma & Ba (2017) Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization, January 2017.  URL http://arxiv.org/abs/1412.6980.  arXiv:1412.6980 [cs].
  * Lee et al. (2023) Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos.  Teaching Arithmetic to Small Transformers, July 2023.  URL http://arxiv.org/abs/2307.03381.  arXiv:2307.03381 [cs].
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.  Solving Quantitative Reasoning Problems with Language Models, June 2022.  URL http://arxiv.org/abs/2206.14858.  arXiv:2206.14858 [cs].
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s Verify Step by Step, May 2023.  URL http://arxiv.org/abs/2305.20050.  arXiv:2305.20050 [cs].
  * Lin et al. (2023) Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren.  SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks, May 2023.  URL http://arxiv.org/abs/2305.17390.  arXiv:2305.17390 [cs].
  * Liu & Low (2023) Tiedong Liu and Bryan Kian Hsiang Low.  Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks, May 2023.  URL http://arxiv.org/abs/2305.14201.  arXiv:2305.14201 [cs].
  * Marcus (2018) Gary Marcus.  Deep Learning: A Critical Appraisal, January 2018.  URL http://arxiv.org/abs/1801.00631.  arXiv:1801.00631 [cs, stat].
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J. Cohen.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem.  In Gordon H. Bower (ed.), _Psychology of Learning and Motivation_ , volume 24, pp. 109–165. Academic Press, January 1989.  doi: 10.1016/S0079-7421(08)60536-8.  URL https://www.sciencedirect.com/science/article/pii/S0079742108605368.
  * Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena.  Show Your Work: Scratchpads for Intermediate Computation with Language Models, November 2021.  URL http://arxiv.org/abs/2112.00114.  arXiv:2112.00114 [cs].
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, March 2022.  URL http://arxiv.org/abs/2203.02155.  arXiv:2203.02155 [cs].
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language Models are Unsupervised Multitask Learners.  pp. 24, 2019.
  * Schrittwieser et al. (2019) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, and others.  Mastering atari, go, chess and shogi by planning with a learned model.  _arXiv preprint arXiv:1911.08265_ , 2019.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language Agents with Verbal Reinforcement Learning, June 2023.  URL http://arxiv.org/abs/2303.11366.  arXiv:2303.11366 [cs].
  * Shumailov et al. (2023) Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson.  The Curse of Recursion: Training on Generated Data Makes Models Forget, May 2023.  URL http://arxiv.org/abs/2305.17493.  arXiv:2305.17493 [cs].
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, January 2016.  ISSN 0028-0836, 1476-4687.  doi: 10.1038/nature16961.  URL http://www.nature.com/articles/nature16961.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, and others.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.  Publisher: Nature Publishing Group.
  * Tamari et al. (2020) Ronen Tamari, Chen Shani, Tom Hope, Miriam R L Petruck, Omri Abend, and Dafna Shahaf.  Language (Re)modelling: Towards Embodied Language Understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6268–6281, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.559.  URL https://aclanthology.org/2020.acl-main.559.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023\.  URL https://arxiv.org/abs/2307.09288v2.
  * Vaswani et al. (2023) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need, August 2023.  URL http://arxiv.org/abs/1706.03762.  arXiv:1706.03762 [cs].
  * Villalobos et al. (2022) Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho.  Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning, October 2022.  URL https://arxiv.org/abs/2211.04325v1.
  * Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An Open-Ended Embodied Agent with Large Language Models, May 2023a.  URL http://arxiv.org/abs/2305.16291.  arXiv:2305.16291 [cs].
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-Consistency Improves Chain of Thought Reasoning in Language Models, March 2023b.  URL http://arxiv.org/abs/2203.11171.  arXiv:2203.11171 [cs].
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv, 2022.  doi: 10.48550/arXiv.2201.11903.  URL http://arxiv.org/abs/2201.11903.  arXiv:2201.11903 [cs].
  * Wu et al. (2023) Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang.  An Empirical Study on Challenging Math Problem Solving with GPT-4, June 2023.  URL http://arxiv.org/abs/2306.01337.  arXiv:2306.01337 [cs, stat].
  * Xue et al. (2022) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel.  ByT5: Towards a token-free future with pre-trained byte-to-byte models, March 2022.  URL http://arxiv.org/abs/2105.13626.  arXiv:2105.13626 [cs].
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of Thoughts: Deliberate Problem Solving with Large Language Models, May 2023.  URL http://arxiv.org/abs/2305.10601.  arXiv:2305.10601 [cs].
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman.  STaR: Bootstrapping Reasoning With Reasoning, May 2022.  URL http://arxiv.org/abs/2203.14465.  arXiv:2203.14465 [cs].
  * Zhang et al. (2023a) Mengxue Zhang, Zichao Wang, Zhichao Yang, Weiqi Feng, and Andrew Lan.  Interpretable Math Word Problem Solution Generation Via Step-by-step Planning, June 2023a.  URL http://arxiv.org/abs/2306.00784.  arXiv:2306.00784 [cs].
  * Zhang et al. (2023b) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith.  How Language Model Hallucinations Can Snowball, May 2023b.  URL http://arxiv.org/abs/2305.13534.  arXiv:2305.13534 [cs].
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, April 2023.  URL http://arxiv.org/abs/2205.10625.  arXiv:2205.10625 [cs].
  * Zhou et al. (2022) Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi.  Teaching Algorithmic Reasoning via In-context Learning, November 2022.  URL http://arxiv.org/abs/2211.09066.  arXiv:2211.09066 [cs].
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-Tuning Language Models from Human Preferences, January 2020\.  URL http://arxiv.org/abs/1909.08593.  arXiv:1909.08593 [cs, stat].