# Self-Rag: Self-reflective Retrieval augmented Generation

Akari Asai†, Zeqiu Wu†, Yizhong Wang†§, Avirup Sil‡, Hannaneh Hajishirzi†§
†University of Washington §Allen Institute for AI ‡IBM Research AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>

#  Self-rag: Learning to Retrieve, Generate, and Critique through Self-
Reflection

Akari Asai†, Zeqiu Wu†, Yizhong Wang†§, Avirup Sil‡, Hannaneh Hajishirzi†§
†University of Washington §Allen Institute for AI ‡IBM Research AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Despite their remarkable capabilities, large language models (LLMs) often
produce responses containing factual inaccuracies due to their sole reliance
on the parametric knowledge they encapsulate. Retrieval-Augmented Generation
(RAG), an ad hoc approach that augments LMs with retrieval of relevant
knowledge, decreases such issues. However, indiscriminately retrieving and
incorporating a fixed number of retrieved passages, regardless of whether
retrieval is necessary, or passages are relevant, diminishes LM versatility or
can lead to unhelpful response generation. We introduce a new framework called
Self-Reflective Retrieval-Augmented Generation (Self-Rag) that enhances an
LM’s quality and factuality through retrieval and self-reflection. Our
framework trains a single arbitrary LM that adaptively retrieves passages on-
demand, and generates and reflects on retrieved passages and its own
generations using special tokens, called reflection tokens. Generating
reflection tokens makes the LM controllable during the inference phase,
enabling it to tailor its behavior to diverse task requirements. Experiments
show that Self-Rag (7B and 13B parameters) significantly outperforms state-of-
the-art LLMs and retrieval-augmented models on a diverse set of tasks.
Specifically, Self-Rag outperforms ChatGPT and retrieval-augmented Llama2-chat
on Open-domain QA, reasoning and fact verification tasks, and it shows
significant gains in improving factuality and citation accuracy for long-form
generations relative to these models.111Our code and trained models are
available at https://selfrag.github.io/.

##  1 Introduction

State-of-the-art LLMs continue to struggle with factual errors (Mallen et al.,
2023; Min et al., 2023) despite their increased model and data scale (Ouyang
et al., 2022). Retrieval-Augmented Generation (RAG) methods (Figure 1 left;
Lewis et al. 2020; Guu et al. 2020) augment the input of LLMs with relevant
retrieved passages, reducing factual errors in knowledge-intensive tasks (Ram
et al., 2023; Asai et al., 2023a). However, these methods may hinder the
versatility of LLMs or introduce unnecessary or off-topic passages that lead
to low-quality generations (Shi et al., 2023) since they retrieve passages
indiscriminately regardless of whether the factual grounding is helpful.
Moreover, the output is not guaranteed to be consistent with retrieved
relevant passages (Gao et al., 2023) since the models are not explicitly
trained to leverage and follow facts from provided passages. This work
introduces Self-Reflective Retrieval-augmented Generation (Self-Rag) to
improve an LLM’s generation quality, including its factual accuracy without
hurting its versatility, via on-demand retrieval and self-reflection. We train
an arbitrary LM in an end-to-end manner to learn to reflect on its own
generation process given a task input by generating both task output and
intermittent special tokens (i.e., reflection tokens). Reflection tokens are
categorized into retrieval and critique tokens to indicate the need for
retrieval and its generation quality respectively (Figure 1 right). In
particular, given an input prompt and preceding generations, Self-Rag first
determines if augmenting the continued generation with retrieved passages
would be helpful. If so, it outputs a retrieval token that calls a retriever
model on demand (Step 1). Subsequently, Self-Rag concurrently processes
multiple retrieved passages, evaluating their relevance and then generating
corresponding task outputs (Step 2). It then generates critique tokens to
criticize its own output and choose best one (Step 3) in terms of factuality
and overall quality. This process differs from conventional RAG (Figure 1
left), which consistently retrieves a fixed number of documents for generation
regardless of the retrieval necessity (e.g., the bottom figure example does
not require factual knowledge) and never second visits the generation quality.
Moreover, Self-Rag provides citations for each segment with its self-
assessment of whether the output is supported by the passage, leading to
easier fact verification.

Self-Rag trains an arbitrary LM to generate text with reflection tokens by
unifying them as the next token prediction from the expanded model vocabulary.
We train our generator LM on a diverse collection of text interleaved with
reflection tokens and retrieved passages. Reflection tokens, inspired by
reward models used in reinforcement learning (Ziegler et al., 2019; Ouyang et
al., 2022), are inserted offline into the original corpus by a trained critic
model. This eliminates the need to host a critic model during training,
reducing overhead. The critic model, in part, is supervised on a dataset of
input, output, and corresponding reflection tokens collected by prompting a
propriety LM (i.e., GPT-4; OpenAI 2023). While we draw inspiration from
studies that use control tokens to start and guide text generation (Lu et al.,
2022; Keskar et al., 2019), our trained LM uses critique tokens to assess its
own predictions after each generated segment as an integral part of the
generation output.

Self-Rag further enables a customizable decoding algorithm to satisfy hard or
soft constraints, which are defined by reflection token predictions. In
particular, our inference-time algorithm enables us to (1) flexibly adjust
retrieval frequency for different downstream applications and (2) customize
models’ behaviors to user preferences by leveraging reflection tokens through
segment-level beam search using the weighted linear sum of the reflection
token probabilities as segment score.

Figure 1: Overview of Self-Rag. Self-Rag learns to retrieve, critique, and
generate text passages to enhance overall generation quality, factuality, and
verifiability.

Empirical results on six tasks, including reasoning and long-form generation,
demonstrate that Self-Rag significantly outperforms pre-trained and
instruction-tuned LLMs that have more parameters and widely adopted RAG
approaches with higher citation accuracy. In particular, Self-Rag outperforms
retrieval-augmented ChatGPT on four tasks, Llama2-chat (Touvron et al., 2023)
and Alpaca (Dubois et al., 2023) on all tasks. Our analysis demonstrates the
effectiveness of training and inference with reflection tokens for overall
performance improvements as well as test-time model customizations (e.g.,
balancing the trade-off between citation previsions and completeness).

##  2 Related Work

Retrieval-Augmented Generation. Retrieval-Augmented Generation (RAG) augments
the input space of LMs with retrieved text passages (Guu et al., 2020; Lewis
et al., 2020), leading to large improvements in knowledge-intensive tasks
after fine-tuning or used with off-the-shelf LMs (Ram et al., 2023). A more
recent work (Luo et al., 2023) instruction-tunes an LM with a fixed number of
retrieved passages prepended to input, or pre-train a retriever and LM
jointly, followed by few-shot fine-tuning on task datasets (Izacard et al.,
2022b). While prior work often retrieves only once at the beginning, Jiang et
al. (2023) propose to adaptively retrieve passages for generation on top of a
proprietary LLM or Schick et al. (2023) train an LM to generate API calls for
named entities. Yet, the improved task performance of such approaches often
comes at the expense of runtime efficiency (Mallen et al., 2023), robustness
to irrelevant context (Shi et al., 2023), and lack of attributions (Liu et
al., 2023a; Gao et al., 2023). We introduce a method to train an arbitrary LM
to learn to use retrieval on-demand for diverse instruction-following queries
and introduce controlled generation guided by reflections tokens to further
improve generation quality and attributions.

Concurrent RAG work.  A few concurrent works222All work is arXived within a
week of this preprint. on RAG propose new training or prompting strategies to
improve widely-adopted RAG approaches. Lin et al. (2023) fine-tune both the
retriever and LM on instruction-tuning datasets in two steps. While we also
train our model on diverse instruction-following datasets, Self-Rag enables
retrieval on demand and selection of the best possible model output via fine-
grained self-reflection, making it widely applicable and more robust and
controllable. Yoran et al. (2023) use a natural language inference model and
Xu et al. (2023) use a summarization model to filter out or compress retrieved
passages before using them to prompt the LM to generate the output. Self-Rag
processes passages in parallel and filters out irrelevant ones through self-
reflection, without relying on external models at inference. Moreover, our
self-reflection mechanism also evaluates other aspects of the model output
quality including factuality. LATS (Zhou et al., 2023) prompt off-the-shelf
LMs to search for relevant information for question answering tasks and to
generate with tree search, guided by LM-generated value scores. While their
value function simply indicates an overall score of each generation, Self-Rag
trains to an arbitrary LM to learn to generate fine-grained self-reflection
and customizable inference.

Training and generating with critics. Training LLMs with reinforcement
learning (e.g., Proximal Policy Optimization or PPO; Schulman et al. 2017)
from human feedback (RLHF) has proven effective in aligning LLMs with human
preferences (Ouyang et al., 2022). Wu et al. (2023) introduce fine-grained
RLHF with multiple reward models. Though our work also studies fine-grained
critique on retrieval and generation, we train our target LM on task examples
augmented with reflection tokens from a critic model offline, with a far lower
training cost compared to RLHF. In addition, reflection tokens in Self-Rag
enable controllable generation at inference, while RLHF focuses on human
preference alignment during training. Other works use general control tokens
to guide LM generation (Lu et al., 2022; Korbak et al., 2023), while Self-Rag
uses reflection tokens to decide the need for retrieval and to self-evaluate
generation quality. Xie et al. (2023) propose a self-evaluation-guided
decoding framework, but they focus only on reasoning tasks with one evaluation
dimension (reasoning path consistency) and without retrieval. Recent work on
LLM refinement (Dhuliawala et al., 2023; Madaan et al., 2023; Paul et al.,
2023) prompts a model to generate task output, natural language feedback and
refined task output iteratively, but at the cost of inference efficiency.

##  3 Self-Rag: Learning to Retrieve, Generate and Critique

We introduce Self-Reflective Retrieval-Augmented Generation (Self-Rag), shown
in Figure 1. Self-Rag is a framework that enhances the quality and factuality
of an LLM through retrieval and self-reflection, without sacrificing LLM’s
original creativity and versatility. Our end-to-end training lets an LM
$\mathcal{M}$ generate text informed by retrieved passages, if needed, and
criticize the output by learning to generate special tokens. These reflection
tokens (Table 1) signal the need for retrieval or confirm the output’s
relevance, support, or completeness. In contrast, common RAG approaches
retrieve passages indiscriminately, without ensuring complete support from
cited sources.

###  3.1 Problem Formalization and Overview

Formally, given input $x$, we train $\mathcal{M}$ to sequentially generate
textual outputs $y$ consisting of multiple segments $y=[y_{1},\dots,y_{T}]$,
where $y_{t}$ indicates a sequence of tokens for the $t$-th segment.333 In
this paper, we treat one sentence as a segment in our experiments, but our
framework is applicable to any segment unit (i.e., sub-sentence). Generated
tokens in $y_{t}$ include text from the original vocabulary as well as the
reflection tokens (Table 1).

Type |  Input |  Output |  Definitions
---|---|---|---
Retrieve |  $x$ / $x,y$ |  {yes, no, continue} |  Decides when to retrieve with $\mathcal{R}$
IsRel |  $x,d$ |  {relevant, irrelevant} |  $d$ provides useful information to solve $x$.
IsSup |  $x,d,y$ |  {fully supported, partially supported, no support} |  All of the verification-worthy statement in $y$ is supported by $d$.
IsUse |  $x,y$ |  {5, 4, 3, 2, 1} |  $y$ is a useful response to $x$.
Table 1: Four types of reflection tokens used in Self-Rag. Each type uses
several tokens to represent its output values. The bottom three rows are three
types of Critique tokens, and the bold text indicates the most desirable
critique tokens. $x,y,d$ indicate input, output, and a relevant passage,
respectively.  Algorithm 1 Self-Rag Inference

1:Generator LM $\mathcal{M}$, Retriever $\mathcal{R}$, Large-scale passage
collections $\\{d_{1},\ldots,d_{N}\\}$

2:Input: input prompt $x$ and preceding generation $y_{<t}$, Output: next
output segment $y_{t}$

3:$\mathcal{M}$ predicts Retrieve given $(x,y_{<t})$

4:if Retrieve == Yes then

5: Retrieve relevant text passages $\mathbf{D}$ using $\mathcal{R}$ given
$(x,y_{t-1})$ $\triangleright$ Retrieve

6: $\mathcal{M}$ predicts IsRel given $x,d$ and $y_{t}$ given $x,d,y_{<t}$ for
each $d\in\mathbf{D}$ $\triangleright$ Generate

7: $\mathcal{M}$ predicts IsSup and IsUse given $x,y_{t},d$ for each
$d\in\mathbf{D}$ $\triangleright$ Critique

8: Rank $y_{t}$ based on IsRel , IsSup , IsUse $\triangleright$ Detailed in
Section 3.3

9:else if Retrieve == No then

10: $\mathcal{M}_{gen}$ predicts $y_{t}$ given $x$ $\triangleright$ Generate

11: $\mathcal{M}_{gen}$ predicts IsUse given $x,y_{t}$$\triangleright$
Critique

Inference overview. Figure 1 and Algorithm 1 present an overview of Self-Rag
at inference. For every $x$ and preceding generation $y_{<t}$, the model
decodes a retrieval token to evaluate the utility of retrieval. If retrieval
is not required, the model predicts the next output segment, as it does in a
standard LM. If retrieval is needed, the model generates: a critique token to
evaluate the retrieved passage’s relevance, the next response segment, and a
critique token to evaluate if the information in the response segment is
supported by the passage. Finally, a new critique token evaluates the overall
utility of the response.444We follow Liu et al. (2023a) in using a “perceived”
utility value that is independent of retrieved passages. To generate each
segment, Self-Rag processes multiple passages in parallel and uses its own
generated reflection tokens to enforce soft constraints (Section 3.3) or hard
control (Algorithm 1) over the generated task output. For instance, in Figure
1 (right), the retrieved passages $d_{1}$ is selected at the first time step
since $d_{2}$ does not provide direct evidence ( IsRel is Irrelevant) and
$d_{3}$ output is only partially supported while $d_{1}$ are fully supported.

Training overview. Self-Rag enables an arbitrary LM to generate text with
reflection tokens by unifying them as next token predictions from the expanded
model vocabulary (i.e., the original vocabulary plus reflection tokens).
Specifically, we train the generator model $\mathcal{M}$ on a curated corpus
with interleaving passages retrieved by a retriever $\mathcal{R}$ and
reflection tokens predicted by a critic model $\mathcal{C}$ (summarized in
Appendix Algorithm 2). We train $\mathcal{C}$ to generate reflection tokens
for evaluating retrieved passages and the quality of a given task output
(Section 3.2.1). Using the critic model, we update the training corpus by
inserting reflection tokens into task outputs offline. Subsequently, we train
the final generator model ($\mathcal{M}$) using the conventional LM objective
(Section 3.2.2) to enable $\mathcal{M}$ to generate reflection tokens by
itself without relying on the critic at inference time.

###  3.2 Self-Rag Training

Here, we describe the supervised data collection and training of two models,
the critic $\mathcal{C}$ (Section 3.2.1) and the generator $\mathcal{M}$
(Section 3.2.2).

####  3.2.1 Training the Critic Model

Data collection for critic model. Manual annotation of reflection tokens for
each segment is expensive (Wu et al., 2023). A state-of-the-art LLM like GPT-4
(OpenAI, 2023) can be effectively used to generate such feedback (Liu et al.,
2023b). However, depending on such proprietary LMs can raise API costs and
diminish reproducibility (Chen et al., 2023). We create supervised data by
prompting GPT-4 to generate reflection tokens and then distill their knowledge
into an in-house $\mathcal{C}$. For each group of reflection tokens, we
randomly sample instances from the original training data:
$\\{X^{sample},Y^{sample}\\}\sim\\{X,Y\\}$. As different reflection token
groups have their own definitions and input, as shown in Table 1, we use
different instruction prompts for them. Here, we use Retrieve as an example.
We prompt GPT-4 with a type-specific instruction (“Given an instruction, make
a judgment on whether finding some external documents from the web helps to
generate a better response.”) followed by few-shot demonstrations $I$ the
original task input $x$ and output ${y}$ to predict an appropriate reflection
token as text: $p(r|I,x,y)$. Manual assessment reveals that GPT-4 reflection
token predictions show high agreement with human evaluations. We collect
4k-20k supervised training data for each type and combine them to form
training data for $\mathcal{C}$. Appendix Section D shows the full list of
instructions, and A.1 contains more details and our analysis.

##### Critic learning.

After we collect training data $\mathcal{D}_{critic}$, we initialize
$\mathcal{C}$ with a pre-trained LM and train it on $\mathcal{D}_{critic}$
using a standard conditional language modeling objective, maximizing
likelihood:

$\max_{\mathcal{C}}\mathbb{E}_{((x,y),r)\sim\mathcal{D}_{critic}}\log
p_{\mathcal{C}}(r|x,y),\text{ $r$ for reflection tokens. }$ (1)

Though the initial model can be any pre-trained LM, we use the same one as the
generator LM (i.e., Llama 2-7B; Touvron et al. 2023) for $\mathcal{C}$
initialization. The critic achieves a higher than 90% agreement with
GPT-4-based predictions on most reflection token categories (Appendix Table
4).

Figure 2: Self-Rag training examples. The left example does not require
retrieval while the right one requires retrieval; thus, passages are inserted.
More examples are in Appendix Table 4.

####  3.2.2 Training the Generator Model

##### Data collection for generator.

Given an input-output pair $(x,y)$, we augment the original output $y$ using
the retrieval and critic models to create supervised data that precisely
mimics the Self-Rag inference-time process (Section 3.1). For each segment
$y_{t}\in y$, we run $\mathcal{C}$ to assess whether additional passages could
help to enhance generation. If retrieval is required, the retrieval special
token Retrieve =Yes is added, and $\mathcal{R}$ retrieves the top $K$
passages, $\mathbf{D}$. For each passage, $\mathcal{C}$ further evaluates
whether the passage is relevant and predicts IsRel . If a passage is relevant,
$\mathcal{C}$ further evaluates whether the passage supports the model
generation and predicts IsSup . Critique tokens IsRel and IsSup are appended
after the retrieved passage or generations. At the end of the output, $y$ (or
$y_{T}$), $\mathcal{C}$ predicts the overall utility token IsUse , and an
augmented output with reflection tokens and the original input pair is added
to $\mathcal{D}_{gen}$. See the example training data in Figure 2.

Generator learning. We train the generator model $\mathcal{M}$ by training on
the curated corpus augmented with reflection tokens $\mathcal{D}_{gen}$ using
the standard next token objective:

$\max_{\mathcal{M}}\mathbb{E}_{(x,y,r)\sim\mathcal{D}_{gen}}\log
p_{\mathcal{M}}(y,r|x).$ (2)

Unlike $\mathcal{C}$ training (Eq. 1), $\mathcal{M}$ learns to predict the
target output as well as the reflection tokens. During training, we mask out
the retrieved text chunks (surrounded by <p> and </p> in Figure 2) for loss
calculation and expand the original vocabulary $\mathcal{V}$ with a set of
reflection tokens $\\{\mathchoice{\leavevmode\hbox to40.66pt{\vbox
to13.58pt{\pgfpicture\makeatletter\hbox{\hskip 20.33058pt\lower-4.34442pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to40.66pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
20.33058pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.9pt{\vbox to10.94pt{\pgfpicture\makeatletter\hbox{\hskip
14.95139pt\lower-3.76108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.55139pt}{-3.36108pt}{29.10278pt}{10.14441pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.55139pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to22.73pt{\vbox to9.19pt{\pgfpicture\makeatletter\hbox{\hskip
11.36526pt\lower-3.3722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.96527pt}{-2.9722pt}{21.93054pt}{8.38885pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.96527pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}},\mathchoice{\leavevmode\hbox
to41.08pt{\vbox to11.63pt{\pgfpicture\makeatletter\hbox{\hskip
20.5389pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-20.1389pt}{-2.0pt}{40.2778pt}{10.83331pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-18.1389pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.7,0.3,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}Retrieve}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to41.08pt{\vbox to11.63pt{\pgfpicture\makeatletter\hbox{\hskip
20.5389pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-20.1389pt}{-2.0pt}{40.2778pt}{10.83331pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-18.1389pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.7,0.3,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}Retrieve}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to30.19pt{\vbox to9.58pt{\pgfpicture\makeatletter\hbox{\hskip
15.09723pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.69724pt}{-2.0pt}{29.39447pt}{8.78333pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.69724pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.7,0.3,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}Retrieve}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to22.94pt{\vbox to8.22pt{\pgfpicture\makeatletter\hbox{\hskip
11.46942pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}\pgfsys@color@rgb@stroke{0.7}{0.3}{0.0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-11.06943pt}{-2.0pt}{22.13885pt}{7.41666pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-9.06943pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.7,0.3,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{0.7,0.3,0.0}Retrieve}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\\}$.

##### Connections to prior work on learning with critique.

Recent work incorporates additional critique (feedback) during training, e.g.,
RLHF (Ouyang et al. 2022) via PPO. While PPO relies on separate reward models
during training, we compute critique offline and directly insert them into the
training corpus, where the generator LM is trained with a standard LM
objective. This significantly reduces training costs compared to PPO. Our work
also relates to prior work that incorporates special tokens to control
generation (Keskar et al., 2019; Lu et al., 2022; Korbak et al., 2023). Our
Self-Rag learns to generate special tokens to evaluate its own prediction
after each generated segment, enabling the use of a soft re-ranking mechanism
or hard constraints at inference (discussed next).

###  3.3 Self-Rag Inference

Generating reflection tokens to self-evaluate its own output makes Self-Rag
controllable during the inference phase, enabling it to tailor its behavior to
diverse task requirements. For tasks demanding factual accuracy (Min et al.,
2023), we aim for the model to retrieve passages more frequently to ensure
that the output aligns closely with the available evidence. Conversely, in
more open-ended tasks, like composing a personal experience essay, the
emphasis shifts towards retrieving less and prioritizing the overall
creativity or utility score. In this section, we describe approaches to
enforce control to meet these distinct objectives during the inference
process.

Adaptive retrieval with threshold. Self-Rag dynamically decides when to
retrieve text passages by predicting Retrieve. Alternatively, our framework
allows a threshold to be set. Specifically, if the probability of generating
the Retrieve=Yes token normalized over all output tokens in Retrieve surpasses
a designated threshold, we trigger retrieval (details in Appendix Section
A.3).

Tree-decoding with critique tokens. At each segment step $t$, when retrieval
is required, based either on hard or soft conditions, $\mathcal{R}$ retrieves
$K$ passages, and the generator $\mathcal{M}$ processes each passage in
parallel and outputs $K$ different continuation candidates. We conduct a
segment-level beam search (with the beam size=$B$) to obtain the top-$B$
segment continuations at each timestamp $t$, and return the best sequence at
the end of generation. The score of each segment $y_{t}$ with respect to
passage $d$ is updated with a critic score $\mathcal{S}$ that is the linear
weighted sum of the normalized probability of each Critique token type. For
each critique token group $G$ (e.g., IsRel ), we denote its score at timestamp
$t$ as $s_{t}^{G}$, and we compute a segment score as follows:

$f(y_{t},d,\mathchoice{\leavevmode\hbox to40.66pt{\vbox
to13.58pt{\pgfpicture\makeatletter\hbox{\hskip 20.33058pt\lower-4.34442pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to40.66pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
20.33058pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.9pt{\vbox to10.94pt{\pgfpicture\makeatletter\hbox{\hskip
14.95139pt\lower-3.76108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.55139pt}{-3.36108pt}{29.10278pt}{10.14441pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.55139pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to22.73pt{\vbox to9.19pt{\pgfpicture\makeatletter\hbox{\hskip
11.36526pt\lower-3.3722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.96527pt}{-2.9722pt}{21.93054pt}{8.38885pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.96527pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}})=p(y_{t}|x,d,y_{<t}))+\mathcal{S}(\mathchoice{\leavevmode\hbox
to40.66pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
20.33058pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to40.66pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
20.33058pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.9pt{\vbox to10.94pt{\pgfpicture\makeatletter\hbox{\hskip
14.95139pt\lower-3.76108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.55139pt}{-3.36108pt}{29.10278pt}{10.14441pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.55139pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to22.73pt{\vbox to9.19pt{\pgfpicture\makeatletter\hbox{\hskip
11.36526pt\lower-3.3722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.96527pt}{-2.9722pt}{21.93054pt}{8.38885pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.96527pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}){\rm,where}$ (3)
$\mathcal{S}(\mathchoice{\leavevmode\hbox to40.66pt{\vbox
to13.58pt{\pgfpicture\makeatletter\hbox{\hskip 20.33058pt\lower-4.34442pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to40.66pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
20.33058pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-19.93059pt}{-3.94443pt}{39.86118pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.93059pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.9pt{\vbox to10.94pt{\pgfpicture\makeatletter\hbox{\hskip
14.95139pt\lower-3.76108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.55139pt}{-3.36108pt}{29.10278pt}{10.14441pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.55139pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to22.73pt{\vbox to9.19pt{\pgfpicture\makeatletter\hbox{\hskip
11.36526pt\lower-3.3722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.96527pt}{-2.9722pt}{21.93054pt}{8.38885pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.96527pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}Critique}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}})=\sum_{G\in\mathcal{G}}w^{G}s_{t}^{G}\mbox{
for }\mathcal{G}=\\{\mathchoice{\leavevmode\hbox to26.94pt{\vbox
to11.74pt{\pgfpicture\makeatletter\hbox{\hskip 13.46944pt\lower-2.4pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-13.06944pt}{-2.0pt}{26.13889pt}{10.94443pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-11.06944pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsRel}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to26.94pt{\vbox to11.74pt{\pgfpicture\makeatletter\hbox{\hskip
13.46944pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-13.06944pt}{-2.0pt}{26.13889pt}{10.94443pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-11.06944pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsRel}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to20.3pt{\vbox to9.66pt{\pgfpicture\makeatletter\hbox{\hskip
10.1486pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-9.74861pt}{-2.0pt}{19.49722pt}{8.86108pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-7.74861pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsRel}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to15.87pt{\vbox to8.27pt{\pgfpicture\makeatletter\hbox{\hskip
7.93471pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-7.53471pt}{-2.0pt}{15.06943pt}{7.4722pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.53471pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsRel}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}},\mathchoice{\leavevmode\hbox
to29.02pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
14.51112pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.11113pt}{-3.94443pt}{28.22226pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.11113pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsSup}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.02pt{\vbox to13.58pt{\pgfpicture\makeatletter\hbox{\hskip
14.51112pt\lower-4.34442pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-14.11113pt}{-3.94443pt}{28.22226pt}{12.77774pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.11113pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsSup}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to21.76pt{\vbox to10.94pt{\pgfpicture\makeatletter\hbox{\hskip
10.87778pt\lower-3.76108pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.47778pt}{-3.36108pt}{20.95557pt}{10.14441pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.47778pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsSup}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to16.91pt{\vbox to9.19pt{\pgfpicture\makeatletter\hbox{\hskip
8.45555pt\lower-3.3722pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-8.05556pt}{-2.9722pt}{16.11111pt}{8.38885pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-6.05556pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsSup}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}},\mathchoice{\leavevmode\hbox
to28.24pt{\vbox to11.63pt{\pgfpicture\makeatletter\hbox{\hskip
14.12222pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-13.72223pt}{-2.0pt}{27.44446pt}{10.83331pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-11.72223pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsUse}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to28.24pt{\vbox to11.63pt{\pgfpicture\makeatletter\hbox{\hskip
14.12222pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-13.72223pt}{-2.0pt}{27.44446pt}{10.83331pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-11.72223pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsUse}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to21.21pt{\vbox to9.58pt{\pgfpicture\makeatletter\hbox{\hskip
10.60555pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-10.20555pt}{-2.0pt}{20.4111pt}{8.78333pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-8.20555pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsUse}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to16.52pt{\vbox to8.22pt{\pgfpicture\makeatletter\hbox{\hskip
8.2611pt\lower-2.4pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} {\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}\pgfsys@color@rgb@stroke{0.2}{0.3}{0.6}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.8pt}\pgfsys@invoke{
}{}\pgfsys@rect{-7.8611pt}{-2.0pt}{15.7222pt}{7.41666pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-5.8611pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\textbf{{\color[rgb]{0.2,0.3,0.6}\definecolor[named]{pgfstrokecolor}{rgb}{0.2,0.3,0.6}{IsUse}}}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\\},$ (4)

where $s_{t}^{G}=\frac{p_{t}(\hat{r})}{\sum_{i=1}^{N^{G}}p_{t}(r_{i})}$ stands
for the generation probability of the most desirable reflection token
$\hat{r}$ (e.g., IsRel =Relevant) for the critique token type $G$ with $N^{G}$
distinct tokens (that represent different possible values for $G$). The
weights $w^{G}$ in Eq. 4 are hyperparameters that can be adjusted at inference
time to enable customized behaviors at test time. For instance, to ensure that
result $y$ is mostly supported by evidence, we can set a weight term for the
IsSup score higher, while relatively lowering weights for other aspects.
Alternatively, we could further enforce hard constraints during decoding using
Critique. Instead of using a soft reward function in Eq. 4, we could
explicitly filter out a segment continuation when the model generates an
undesirable Critique token (e.g., IsSup =No support) . Balancing the trade-off
between multiple preferences has been studied in RLHF (Touvron et al., 2023;
Wu et al., 2023), which often requires training to change models’ behaviors.
Self-Rag tailors an LM with no additional training.

##  4 Experiments

###  4.1 Tasks and Datasets

We conduct evaluations of our Self-Rag and diverse baselines on a range of
downstream tasks, holistically evaluating outputs with metrics designed to
assess overall correctness, factuality, and fluency. Throughout these
experiments, we conduct zero-shot evaluations, where we provide instructions
describing tasks without few-shot demonstrations (Wei et al., 2022; Sanh et
al., 2022). Details of our experiments’ settings, including test-time
instructions, are available in the Appendix Section B.1.

Closed-set tasks include two datasets, i.e., a fact verification dataset about
public health (PubHealth; Zhang et al. 2023) and a multiple-choice reasoning
dataset created from scientific exams (ARC-Challenge; Clark et al. 2018). We
use accuracy as an evaluation metric and report on the test set. We aggregate
the answer probabilities of target classes for both of these datasets
(Appendix Section B.2).

Short-form generations tasks include two open-domain question answering (QA)
datasets, PopQA (Mallen et al., 2023) and TriviaQA-unfiltered (Joshi et al.,
2017), where systems need to answer arbitrary questions about factual
knowledge. For PopQA, we use the long-tail subset, consisting of 1,399 rare
entity queries whose monthly Wikipedia page views are less than 100. As the
TriviaQA-unfiltered (open) test set is not publicly available, we follow prior
work’s validation and test split (Min et al., 2019; Guu et al., 2020), using
11,313 test queries for evaluation. We evaluate performance based on whether
gold answers are included in the model generations instead of strictly
requiring exact matching, following Mallen et al. (2023); Schick et al.
(2023).

Long-form generation tasks include a biography generation task (Min et al.,
2023) and a long-form QA task ALCE-ASQA Gao et al. (2023); Stelmakh et al.
(2022). We use FactScore (Min et al., 2023) to evaluate biographies, and we
use official metrics of correctness (str-em), fluency based on MAUVE (Pillutla
et al., 2021), and citation precision and recall (Gao et al., 2023) for ASQA.
555https://github.com/princeton-nlp/ALCE

###  4.2 Baselines

Baselines without retrievals. We evaluate strong publicly available pre-
trained LLMs, Llama2${}_{\textsc{7b},\textsc{13b}}$ (Touvron et al., 2023),
instruction-tuned models, Alpaca${}_{\textsc{7b},\textsc{13b}}$ (Dubois et
al., 2023) (our replication based on Llama2); and models trained and
reinforced using private data, ChatGPT (Ouyang et al., 2022) and
Llama2-chat${}_{\textsc{13b}}$. For instruction-tuned LMs, we use the official
system prompt or instruction format used during training if publicly
available. We also compare our method to concurrent work,
CoVE${}_{\textsc{65b}}$ (Dhuliawala et al., 2023), which introduces iterative
prompt engineering to improve the factuality of LLM generations.

Baselines with retrievals. We evaluate models augmented with retrieval at test
time or during training. The first category includes standard RAG baselines,
where an LM (Llama2, Alpaca) generates output given the query prepended with
the top retrieved documents using the same retriever as in our system. It also
includes Llama2-FT, where Llama2 is fine-tuned on all training data we use
without the reflection tokens or retrieved passages. We also report the result
of retrieval-augmented baselines with LMs trained with private data: Ret-
ChatGPT and Ret-Llama2-chat, which deploy the same augmentation technique
above, as well as perplexity.ai, an InstructGPT-based production search
system. The second category includes concurrent methods that are trained with
retrieved text passages, i.e., SAIL (Luo et al., 2023) to instruction-tune an
LM on the Alpaca instruction-tuning data with top retrieved documents inserted
before instructions, and Toolformer (Schick et al., 2023) to pre-train an LM
with API calls (e.g., Wikipedia APIs).666We report numbers using the results
reported in the paper as the implementations are not available.

###  4.3 Experimental settings

Training data and settings. Our training data consists of diverse instruction-
following input-output pairs. In particular, we sample instances from Open-
Instruct processed data (Wang et al., 2023) and knowledge-intensive datasets
(Petroni et al., 2021; Stelmakh et al., 2022; Mihaylov et al., 2018). In
total, we use 150k instruction-output pairs. We use Llama2 7B and 13B (Touvron
et al., 2023) as our generator base LM, and we use Llama2 7B as our base
critic LM. For the retriever model $\mathcal{R}$, we use off-the-shelf
Contriever-MS MARCO (Izacard et al., 2022a) by default and retrieve up to ten
documents for each input. More training details are in the Appendix Section
B.1.

Inference settings. As a default configuration, we assign the weight terms
IsRel , IsSup , IsUse values of 1.0, 1.0 and 0.5, respectively. To encourage
frequent retrieval, we set the retrieval threshold to 0.2 for most tasks and
to 0 for ALCE (Gao et al., 2023) due to citation requirements. We speed up
inference using vllm (Kwon et al., 2023). At each segment level, we adopt a
beam width of 2. For a token-level generation, we use greedy decoding. By
default, we use the top five documents from Contriever-MS MARCO (Izacard et
al., 2022a); for biographies and open-domain QA, we use additional top five
documents retrieved by a web search engine, following Luo et al. (2023); for
ASQA, we use the author-provided top 5 documents by GTR-XXL (Ni et al., 2022)
across all baselines for a fair comparison.

##  5 Results and Analysis

Table 2:  Overall experiment results on six tasks. Bold numbers indicate the best performance among non-proprietary models, and gray-colored bold text indicates the best proprietary model when they outperforms all non-proprietary models. ∗ indicates concurrent or recent results reported by concurrent work. – indicates numbers that are not reported by the original papers or are not applicable. Models are sorted based on scale. FS, em, rg, mau, prec, rec denote FactScore (factuality); str-em, rouge (correctness); MAUVE (fluency); citation precision and recall, respectively.  | Short-form | Closed-set | Long-form generations (with citations)
---|---|---|---
| PopQA | TQA | Pub | ARC | Bio | ASQA
LM | (acc) | (acc) | (acc) | (acc) | (FS) | (em) | (rg) | (mau) | (pre) | (rec)
LMs with proprietary data
Llama2-c${}_{\textsc{13b}}$ | 20.0 | 59.3 | 49.4 | 38.4 | 55.9 | 22.4 | 29.6 | 28.6 | – | –
Ret-Llama2-c${}_{\textsc{13b}}$ | 51.8 | 59.8 | 52.1 | 37.9 | 79.9 | 32.8 | 34.8 | 43.8 | 19.8 | 36.1
ChatGPT | 29.3 | 74.3 | 70.1 | 75.3 | 71.8 | 35.3 | 36.2 | 68.8 | – | –
Ret-ChatGPT | 50.8 | 65.7 | 54.7 | 75.3 | – | 40.7 | 39.9 | 79.7 | 65.1 | 76.6
Perplexity.ai | – | – | – | – | 71.2 | – | – | – | – | –
Baselines without retrieval
Llama2${}_{\textsc{7b}}$ | 14.7 | 30.5 | 34.2 | 21.8 | 44.5 | 7.9 | 15.3 | 19.0 | – | –
Alpaca${}_{\textsc{7b}}$ | 23.6 | 54.5 | 49.8 | 45.0 | 45.8 | 18.8 | 29.4 | 61.7 | – | –
Llama2${}_{\textsc{13b}}$ | 14.7 | 38.5 | 29.4 | 29.4 | 53.4 | 7.2 | 12.4 | 16.0 | – | –
Alpaca${}_{\textsc{13b}}$ | 24.4 | 61.3 | 55.5 | 54.9 | 50.2 | 22.9 | 32.0 | 70.6 | – | –
CoVE${}_{\textsc{65b}}$ * | – | – | – | – | 71.2 | – | – | – | – | –
Baselines with retrieval
Toolformer*${}_{\textsc{6b}}$ | – | 48.8 | – | – | – | – | – | – | – | –
Llama2${}_{\textsc{7b}}$ | 38.2 | 42.5 | 30.0 | 48.0 | 78.0 | 15.2 | 22.1 | 32.0 | 2.9 | 4.0
Alpaca${}_{\textsc{7b}}$ | 46.7 | 64.1 | 40.2 | 48.0 | 76.6 | 30.9 | 33.3 | 57.9 | 5.5 | 7.2
Llama2-FT${}_{\textsc{7b}}$ | 48.7 | 57.3 | 64.3 | 65.8 | 78.2 | 31.0 | 35.8 | 51.2 | 5.0 | 7.5
SAIL*${}_{\textsc{7b}}$ | – | – | 69.2 | 48.4 | – | – | – | – | – | –
Llama2${}_{\textsc{13b}}$ | 45.7 | 47.0 | 30.2 | 26.0 | 77.5 | 16.3 | 20.5 | 24.7 | 2.3 | 3.6
Alpaca${}_{\textsc{13b}}$ | 46.1 | 66.9 | 51.1 | 57.6 | 77.7 | 34.8 | 36.7 | 56.6 | 2.0 | 3.8
Our Self-Rag ${}_{\textsc{7b}}$ | 54.9 | 66.4 | 72.4 | 67.3 | 81.2 | 30.0 | 35.7 | 74.3 | 66.9 | 67.8
Our Self-Rag ${}_{\textsc{13b}}$ | 55.8 | 69.3 | 74.5 | 73.1 | 80.2 | 31.7 | 37.0 | 71.6 | 70.3 | 71.3

###  5.1 Main Results

Comparison against baselines without retrieval.  Table 2 (top) presents the
baselines without retrieval. Our Self-Rag (bottom two rows) demonstrates a
substantial performance advantage over supervised fine-tuned LLMs in all tasks
and even outperforms ChatGPT in PubHealth, PopQA, biography generations, and
ASQA (Rouge and MAUVE). Our approach also significantly outperforms a
concurrent method that employs sophisticated prompt engineering; specifically,
on the bio generation task, our 7B and 13B models outperform the concurrent
CoVE (Dhuliawala et al., 2023), which iteratively prompts
Llama2${}_{\textsc{65b}}$ to refine output.

Comparison against baselines with retrieval. As shown in Tables 2 (bottom),
our Self-Rag also outperforms existing RAG in many tasks, obtaining the best
performance among non-proprietary LM-based models on all tasks. While our
method outperforms other baselines, on PopQA or Bio, powerful instruction-
tuned LMs with retrieval (e.g., LLama2-chat, Alpaca) show large gains from
their non-retrieval baselines. However, we found that these baselines provide
limited solutions for tasks where we cannot simply copy or extract sub-strings
of retrieved passages. On PubHealth and ARC-Challenge, baselines with
retrieval do not improve performance notably from their no-retrieval
counterparts. We also observe that most baselines with retrieval struggle to
improve citation accuracy. On ASQA, our model shows significantly higher
citation precision and recall than all models except ChatGPT. Gao et al.
(2023) found that ChatGPT consistently exhibits superior efficacy in this
particular task, surpassing smaller LMs. Our Self-Rag bridges this performance
gap, even outperforming ChatGPT in citation precision, which measures whether
the model-generated claim is fully supported by cited evidence. We also found
that on the metrics for factual precision, Self-Rag 7B occasionally
outperforms our 13B due to the tendency of smaller Self-Rag to often generate
precisely grounded yet shorter outputs. Llama2-FT${}_{\textsc{7b}}$, which is
the baseline LM trained on the same instruction-output pairs as Self-Rag
without retrieval or self-reflection and is retrieval-augmented at test time
only, lags behind Self-Rag. This result indicates Self-Rag gains are not
solely from training data and demonstrate the effectiveness of Self-Rag
framework.

| PQA | Med | AS
---|---|---|---
| (acc) | (acc) | (em)
Self-Rag (50k) | 45.5 | 73.5 | 32.1
Training
No Retriever $\mathcal{R}$ | 43.6 | 67.8 | 31.0
No Critic $\mathcal{C}$ | 42.6 | 72.0 | 18.1
Test
No retrieval | 24.7 | 73.0 | –
Hard constraints | 28.3 | 72.6 | –
Retrieve top1 | 41.8 | 73.1 | 28.6
Remove IsSup | 44.1 | 73.2 | 30.6

(a)  Ablation

(b) Customization

(c) Retrieval

(d) Analysis on Self-Rag: (a) Ablation studies for key components of Self-Rag
training and inference based on our 7B model. (b) Effects of soft weights on
ASQA citation precision and Mauve (fluency). (c) Retrieval frequency and
normalized accuracy on PubHealth and PopQA.

###  5.2 Analysis

##### Ablation studies.

We conduct a set of ablations of our framework to identify which factors play
key roles. We evaluate two model variants trained differently than our model:
No Retriever trains an LM using the standard instruction-following method
given instruction-output pairs, without retrieved passages; No Critic trains
an LM trained with input-output pairs that are always augmented with the top
one retrieved document without reflection tokens. This is similar to SAIL (Luo
et al., 2023), and we use our instruction-output data instead of using the
Alpaca dataset (Dubois et al., 2023), as in SAIL. We also conduct ablation on
our inference-time algorithm, including No retrieval disables retrieval during
inference; Hard constraints indicates the model performance that retrieves
when Retrieve=Yes instead of using the adaptive threshold; Retrieve top 1
always retrieves and uses the top one document only, similar to standard RAG
approaches; Remove IsSup indicates the model performance that removes IsSup
score only during critique-guided beam search in Eq. 4. In this ablation
experiment, we use a training instance size of 50k for a more efficient
exploration of training variations. Later in this section, we conduct an
analysis of the effect of training data size. We conduct the ablation studies
on three datasets, PopQA, PubHealth, and ASQA. On ASQA, we evaluate models on
sampled 150 instances and exclude ablations involving adaptive or no retrieval
processes.

We show in Table 5.1 the ablation results. The top part of the table shows
results for training ablations, and the bottom part is for inference
ablations. We see that all components play important roles. We also observe a
large performance gap between Self-Rag and No Retriever or Critic baselines
across tasks, indicating that training an LM with those models largely
contributes to the performance gain of Self-Rag. Using the top passages
regardless of their relevance (Retrieve top 1) as in conventional RAG
approaches causes a large drop in PopQA and ASQA, and removing IsSup during
the beam search results hurts performance on ASQA. This demonstrates the
effectiveness of Self-Rag’s capabilities of carefully selecting generations
based fine-grained multiple criterion, instead of naively using all of the top
passages from the retrieval model or solely depending on relevance scores.

Effects of inference-time customization. One key benefit of our proposed
framework is that it enables us to control how much each critique type affects
the final generation sampling. We analyze the effects of different parameter
weights on the top of our 7B model during inference time on ASQA, where
multiple evaluation aspects are considered. Figure 3(b) shows the effects of
changing the weighting term for IsSup , which criticizes how supported the
output is by the text passage. As the figure shows, increasing the weight
leads to positive effects on the models’ citation precision since this puts
more emphasis on whether model generation is supported by the evidence. On the
contrary, a larger weight results in lower MAUVE scores: when generation gets
longer and more fluent, there are often more claims that are not fully
supported by citations, consistent with findings by Liu et al. (2023a). Our
framework lets practitioners choose and customize models’ behaviors at test
time by adjusting such parameters without requiring additional training.

Efficiency and accuracy trade-off. Using our framework, practitioners can
adjust how often retrieval occurs using the token probability of reward
tokens. We evaluate how this adaptive threshold affects overall accuracy and
frequency of retrieval, and we evaluate the performance with varying numbers
of threshold $\delta$ (larger $\delta$ results in less retrieval) on PubHealth
and PopQA. Figure 3(c) shows that the model’s retrieval frequencies
dramatically change on both datasets. as $\delta$ varies. On one hand,
performance deterioration by retrieving less is smaller on PubHealth but
larger in PopQA.

Effects of training data size. We conduct an analysis of how the data scale
affects the model’s performance. In particular, we randomly sample 5k, 10k,
20k, and 50k instances from our original 150k training instances, and fine-
tune four Self-Rag ${}_{\textsc{7b}}$ variants on those subsets. Then, we
compare the model performance on PopQA, PubHealth, and ASQA (citation
precision) with our final Self-Rag trained on the full 150k instances. We also
evaluate Figures 3(e), 3(f) and 3(g) shows the models’ performance trained on
different amount of data. Across all datasets, increasing data size often
shows upward trajectories and the improvements are significantly larger in
PopQA and ASQA, while we do not observed such significant improvements on
Llama2-FT${}_{\textsc{7b}}$ when increasing the training data from 50k to
150k. These results also indicate that further expanding the training data of
Self-Rag may lead to further improvements, although in this work we limit our
training data size to 150k.

Human evaluations. We conduct small human evaluations on Self-Rag outputs, as
well as the reliability of predicted reflection tokens. In particular, we
sampled 50 samples from PopQA and Bio results. Following Menick et al. (2022),
human annotators evaluate S&P, which indicates whether the model output is
plausible (i.e., the output is a reasonable and on-topic response to the
question as if it were occurring in a conversation) and supported (i.e., the
provided evidence is sufficient to verify the validity of the answer). For
S&P, we do not consider the instances where Self-Rag predicts irrelevant or no
support. We then ask our annotators whether the model-predicted reflection
tokens about IsRel and IsSup match their inspections (e.g., whether the fully
supported output is supported by the cited evidence). Human annotators find
Self-Rag answers are often plausible and supported by relevant passages with
higher S&P scores on short-form PopQA, which is consistent with Menick et al.
(2022). Human annotators also find IsRel and IsSup reflection token
predictions are mostly aligned with their assessments. Appendix Table 6 shows
several annotated examples and explanations on assessments.

(e) PopQA

(f) PubHealth

(g) ASQA (prec)

| Pop | Bio.
---|---|---
S & P | 92.5 | 70.0
IsRel | 95.0 | 90.0
IsSup | 90.0 | 85.0
(h) Human evaluation on PopQA and Bio generation.

Figure 3: Training scale and Human analysis: (a) (b) (c) Training scale
analysis shows the effect of the training data scale on PopQA, PubHealth and
ASQA (citation precision), respectively. (d) Human analysis on Self-Rag
outputs as well as reflection tokens.

##  6 Conclusion

This work introduces Self-Rag, a new framework to enhance the quality and
factuality of LLMs through retrieval on demand and self-reflection. Self-Rag
trains an LM to learn to retrieve, generate, and critique text passages and
its own generation by predicting the next tokens from its original vocabulary
as well as newly added special tokens, called reflection tokens. Self-Rag
further enables the tailoring of LM behaviors at test time by leveraging
reflection tokens. Our holistic evaluations on six tasks using multiple
metrics demonstrate that Self-Rag significantly outperforms LLMs with more
parameters or with conventional retrieval-augmented generation approaches.

## Ethical Concerns

This work aims to improve the factuality of LLM outputs, the lack of which
continues to cause numerous real-world problems (e.g., spread of
misinformation and provision of incorrect and dangerous advice). While our
method shows significant improvements in terms of performance, factuality, and
citation accuracy, it can still generate outputs that are not fully supported
by the citations. We hope that explicit self-reflection and fine-grained
attribution may help users verify factual errors in the model outputs.

#### Acknowledgments

We thank Sewon Min, Scott Wen-tau Yih, Sean Welleck, and Kawin Ethayarajh for
fruitful discussions in the early stages of this work. We thank Sewon Min,
Joongwon (Daniel) Kim, and Sandy Kaplan for valuable feedback on the paper,
and Tianyu Gao and Weijia Shi for their help on evaluations. Akari Asai is
supported by the IBM Fellowship. We thank Stability AI for providing computing
to train and evaluate the LMs in this work, and Microsoft Accelerate
Foundation Models Research Program for the access to OpenAI APIs. This work
was funded in part by the DARPA MCS program through NIWC Pacific
(N66001-19-2-4031), NSF IIS-2044660, and gifts from AI2.

## References

  * Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SJgVHkrYDH.
  * Asai et al. (2023a) Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen.  Retrieval-based language models and applications.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Tutorial)_ , 2023a.  URL https://aclanthology.org/2023.acl-tutorials.6.
  * Asai et al. (2023b) Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih.  Task-aware retrieval with instructions.  In _Findings of the Association for Computational Linguistics_ , 2023b.  URL https://aclanthology.org/2023.findings-acl.225.
  * Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al.  Attributed question answering: Evaluation and modeling for attributed large language models.  _arXiv preprint arXiv:2212.08037_ , 2022.  URL https://arxiv.org/abs/2212.08037.
  * Chen et al. (2023) Lingjiao Chen, Matei Zaharia, and James Zou.  How is chatgpt’s behavior changing over time?  _arXiv preprint arXiv:2307.09009_ , 2023.  URL https://arxiv.org/abs/2307.09009.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.  URL https://arxiv.org/abs/1803.05457.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  In _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=H4DqfPSibmx.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston.  Chain-of-verification reduces hallucination in large language models.  _arXiv preprint arXiv:2309.11495_ , 2023.  URL https://arxiv.org/abs/2309.11495.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=r1l73iRqKm.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint arXiv:2305.14387_ , 2023.  URL https://arxiv.org/abs/2305.14387.
  * Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen.  Enabling large language models to generate text with citations.  _arXiv preprint arXiv:2305.14627_ , 2023.  URL https://arxiv.org/abs/2305.14627.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_ , 2020.  URL https://dl.acm.org/doi/pdf/10.5555/3524938.3525306.
  * Izacard et al. (2022a) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_ , 2022a.  URL https://openreview.net/forum?id=jKN1pXi7b0.
  * Izacard et al. (2022b) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_ , 2022b.  URL https://arxiv.org/abs/2208.03299.
  * Jiang et al. (2023) Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig.  Active retrieval augmented generation.  _arXiv preprint arXiv:2305.06983_ , 2023.  URL https://arxiv.org/abs/2305.06983.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017.  URL https://aclanthology.org/P17-1147.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_ , 2019.  URL https://arxiv.org/abs/1909.05858.
  * Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez.  Pretraining language models with human preferences.  In _International Conference on Machine Learning_ , 2023.  URL https://openreview.net/forum?id=AT8Iw8KOeC.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 2019\.  URL https://aclanthology.org/Q19-1026.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_ , 2023.  URL https://arxiv.org/abs/2309.06180.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.
  * Lin et al. (2023) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih.  Ra-dit: Retrieval-augmented dual instruction tuning, 2023.  URL https://arxiv.org/abs/2310.01352.
  * Liu et al. (2023a) Nelson F Liu, Tianyi Zhang, and Percy Liang.  Evaluating verifiability in generative search engines.  _arXiv preprint arXiv:2304.09848_ , 2023a.  URL https://arxiv.org/abs/2304.09848.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ , 2023b.  URL https://arxiv.org/abs/2303.16634.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi.  QUARK: Controllable text generation with reinforced unlearning.  In _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=5HaIds3ux5O.
  * Luo et al. (2023) Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass.  Sail: Search-augmented instruction learning.  _arXiv preprint arXiv:2305.15225_ , 2023.  URL https://arxiv.org/abs/2305.15225.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.  URL https://arxiv.org/abs/2303.17651.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2023.  URL https://aclanthology.org/2023.acl-long.546.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_ , 2022.  URL https://arxiv.org/abs/2203.11147.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018.  URL https://aclanthology.org/D18-1260.
  * Min et al. (2019) Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer.  A discrete hard EM approach for weakly supervised question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019.  URL https://aclanthology.org/D19-1284.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  _arXiv preprint arXiv:2305.14251_ , 2023.  URL https://arxiv.org/abs/2305.14251.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.  URL https://arxiv.org/abs/2112.09332.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022.  URL https://aclanthology.org/2022.emnlp-main.669.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.  URL https://arxiv.org/abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=TG8KACxEON.
  * Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings.  Refiner: Reasoning feedback on intermediate representations.  _arXiv preprint arXiv:2304.01904_ , 2023.  URL https://arxiv.org/abs/2304.01904.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel.  KILT: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021.  URL https://aclanthology.org/2021.naacl-main.200.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui.  MAUVE: Measuring the gap between neural text and human text using divergence frontiers.  In _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=Tqx7nJp7PR.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2020.  URL https://dl.acm.org/doi/10.5555/3433701.3433727.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 2023\.  URL https://arxiv.org/abs/2302.00083.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=9Vrb9D0WI4.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.  URL https://arxiv.org/abs/2302.04761.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.  URL https://arxiv.org/abs/1707.06347.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou.  Large language models can be easily distracted by irrelevant context.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.  URL https://proceedings.mlr.press/v202/shi23a.html.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang.  ASQA: Factoid questions meet long-form answers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022.  URL https://aclanthology.org/2022.emnlp-main.566.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018.  URL https://aclanthology.org/N18-1074.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.  URL https://arxiv.org/abs/2307.09288.
  * Wang et al. (2023) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al.  How far can camels go? exploring the state of instruction tuning on open resources.  _arXiv preprint arXiv:2306.04751_ , 2023.  URL https://arxiv.org/abs/2306.04751.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Wu et al. (2023) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  _arXiv preprint arXiv:2306.01693_ , 2023.  URL https://arxiv.org/abs/2306.01693.
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie.  Decomposition enhances reasoning via self-evaluation guided decoding.  _arXiv preprint arXiv:2305.00633_ , 2023.  URL https://arxiv.org/abs/2305.00633.
  * Xu et al. (2023) Fangyuan Xu, Weijia Shi, and Eunsol Choi.  Recomp: Improving retrieval-augmented lms with compression and selective augmentation, 2023.  URL https://arxiv.org/abs/2310.04408.
  * Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant.  Making retrieval-augmented language models robust to irrelevant context, 2023.  URL https://arxiv.org/abs/2310.01558.
  * Yue et al. (2023) Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun.  Automatic evaluation of attribution by large language models.  _arXiv preprint arXiv:2305.06311_ , 2023.  URL https://arxiv.org/abs/2305.06311.
  * Zhang et al. (2023) Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass.  Interpretable unified language checking.  _arXiv preprint arXiv:2304.03728_ , 2023.  URL https://arxiv.org/abs/2304.03728.
  * Zhou et al. (2023) Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang.  Language agent tree search unifies reasoning acting and planning in language models, 2023.  URL https://arxiv.org/abs/2310.04406.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.  URL https://arxiv.org/abs/1909.08593.

## Appendix

[sections] [sections]l1

##  Appendix A Self-Rag Details

###  A.1 Reflection Tokens.

##### Definitions of reflection tokens.

Below, we provide a detailed definition of reflection type and output tokens.
The first three aspects will be provided at each segment level, while the
final aspect is only given at each output level.

  * •

Retrieval-on-demand ( Retrieve): Given an input and previous-step generation
(if applicable), an LM determines whether the continuation requires factual
grounding. No indicates retrieval is unnecessary as the sequence does not
require factual grounding or may not be enhanced by knowledge retrieval, Yes
indicates retrieval is necessary. We additionally have continue to use
evidence, which indicates that a model can continue to use the evidence
retrieved previously. For instance, a passage may contain rich factual
information, and thus Self-Rag generates multiple segments based on the
passage.

  * •

Relevant ( IsRel ): Retrieved knowledge may not be always relevant to the
input. This aspect indicates whether the evidence provides useful information
(Relevant) or not (Irrelevant).

  * •

Supported ( IsSup ): Attribution is the concept of whether the output is fully
supported by certain evidence (Menick et al., 2022; Bohnet et al., 2022). This
aspect judges how much information in the output is entailed by the evidence.
We evaluate attributions in three scale, Fully supported, Partially supported,
and No support / Contradictory, following Yue et al. (2023); Nakano et al.
(2021).

  * •

Useful ( IsUse ): Following the definitions from Liu et al. (2023a), we define
the perceived utility as whether the response is a helpful and informative
answer to the query, independently from whether it is in fact factual or not.
This can be also viewed as plausibility in Menick et al. (2022). For
usefulness, we use a five-scale evaluation (1 is the lowest and 5 is the
highest).

##### Details of GPT-4-based data collections.

We use the instruction and demonstration pairs to prompt GPT-4, listed in
Section D. Following an official recommendation, we separate instructions and
outputs with “##”. We use the temperature 1 and set the maximum output token
counts to be 200. We discard instances where GPT-4 does not follow the
designated output formats or output sequences that do not match our expected
category names. As a result, we collected 1,2594 for Retrieve, 11,181 for
IsSup , 19,317 for relevance, 3,831 for utility.

##### Manual analysis of the GPT-4 predictions.

The authors of this paper manually assess randomly sampled 20 instances for
each aspect and check if GPT-4 predictions match their assessments given the
same instruction, demonstrations, and test instances. We found our assessments
show high agreement with GPT-4 predictions, especially for relevance (95%),
retrieval necessity (95%), and the degree of support (90%). Agreement was
slightly lower in usefulness (80%), mostly due to the disagreement between 1
and 2 or 4 and 5.

###  A.2 Self-Rag Training

##### Overview of training.

Algorithm 2 provides a high-level overview of our training.

Algorithm 2 Self-Rag Training

1:Input input-output data $\mathcal{D}=\\{X,Y\\}$, generator $\mathcal{M}$,
$\mathcal{C}$ $\theta$

2:Initialize $\mathcal{C}$ with a pre-trained LM

3:Sample data $\\{X^{sample},Y^{sample}\\}\sim\\{X,Y\\}$ $\triangleright$
Training Critic LM (Section 3.2.1)

4:for $(x,y)\in(X^{sample},Y^{sample})$ do $\triangleright$ Data collections
for $\mathcal{C}$

5: Prompt GPT-4 to collect a reflection token $r$ for $(x,y)$

6: Add $\\{(x,y,r)\\}$ to $\mathcal{D}_{critic}$

7:Update $\mathcal{C}$ with next token prediction loss $\triangleright$ Critic
learning; Eq. 1

8:Initialize $\mathcal{M}$ with a pre-trained LM $\triangleright$ Training
Generator LM (Section 3.2.2)

9:for $(x,y)\in(X,Y)$ do $\triangleright$ Data collection for $\mathcal{M}$
with $\mathcal{D}_{critic}$

10: Run $\mathcal{C}$ to predict $r$ given $(x,y)$

11: Add $(x,y,r)$ to $\mathcal{D}_{gen}$

12:Update $\mathcal{M}$ on $\mathcal{D}_{gen}$ with next token prediction loss
$\triangleright$ Generator LM learning; Eq. 2

##### Full list of seed datasets.

To sample diverse input-output pairs, we sample instances of the Open-Instruct
(Wang et al., 2023) dataset. In particular, we use their ShareGPT, GPT-4
Alpaca, Alpaca, OpenAssistant, and FLAN subsets subsets. We also sample
instances from a couple of knowledge-intensive datasets, Natural Questions
(Kwiatkowski et al., 2019), Wizard of Wikipedia (Dinan et al., 2019) and FEVER
(Thorne et al., 2018) from the KILT benchmark (Petroni et al., 2021), ASQA
(Stelmakh et al., 2022) and multiple QA datasets including ARC-Easy and
OpenBookQA (Mihaylov et al., 2018). Table 3 shows the full list of training
instances, and in total, we use 145,619 instances.

Dataset name | category | Data source | the number of instances
---|---|---|---
GPT-4 Alpaca | Instruction-following | Open-Instruct | 26,168
Stanford Alpaca | Instruction-following | Open-Instruct | 25,153
FLAN-V2 | Instruction-following | Open-Instruct | 17,817
ShareGPT | Instruction-following | Open-Instruct | 13,406
Open Assistant 1 | Instruction-following | Open-Instruct | 9,464
Wizard of Wikipedia | Knowledge-intensive | KILT | 17,367
Natural Questions | Knowledge-intensive | KILT | 15,535
FEVER | Knowledge-intensive | KILT | 9,966
OpenBoookQA | Knowledge-intensive | HF Dataset | 4,699
Arc-Easy | Knowledge-intensive | HF Dataset | 2,147
ASQA | Knowledge-intensive | ASQA | 3,897
Table 3: The generator LM $\mathcal{M}$ training data statistics.

##### Performance of the Critic $\mathcal{C}$.

We evaluate the accuracy of reward predictions by splitting GPT-4 generated
feedback into training, development, and test sets. The accuracy of the reward
model is as follows. Table 4 shows the model performance of predicting GPT-4
judgments. As you can see, overall our fine-tuned reward model shows high
prediction matching with GPT-4 predicted feedback. While our final model uses
Llama2-7B as a base LM, we also train and compare FLAN-3B (Wei et al., 2022)
model on the same data, to investigate the effectiveness of different data
sizes affect final reward predictions. In most aspects, our reward model shows
higher than 80% accuracy, indicating the powerful ability of fine-tuned
specialized LMs to evaluate text. While both models show relatively lower
performance on IsUse , this is because both models often confuse between the
two highest cases (5 and 4), where human annotators can also disagree.

base LM | Retrieve | IsSup | IsRel | IsUse
---|---|---|---|---
Llama2-7B | 93.8 | 93.5 | 80.2 | 73.5
FLAN-3B | 85.6 | 73.1 | 82.0 | 72.1
Figure 4: Reward prediction accuracy using GPT-4 predictions as ground-truth
predictions.

##### Details of $\mathcal{M}$ data creation.

Here, we provide detailed data creation procedures. Algorithm 3 summarizes the
process. Here we set $y_{t}$ to $y$ for simplification. Once we train the
critic model, we first run it on input data from the aforementioned datasets,
to predict whether retrieval is needed or not. For the instances where the
critic predicts Retrieve=No, we only predict the IsUse given input and output.
For the instances where the critic predicts Retrieve=Yes, we first retrieve
passages using the input and the entire output as queries, to find passages
that are relevant to the entire output. We then split output sentences using
Spacy.777https://spacy.io/ For each sentence, we run $\mathcal{C}$ to predict
whether the retrieval is necessary or not, given the input, preceding
segments, and the initial retrieved passage. If $\mathcal{C}$ predicts
Retrieve=No, then do not insert any paragraph at the $t$th segment. If
$\mathcal{C}$ predicts Retrieve=Yes, then we use the original input and the
$t$th segment as a retrieval query to find relevant passages for the $t$-th
segment. For each retrieved passage, we predict IsRel and IsSup . If there is
any passage and continuation with IsRel =Relevant and IsSup =Fully Supported /
IsSup =Partially Supported, then we sample it as the continuation. If there is
more than one passage satisfying this criterion, we use the one with the
highest retrieval score. If there are only IsRel =Irrelevant or IsSup =No
Support passages, we randomly sample one passage.

Algorithm 3 $\mathcal{M}_{gen}$ Data creation

1:Input Input-output data $\mathcal{D}$ = ${X,Y}$

2:for $(x,y)\in\\{X,Y\\}$ do

3: Given $(x,y)$ $\mathcal{C}$ predicts Retrieve

4: if Retrieve is predicted then

5: Retrieve relevant passages $\mathbf{D}$ using $\mathcal{R}$ given $(x,y)$
$\triangleright$ Retrieve passages

6: for $d\in\mathbf{D}$ do

7: $\mathcal{C}$ predicts IsRel for each $d$ $\triangleright$ Predict
relevance of passages

8: $\mathcal{C}$ predicts IsSup for each $(y,d)$ $\triangleright$ Predict
supports of outputs

9: $\mathcal{C}$ predicts IsUse for each $d$ $\triangleright$ Predict overall
utility ($t=\mathcal{T}$ only)

10: Sample $d$

11: else if Retrieve is not predicted then

12: $\mathcal{C}$ predicts IsUse given $x,y$ Add augmented $(x,y,d,r)$ to
$\mathcal{D}_{gen}$

##### Training examples.

Table 4 show several training examples used for $\mathcal{M}$ training.

Input Identify the two types of computer memory. (from gpt4_alpaca)
---
Output [ Retrieve=Yes]<p>Computer memory Computer memory – In computing,
memory refers to the computer hardware integrated circuits that store
information for immediate use in a computer; it is synonymous with the term
“primary storage”. Computer memory operates at a high speed, for example
random-access memory (RAM), as a distinction from storage that provides slow-
to-access information but offers higher capacities. If needed, contents of the
computer memory can be transferred to secondary storage; a very common way of
doing this is through a memory management technique called “virtual memory”.
An archaic synonym for memory is store. The term “memory”, meaning “primary
storage” or “main memory”, is<p>[ IsRel =Relevant]Primary memory, also known
as main memory, is the memory that the computer can directly access and
manipulate. [ IsSup =Fully Supported] [ Retrieve=Yes]<p>Computer memory–often
associated with addressable semiconductor memory, i.e. integrated circuits
consisting of silicon-based transistors, used for example as primary storage
but also other purposes in computers and other digital electronic devices.
There are two main kinds of semiconductor memory, volatile and non-volatile.
Examples of non-volatile memory are flash memory (used as secondary memory)
and ROM, PROM, EPROM and EEPROM memory (used for storing firmware such as
BIOS). Examples of volatile memory are primary storage, which is typically
dynamic random-access memory (DRAM), and fast CPU cache memory, which is
typically static random-access memory (SRAM) that is fast but energy-
consuming, offering lower memory<p>[ IsRel =Relevant]Primary memory can be
accessed quickly, but it is volatile, meaning that the data is lost if the
power is turned off.[ IsSup =Partially Supported]…[ IsUse :5]
Input Give an example of a situation where you have to delegate tasks (from
gpt4_alpaca)
Output [ Retrieve=No]The manager realizes that there is an extensive list of