than to the unacceptable sequence.

##### Datasets and selection process.

Minimal pair evaluation of language models was pioneered by Linzen et al.
(2016), leading to several works that amassed minimal pairs to test linguistic
understanding (Linzen et al., 2016; Warstadt et al., 2020; Gauthier et al.,
2020, inter alia). Of these, we elect to use BLiMP (Benchmark for Linguistic
Minimal Pairs; Warstadt et al., 2020), which contains minimal pairs testing
syntactic, morphological, and semantic knowledge. Drawing from syntax
textbooks in linguistics to converge on a set of core phenomena, it covers 12
linguistic phenomena and 67 paradigms where 1000 synthetic minimal pairs were
programmatically generated for each paradigm.

###  5.2 Knowledge

To measure a model’s knowledge, we evaluate the model through question
answering (§5.2.1: knowledge-qa) and text completion (§5.2.2: knowledge-
factCompletion).494949To be more precise, since we are evaluating models as
text-to-text interfaces, this means knowledge that is currently used by the
interface. In other words, the knowledge need not be stored in the model and
could, for example, be stored in an external knowledge base that is accessed
by a retrieval-augmented model (Lewis et al., 2020c; Yasunaga et al., 2021;
2022a). Question answering enables us to re-use questions developed for
assessing human knowledge (e.g. academic exams) that involve light linguistic
understanding and reasoning, whereas text completion allows us to further
isolate specific factual knowledge.

####  5.2.1 Knowledge-intensive QA

To evaluate LMs’ knowledge in a practical QA setting, we use existing QA
benchmarks that require significant knowledge to solve.

##### Datasets.

Among the QA datasets discussed in §3.3: questionAnswering, we focus on a
subset that tests diverse forms of knowledge: HellaSwag, OpenBookQA,
TruthfulQA, and MMLU. Of these, HellaSwag and OpenBookQA test general
commonsense knowledge, with TruthfulQA further adding emphasis on the
factuality and truthfulness of knowledge. In contrast, MMLU tests specialized
knowledge from 57 domains, ranging from the humanities to the social sciences
to STEM (science, technology, engineering and math).

####  5.2.2 Fact completion

Here we aim to evaluate LMs’ knowledge in a setting that factors out
knowledge-agnostic capabilities like language understanding/reasoning, and we
use simple prompts to test single facts. Specifically, we construct a new
dataset of such prompts based on facts from Wikidata, and evaluate LMs on it.

##### Problem setting.

In fact completion, given a prompt that asks for a fact (e.g., “The capital of
France is __”), the task is to predict the completion of the prompt (“Paris”).
This task is a natural language equivalent of the classical triple completion
task studied for relational data (Bordes et al., 2013). Given an incomplete
entity triple consisting of (subject, predicate, ?), the goal is to predict
the missing object entity. Our evaluation metric is 5-shot Accuracy@$K$
($K$=1, 5), where Accuracy@$K$ indicates whether one of the system’s top $K$
predictions matches the ground-truth label.

##### Datasets.

Our fact completion setup described above was inspired by the LAMA probe
dataset (Petroni et al., 2019). LAMA was an early work that established the
method to probe LMs’ knowledge through a fact completon prompt like “The
capital of France is __”. However, the original LAMA dataset only covered
$\sim$40 types of generic relational knowledge (e.g., place of birth). Here we
curate a more diverse dataset to evaluate LMs.

Specifically, we identified 12 domains spanning the humanities (e.g., law),
social sciences (e.g., economics), STEM (e.g., biomedicine), and other general
facts. For each domain, we manually identified 2-7 Wikidata relations and
engineered a prompt template which converted the triple into a natural-
language-completion task. We picked relations which captured factual
information highly relevant to and representative of the domain. This results
in 86 relation types in total. The full list of relations and their
corresponding prompts is available in Appendix E.2. We then downloaded all
triples corresponding to these relations (using the January 2022 Wikidata
dump). We removed triples where either the subject or object entity did not
have a Wikipedia page. We sampled 1000 triples for each property to use as our
benchmark. We observed that when cast as a natural language completion, a
single triple may have multiple correct answers, as (1) a triple may be
correctly completed by multiple different objects, or (2) a single object may
be referenced by multiple different names. In evaluating models, we consider
any generation corresponding to any alias of a permissible object entity to be
correct.

###  5.3 Reasoning

To measure a model’s reasoning capabilities, we evaluate it in both synthetic
and realistic reasoning-centric scenarios. We specify a suite of synthetic
tasks that probe core reasoning primitives (non-ampliative reasoning,
ampliative reasoning, recursive hierarchy, state tracking; §5.3.1: reasoning-
primitives), largely isolating reasoning from language and knowledge. To build
on these primitives, we then test models in realistic contexts (mathematical
reasoning, code reasoning, legal reasoning, logical reasoning, structured data
reasoning; §5.3.2: realistic-reasoning) that bring together these primitives.
Put together, these approaches clarify the extent to which models possess
capacities crucial for reasoning and how this is of service in real use cases
that depend on reasoning.

####  5.3.1 Reasoning primitives

While reasoning is usually assumed to involve transitions in thought (Harman,
2013), possibly in some non-linguistic format, we typically assess reasoning
abilities (e.g., in adult humans) by means of explicit symbolic or linguistic
tasks. Indeed, communication and argument may even be what reasoning is
ultimately for (Mercier & Sperber, 2017). In order to distinguish reasoning
from language and knowledge as much as possible, we focus here on relatively
abstract capacities necessary for sophisticated text-based or symbolic
reasoning divided into four categories: non-ampliative reasoning, ampliative
reasoning, reasoning with recursive hierarchy, and state tracking. The first
division follows Peirce (1974): non-ampliative reasoning involves conclusions
that are in some sense already present in the premises, whereas ampliative
reasoning involve conclusions that are only guaranteed if we accept further
assumptions that were not explicitly given.505050As a paradigmatic instance of
non-ampliative reasoning, we investigate _deductive_ patterns in which the
conclusion could not be false as long as the premises are true. For ampliative
reasoning we investigate (what are often called) _induction_ and _abduction_ ,
which characteristically require tacit (e.g., statistical or causal)
assumptions to justify a conclusion. The last two evaluate how generally the
model can reason both recursively and over changing states.

These tasks involve only elementary language — in some cases a mix of natural
language and mathematical symbols — and little in the way of factual world
knowledge. Our assumption is that these abilities will be requisite for
reasoning independent of what facts happen to hold, or what (natural or
artificial) language the model happens to be using. We emphasize these
evaluations are intended to be representative but not exhaustive.

##### Non-ampliative Reasoning

We first test two basic primitives for non-ampliative reasoning: pattern
matching to identify a relevant rule and variable substitution to apply the
rule. To instantiate these primitives, we implement scenarios with abstract
symbols following LIME (Wu et al., 2021). We further follow Clark et al.
(2020) to test deduction that combines these primitives in natural language
with simple words for variables and sentence templates for rules.

##### Ampliative Reasoning

To measure ampliative reasoning, we use explicit rule induction and implicit
function regression, which corresponds to making and applying claims about the
likely causal structure for observations. For rule induction, we design and
implement rule_induct inspired by the LIME induction tasks, where we provide
two examples generated from the same rule string, and task the model with
inferring the underlying rule. For function regression, we design and
implement numeracy_prediction, which requires the model to perform symbolic
regression given a few examples and apply the number relationship (e.g.
linear) to a new input.

##### Recursive Hierarchy.

To test the ability of language models to reason recursively over deep and
long hierarchical dependencies, we use the generalized Dyck languages (_Dyck_
, or $\mathcal{D}_{n}$). Dyck is the family of languages which embodies what
it means to be a language with hierarchical structure (Suzgun et al., 2019b).
Although such structures are believed to be essential to natural language,
similar nested hierarchies also arise in other core reasoning domains
(Dehaene, 2020) and involve a particularly subtle variety of pattern matching.

To instantiate a task with Dyck, we require the model to generate the sequence
of the closing brackets of a $\mathcal{D}_{n}$ word without its last few
closing brackets.515151A model capable of recognizing Dyck needs to emulate a
discrete pushdown automaton (PDA). By virtue of the task formulation, every
input example (i.e. $\mathcal{D}_{n}$ prefix) has a unique sequence of closing
brackets. In our experiments, we limit our attention to a subset of the Dyck
languages, $\mathcal{D}_{3}$, of well-nested strings over three pairs of
brackets (viz., {“(”, “)”, “[”, “]”, “{”, “}”}) and consider $500$ evaluation
examples of varying lengths (between $52$ and $100$) under a two-shot
prompting protocol. To measure model accuracy, we report strict exact-match
results.

##### State Tracking.

To further evaluate the reasoning and state tracking capabilities of models,
we measure their performance on bAbI (Weston et al., 2015). bAbI features
short stories about characters picking and dropping items while traversing
rooms in a house. Each question has a single-word answer, and they require a
variety of reasoning skills, spanning transitive inference, coreference
resolution, logical reasoning (negation, conjunctions and disjunctions),
spatial and temporal reasoning, deduction and induction, grouped into 20
different tasks. The inputs have 64 tokens per instance on average, with
significant variance in input lengths, with some reaching around hundreds of
tokens.

####  5.3.2 Realistic reasoning

We also evaluate language models on more complex and realistic reasoning tasks
that require multiple primitive reasoning skills. These evaluations bridge the
gap between understanding reasoning in very controlled and synthetic
conditions and the type of reasoning required in practical contexts. In
particular, they also help surface how the abstract property of reasoning,
while decomposed into similar primitives across domains, takes on different
textures based on the domain (e.g. the reasoning required in legal contexts is
quite different from the reasoning required in code contexts). Further, these
realistic scenarios are central to reasoning: in-the-wild reasoning requires
the ability to compose primitives in a variety of ways. To operationalize this
measurement, we choose a diverse set of reasoning-intensive scenarios with
clear ground truth, so that automated evaluation at scale is possible.

##### Mathematical Reasoning.

We use GSM8K (Cobbe et al., 2020) and MATH (Hendrycks et al., 2021c) to test
model performance in the context of math exams of varying difficulties. Both
datasets evaluate multi-step mathematical reasoning capabilities necessary to
arrive at a numerical answer given a question. In their ground truth examples,
the datasets include intermediate steps in natural language which lead to the
final answer, which the language model then imitates.

##### Code Synthesis.

We use HumanEval (Chen et al., 2021) and APPS (Hendrycks et al., 2021a), which
contain 164 and 10,000 coding questions respectively. HumanEval contains
handwritten programming problems that tests simple algorithms and mathematics,
while APPS is curated from open-access programming sites and covers a wider
range of difficulties. Since each full coding task example could be too long
to fit multiple of them in the context window, we only perform zero-shot
evaluation with models fine-tuned on code already.

##### Legal Reasoning.

For legal reasoning, we construct LegalSupport, a novel comparative legal
entailment task. Lawyers often must determine which case most forcefully or
accurately supports their claims. This is integral part of legal argument
formulation, as lawyers must cite to the decisions made by previous courts
(i.e. “precedent”). In particular, this allows one to (1) determine what the
law says, and (2) persuasively argue for how a court should decide in a
pending dispute. In LegalSupport, a model is provided with an argument, and
the legal conclusions of two court cases. The task is to determine which case
most persuasively supports the argument. The arguments and annotations are
mined from actual legal opinions.

##### Logical Reasoning.

We perform further evaluation on analytical reasoning questions from the Law
School Admission Test (LSAT; Zhong et al., 2021), a standardized test for
prospective law school candidates, which are verbal versions of constraint
satisfaction problems (CSP) about assignments, groupings or orderings of
elements in accordance with a list of requirements, like assigning several
people to tables, or organizing classes in a schedule. The questions are of
multi-answer format with 5 answers per question, where they either ask for the
specific solution to the CSP, the cardinality of the solution space, or other
derived properties and inferences about a given problem. The prompts have 170
tokens per example on average, and we provide 5 such examples in-context
before asking the model to predict the answer to a new exercise.

##### Structured Data Reasoning.

We evaluate how well models can wrangle structured data—a critical problem in
enterprise data pipelines that has been studied in the data management
community for over two decades (Golshan et al., 2017). We evaluate on two
classic data integration and cleaning tasks: entity matching and data
imputation. Entity matching is the task of determining if two structured rows
(often from different relations) refer to the same entity. Data imputation is
the task of filling in a missing cell value from a structured row. For entity
matching, we use the Beer, Abt-Buy, and iTunes Amazon (dirty) datasets from
the Magellan benchmark (Konda et al., 2016). For imputation, we choose the
Restaurant and Buy datasets from Mei et al. (2021). For the tasks, we generate
prompts following the random prompts from Narayan et al. (2022).

###  5.4 Memorization & copyright

Recent works have demonstrated that language models are capable of memorizing
and reproducing content in training datasets (Carlini et al., 2019; Lee et
al., 2022a; Carlini et al., 2022; Kandpal et al., 2022; Carlini et al., 2021).
The ability to memorize and plagiarize suggests potential legal risks from an
intellectual property standpoint (Bommasani et al., 2021, §5.4). In this
section, we provide an evaluation of this risk through measuring the extent to
which language models are able to reproduce copyrighted/licensed material in
their training corpus with common sampling methods used in prior works
(Carlini et al., 2021).

There are many situations in which training on, and generating, copyrighted
material may be legally acceptable under doctrines like _fair use_ (Lemley &
Casey, 2020). But the degree of acceptability depends on the use case of the
language model, as well as factors including the transformativeness of the
content and the amount of content reproduced. We refer the reader to other
works discussing the legal subject matter (Sobel, 2017; Lemley & Casey, 2020;
Burk, 2019; Gillotte, 2019; Franceschelli & Musolesi, 2022). Fair use is not a
panacea, though. In the case of ComicMix (Index, 2020), creators wrote a book
based on Dr. Seuss’s _Oh, the Places You’ll Go!_ titled _Oh, the Places You’ll
Boldly Go!_ The book mimicked the style of Dr. Seuss, but replaced the text
and imagery with a story in a Star Trek theme. This case was not covered by
fair use because the book was a derivative product in a similar market that
matched the original work too closely. As in the ComicMix case, machine
learning models can memorize and output derivative content that is not
necessarily protected by fair use. It is thus important to measure the
memorization of copyrighted material in models to guide the investigative
resources placed to address this risk.

Here, we introduce experiments to examine models’ abilities to generate
verbatim content. We emphasize that our numerical results do not
(unconditionally) quantify the degree of legal risk. At a high level, given a
model and a prompt (typically some initial portion of a copyrighted/licensed
material), we measure the extent to which the model is able to reproduce the
completion. This is a simple test of models’ regurgitation capabilities.
Specifically, we compiled prompts from three sources: (i) 1k randomly sampled
books from the BooksCorpus (copyrighted), (ii) 20 books from the BooksCorpus
that also appear in a list of bestsellers
(copyrighted),525252https://www.theguardian.com/news/datablog/2012/aug/09/best-
selling-books-all-time-fifty-shades-grey-compare and (iii) 2k randomly sampled
functions from the linux kernel source (GPL licensed).535353GPL licenses
generally require that the source code of derivative work also be distributed
under the same license, compared to other licenses (e.g. MIT) which don’t
enforce such distribution. For (i), we took varying numbers of tokens at the
beginning of randomly sampled paragraphs as prompts. For (ii), we repeated the
previous procedure—this time only considering the first paragraph of each
bestseller. Lastly, for (iii), we took varying numbers of lines starting from
the top of each function to form prompts. (i) and (ii) respectively test
models’ ability to reproduce the “average” content and content that is likely
to be repeated in the training corpus. The comparison of the two is motivated
by observations in past work that extraction of duplicated content is on
average more likely (Kandpal et al., 2022).545454We expect portions of some of
the bestsellers to appear repeatedly in the training corpus. Our evaluation
metrics measure both exact regurgitation (longest common sequence normalized
by length of given prefix) and near-exact reproduction (edit distance and edit
similarity (Lee et al., 2021) normalized by length of given prefix).

Due to token credit constraints, we only sample one completion per prompt to
test extraction. Thus, the results here may be underpowered as ideally one
would generate many samples for the same prefix to approximate the worst-case
behavior. In addition, given that we only focus on extracting selected
contents (books and linux source code), our results may not fully reflect the
qualitative behavior of extracting other sources. Nonetheless, we find that
some models, particularly larger models and models trained on source code,
generate verbatim content in certain situations. Future work with black-box
model access can explore this further. We discuss additional nuances of this
evaluation in the appendix.

###  5.5 Disinformation

Disinformation refers to false information that is disseminated by an actor
with the intent to deceive, mislead, or otherwise influence the behavior of
the target. Disinformation is of social concern: it has been used to disrupt
democratic processes, undermine public health campaigns, and incite genocidal
violence (Partnership, 2021; Zarocostas, 2020; Whitten-Woodring et al., 2020).
Structurally, effective disinformation relies on both (i) convincing content
and (ii) network behavior to propagate disinformation (e.g. via social media
platforms) (Benkler et al., 2018).

To understand disinformation, we begin by considering approaches used for
disinformation that pre-date modern language models. Disinformation actors
generally have two approaches outlined in DiResta et al. (2022). One option is
to hire people in-house, which is good for operational security, but these
employees may lack the cultural or linguistic knowledge required to make
effective disinformation. This is particularly relevant when discussing
geopolitical disinformation that may involve parties of very different
cultural/linguistic backgrounds (e.g. different nations). The other option is
to hire freelancers in the target country; freelance authors have the cultural
context to make effective disinformation, but can more easily compromise the
operation’s security.

Given the growing generative capabilities of language models, malicious use
for disinformation is a specific risk arising from the potential dual use of a
language model. Specifically, several works have identified that language
models can be a secure, efficient, and effective means for producing content
for disinformation operations (Radford et al., 2019; Buchanan et al., 2021;
Bommasani et al., 2021, §5.2). Relative to existing approaches, models can be
created and stored in-house, matching the operational security of in-house
operations, and can be trained on data from the foreign population, providing
the effectiveness of remote operations (see Horvitz, 2022).

Beyond existing approaches that may be bottle-necked by the speed of human
authorship, machine-generated text is highly scalable. Furthermore, when
teamed with human editors who iteratively edit the model’s generations, refine
their inputs, and fine-tune the system, language models may generate outputs
that perform better than either the human or the model can manage by
themselves (Buchanan et al., 2021) Thus, the bottle-neck for using model
generations for disinformation is reliability: if model generations need
extensive human post-editing, the cost and risk of using them might be
comparable to hiring humans to author text in the first place (Goldstein et
al., Forthcoming).

##### Problem Setting.

In foundational work on the relationship between language models and
disinformation, Buchanan et al. (2021) introduce a taxonomy of six phenomena
where language models may prove to be service. Of these six phenomena, we
choose to focus on two: narrative reiteration and narrative wedging. We do so
because the threats posed by these specifically align most strongly with the
settings that concern disinformation researchers (Pennycook et al., 2021), and
because they prompt models for short generations, so a larger number can be
efficiently evaluated.

Narrative reiteration tests the ability of a language model to advance a
specified narrative. We test this by adapting models to generate headlines
that support a given thesis statement. In this sense, narrative reiteration
reuses core capabilities related to the ability to paraphrase and the
controllability of a language model’s generation.

Narrative wedging, on the other hand, tests the ability of a language model to
generate messages that divide people based on group identity (e.g. race or
religion). Such wedging can be clearly divisive, fragmenting or polarizing a
community by exerting pressure on individuals’ sense of belonging to their
social groups. We test this by adapting models to generate social media posts
to amplify social divisions as well as to encourage a certain group to take a
certain action. Further, divisive language may be either overtly hostile or
only implicitly hostile (such as code words, microaggressions, stereotypes,
and dogwhistles; Fogal et al., 2018; Quaranto, 2022). Hence, we distinguish
between covert and overt hostility in the models’ generations.

##### Datasets.

Since we follow Buchanan et al. (2021), we begin by considering their dataset
design for both reiteration and wedging. For reiteration, we choose to deviate
from their work because they use only a single prompt in their evaluation.
Instead, we use the Misinformation Reaction Frames dataset of Gabriel et al.
(2022), which provides headlines and theses on COVID-19 and climate change.
From this dataset, we manually cluster 114 headlines about COVID and 147
headlines about climate change from the Misinformation Reaction Frames dataset
into 38 and 49 clusters respectively. For each cluster, we write a thesis
statement that all of the headlines in that cluster support (e.g. “COVID-19 is
a man-made disease.”). For wedging, we use the same 11 prompts introduced by
Buchanan et al. (2021). These prompts encourage certain voting behavior (vote
Democratic, Republican, or not at all) targeting religious groups (Christians,
Jews, and Muslims), as well encourage division (e.g. anti-Black racism).

###  5.6 Bias

In §4.7: metrics-bias, we discuss social bias measurement in the context of
realistic use cases, namely in the context of model generations. We believe
such measurement is most directly useful for measuring the bias-related harms
associated with language models. However, most of the literature on bias
measurement in NLP, which can be traced back to Bolukbasi et al. (2016),
instead focuses on biases that are more intrinsic and less directly grounded
in extrinsically relevant use cases. As we note in §4.7: metrics-bias, the
predictive validity of these measurements (that is, the extent to which they
predict biases in downstream behavior) has been questioned in several works
(e.g. Goldfarb-Tarrant et al., 2021; Steed et al., 2022). As Bommasani et al.
(2021, §5.1) describe, bias evaluation for foundation models can either target
intrinsic properties of the model or extrinsic behavior of the model once
adapted for a particular downstream use case. Here, we complement our existing
evaluation with a finer-grained assessment of bias, noting that this also
helps ensure coverage of biases that may be pertinent even for tasks that
involve minimal/no generation.

##### Dataset Selection.

Nadeem et al. (2020) and Nangia et al. (2020) each introduced minimal pair
evaluations for assessing the biases of language models. To date, these two
datasets have been the primary datasets for evaluating biases in language
models, as they provide significant control due to the minimal pair design
(which is the same design used for the linguistic evaluations that we describe
in §5.1.2: language-minimal-pairs). However, Blodgett et al. (2021) provides
an extensive critique of these two datasets, providing comprehensive evidence
for their inadequate validity.

For this reason, we instead turn to the more recently introduced BBQ dataset
of Parrish et al. (2022). We note that the BBQ dataset may still suffer from
some of the concerns discussed by Blodgett et al. (2021), but we expect it is
comparatively better than the other options.555555Parrish et al. (2022) do
provide some discussion on this front. In particular, the BBQ dataset frames
bias evaluation in the context of question answering, which also aligns with
our more general approach of preferring evaluations that are more realistic
and less synthetic when discussing social harms. BBQ measures biases
associated with nine demographic categories, following the US Equal Employment
Opportunities Commission, of age, disability status, gender, nationality,
physical appearance, race/ethnicity, religion, socio-economic status, and
sexual orientation. BBQ uses data generated via templates, with attested
biases (with documented evidence that establishes the bias), that are then
vetted by crowdworkers on Amazon Mechanical Turk.

##### Problem Setting.

BBQ involves multiple-choice question answering, where each question is paired
with a context and three answer choices (two of which reference different
social groups of the same demographic category, and the third of which is
always “Unknown”). To facilitate bias measurement, each example in the dataset
is associated with a $2\times 2$ pattern of instances: questions appear in
pairs (one which is negative, i.e. a social value in the US is violated and
the bias it reflects is harmful to certain groups, and the other of which is
negative) and contexts appear in pairs (one which is ambiguous, and one which
is disambiguating). Due to these alterations, in additional to measuring the
exact match accuracy of the model on all data, the authors also introduces
measures for the bias of the model both on ambiguous and disambiguated
contexts.

###  5.7 Toxicity

In §4.8: metrics-toxicity, we discuss toxicity measurement in the context of
realistic use cases, namely in the context of model generations. In these use
cases, we empirically find very little incidence of toxicity, which we take to
be a generally positive sign regarding the risks of toxicity-related harms
arising from language model deployment. On the other hand, Abid et al. (2021)
demonstrate models have strong tendencies towards toxicity even with innocuous
prompts, including in ways that reflect abhorrent social biases (e.g.
associating Islam with violence). Further, it has been made clear by Kilcher
(2022) that these models can be further intensified in their toxicity, which
has created controversy in the AI research community over potentially reckless
deployment (Liang & Reich, 2022). In light of this, we complement our existing
evaluation with a finer-grained assessment of toxicity, noting that this also
poses interesting connections with the models ability to detect toxicity
(§3.7: toxicity-detection).

##### Dataset Selection.

Gehman et al. (2020) introduced the main dataset used to evaluate toxicity in
RealToxicityPrompts. Shortly thereafter, Dhamala et al. (2021) introduced
BOLD, which follows a similar structure as RealToxicityPrompts but uses more
innocuous input prompts. Given this, we choose to evaluate on both datasets to
understand the relationship between properties of the prompt and properties of
the model generation. We note that other works, especially Abid et al. (2021),
also demonstrate language model toxicity, but do not release a standardized
dataset that has been used for measuring toxicity more broadly.

##### Problem Setting.

For toxicity evaluation, models are presented with a prompt and generate a
completion. In the case of RealToxicityPrompts, these prompts are drawn from
OpenWebText (Gokaslan & Cohen, 2019), which is a collection of Internet text
that replicates the training data of GPT-2 (Radford et al., 2019). To provide
a stratified sample of different prompt toxicities, Gehman et al. (2020) ran
the text in the corpus through PerspectiveAPI, binning into 4 bins ([0, .25),
[0.25, 0.50), [0.50, 0.75), [0.75, 1.00]) and sampling 25k sentences from
each. For BOLD, the prompts are instead drawn from Wikipedia, taking the first
6-9 words of an article that mention a (profession, gender, race, religion, or
political ideology), which also implies that toxicity measurements for BOLD
can be stratified by the associated social category. We note that in
comparison to RealToxicityPrompts, the prompts in BOLD tend to be more
neutral, so generating toxic text is even less justifiable in these contexts.
We use PerspectiveAPI to measure toxicity in the model completions,
reiterating the extensive caveats regarding the validity of PerpsectiveAPI
that we discussed in §4.8: metrics-toxicity.

##  6 Models

Model | Model Creator | Modality | # Parameters | Tokenizer | Window Size | Access | Total Tokens | Total Queries | Total Cost
---|---|---|---|---|---|---|---|---|---
J1-Jumbo v1 (178B) | AI21 Labs | Text | 178B | AI21 | 2047 | limited | 327,443,515 | 591,384 | $10,926
J1-Grande v1 (17B) | AI21 Labs | Text | 17B | AI21 | 2047 | limited | 326,815,150 | 591,384 | $2,973
J1-Large v1 (7.5B) | AI21 Labs | Text | 7.5B | AI21 | 2047 | limited | 342,616,800 | 601,560 | $1,128
Anthropic-LM v4-s3 (52B) | Anthropic | Text | 52B | GPT-2 | 8192 | closed | 767,856,111 | 842,195 | -
BLOOM (176B) | BigScience | Text | 176B | BLOOM | 2048 | open | 581,384,088 | 849,303 | 4,200 GPU hours
T0++ (11B) | BigScience | Text | 11B | T0 | 1024 | open | 305,488,229 | 406,072 | 1,250 GPU hours
Cohere xlarge v20220609 (52.4B) | Cohere | Text | 52.4B | Cohere | 2047 | limited | 397,920,975 | 597,252 | $1,743
Cohere large v20220720 (13.1B) 565656Deprecated by Cohere as of December 2, 2022. | Cohere | Text | 13.1B | Cohere | 2047 | limited | 398,293,651 | 597,252 | $1,743
Cohere medium v20220720 (6.1B) | Cohere | Text | 6.1B | Cohere | 2047 | limited | 398,036,367 | 597,252 | $1,743
Cohere small v20220720 (410M) 575757Deprecated by Cohere as of December 2, 2022. | Cohere | Text | 410M | Cohere | 2047 | limited | 399,114,309 | 597,252 | $1,743
GPT-J (6B) | EleutherAI | Text | 6B | GPT-J | 2048 | open | 611,026,748 | 851,178 | 860 GPU hours
GPT-NeoX (20B) | EleutherAI | Text | 20B | GPT-NeoX | 2048 | open | 599,170,730 | 849,830 | 540 GPU hours
T5 (11B) | Google | Text | 11B | T5 | 512 | open | 199,017,126 | 406,072 | 1,380 GPU hours
UL2 (20B) | Google | Text | 20B | UL2 | 512 | open | 199,539,380 | 406,072 | 1,570 GPU hours
OPT (66B) | Meta | Text | 66B | OPT | 2048 | open | 612,752,867 | 851,178 | 2,000 GPU hours
OPT (175B) | Meta | Text | 175B | OPT | 2048 | open | 610,436,798 | 851,178 | 3,400 GPU hours
TNLG v2 (6.7B) | Microsoft/NVIDIA | Text | 6.7B | GPT-2 | 2047 | closed | 417,583,950 | 590,756 | -
TNLG v2 (530B) | Microsoft/NVIDIA | Text | 530B | GPT-2 | 2047 | closed | 417,111,519 | 590,756 | -
davinci (175B) | OpenAI | Text | 175B | GPT-2 | 2048 | limited | 422,001,611 | 606,253 | $8,440
curie (6.7B) | OpenAI | Text | 6.7B | GPT-2 | 2048 | limited | 423,016,414 | 606,253 | $846
babbage (1.3B) | OpenAI | Text | 1.3B | GPT-2 | 2048 | limited | 422,123,900 | 606,253 | $211
ada (350M) | OpenAI | Text | 350M | GPT-2 | 2048 | limited | 422,635,705 | 604,253 | $169
text-davinci-002 | OpenAI | Text | Unknown | GPT-2 | 4000 | limited | 466,872,228 | 599,815 | $9,337
text-curie-001 | OpenAI | Text | Unknown | GPT-2 | 2048 | limited | 420,004,477 | 606,253 | $840
text-babbage-001 | OpenAI | Text | Unknown | GPT-2 | 2048 | limited | 419,036,038 | 604,253 | $210
text-ada-001 | OpenAI | Text | Unknown | GPT-2 | 2048 | limited | 418,915,281 | 604,253 | $168
code-davinci-002 | OpenAI | Code | Unknown | GPT-2 | 4000 | limited | 46,272,590 | 57,051 | $925
code-cushman-001 (12B) | OpenAI | Code | 12B | GPT-2 | 2048 | limited | 42,659,399 | 59,751 | $85
GLM (130B) | Tsinghua University | Text | 130B | ICE | 2048 | open | 375,474,243 | 406,072 | 2,100 GPU hours
YaLM (100B) | Yandex | Text | 100B | Yandex | 2048 | open | 378,607,292 | 405,093 | 2,200 GPU hours

Table 5: Models. Description of the 30 models evaluated in this effort:
provenance for this information is provided in Appendix I. The total tokens,
queries, and cost refer to the overall costs we incur to evaluate the given
model. For this reason, the costs for open models are measured in GPU hours,
the costs for commercial models are measured in the monetary cost calculated
using the current pricing scheme for each model, and the costs for the private
models were covered by the model providers.

We evaluated 30 state-of-the-art language models varying in size, training
procedure, organization, and access. These 30 can be grouped into 16 families
(models within a family are identical up to model scale), and further into 12
organizations that created the model. Table 5 documents the models and their
properties; further description is deferred to Appendix I.

##### Model selection.

As one of the pillars of this work, our aim was common understanding of
language models. To achieve this ambition, we had to navigate the access
conditions of different models (Liang et al., 2022). Some models were open
(full model weights are available for download) (e.g. GPT-NeoX (20B), OPT
(175B), BLOOM (176B)), some were limited API access (e.g. davinci (175B),
J1-Jumbo v1 (178B), Cohere xlarge v20220609 (52.4B)), and some were closed,
but the model owner provided us with access for this research effort
(Anthropic-LM v4-s3 (52B), TNLG v2 (530B)). Unfortunately, we did not evaluate
models that we could not access (e.g. Google’s PaLM (Chowdhery et al., 2022)
and LaMDA (Thoppilan et al., 2022), DeepMind’s Gopher (Rae et al., 2021) and
RETRO (Borgeaud et al., 2022)). We more extensively discuss missing models in
§10.4: missing-models. We encourage future work to evaluate these models to
allow for common understanding, and to develop standards for research access
to these models (Liang et al., 2022).

Model | Hardware
---|---
GPT-J (6B) | 2$\times$A100 (10.4%); 4$\times$2080 Ti (89.6%)
GPT-NeoX (20B) | 2$\times$A100 (73.9%); 11$\times$2080 Ti (26.1%)
T5 (11B) | 2$\times$A100 (59.1%); 8$\times$2080 Ti (40.9%)
T0++ (11B) | 2$\times$A100 (1.1%); 8$\times$2080 Ti (98.9%)
UL2 (20B) | 2$\times$A100 (3.5%); 16$\times$2080 Ti (96.5%)
YaLM (100B) | 8$\times$A100
GLM (130B) | 8$\times$A100
OPT (66B) | 8$\times$A100
OPT (175B) | 8$\times$A100
BLOOM (176B) | 8$\times$A100

Table 6: Hardware and compute for public models. To perform inference on the
public models, we used the Together Research Computer. At the time of this
work, Together Research Computer connects clusters at several institutions. We
mainly use NVIDIA GeForce RTX 2080 Ti GPUs and NVIDIA A100 GPUs to perform
inference. If jobs were run on multiple hardware configurations, we report all
configurations separated by “;” (with the percentage of GPU hours spent on
each configuration).

##### Computational Resources.

For models that required we provide computational resources, we used the
Together Research Computer (Table 6).585858Together Research Computer:
https://together.xyz Specifically, Together Research Computer employs pipeline
parallelism to achieve high-throughput inference: the model is partitioned
across GPUs and the GPUs communicate with each other the activations of the
model to accomplish forward passes to get the predictions. Generally, Together
Research Computer will dispatch inference jobs to idle GPUs. For models hosted
via commercial APIs, the providers of these models graciously provided us
sufficient API credits to evaluate their models as part of this effort.

##### Live systems.

For both the private models and commercial APIs, we are evaluating live
systems that may be regularly updated in some cases. Unfortunately, the
practices of (publicly disclosed) version control and the corresponding change
logs are uneven across model providers. For this reason, to the extent
available, we report model versions (e.g. the version date for Cohere models).

The results we produce are specific to the model versions at the time we
queried them: we record these timestamps in the results we release. Further,
due the scale of our evaluation, it is possible that models change over the
duration of our evaluation, but we assume they do not change.595959This may
relate to observations of unexpected non-determinism in model behavior as
reported in https://twitter.com/OfirPress/status/1542610741668093952 and
studied in Ruis et al. (2022). To the extent the models/APIs change, we
anticipate the change is fairly gradual, therefore the results are not subject
to large shocks, but we encourage efforts to longitudinally monitor such
changes to clarify their nature (cf. Chen et al., 2022). Similarly, we note
that the models we evaluate may be deprecated at some point after our
evaluation.

##### Isolated technical issues.

In general, we evaluate every model on every scenario (see Figure 4), and for
every scenario we measure every metric (see Table 4). However, in a few
isolated cases, we encounter issues/concerns that are generally attributable
to specific model providers. We believe these issues are quite marginal in
influencing the overall results (e.g. most models are not affected at all, and
even affected models are affected on few scenarios).

First, for scenarios which require probabilities and not just completions
(e.g. HellaSwag, MS MARCO (regular), The Pile), we currently are not able to
evaluate (possibly because the model itself fundamentally does not yield such
probabilities) for T0++ (11B), T5 (11B), UL2 (20B), GLM (130B), and YaLM
(100B). Second, for two models (J1-Jumbo v1 (178B), J1-Grande v1 (17B)), the
API yields a tokenization error which currently prevents us from reporting
results for NaturalQuestions (closed-book), though we expect this to be
resolved shortly based on correspondence with the model providers. Third, the
model probabilities for completions sampled with temperature zero are
incorrect for models from AI21 Labs: for this reason, we do not report results
for calibration, MS MARCO (regular), and MS MARCO (TREC). We explicitly
acknowledge this in Figure 4, which fully explains why we evaluate 96% of the
(model, scenario) pairs instead of 100%.

##### Model contamination.

While we provide some information on the models we evaluate in Table 5, with
further details in Appendix I, we emphasize that we have uneven and usually
limited information on the training procedure for these models (see Liang et
al., 2022). Consequently, we generally lack a precise characterization of, and
access to, the training data for these models with notable exceptions
including the ROOTS corpus for BLOOM (176B) 606060See
https://huggingface.co/spaces/bigscience-data/roots-search. as well as the
Pile corpus along with further training details for GPT-J (6B) and GPT-NeoX
(20B) (see Biderman et al., 2022). Without such access and/or disclosure of
contamination by model providers, we cannot verify whether the model was
trained on the data we evaluate it on. This concern is heightened given we
evaluate in a few-shot manner (as we discuss in §7: prompting), so the model
being trained on data drawn from this evaluation distribution, even beyond the
exact evaluation instances, could compromise the legitimacy/purity of such
few-shot evaluation. For this reason, we flag known instances of model
contamination (i.e. where the model creators have disclosed possible
contamination for a given scenario) when we report results, and compile this
known evidence in Appendix G, though overall it is still largely limited. We
encourage more proactive documentation/disclosure of model contamination, as
well as the use of canaries or other methods for minimizing contamination, for
future work on language model development to maintain the health of the
evaluation ecosystem.

##### Fairness of evaluation.

We prioritize standardization: models are evaluated on the same scenarios, for
the same metrics, and with the same prompts for 5-shot prompting. However, the
models themselves can require vastly different resources to produce, which are
only partially accounted for by our measurements of efficiency as one class of
costs. In this sense alone, the evaluation can be seen as unfair (cf. Linzen,
2020; Bommasani et al., 2021, §4.4). Even beyond this, particular models may
be better suited for specific scenarios, metrics, prompts, and broader
adaptation methods. As an example, we evaluate T0++ (11B) in a 5-shot fashion,
whereas the model was designed for 0-shot prompting, and T5 (11B) similarly
was neither designed for in-context learning nor to follow instructions in
prompts. As one concrete recommendation as the language modeling space
matures, we recommend model developers explicitly declare how their models
should be evaluated and what the scope is of their generality/when models
should be preferred. We believe this disclosure will help to strike a
productive balance between the general-purpose possibilities for language
models in the abstract and what is concretely sensible for a given model.

##  7 Adaptation via prompting

Figure 23: Prompt formatting. An example of how we structure and format the prompt for querying the language model. | Parameter | Language Modeling | TruthfulQA | CNN/DailyMail
---|---|---|---|---
Prompt format §J.1: prompting-test §J.2: prompting-remainder | Instructions | None | None | Summarize the given documents.
Input prefix | None | Question: | Document:
Reference prefix | None | None | None
Output prefix | None | Answer: |  Summary: {
Instance prefix | None | None | None
Max training instances | 0 | 5 | 5
Decoding parameters §J.3: decoding-parameters | Temperature | 0 | 0 | 0.3
Max tokens | 0 | 5 | 128
Stop sequence(s) | None | \n | }
Num. outputs | 0 | 1 | 1
Evaluation parameters | Num. runs | 3 | 3 | 3
Max evaluation instances | 1000 | 1000 | 1000
Table 7: Adaptation parameters. Example of the adaptation parameters specified
for (i) language modeling scenarios, (ii) the TruthfulQA question answering
scenario, and (iii) the CNN/DailyMail summarization scenario. We also include
additional parameters required to fully specify evaluation (i.e. the number of
evaluation instances and runs, which influence the statistical validity and
reliability of the results), though they are not strictly part of the
adaptation process.

Adaptation transforms a raw language model into a system that makes
predictions on test instances. In short, we use prompting as our adaptation
method with 5 in-context examples (when in-context examples are included) as
depicted in Figure 23. However, many lower-level details must be specified to
fully define the adaptation procedure: we discuss important conceptual details
here and defer the remainder to Appendix J. Table 7 provides example scenarios
along with the full set of adaptation parameters.

##### In-context examples.

Since we include 5 in-context examples, two core decisions are why we use this
number of examples and how we select the examples. We follow Brown et al.
(2020) in their choice of the number, testing in §8.2: prompting-analysis how
the number of examples influences performance. To select examples, we sample
examples to ensure class coverage (for classification coverage) in order of
class frequency, so the 5 most frequent classes will be represented in-
context. Critically, we choose to fix the in-context examples across all
evaluation instances, in contrast to prior work (e.g. Brown et al., 2020), to
more accurately perform few-shot evaluation (Perez et al., 2021). While more
realistic, this also implies our model performance can be more sensitive to
the random choice of in-context examples. Therefore, we re-run all experiments
3 times, only varying the randomness in in-context sample selection, to
estimate this variance: in §8.2: prompting-analysis, we show results can be
especially high variance for some scenarios.

##### Prompt formatting.

Beyond the in-context examples and evaluation instance, there are many other
mechanistic nuances involved in formatting the exact string that submitted to
the language model. Prior work has extensively shown that the design of
prompts matters (Le Scao & Rush, 2021; Liu et al., 2022b), with Bach et al.
(2022) introducing a repository for
prompts.616161https://github.com/bigscience-workshop/promptsource Since we
more broadly view language models as interfaces, we treat prompts as user
behavior rather than explicitly optimizing prompts (Shin et al., 2020; Zhou et
al., 2022). Moving forward, we further expect that it may be desirable for
language models to be interoperable meaning different models should be
standardized such that they all work well for similar user prompting behavior.
We provide initial evidence of how model performance varies in the exact
string formatting of the inputs in §8.2: prompting-analysis.

##### Adaptation strategy.

For many scenarios (e.g. language modeling, text summarization), the way to
interact with a language model may appear clear. However, in the case of
multiple choice classification, each instance has several answer choices as
references (usually with one marked as correct). This structure affords
flexibility in how to use a language model that traditionally does not exist
in classification (where a model necessarily predicts a distribution over
answer choices with the prediction being the argmax).

Prior work introduces two classes of approaches to address adaptation for
multiple choice scenarios. The first is the separate approach employed by
Brown et al. (2020), where each choice is independently scored by
concatenating it with the question and computing its probability according to
the LM. In this approach, the probabilities may be further calibrated by the
probability the LM assigns to the answer choice alone. The second is the joint
approach employed by Hendrycks et al. (2021c), where all the choices are
concatenated with the question to form a prompt (e.g., “<question> A. <choice
1> B. <choice 2> Answer:”) and the LM predicts the choice index (e.g., A or
B). This resembles how one might see a multiple choice question presented on
an exam or test. In §8.2: prompting-analysis, we vary this adaptation strategy
to test the influence on performance: we find the strategy significantly
influences accuracy (among other metrics), and the most accurate strategy
depends on the scenario (i.e. no strategy is uniformly more accurate).

##  8 Experiments and results

Our evaluation offers an immense array of model predictions along with
quantitative metrics for these predictions. To understand these results, we
provide a web interface.626262https://crfm.stanford.edu/helm/v0.1.0 This
interface not only provides the quantitative results, as is customary for
other benchmarks and leaderboards, but also the underlying model predictions
and the exact inputs and prompts that yielded these predictions. We encourage
exploration of these results interactively: we believe grounding and
interrogating various quantitative trends by mapping them to explicit model
behaviors is necessary for the community to have common understanding of these
models. Here, we provide a succinct analysis, foregrounding unique aspects of
our evaluation that are made possible by its broad, holistic, and systematic
nature.

###  8.1 Meta-analysis

To summarize, we evaluate 30 language models (e.g. davinci (175B), BLOOM
(176B)) on 16 core scenarios (e.g. NaturalQuestions, IMDB) for 7 metric
categories (e.g. accuracy, fairness). Here, we use our comprehensive
evaluation to provide answers to important questions in the field like whether
model accuracy correlates with scale or if more robust models are less biased.

Figure 24: Accuracy vs. X. The relationship between accuracy (x-axis) and each
of the 6 metrics (calibration, robustness, fairness, social bias, toxicity,
efficiency) we study in this work across all core scenarios and for all
models. For calibration error, we measure ECE-10; for bias, we measure bias in
gender representation; and for efficiency, we measure denoised inference time.
Figure 25: Correlation between metrics. The Pearson correlation between each
metric and every other metric (x-axis). The small grey dots denote the
correlation on each individual scenario. Trends are qualitatively similarly
for other correlation measures (e.g. Spearman correlation). For calibration
error, we measure ECE-10; for bias, we measure bias in gender representation;
and for efficiency, we measure denoised inference time.

##### Inter-metric relationships.

A central tenet of our holistic evaluation is to foreground metrics beyond
accuracy, ensuring that the many desiderata we should expect of systems are
indeed measured for each scenario (when possible; see Table 4). As a
consequence, this clarifies how different metrics relate to each other. Many
prior works have explored these types of relationships extensively for
specific metric pairs: for example, accuracy-robustness (Miller et al., 2021;
Koh et al., 2021), accuracy-efficiency (Coleman et al., 2017), calibration-
fairness (Pleiss et al., 2017; Jones et al., 2021), and bias-toxicity (Sap et
al., 2019a; Halevy et al., 2021). In the context of language models, we may
posit the relationships found in prior works will continue to bear, but many
pairs have not been studied and we do not have definitive evidence that these
relationships hold up reliably.

Therefore, we harness our evaluation’s coverage of these metrics to provide
two resources. First, in Figure 24, we depict the relationship between
accuracy and each of the 6 other desiderata. Among other interpretations, this
figure helps clarify when more accurate systems coincide with improvements in
other key desiderata and when there are important trade-offs. Further, by
plotting these on a per-scenario basis, this helps to showcase the underlying
heterogeneity in the trends: when are metric relationships consistent across
scenarios, when are they highly scenario-dependent, and what are the
anomalies? Second, in Figure 25, we consolidate the relationship across models
for specific scenarios. Concretely, in each subfigure we consider a pair of
metrics, where each grey point corresponds to the Pearson correlation (i.e.
linear fit) between these metrics across all models for this scenario.636363We
also considered Spearman correlation (i.e. monotonic fit), given this may be
more appropriate for some metric relationships, but found the qualitative
trends to be largely invariant across these two choices of correlations
metrics. Once again, this helps to showcase the spread and heterogeneity in
pairwise metric relationships as well as the macro-level trends.

While these figures together provide a wealth of information, we step through
them to state specific takeaways. Most strikingly, we find that across all
scenarios, accuracy, robustness, and fairness are extremely strong correlated
(see Figure 25). In part, we believe this is a finding contingent on how we
chose to measure robustness/fairness. For robustness, this aligns with
findings for other forms of (non-adversarial) robustness, such as the work of
Miller et al. (2021) for distributional robustness, which shows in-domain and
out-of-domain accuracy lie on a line. On the other hand, for fairness, we find
this to be quite surprising. We do emphasize several prior works instead find
trade-offs between fairness and accuracy, but we do not believe that our work
should be interpreted as contradicting their findings. Not only do we measure
fairness differently from these works, but the setting of few-shot prompting
of language models is considerably different from the settings studied in
these prior works (Zhang et al., 2019a; Raghunathan et al., 2020; Dutta et
al., 2020; Wang et al., 2021c; Zhao & Gordon, 2022).

Shifting focus to calibration, the trends are more scenario-dependent, as can
be seen for accuracy by the visual clustering of points for a given scenario
in Figure 24. Strikingly, we find that the relationship between accuracy and
calibration can be quite different for very similar scenarios: for two
commonsense-centric QA scenarios, we see accuracy and calibration are highly
correlated in OpenBookQA (correlation of greater than 0.8) whereas accuracy
and calibration error are highly correlated in HellaSwag (correlation greater
than 0.85). Further, while not unsurprising given the strong correlations
between accuracy, robustness, and fairness, the finding that more robust and
more fair models can be less well-calibrated is counter-intuitive and
surprising.

For generative harms, we also see significant variation in the relationship
with other metrics. For toxicity, the correlations on average with other
metrics are near zero as the toxicity rates themselves for toxicity are
relatively constant and near zero, with the clear exception of NarrativeQA in
Figure 24. In contrast for bias, we see some positive correlation between
accuracy and gender representation bias (lower is better), with an even more
striking average correlation of scenarios of more the 0.5 between fairness
(higher is better) and gender representation bias (lower is better). In other
words, we see a clear and surprising trade-off: models that tend to have
better fairness performance, which depends on task-specific performance, tend
to have worse gender bias, which depends on model generations but not task-
specific performance. As to the two generative harms, we see there are some
scenarios where they are fairly anti-correlated, which indicates the less
gender biased models tend to be more toxic for these scenarios. These findings
indicate it may not suffice to only measure one of these types of harms, and
that efforts to reduce one may have side-effects for the other (cf. Xu et al.,
2021), which is especially relevant given the harms of toxicity are more
straightforward/immediate (e.g. more likely to draw media/journalistic
attention) whereas the harms of bias may be more subtle/gradual.

Finally, for efficiency, we see fairly weak correlations with other metrics.
In particular, we do not see a strong overarching trade-off between accuracy
and efficiency, which we may have anticipated a priori. This is made clear by
Figure 24, which instead showcases that the trends for efficiency tend to be
quite scenario-dependent. This is especially true for the denoised metric,
given a lot of what drives the measurements relates to the fundamental
distributional properties of the scenario (i.e. the length of inputs and
outputs).

##### Direct model comparisons.

A central objective for our evaluation is to achieve common and unified
understanding of all the models available. In particular, as we document in
Appendix F, the 30 models we evaluate in this work have been previously
evaluated under different conditions for different evaluation suites. In
particular, the degree of overlap is surprisingly low and uneven across
different models. Consequently, we lack clarity as a field on the relationship
between different models.

Figure 26: Head-to-head win rate per each model. We report the fraction of
head-to-head comparisons between the given model and all other models, across
all scenarios, where the given model is higher along the metric (e.g. more
accurate in the accuracy subfigure). If a model was the highest for the given
metric for every scenario, it would receive a score of 1.0; if a model
received a score of 0.5, then if a scenario and second model were chosen at
random, the outcome of the comparison would be a coin flip. For metrics where
higher is better (Accuracy, Robustness, Fairness), the better models are
towards the top, and vice versa. For calibration error, we measure ECE-10; for
bias, we measure bias in gender representation; and for efficiency, we believe
these comparisons should not be made without consideration of accuracy and
they are further discussed in Narayanan et al. (Forthcoming).

In Figure 26, we report how different models would fare in a head-to-head
comparison for each metric across all the core scenarios.646464We exclude
efficiency as we believe head-to-head comparisons based solely on efficiency
are not meaningful; instead efficiency needs to be considered in conjunction
with other metrics like accuracy. Forthcoming work (Narayanan et al.,
Forthcoming) examines the tradeoffs between accuracy and efficiency.  We find
that text-davinci-002 is clearly the most accurate model, winning in these
comparisons more than 90% of the time. Of the remaining models, we see that
the largest model TNLG v2 (530B) is the second most accurate, followed by
Anthropic-LM v4-s3 (52B). In light of the significant difference in model
scale of a factor of more than 10 between TNLG v2 (530B) and Anthropic-LM
v4-s3 (52B), given text-davinci-002 and Anthropic-LM v4-s3 (52B) share
instruction-tuning in common (which other models, beyond smaller OpenAI text
variants, do not), this suggests that instruction-tuning and the use of human
feedback is an efficient and effective means for improving model accuracy.

Further, while we will later see the overall relationship between model scale
and accuracy can be complicated (Figure 29), here we see a clear thresholding
effect in terms of model scale. All of the models with a win rate for accuracy
clearly above chance (e.g. 55% or more) have more than 50B parameters, whereas
the sole exception of a large model outside this group of the top 10 is YaLM
(100B). YaLM (100B) has a surprisingly poor accuracy win rate below 25%,
perhaps because of significant training on Russian instead of English). Within
the top 10, we see model scale appears to be less predictive of rank, as two
of the largest models (BLOOM (176B) and J1-Jumbo v1 (178B); both 100B+
parameters) are at the bottom of this tier, whereas Anthropic-LM v4-s3 (52B)
and Cohere xlarge v20220609 (52.4B) (the two smallest models in the tier) are
in the top half of this tier. Similarly, within a model family, we see model
scale is perfectly monotonically correlated with model accuracy win rate (e.g.
see the OpenAI and Cohere model variants). We emphasize this certainly does
not imply the models that fare poorly, or the small models, are inaccurate
under all conditions: we expect that the prompting approaches we used are
poorly suited for these smaller models and, perhaps, it is better to fine-tune
these models (cf. Liu et al., 2022a).

Consistent with previous findings of accuracy being correlated with robustness
and fairness, we see similar rankings of models for those scenarios. However,
we note that BLOOM (176B) notably performs considerably better (in a relative
sense) for robustness/fairness than its accuracy would suggest, with OPT
(175B) and GLM (130B) roughly interchanging positions for robustness and
accuracy (i.e. GLM (130B) is more robust and less accurate in a relative sense
compared to OPT (175B)). In contrast, perhaps also as expected given the weak
correlations of generative harms with other metrics, we see the rankings for
toxicity and bias are notably quite different. We highlight how T0++ (11B) and
davinci (175B) in particular demonstrate the opposite trends when comparing
their bias win rates to their toxicity win rates. That is, T0++ (11B) is the
most toxic when compared to all other models, but one of the three least
(gender) biases, whereas davinci (175B) is one of the four most biased head-
to-head, but one of the less toxic models.

Figure 27: Cumulative accuracy over time. The relationship between time
(x-axis) and the accuracy of the most accurate model released up to that point
(y-axis) across 16 core scenarios. That is, the graph tracks the progress in
the state-of-the-art (SOTA) accuracy over time for each scenario.  Figure 28:
Accuracy as a function of model access. The relationship between access (open
vs. limited vs. closed) and model accuracy for each of the 16 core scenarios.
Shaded bars indicate the performance of the best model for that scenario,
whereas the solid bars indicate the performance of the overall most accurate
model across all core scenarios based on Figure 26.

##### Accuracy as a function of other variables.

To build on these model comparisons, we now consider trends in model
accuracies as a function of other relevant variables. In Figure 27, we report
model accuracies as a function of time (i.e. model release date). As a
function of time, we see that the release of GPT-3 (Brown et al., 2020)
clearly establishes a strong baseline for future models across all scenarios,
with a distinctive improvement over T5 (11B). To the extent there is upward
movement in accuracy since then, we often see it coming with the introduction
of Anthropic-LM v4-s3 (52B) roughly 18 months later in December, 2021 (the
first model to use reinforcement learning with human feedback of those we
evaluate), though we see a surprisingly clear monotonic improvement over the
intermediary period for NaturalQuestions.

In Figure 28, we stratify the results for accuracy based on model
accessibility (see Liang et al., 2022) as we discuss in §6: models. For each
access category, we report the per-scenario accuracy of the model in that
category that is the most accurate (full bar) and that is generally the most
accurate (smaller dark sub-bar; based on Figure 26). This means the sub-bar is
always the accuracy of OPT (175B) for open models, text-davinci-002 for
limited access models, and TNLG v2 (530B) for closed models. We see that these
models for limited access are consistently the best in their categories, but
OPT (175B) is sometimes improved over (e.g. 10 point improvement for MMLU and
CivilComments) as is TNLG v2 (530B) (e.g. more than 5 point improvement for
NaturalQuestions (open-book) and TruthfulQA).

Overall, we see the accuracies generally fall in the direction limited access
> closed > open, which aligns with Figure 26. With that said, we see some
reason for optimism regarding open science through open-sourced models, in
that the best public model is usually somewhat competitive in accuracy (i.e.
within 5 points of the best non-open model), with the notable exceptions of
more knowledge-intensive scenarios of MMLU, NaturalQuestions (closed-book) and
TruthfulQA as well as both information retrieval scenarios. This is especially
noteworthy given we have yet to see models being open-sourced with significant
use of human preferences and reinforcement learning from human feedback, which
may further bridge this gap. However, we do highlight there are other
confounds: some models may not be disclosed to the public (let alone
released), some models may be disclosed after significant delay, and some
highly accurate models are not yet evaluated in HELM (e.g. Gopher, Chinchilla,
PaLM; see §10.4: missing-models). We hope our findings on the relationship
between model accessiblity and model performance (e.g. accuracy) will inform
the development of community norms and standards around model access/release
(Liang et al., 2022).

Figure 29: Cumultative Scale vs. Accuracy. The relationship between model
parameter size (x-axis) and the accuracy of the most accurate model released
up to that scale on each core scenario. That is, the graph tracks the progress
in the state-of-the-art (SOTA) accuracy as a function of scale for each
scenario.  Figure 30: The Pile loss vs. Accuracy. The relationship between log
bits-per-byte (BPB) on The Pile and the accuracy on each core scenario.

##### Predicting accuracy.

Given the resource-intensive nature of producing language models, especially
the most capable models (for which we take accuracy for different scenarios as
a proxy), the ability to predict model capabilities656565It would be similarly
interesting to predict our properties (e.g. those related to harms), though
here we focus on accuracy. We do note that our evaluation deliberately
surfaces all the necessary metrics required to project other properties. is
valuable for orienting decision-making. Prior work demonstrates model scale
(whether measured in model size, data size, or compute/training FLOPs)
reliably predicts model loss for language models, establishing scaling laws
(Kaplan et al., 2020; Hoffmann et al., 2022). On the other hand, work has also
shown that certain qualitative capabilities (e.g. in-context learning; Brown
et al., 2020) emerges unpredictably beyond a critical scale threshold (see Wei
et al., 2022b). And many works have implicitly assumed and demonstrated that
improvements in language modeling loss (i.e. perplexity) translate to
improvements on downstream use cases. In Figure 29, we report on the
relationship between model scale and accuracy, while in Figure 30, we report
perplexity (measured on The Pile) vs. accuracy.

In light of the very consistent picture that model scale within a family led
to reliable improvements in accuracy (Figure 26), we see that this does not
generalize across model families. If we are to plot model scale vs. accuracy,
we find the relationship is very chaotic; model scale (agnostic to model
family) as measured in parameters is a poor predictor of trends in accuracy.
Unfortunately, the data on different models was too sparse to attempt a
similar analysis in terms of training FLOPs, but we expect such an analysis
may be more helpful in demonstrating a crisper relationship between training
compute and downstream accuracy. When we accumulate the performance of models
in Figure 29, so as to understand if scale appears more necessary to improve
accuracy, we do see some notable jumps for many scenarios both in the range of
12B parameters and 50B parameters. These jumps are specifically attributable
to T0++ (11B) and Anthropic-LM v4-s3 (52B), which likely instead indicates
that attributing these jumps to scale is confounded by other changes (i.e. the
training procedure of T0++ (11B) to deliberately improve few-shot prompting
performance and the RLHF involved in Anthropic-LM v4-s3 (52B)). And we
similarly see the relationship between perplexity (or, more accurately, log
bits-per-byte) on The Pile and downstream accuracy is similarly messy (Figure
30), though we expect the confound of contamination (some models are trained
on The Pile and others are not) exacerbates the messiness of these findings.

###  8.2 Prompting analysis

While the benchmark we design is general, we evaluate 30 models by adapting
them through few-shot prompting (see §7: prompting). Consequently, here we
study variations across several of these models with respect to different
design decisions involved in prompting.666666We do not conduct all of these
analyses on every model due to cost. We exclude all commercial models due to
monetary cost, as well as TNLGv2 models due to long wait times given the scale
of the model. Together, these analyses help clarify the potential brittleness
of model behavior to prompt design, which bears on the legitimacy and
reliability of using prompting to build practical systems.

##### Choice of in-context examples.

Given that we adapt language models through few-shot prompting, the selection
of the specific in-context examples could significantly influence model
performance. In particular, unlike prior work (e.g. Brown et al., 2020), we
use the same examples across all evaluation instances, rather than selecting
different in-context examples for each evaluation instance, as this better
reflects the realities of few-shot adaptation (Perez et al., 2021). (We
describe our specific selection algorithm which, to first-approximation, is
random selection from the scenario’s training set while ensuring coverage of
different classes for classification problems, in §J.2: prompting-remainder.)
As a consequence, this exacerbates the variation observed for different
choices of in-context examples (consistent with the high variability one might
witness in practice if they truly only have a few examples).

To measure this sensitivity, we repeat all evaluations for 3 runs, meaning 3
choices of the in-context examples. To explore the sensitivity of model
results to this choice, hover over a specific entry to see the variance across
runs on the website. Further, the predictions across the 3 runs can be found
for a given evaluation instance. Additionally, we visualize the accuracy range
(best minus worst performance) across seeds in Figure 31. We observe that the
performance of most models tends to be relatively consistent: the median range
(across scenarios) is below 0.03 for 24 out of 28 models (exceptions are YaLM
(100B), davinci (175B), curie (6.7B), ada (350M)). Interestingly, we find that
there are scenarios where all models tend to exhibit higher variance. A
prominent example is NaturalQuestions (open-book), where the median range is
0.1730 (e.g., davinci (175B) obtains F1 scores of 0.376, 0.611, and 0.636
across the three sets of in-context examples).

Figure 31: Variance across seeds. For a subset of models and scenarios, we
evaluate each scenario with three different random sets of in-context
examples. We compute the range of the accuracy metric (maximum minus minimum
value over the three random seeds) and visualize across models and scenarios.

##### Number of in-context examples.

By default, we either use 5 in-context examples, or fewer examples for
scenarios where 5 examples do not fit within in the context window. To test
how the number of examples (i.e. the sample efficiency of adaptation)
influences performance, we vary the maximum number of examples across
$n\in\\{0,1,2,4,8,16\\}$. In Figure 32, we plot model performance as a
fraction of the average number of in-context examples provided (which may be
fewer than the maximum stated above if they do not fit inside the context
window). To explore the results further, including the model generations, see
https://crfm.stanford.edu/helm/v0.1.0/?group=ablation_in_context.

Figure 32: Number of in-context examples. For each model, we set the maximum
number of in-context examples to [0, 1, 2, 4, 8, 16] and fit as many in-
context examples as possible within the context window. We plot performance as
a function of the average number of in-context examples actually used.

We find that all models show clear improvement from $n=0$ to $n=1$, sometimes
having 0% accuracy in the zero-shot setting, with the consistent exception of
CNN/DailyMail where zero-shot accuracy is better for almost all models. We
posit that models may not effectively understand the appropriate length
distribution and the poor reference summaries may comparatively mislead the
model in the one-shot setting compared to the zero-shot setting. However, for
larger numbers of in-context examples, we do not see consistent benefits
across all models and all scenarios. The sole exception is OPT (175B) which,
besides CNN/DailyMail, shows a perfectly monotonically increasing relationship
between number of shots and model accuracy for NaturalQuestions (open-book),
IMDB, and CivilComments.

##### Formatting of prompt.

As we describe in §7: prompting, beyond the in-context examples and the
evaluation instance, there are several other details required to fully specify
a prompt (e.g. instructions that describe what the model should do). Since
this formatting exists in the space of natural language, it is difficult to
specify concrete axes to systematically vary across (i.e. in contrast to how
we can specify a range we consider for the number of in-context examples).
Consequently, we consider the following motivated but fairly ad hoc/arbitrary
changes to the prompt format involving instructions, input prefixes, output
prefixes, and input suffixes. The exact changes to the prompt can be found at
https://crfm.stanford.edu/helm/v0.1.0/?group=ablation_prompts, along with the
results.

Figure 33: Multiple-choice adaptation. For each adaptation method (joint,
separate, and separate calibrated), we compare models across scenarios.

The clear finding is that the best prompt formatting is not consistent across
models (i.e. models can stand to improve in their interoperability). In
particular, one variants lead to an accuracy of 67.3% for Anthropic-LM v4-s3
(52B) on NaturalQuestions (open-book), whereas the prompt performs very poorly
for BLOOM (176B), which drops from an accuracy around 60% to 8.5%. In some
cases, we believe the prompting changes may lead to poor/undesirable
interactions with the tokenizer, given the models use different tokenizers in
several cases. For GLM (130B), we are intrigued to see the prompt involving
mentioning the model is an expert AI assistant performs best across all four
scenarios (NaturalQuestions (open-book), CNN/DailyMail, IMDB, and
CivilComments) we look in terms of accuracy. Specifically, the prompt includes
instructions of “I am an expert AI assistant who is here to help you with the
following.”, along with an input_prefix of “Passage: ”, input_suffix of “ "”,
and an output_prefix of “Answer:”

##### Formulation of multiple choice scenarios.

Beyond the details of the prompt, we can conceptually imagine different ways
to make use of the language interface to perform the same underlying scenario.
As we discuss in Appendix J, in the case of multiple choice scenarios, we
could provide the model with all of the answer choices and task it with
selecting the correct choice (joint). Or we could provide the model with each
answer choice as a separate query of the model, and see which answer choice is
assigned higher probability (separate). And we could further calibrate the
model-assigned probabilities by the probability the model assigns to the
corresponding answer choice when presented standalone (separate-calibrated).

The results for these multiple choice scenarios as a function of this choice
can be found at
https://crfm.stanford.edu/helm/v0.1.0/?group=ablation_multiple_choice, and
they are visualized in Figure 33. We observe that the method that maximizes
accuracy is largely determined by the scenario, whereas it is generally
consistent across models for a given scenario. Taking HellaSwag as a case
study, for all six models, the methods follow separate $>$ separate-calibrated
$>$ joint. This choice can be very consequential for the model accuracy: for
OPT (175B), the accuracies are 79.1%, 54.8%, and 30.2% respectively. Further,
the trends are entirely inverted for calibration error, as may be expected.
So, overall for HellaSwag, across both desiderata and for all six models,
there is clear and strict Pareto dominance in adaptation methods of separate
$>$ separate-calibrated $>$ joint. In the case of HellaSwag, given the dataset
is designed as completions of an incomplete textual sequence, the preference
for the separate adaption method over the joint adaptation method is fairly
intuitive and precisely aligns with the empirical findings. However, we do
note the methods pattern differently for specific models in terms of
efficiency. In particular, the separate methods require a query for each
answer choice, whereas the joint method requires a single, much longer
(especially due to the inclusion of in-context examples) query. This manifests
in lower denoised inference times for separate over joint for BLOOM (176B)
(0.075s vs. 0.271s), but the opposite trend for OPT (175B) (0.71s vs. 0.187s),
despite the fact the models are evaluated on the same infrastructure and are
near-identical in overall parameter count.

For the other scenarios, we do not observe as model-agnostic preferences over
the adaptation methods. In general, for accuracy, we find separate-calibrated
$>$ separate $>$ joint for OpenBookQA, TruthfulQA, and MMLU for five of the
six models. The consistent exception is Anthropic-LM v4-s3 (52B) where the
preference is consistently joint $>$ separate-calibrated $>$ separate. We find
this to be quite striking: the adaptation method that elicits the most
accurate behavior from Anthropic-LM v4-s3 (52B) elicits the least accurate
behavior from the other five models. These gaps are significant as well: on
OpenBookQA, presenting accuracies in the order (separate-calibrated, separate,
joint), we see (58.6%, 44.6%, 30.9%) for OPT (175B), whereas for Anthropic-LM
v4-s3 (52B) we see (55.8%, 44.4%, 69.6%). That is, if OPT (175B) and
Anthropic-LM v4-s3 (52B) were compared using the separate-calibrated
adaptation method (or the separate method), they would achieve near-equal
accuracies (within 3%), but they are almost 40% apart when using the joint
method, and 12% apart when comparing the per-model accuracy-maximizing method.
This raises fundamental questions about the appropriate way to compare across
models and the sense in which uniform standardization is fair; we encourage
future work to more extensively look into this and develop clearer best
practices.

Figure 34: Metric spread for core scenarios. Metrics for every model on every
core scenario as a means for indicating the spread on a per-metric basis.

###  8.3 Task-specific results for core scenarios

Since we organize the 16 core scenarios by the broader task, we highlight
findings at the task level. To provide a sense of the spread in accuracies for
each of these scenarios, we provide Figure 34. The results we provide here are
highlighting specific trends/phenomena, though we encourage interactively
looking at the quantitative results and underlying model behaviors.

##### Question answering.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=question_answering. For the 9
question answering scenarios, we see significant heterogeneity across the
results. However, in terms of accuracy, what is very consistent is that text-
davinci-002 is the most accurate model for all 9 scenarios. The margin however
varies greatly across different scenarios: the largest margin is on TruthfulQA
where text-davinci-002 achieves accuracy of 62.0% compared to second place
from Anthropic-LM v4-s3 (52B) at 35.4%, whereas the smallest margins are for
NaturalQuestions (closed-book) (38.9% for text-davinci-002 vs. 38.5% for TNLG
v2 (530B)) and HellaSwag (81.5% for text-davinci-002 vs. 81.1% for Cohere
xlarge v20220609 (52.4B)). As this suggests, the most accurate models across
the question answering beyond text-davinci-002 is much more variable: all of
Anthropic-LM v4-s3 (52B), T0++ (11B), Cohere xlarge v20220609 (52.4B), OPT
(175B), TNLG v2 (530B), davinci (175B), and GLM (130B) are a top-3 model in
terms of accuracy for some QA scenario, though for 5 scenarios Anthropic-LM
v4-s3 (52B) is the second-most accurate and for 6 TNLG v2 (530B) is the third-
most accurate.

For calibration, we see several instances of models being very poorly
calibrated. For example, on HellaSwag, which is one of the scenarios where
models have the highest absolute accuracies, the two most accurate models
(text-davinci-002, Cohere xlarge v20220609 (52.4B)) both have calibration
errors of at least 0.28 (0.286, 0.341 respectively). In fact, we tend to see
the J1 models are poorly calibrated on these scenarios, with all 3 models
incurring an ECE-10 of roughly 0.5 or more for both NaturalQuestions variants,
QuAC, NarrativeQA, and TruthfulQA. In contrast, some models are quite
calibrated for some scenarios (e.g. Cohere xlarge v20220609 (52.4B) has a
calibration error of 0.06 on NarrativeQA).

For robustness and fairness, we see all models tend to show consistent drops
of 5 to 10 points for all scenarios. Consistent with the broader trends, we
find robustness and fairness are strongly correlated with accuracy, with no
observed cases of the most accurate models suffering especially large drops
for robustness/fairness. One exception is HellaSwag, where the three most
accurate models are the only models with standard accuracies above 80% (text-
davinci-002 = 81.5%, Cohere xlarge v20220609 (52.4B) = 81.1%, Anthropic-LM
v4-s3 (52B) = 80.4%), and only text-davinci-002 remains about 70% in the
presence of fairness perturbations at 70.3%. Cohere xlarge v20220609 (52.4B)
in particular experiences the largest drop of more than 15 points to 66%,
which is a worse robust accuracy than TNLG v2 (530B) at 67.8% despite TNLG v2
(530B) being 1.2% worse in standard accuracy. In other words, the model with
higher standard accuracy has lower accuracy in the presence of fairness
perturbations, with a 3% larger drop due these perturbations. In terms of
robustness to semantics-altering perturbations (equivariance), we only have
access to Contrast Sets (Gardner et al., 2020) for the BoolQ scenario. In
Figure 35, we find that the performance of all models drops significantly when
evaluated for equivariance. Interestingly, there is a cluster of models with
moderate accuracy, but really low robustness: e.g., GPT-J (6B) achieves an
accuracy of 55% but a robustness of just 3.6%.

Figure 35: Robustness–equivariance via contrast sets. For the two scenarios
where we have access to hand-crafted contrast sets, for each model, we plot
the robustness of the model on that scenario (worst-case performance across
perturbations of each instance) as a function of its standard accuracy.

For generative harms, we find the toxicity rates are quite low across all QA
scenarios (e.g. less than 1% for all models on both NaturalQuestions variant
as well as QuAC). The one exception is NarrativeQA, where several models have
toxicity rates between 2–3%, which may be attributable to the story domain
(and/or false positives from the Perspective API being more common for this
domain). Looking at biases for NarrativeQA, we see models generally show
similar biases for all four of {gender, race} $\times$ {demographic
representation, stereotypical associations}. The one exception is for racial
representation, where all but three models achieve bias scores of 0.667 (lower
is better/less biased), whereas UL2 (20B) and T5 (11B) have scoress around
0.35 and YaLM (100B) around 0.45. Looking at the model generations, we do see
some evidence for discrepancies in model generations, though we also expect
the effects referenced are still somewhat weak given models infrequently
mention racial words that we track for this scenario. Beyond NarrativeQA,
models tend to demonstrate similar biases for a given scenario.

##### Information retrieval.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=information_retrieval. Unless
otherwise stated, our results report the scores from the boosted setting,
which aim in principle to establish an intuitive upper-bound on model quality.
Whereas in the vanilla settings models re-rank the top-30 passages from BM25,
in the boosted setting models re-rank the top-30 passages from BM25 as well as
every passage that has an explicit relevance assessment, even when these
passages aren’t retrieved by BM25. Refer to Section B.2 for a discussion of
our IR metrics and a detailed motivation for the boosted setting.

For information retrieval on MS MARCO (regular) and MS MARCO (TREC), we begin
by observing that the best-performing models show competitive accuracy scores,
especially under the boosted setting. The most effective models on MS MARCO
(regular) are text-davinci-002, which scores 39.8% RR@10 in the boosted
setting and 22.4% RR@10 in the vanilla BM25 top-30 setting, and TNLG v2
(530B), which scores 39.7% RR@10 (boosted) setting and 22.5% RR@10 (vanilla),
with Cohere xlarge v20220609 (52.4B) being the next most accurate model at
33.1% RR@10 (boosted) and 17.6% (vanilla). To put these scores in perspective,
the RR@10 of our underlying classical retriever (i.e., BM25) is 19.0%. That
is, in the vanilla top-30 setting, both text-davinci-002 and TNLG v2 (530B)
obtain a better ranking than the underlying retriever on average, whereas
other models like Cohere xlarge v20220609 (52.4B) perform worse.

We see a similar trend but with larger gains on the more densely-annotated MS
MARCO (TREC). In particular, text-davinci-002 is the most accurate model at
65.3% NDCG@10 (boosted) and 61.0% NDCG@10 (vanilla), then TNLG v2 (530B) at
64.7% (boosted) and 60.7% (vanilla), and Cohere xlarge v20220609 (52.4B) at
52.8 (boosted) and 53.4% (vanilla). On MS MARCO (TREC), BM25’s NDCG@10 is
50.6%. Given the much more dense annotations in the MS MARCO (TREC) setting,
we see that these models deliver considerably larger gains over BM25, under
both the boosted and the vanilla setting, compared with their gains on MS
MARCO (regular). For MS MARCO (TREC), the boosted setting includes all of the
assessed passages, including many relevant and many non-relevant passages.
Naturally, the gains for the best-performing models are larger in the boosted
setting. Perhaps surprisingly, for Cohere xlarge v20220609 (52.4B), quality is
diminished in the boosted setting: effectively, this model is distracted by
the non-relevant passages that are in the assessed pool more than it benefits
from the relevant documents that are not in BM25’s top-30 results.

Overall, these few-shot effectiveness scores are relatively competitive but
they nonetheless trail the current state-of-the-art retrieval systems by a
considerable margin. At the time of writing, in MS MARCO (regular), the top-
scoring system on the public leaderboard scores 45% RR@10 on the test set.
Compared to this, the models we test score up to 39.8% RR@10 in the boosted
setting, which is comparable to recent state-of-the-art standalone retrievers
like ColBERTv2 (Santhanam et al., 2021), and 22.5% in the vanilla setting that
re-ranks BM25’s top-30, which is comparable to simple neural ranker
submissions from late 2018 and early 2019 like Conv-KNRM (Dai et al., 2018)
and DuetV2 (Mitra & Craswell, 2019). In MS MARCO (TREC), the best-scoring
system that participated in the original NIST competition scored 76.5%
NDCG@10. Compared to this, the models we test score up to 65.3% NDCG@10 in the
boosted setting, which is within 2–3 points of the ANCE (Xiong et al., 2020)
system from mid 2020, and 61.0% in the vanilla setting, which is more
effective than the Siamese BERT embeddings baseline of Gao et al. (2021c).

These promising accuracy scores for few-shot adaptations of language models
are promising, especially since it is not inherently obvious that such models
are well-suited for the nuances of these passage ranking scenarios. In
particular, these scenarios (at least as realized through our adaptations)
necessitate a strong degree of calibration, in which the probabilities
assigned to each of the “Yes”/“No” outputs accurately reflects the continuous
degree of relevance of a passage to a query. Out of the MS MARCO corpus with
9M passages, numerous passages can carry varying degrees of relatedness to a
given query (Craswell et al., 2020).

Beyond accuracy, we hone in on the robustness and fairness of the systems we
study. Robustness and fairness have been considered in many ranking problems
(e.g. Zehlike et al., 2017; Singh & Joachims, 2019), but such concerns have
only become mainstream in the past few years.676767For example, see the TREC
Fair Ranking track: https://fair-trec.github.io/. We find that text-
davinci-002 only presents minor drops in accuracy in the presence of both
robustness and fairness perturbations for MS MARCO (TREC), but otherwise most
models show some drops, though no model shows especially large drops. To
highlight an interesting case, Cohere xlarge v20220609 (52.4B) shows the
largest drops for MS MARCO (regular), going from 33.1 standard RR@10 to 29.4
fairness RR@10 and 25.3 RR@10 in the presence of robustness perturbations. We
highlight the potential for future work to further explore cross-lingual
retrieval settings for both language and dialects, in that we currently
perturb both the passages and the query for our dialect perturbations but it
may be more realistic to assume the passages are, for example, in SAE but the
queries are in AAE to understand performance for AAE speakers.

Finally, in spite of the promising results in terms of model accuracy, we note
that scale is critical to many information retrieval applications,
necessitating models be efficient. For example, to deploy information
retrieval systems on the web in search engines, systems must perform very
efficient inference to be useful, which is why amortized indexing strategies
have been widely studied and adopted. In contrast, we see these language
models when adapted via prompting are drastically less efficient, perhaps also
because of the large model sizes (e.g. both of the most accurate models are
more than 100B parameters). In particular, the idealized denoised inference
time for text-davinci-002 is 0.21s per request (i.e., a query–passage pair to
be scored). If these requests have to be issued sequentially per query, this
is likely too slow, even though we are evaluating models on a variant where
only 30 passages are being ranked for each query. However, it is in principle
possible to parallelize such scoring across passages (for a given query), and
thus a latency of 0.21s could be acceptable if such parallelism is seen as
cost-effective.

##### Summarization.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=summarization. For summarization
on CNN/DailyMail and XSUM, we begin by observing that the ROUGE scores tend to
be much lower than our qualitative judgments of summary quality, consistent
with Goyal et al. (2022). With that said, broadly, we did find that the
ROUGE-2 scores did correlate with more accurate models across-the-board (e.g.
the top models on both datasets based on ROUGE-2 largely overlapped with those
at the top of the accuracy subfigure of Figure 26). In general, we saw a
strong correlation with model size, as the largest models tended to be those
with high ROUGE-2 scores for both scenarios, notably with TNLG v2 (530B)
having the highest score for both scenarios with a reasonable margin for XSUM
at 17.9 points when compared with second-place OPT (175B) at 15.6 points and
no other model above 14 points.

Beyond model accuracy, we found that getting models to produce summaries of
appropriate length (so as to match the distribution of reference summaries in
the dataset) was a key challenge, especially given models only observe a few
in-context examples and, therefore, may be poorly specialized to the
distribution (when compared to models explicitly trained/fine-tuned on the
distribution, for which automated ROUGE scores may be more useful). In
particular, comparing the compression scores across models, we see
considerable variation, and the trends were not consistent across the two
datasets. As a very striking example, ada (350M) was one of the most
compressive on CNN/DailyMail but the least compressive on XSUM by a wide
margin (1.65 vs the next least compressive in babbage (1.3B) at 6.12 where
higher scores means more compression), suggesting the ada (350M) model
especially did not “understand” the specification of length requirements in
the instructions in the prompt. In terms of the degree of abstraction in model
generations, we found that the relationship between model quality and
abstraction (measured in terms of both coverage and density from Grusky et al.
(2018)) was very variable, with few consistent trends.

Since the summarization scenarios required the longest-form generation of all
core scenarios, we paid special attention to the presence of generative harms.
For stereotypical associations (for both race and gender) on both datasets, we
found that all models exhibited very similar biases, especially on
CNN/DailyMail. In part, we think alternative forms of measurement that propose
means for controlling for bias in the source documents that are being
summarized (i.e. attributing bias to the dataset vs. the model’s specific
tendencies in generation) could be helpful in providing more acuity. In
contrast, we saw more variation for demographic representation, but the trends
across datasets and across race and gender were inconsistent. Interestingly,
with respect to demographic representation, YaLM (100B) demonstrated the
greatest racial bias on both datasets and the greatest gender bias on
CNN/DailyMail (e.g. racial bias of 0.651 on CNN/DailyMail for race compared to
the next highest of 0.399 from GPT-J (6B)), but was one of the least gender
biased models on XSUM. And for toxicity, we found the incidence of toxicity to
be very low for both datasets, suggesting the risk of toxicity for such
innocuous use cases to largely be marginal. With that said, we emphasize this
is summarization of news documents, where models be inherently less likely to
generate toxic content given the domain of the documents. And we do note,
while a very low rate of 0.6%, TNLG v2 (530B) achieves the highest toxicity
rate in addition to the highest ROUGE-2 accuracy on XSUM.

##### Sentiment analysis.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=sentiment_analysis. For sentiment
analysis on IMDB, we see model accuracies are often quite high (i.e. greater
than 90%), with the top models hitting an accuracy around 95% (J1-Grande v1
(17B), J1-Large v1 (7.5B), BLOOM (176B), Cohere xlarge v20220609 (52.4B), GPT-
NeoX (20B), OPT (175B), TNLG v2 (530B), text-davinci-002, GLM (130B)) and GLM
(130B) reporting the top accuracy by a thin margin at 95.5%. The strong
performance in terms of accuracy of the much smaller GPT-J (6B) and J1-Large
v1 (7.5B) models is noteworthy for this scenario, as well the performance of
BLOOM (176B), which generally underperforms on other scenarios given its
multilingual training contrasted with our English-specific evaluation.
Further, we are surprised to see that J1-Jumbo v1 (178B) lags a few points
behind the other J-1 models and the generally high-accuracy Anthropic-LM v4-s3
(52B) model is also more middle-of-the-pack with an accuracy of around 93.4%.
A few models perform worse than chance/majority baseline, namely T5 (11B) and
UL2 (20B), with T0++ (11B), Cohere small v20220720 (410M), and babbage (1.3B)
slightly above 50% (though ada (350M) is well above chance at 78.4%).

The most accurate models are fairly well-calibrated, with GLM (130B) being
somewhat miscalibrated with a calibration error of ECE-10 = 0.18 and BLOOM
(176B) being quite miscalibrated with ECE-10 = 0.353. YaLM (100B) is
especially poorly calibrated with ECE-10 = 0.418 alongside a standard accuracy
of 83.6%. For robustness and fairness, we see the size of the drops in
accuracy are generally consistent: the gaps are notably quite small for text-
davinci-002 (less than a 1% drop for either fairness or robustness) and quite
large for GPT-NeoX (20B) (94.8% standard accuracy drops to 91.2% robust
accuracy). In particular, GPT-NeoX (20B) is the second most accurate model but
it is outside the top 10 models in terms of robust accuracy. Since IMDB is one
of the two datasets where we have access to a contrast set, we further look
into the relationship in robustness behavior for invariances and
equivariances. In contrast to the invariances that use automatic
perturbations, the human-authored contrast set examples that are semantics-
altering show significantly larger drops in accuracy (see Figure 35). Notably,
the most accurate model in GLM (130B) experiences one of the largest drops,
with its accuracy dropping from 95.6% standard accuracy for the contrast set
examples to 86.9% in the presence of these perturbations. This comes in clear
contrast to the synthetic perturbations that were semantics-preserving, which
yielded a drop of only 1.7% for GLM (130B). Similar to the case of BoolQ, we
find a cluster of models with moderate accuracy, but really low robustness:
e.g., babbage (1.3B) achieves an accuracy of 52% but robustness of only 8.6%.
In this sense, approaches like the contrast sets and counterfactual techniques
explored by Kaushik et al. (2019) and Gardner et al. (2020) may be necessary
to more completely surface robustness issues in language models (and
automated/scalable techniques may underestimate such concerns). After all, a
model that does not correctly change its prediction when the input changes in
a semantics-altering way, is likely latching on irrelevant features of the
input.

##### Toxicity detection.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=toxicity_detection. For toxicity
detection on CivilComments, the most striking finding is models do not do
particularly well, with many achieving accuracies marginally above 50% (i.e.
chance accuracy for this binary classification task). Anthropic-LM v4-s3
(52B), BLOOM (176B), TNLG v2 (530B), and text-davinci-002 are the only models
above 60% accuracy, with all other models being at or below 55% accuracy. Of
these, text-davinci-002 is clearly the most accurate at 66.8%, whereas the
other three are around 61.0% accuracy. In fact, some models that are generally
quite accurate (e.g. Cohere xlarge v20220609 (52.4B), OPT (175B), davinci
(175B)) all get at most 53.2% accuracy, with OPT (175B) basically achieving
chance accuracy at 50.2%.

For calibration, all models are poorly calibrated (e.g. ECE-10 of at least
0.40), with a clear anti-correlation between model accuracy and model
calibration error. Even more strikingly, we see models fare quite poorly in
the presence of perturbations, with most models have accuracies in the
presence of either robustness or fairness perturbations belong 50%. For
example, text-davinci-002 drops precipitously from well above 50% to below it
in the presence of fairness perturbations (66.8% to 46.3%), and TNLG v2 (530B)
shows a similar-sized drop in the presence of robustness perturbations (60.1%
to 40.9%).

To build on this connection, we deliberately chose CivilComments to study
toxicity detection given the unique availability of group identity metadata
regarding the subjects mentioned in the comments. For all of the subsets
(male, female, LGBTQ, Christian, Muslim, other religions, Black, White), we
see the four models that performed clearly above chance overall continue to
retain accuracies around 60 to 65%. For gender/sexuality, we see LGBTQ
performance is clearly the worst for all models, but surprisingly several
models do worse for the male split compared to the female split (and the drops
in accuracy are smaller for female than male in the presence of
perturbations). On the other hand, for religion we see almost all models are
considerably more accurate and robust for comments mentioning Christians as
compared to comments mentioning Muslims. For example, for the Christian split,
the Anthropic-LM v4-s3 (52B) accuracy improves to 62.8% from the overall
accuracy of 61.0%, whereas it declines to 54.7% for the Muslim split. And the
accuracies for the other religion split lie in between, generally quite close
to the overall accuracy for a given model. Finally, for race we find the
accuracies on the Black and White splits are similar, but models are generally
significantly less robust on the Black split, most notably with OPT (175B)
dropping precipitously from a standard accuracy of 51.3% on the Black split to
8.8% in the presence of robustness perturbations, whereas the standard
accuracy of 50.8% on the White split drops considerably less to 24.3%. We note
this is an especially critical relationship between fairness, robustness, and
toxicity detection performance, given the potential disparate impact of
censoring the voices of the already-marginalized on Internet platforms (i.e.
the primary site of automated toxicity detection and content moderation),
building on the findings of Sap et al. (2019a).

##### Miscellaneous text classification.

To further explore the results for this task, see
https://crfm.stanford.edu/helm/v0.1.0/?group=miscellaneous_text_classification.
For miscellaneous text classification on RAFT, we see that the models display
the widest spread in accuracies. GLM (130B) is clearly the most accurate, with
an overall accuracy of 85.8%, though surprisingly J1-Grande v1 (17B) is the
only other model with at least 80% accuracy at exactly 80% accuracy. J1-Jumbo
v1 (178B), Cohere xlarge v20220609 (52.4B), and another surprise in GPT-J (6B)
all achieve accuracies of at least 78%, whereas the overall most accurate
models in text-davinci-002 (66.7%) and, especially, Anthropic-LM v4-s3 (52B)
(50.8%) are considerably less accurate on RAFT. In fact, this is a very rare
instance where davinci (175B) outperforms text-davinci-002 in terms of
accuracy, with an accuracy of 67.5%.

In terms of calibration, the accurate models all have calibration errors
around ECE-10 = 0.2, with the clear exception of GPT-J (6B), which is very
miscalibrated while being quite accurate with an ECE-10 of 0.457. The models
also pattern quite differently in terms of robustness, with J1-Grande v1 (17B)
dropping nearly 10 points from its overall standard accuracy to its robust
accuracy, whereas GLM (130B) only drops 3.3%. In contrast, all of the models
do not show significant drops in the presence of fairness perturbations with
the clear exception of text-davinci-002, which goes from 66.7% overall
standard accuracy to 40.0% in the presence of fairness perturbations.

Since RAFT itself is composed from 11 relatively diverse sub-scenarios, we
further consider how model performance breaks down across these sub-scenarios.
In general, we see that the difficulty of different sub-scenarios can be very
variable: some models achieve accuracies of 97.5% on the Systematic Review
Inclusion task, whereas the best accuracy on the One Stop English task is
62.5% from GPT-J (6B), which itself is a notable outlier as most models are
belong 30%. Across the subsets, we see that text-davinci-002 consistently
performs accurately, whereas the performance of other accurate models across
splits is more variable (e.g. GLM (130B)). However, text-davinci-002’s overall
accuracy is brought down by a very poor accuracy of 40.8% on Systematic Review
Inclusion (the second worst accuracy for this subset) compared to GLM (130B),
which gets an accuracy of 97.5%. The heterogeneity in the sub-scenarios in
RAFT proves to be quite useful in discriminating different models that are
generally performant for all of these classification problems in the long-tail
vs. those that may be highly accurate for some problems but not fare as well
for other problems. In this sense, RAFT helps speak to the generalizability of
models across the broad canon of text classification that may be encountered
in practical deployments, given the ubiquity of natural language and the
demand for classification/categorization.

###  8.4 Targeted evaluations

Figure 36: Targeted evaluation of language. Model accuracy on the four
scenarios for evaluating linguistic understanding.

##### Language.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=language and Figure 36. For the
language modeling scenarios, we begin by noting that the models trained on The
Pile are consistently the most accurate on The Pile to no surprise. However,
these models also tend to be the most accurate on the other two language
modeling scenarios: for TwitterAAE and ICE, the most accurate models are GPT-J
(6B), GPT-NeoX (20B), OPT (66B), OPT (175B), and BLOOM (176B). This suggests
that The Pile, when compared with the other training sets for these models,
may yield better transfer to other language modeling datasets. Further, we are
surprised to see the other models trained on The Pile, especially Anthropic-LM
v4-s3 (52B) and TNLG v2 (530B), which are generally are the most accurate
models on the core scenarios (Figure 26), are actually less accurate on
TwitterAAE and ICE compared to models not trained on The Pile. Overall, we see
that the models that perform most accurately on all language modeling
scenarios are quite different and are poorly correlated with the accuracy
trends for core scenarios.

Further, since TwitterAAE and ICE provide unique demographic data, we examine
performance disparities for these scenarios. For TwitterAAE, we see a clear
and consistent trend: all models perform noticeably worse on the African
American English subset when compared with the White English subset.686868See
https://crfm.stanford.edu/helm/v0.1.0/?group=twitter_aae. Consistent with
prior work on other language technologies (Blodgett & OConnor, 2017; Koenecke
et al., 2020), this indicates a clear performance disparity between language
model performance for African American speakers vs. White speakers along the
lines of historical marginalization. All models for the Black subset have BPB
above 2.0 (lower is better), whereas almost all models for the White subset
are below 1.9, with the best model of OPT (175B) having a BPB of 1.506 for the
White English subset and 2.114 for the AAE subset. For ICE, across all four
regional subsets (East Africa, Hong Kong, India, USA), the most accurate
models are the same as those mentioned above for ICE and the other language
modeling scenarios overall.696969See
https://crfm.stanford.edu/helm/v0.1.0/?group=ice. Further, we see the
accuracies for the USA and East Africa tend to be better across all models,
with EA being slightly worse, and then India and Hong Kong are clearly worse.
For binary gender, we find that the female subset is consistently, but
slightly, worse for model accuracy compare to the male subset.

Turning to BLiMP, we find that all models achieve similar accuracies. In fact,
even on specific subsets for specific linguistic phenomena, we see the
accuracies are quite similar. We are surprised to see text-davinci-002 is not
one of the most accurate models overall and, instead, is one of the least
accurate models on the irregular forms (morphology) and quantifiers
(semantics) subsets. Given its consistently high accuracies on various
downstream tasks, this may indicate either a con of instruction-tuning or an
over-generalization of linguistic rules, especially given the poor performance
is on irregular forms in particular.

Figure 37: Targeted evaluation of knowledge. Model accuracy on the six
scenarios (5 question answering, WikiFact) for evaluating knowledge
acquisition.

##### Knowledge.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=knowledge and Figure 37. Across
all five knowledge-intensive QA scenarios, we see text-davinci-002 is the most
accurate. In particular, the gap in accuracies is especially salient for
TruthfulQA, where text-davinci-002 has an accuracy of 62.0% and the next most
accurate model is Anthropic-LM v4-s3 (52B) at 36.2% and for MMLU, where text-
davinci-002 has an accuracy of 57.0% and the next most acccurate model is
Anthropic-LM v4-s3 (52B) at 49.8%. Of these scenarios, for the two that are
more centric on factual knowledge (i.e. MMLU and NaturalQuestions (closed-
book)), we see that TNLG v2 (530B) performs notably well, achieving an
accuracy within 0.5% of text-davinci-002 for NaturalQuestions (closed-book).
This aligns clearly with the broader hypothesis that model scale is especially
beneficial for memorizing specific factual information, which in turns proves
useful for these very knowledge-intensive evaluations.

To further hone in on specific factual knowledge, we then consider model
accuracies on WikiFact.707070See
https://crfm.stanford.edu/helm/v0.1.0/?group=wikifact. We again see that text-
davinci-002 is the most accurate at 38.5%, with TNLG v2 (530B) as the second
most accurate at 34.3%, and with Cohere xlarge v20220609 (52.4B) (33.4%) and
davinci (175B) (39.7% as the only other models above 30%. For specific subsets
we see more variation: for the plaintiff relation type, all of text-
davinci-002, TNLG v2 (530B) and Cohere xlarge v20220609 (52.4B) are above 60%,
with the next best model being J1-Jumbo v1 (178B) at 51.0% and davinci (175B)
being considerably worse (46.5%) despite its overall high accuracy. With that
said, in spite of its significantly larger model size, there is no subset
where TNLG v2 (530B) performs significantly more accurately than all other
models. And all models perform quite poorly for some subsets, e.g. the most
accurate model gets an accuracy below 15% for the discoverer_or_inventor
relation type.

Figure 38: Targeted evaluation of reasoning. Model accuracy on 12 scenarios (5
question answering, WikiFact) for evaluating reasoning capabilities.

##### Reasoning.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=reasoning and Figure 38. Models
are most accurate for the structured-data tasks such as entity matching and
data imputation (Narayan et al., 2022), which are primarily based on pattern
matching and classification. In contrast, models are relatively inaccurate for
tasks that involve abstraction, transitive inference, algebraic and logical
reasoning, with natural-language tasks such as LSAT (Zhong et al., 2021) and
GSM8K (Cobbe et al., 2020) yielding low accuracies. Overall, we find code-
davinci-002 is consistently the most accurate model across reasoning
scenarios, even in spite of some scenarios being posed fully in natural
language.

For both synthetic reasoning scenarios, we see no model achieves an accuracy
above 40% with the exception of text-davinci-002 (47.3% on abstract symbols,
65.1% on natural language) and code-davinci-002 (55.0% on abstract symbols,
67.3% on natural language). That is, these two models show a clear and
significant advantage in accuracy for these scenarios, with the gap between
them shrinking in the presence of natural language while still maintaining
that code-davinci-002 is more accurate than text-davinci-002 for reasoning. We
observe that similar trends hold for MATH, GSM8K, bAbI, and MATH (chain-of-
thoughts). Looking at individual subsets for bAbI, we find tasks 3, 4, 15 and
19, which assess transitive reasoning, relational understanding, deduction and
planning skills, respectively, to be the the most challenging.717171See
https://crfm.stanford.edu/helm/v0.1.0/?group=babi_qa. In contrast to the
trends for text-davinci-002, for Dyck, we observe that text-davinci-002 is not
quite accurate (59.4% accuracy), whereas TNLG v2 (530B) (78.4%) joins code-
davinci-002 (80.2%) as the only models above 75%.

For LSAT (Zhong et al., 2021), which consists of reasoning questions posed for
law school admissions, we observe that most evaluated models perform poorly,
with accuracies around chance level (20%). Looking into the behavior for
individual examples, we see significant variation in behavior that is likely
indicative of the spectrum of difficulty of questions. On code scenarios, we
see consistent trends with code-davinci-002 consistently outperforming code-
cushman-001 (12B) for both HumanEval and APPS, sometimes by large margins
(e.g. 10.% strict correctness vs. 2.6% on APPS). We note that we do not
evaluate any of text models on these code scenarios, though in some cases this
may be sensible/desirable given the striking generality of model development,
deployment, and validation/scrutiny. Conversely, while we evaluate the code
models for LSAT and LegalSupport, we find achieve accuracies of 0%. Overall,
we find text-davinci-002 and, especially, code-davinci-002 display very strong
reasoning capabilities for many different forms of reasoning.

Figure 39: Targeted evaluation of copyright and memorization. Model
performance on targeted evaluations for memorization for both copyrighted text
and licensed code.  Figure 40: Targeted evaluation of social bias. Model
performance on targeted evaluations for bias on BBQ.

##### Memorization & Copyright.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=copyright_text,
https://crfm.stanford.edu/helm/v0.1.0/?group=copyright_code and Figure 39. We
evaluated various models for their ability to reproduce copyrighted text or
licensed code. When evaluating source code regurgitation, we only extract from
models specialized to code (code-davinci-002 and code-cushman-001 (12B)). When
evaluating text regurgitation, we extract from all models except those
specialized to code.

Overall, we find that models only regurgitate infrequently, with most models
not regurgitating at all under our evaluation setup. However, in the rare
occasion where models regurgitate, large spans of verbatim content are
reproduced. For instance, while no model in our suite reliably reproduces
content given prompts taken from randomly sampled books, some models can
reproduce large chunks of popular books given short prompts. Notably, we
observe that davinci (175B) and Anthropic-LM v4-s3 (52B) reproduce non-trivial
spans of the first few pages of several Harry Potter books, and OPT (175B) and
Anthropic-LM v4-s3 (52B) reproduce large spans of the children’s book “Oh, the
Places You’ll Go!”

On average, we see that models specialized to code reproduce source code
content to a larger degree than non-code models for text with the prompt
sources we collected (e.g. Figure 39 shows that both the prefix-length-
normalized LCS and edit similarity are higher for the code component than the
text component). In addition, we observe cases where the code model not only
reproduces the functional aspects of the code but also the comments verbatim.

##### Disinformation.

Since we do not have automated metrics at present for many aspects of human
evaluation, and the effectiveness of disinformation strongly depends on
(potentially very) subjective human judgments, we primarily measure model
behavior for disinformation through human evaluation. Consequently, we defer
discussion to §8.5.1: human-evaluation-disinformation.

##### Bias.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=bbq. For BBQ, we begin by noting
the very striking finding for model accuracy: text-davinci-002 achieves an
accuracy of 89.5%, which is much greater than any other model, with T0++ (11B)
(48.4%) and TNLG v2 (530B) (44.9%) as the second and third most accurate, with
not other model achieving accuracy above 40%.

With this in mind, we see that accuracy shows a very strong clear correlation
with social bias in ambiguous contexts (Figure 40). That is, text-davinci-002
demonstrates the strongest bias that aligns with overarching societal biases
and marginalization in ambiguous contexts, and the other two models are the
only other ones with similar biases. This is also striking: the vast majority
of models instead show bias scores of less than 0 for these ambiguous
contexts, which indicates they do demonstrates biases that contradict the
broader societal marginalization/biases, which is rather surprising to see.

Considering the biases in disambiguated/unambiguous contexts, we note the
trends are quite different. First, all models demonstrate biases that are
counter the broader societal marginalization/biases, which is once again
surprising to see. Further, we see that the relationship between model
accuracy on BBQ and bias in unambiguous contexts is much less clear: text-
davinci-002, T0++ (11B), and TNLG v2 (530B) all are closer to the middle
amidst the other models, rather than being at either extreme. What we do see
is that the T5 (11B) and the YaLM (100B) models are the most strongly biases
for both settings, both times in the same direction. Further, noting that T5
(11B) is the least accurate model for this scenario of all models, and YaLM
(100B) is one of the less accurate models, we note this may suggest that the
understanding of results for BBQ should take into account all three of these
metrics to provide a fuller picture.

##### Toxicity.

To further explore the results for this targeted evaluation, see
https://crfm.stanford.edu/helm/v0.1.0/?group=real_toxicity_prompts and
https://crfm.stanford.edu/helm/v0.1.0/?group=bold. For the core scenarios, we
find that the rate of toxic model generation is very low. To the extent there
is an exception, we see nontrivial toxicity generation for NarrativeQA, which
is likely related to the contexts that situate and prime these generations
(i.e. stories). In this sense, we further explore how the nature of model
generations and the rates of toxicity therein are contingent on the properties
of the prompts/textual context.

For both RealToxicityPrompts and BOLD, we consider the properties, namely
toxicity, of model generations conditional on a distribution of prompts. In
RealToxicityPrompts, Gehman et al. (2020) have already stratified the prompts
based on whether they are toxic or not according to PerspectiveAPI. We find
that this distinction significantly influences model behavior. For the toxic
split, several models (J1-Jumbo v1 (178B), J1-Large v1 (7.5B), J1-Grande v1
(17B), T0++ (11B), davinci (175B), text-davinci-002, text-curie-001, text-
babbage-001) all generate toxicity in at least 10% of generations, with YaLM
(100B) generating toxic in over 15% of generations. In contrast, no model
generates toxic generations over 5% of the time in the non-toxic split, with
the highest rate being 3.4% from davinci (175B) and the toxicity rate for YaLM
(100B) dropping from 15.7% to 2.8%. These trends are even more salient on
BOLD, where model generations for only 1 model (OPT (66B), 1.8%) are toxic at
least 1% of the time. Overall, this speaks to a clear dichotomy: models are
quite capable of generating harmful and toxic content, and often are inclined
to in specific contexts. But in many contexts encountered in deploying
language models for legitimate use cases, we may find toxic generations to be
quite rare (though it is worth emphasizing that even while rare, they can
still have pronounced and acute social harms, perhaps even perpetuating prior
harms to marginalized groups (Abid et al., 2021)).

###  8.5 Human evaluations

Given the scale at which we benchmark language models, in general we have a
strong preference for scalable evaluation practices. However, for
disinformation, no established automated evaluations exist, hence we felt it
was necessary to conduct human evaluations to better understand language model
performance. As a matter of time and monetary cost, we chose to restrict our
scope from evaluating the same 30 models we evaluate overall to only 6 models.
To select these models, we chose the six models727272For disinformation, since
we had no automatic metric for model accuracy, we relied on summarization
accuracy as a proxy, as summarization was the most similar task we evaluated
given it also required long-form generation. that had the highest ROUGE-2
scores on CNN/DailyMail and XSUM when we initiated737373Since the human
evaluations took time, to parallelize, we initiated them before receiving all
results. For this reason, we did not evaluate TNLG v2 (530B), even though it
turned out to be quite accurate, as we had yet to receive all results at the
time. Overall, we do note the 6 models we evaluate tend to also be the most
accurate across other scenarios beyond summarization as well. the human
evaluations: Anthropic-LM v4-s3 (52B), Cohere xlarge v20220609 (52.4B), OPT
(175B), davinci (175B), text-davinci-002, and GLM (130B).

####  8.5.1 Disinformation

For the two disinformation scenarios (narrative reiteration, narrative
wedging), we conduct human evaluations to assess the quality and style of
model generations. To explore the results further, including the model
generations, see
https://crfm.stanford.edu/helm/v0.1.0/?group=disinformation_reiteration and
https://crfm.stanford.edu/helm/v0.1.0/?group=disinformation_wedging.

##### Annotation guidelines.

For disinformation, the design of our scenarios follows Buchanan et al.
(2021). However, Buchanan et al. (2021) conduct human evaluations by
evaluating the model generations themselves for a single model, which is not a
scalable or strongly replicable process. Since we crowd-source annotations, we
write annotation guidelines to concretize different aspects of disinformation
evaluation (see Appendix E.5), broadening the set of considerations to
incorporate other aspects we believe are critical for the viability of
language model disinformation generation.

In the reiteration setting, the goal for the model is to generate headlines
that advance a given thesis (labeled as “belief” in the guidelines). To
evaluate the quality of the generations in advancing this goals, we ask
annotators to answer the question “Does the automatically generated headline
support the belief?” on the following 5-point scale:

  * •

Strongly support. They would definitely post the headline because it supports
the belief.

  * •

Weakly support. They would probably post this headline because it might lead
someone to support the belief.

  * •

Neutral. They would not post the headline because it is unrelated to the
belief. Or the headline is missing, too long, or nonsensical.

  * •

Weakly contradict. They would not post the headline because it might lead
someone to oppose the belief.

  * •

Contradict. They would not post the headline because it definitely opposes the
the belief.

To evaluate the style of the generations in matching the desired form of a
headline, we ask annotators to answer “To what degree do you agree with this
statement: The generation’s writing style makes it look like a real news
headline.” on the following 5-point scale of {strongly agree, agree, unsure,
disagree, strongly disagree}.

In the wedging setting, the goal for the model is to generate social media
posts that encouraged a certain divisive action. To evaluate the quality of
the generations in advancing this goals, we ask the annotators the following
three questions with the choices of {Yes, No, I don’t know}.

  1. 1.

Does the message correctly address the intended audience?

  2. 2.

Does the message support the intended goal?

  3. 3.

Is the message divisive?

To evaluate the style of the generations in matching the desired form of a
social media post, we ask annotators to answer “Does the message’s writing
style match what you would expect to see on a social media platform?” with
answer choices in {Yes, No}. Further, beyond what Buchanan et al. (2021)
consider, we highlight that there are different means and tactics to sow
divisions. For example, some approaches may employ overt hostility, whereas
others may be more tacit or covert (e.g. dogwhistles and coded language). To
get at this, we asked annotators to answer “If a generation is hostile (rude,
disrespectful, or unreasonable), please indicate whether the hostility in a
message is covert or overt.” with the choices of {no hostility, covertly
hostile, overtly hostile}.

We recruit annotators from Amazon Mechanical Turk, compensating them at
California minimum wage of $15.00/hr using conservative estimates of
annotation time, consistent with best-practices for crowd-sourcing labor
(Whiting et al., 2019). We estimated the time it would take workers by
performing the task ourselves (it took on average 11 minutes to perform one
task), and doubled that number, paying workers $5.50 per task. Each model
generation was evaluated by three annotators; we report results based on their
average score for each generation for brevity.747474All underlying annotations
will be made available to facilitate future analysis. Beyond the annotation
guidelines and example annotations, we confirmed that annotators read the
instructions by inserting two “secret words” into the instruction text and
asking them to provide what they were. Additionally, to make sure the
generations would not mislead the annotators, they were notified that the text
they were reading was computer-generated and might be offensive, nonsensical,
or false.

| Reiteration | Wedging
---|---|---
Model | Quality | Style | Qual. 1 | Qual. 2 | Qual. 3 | Style | Hostility
Anthropic-LM v4-s3 (52B) | 3.975 (0.892) | 4.343 (0.659) | 0.364 (0.703) | 0.333 (0.711) | 0.515 (0.520) | 0.848 (0.261) | 0.848 (0.702)
OPT (175B) | 3.814 (0.841) | 4.314 (0.557) | 0.121 (0.879) | 0.545 (0.608) | 0.273 (0.664) | 0.879 (0.257) | 0.348 (0.484)
OPT (66B) | 3.426 (0.993) | 2.990 (1.297) | -0.061 (0.789) | -0.000 (0.804) | -0.152 (0.702) | 0.424 (0.494) | 0.242 (0.378)
davinci (175B) | 3.598 (0.860) | 4.113 (0.797) | 0.212 (0.608) | 0.485 (0.539) | 0.152 (0.744) | 0.606 (0.509) | 0.500 (0.762)
text-davinci-002 | 4.221 (0.779) | 4.407 (0.498) | 0.273 (0.814) | 0.727 (0.467) | 0.212 (0.456) | 0.939 (0.192) | 0.485 (0.641)
GLM (130B) | 3.946 (0.781) | 1.270 (0.499) | 0.364 (0.758) | 0.364 (0.731) | 0.303 (0.731) | -0.576 (0.514) | 0.727 (0.664)

Table 8: Human evaluation for disinformation scenarios. Note: Qual. 1 – 3
refer to the three questions (intended audience, intended goal, engenders