division) discussed in the prose for measuring quality for wedging. Values are
mean scores and values in parentheses are standard deviations of scores.
Reiteration values are in the range from 1 to 5, while wedging values are
between -1 to 1, except for Hostility, which is rated from 0 to 2.

##### Results and discussion.

Table 8 displays the results of our human evaluation for disinformation. We
find that for the reiteration scenario, all models received average quality
scores above 3, indicating that they generated text that tended to support the
given thesis statements. When it came to style, there was greater variation
with text-davinci-002, Anthropic-LM v4-s3 (52B), OPT (175B), and even davinci
(175B) receiving scores above 4.0, but OPT (66B) and GLM (130B), received much
lower scores. text-davinci-002 and Anthropic-LM v4-s3 (52B) generate text that
supports the given thesis statements, and looks like real headlines. text-
davinci-002 significantly outperforms Anthropic-LM v4-s3 (52B) ($p=0.028$),
GLM (130B) ($p=0.028$), OPT (175B) ($p=0.002$), and davinci (175B)
($p=0.000)$, and Anthropic-LM v4-s3 (52B) significantly outperforms davinci
(175B) ($p=0.002$) when it comes to supporting the thesis.757575All
significance tests are paired bootstrapped tests with 10,000 samples, and
considered significant if $p<0.05$. Similarly, when it comes to whether the
generations match the style of headlines, text-davinci-002, Anthropic-LM v4-s3
(52B), and OPT (175B) all significantly outperform davinci (175B) and GLM
(130B), but not each other.

The results for the wedging scenario do not tell as clear of a story. There is
no statistically significantly better model when it comes to model generations
addressing the intended audience (Qual. 1), and all of the models received
ratings around $0$, with high standard deviations. For Qual. 2, which asked
about the generations supporting the intended goal, all models did were rated
slightly higher, with text-davinci-002 significantly outperforming all except
OPT (175B). For Qual. 3, which asked about whether the generations are
divisive, Anthropic-LM v4-s3 (52B) was the best at generating divisive text,
being rated significantly higher than text-davinci-002, davinci (175B), and
OPT (66B). For style, all models except GLM (130B) perform well, with text-
davinci-002 rated significantly higher than davinci (175B), OPT (66B), and GLM
(130B) (but not significantly higher than OPT (175B) or Anthropic-LM v4-s3
(52B)). GLM (130B) performs much worse than others, so we investigate why
qualitatively and notice that GLM (130B) often does not stop generations at
the END token. Finally, for evaluating hostility, none of the models are
overtly hostile in the average case—the average hostility ranking is below
1.0. Anthropic-LM v4-s3 (52B) and text-davinci-002 are the only models where
all annotators agree that at least one generation is overtly hostile. The
model rated most hostile, Anthropic-LM v4-s3 (52B), is only significantly more
hostile than OPT (175B) and OPT (66B).

Finally, in order for text generation systems to be useful for disinformation
operations, their generations have to be diverse — if there is too much
overlap among generation, it would be easier to detect. As a proxy for
diversity, we look at the automated measures of self-BLEU and entropy,
estimated across different generations.767676See
https://crfm.stanford.edu/helm/v0.1.0/?group=disinformation_reiteration and
https://crfm.stanford.edu/helm/v0.1.0/?group=disinformation_wedging. Some
models, such as the Cohere models and T5 (11B) have very high self-BLEU
scores, indicating that all of their generations are the same. (Note that the
entropies of these model’s sampling distributions are also quite low,
indicating less uncertainty during sampling.) On the other hand, the models
that were evaluated with human annotators had much lower self-BLEU scores. In
particular, Anthropic-LM v4-s3 (52B), the OPT models, the GPT-3 models, and
the Jurassic models have low self-bleu ($<10.0$), while the Instruct series
have much higher scores in comparison, indicating less diversity.

Put together, our human evaluation demonstrates that models are able to
effectively promote desired arguments that match a style of interest with
diverse generations, but targeted them toward specific audiences is still a
challenge. To situate these findings, we highlight the analysis of Goldstein
et al. (Forthcoming) in that the practical viability of language models for
disinformation hinges on their reliability and the (lack of) need for post-
editing. In our evaluations, generations that are annotated with low quality
and style scores both serve as signals that model generations need to be
edited. Further, we make explicit that our annotators are not disinformation
experts (they are crowd-workers on Amazon Mechanical Turk), and this might
overestimate the true utility of the model generations—trained annotators
might be able to find more fine-grained issues with the generations. On the
other hand, disinformation actors may be able to elicit strong performance
through fine-tuning or more sophisticated prompting, i.e. we should expect
further optimization and encouraging modeling of more adversarial/worst-case
behavior in future work as a result. And we do not currently evaluate all
language models (e.g. PaLM, Gopher), including especially models that are
designed specifically by malicious actors for disinformation generation.
Overall, we conservatively recommend our results be interpreted as a lower
bound of the current disinformation risk posed by language models.

##  9 Related work and discussion

##### The rise of language models.

Language modeling has a long-standing tradition of study across human language
processing and computational language processing (Shannon, 1948; Lounsburg,
1954; Goldman-Eisler, 1958; Baker, 1975b; a; Jelinek, 1976; 1990; Hale, 2001;
Levy, 2008; Merity et al., 2018; Radford et al., 2018; Devlin et al., 2019;
Brown et al., 2020; Chowdhery et al., 2022). Language modeling has also been
seen as a grand challenge for AI, most notably in the Hutter Prize and the
associated enwiki8 benchmark on data
compression.777777https://en.wikipedia.org/wiki/Hutter_Prize However, in
contrast to these prior framings, where language models were viewed as
standalone generative models, the models we study in this work instead are
better understood by situating language models in two broader contexts. First,
given the models function as adaptable foundations for the myriad scenarios
they are tested on, we view language models as foundation models in service of
building performant systems for these downstream use cases (Bommasani et al.,
2021). Second, as we demonstrate in our agnosticity to how the models are
constructed, we view language models as natural language interfaces (see Lee
et al., Forthcoming).

As Bommasani et al. (2021, §1.1) describe, the rise of language models in NLP
initiated the foundation model paradigm. Specifically, ELMo (Peters et al.,
2018), GPT (Radford et al., 2018), and BERT (Devlin et al., 2019) demonstrated
that pretraining using language modeling objectives could produce powerful
general-purpose representations for many downstream use cases, building on
prior evidence of the successes of pretraining (Mikolov et al., 2013;
Pennington et al., 2014). Further, these works, especially GPT and later GPT-2
(Radford et al., 2019), produced models with qualitatively better generative
capabilities than what had been seen previously.

Together, these formative works ushered in significant change in the status of
language modeling in NLP: language models rapidly became the substrate for
almost all modeling work, especially with the advent of open infrastructure
through Hugging Face Transformers (Wolf et al., 2019) and models developed for
languages beyond English (e.g. multilingual-BERT, XLM; Devlin et al., 2019;
Conneau & Lample, 2019). Since then, we have seen a proliferation of different
organizations building language models, often through conceptually similar
means, with a rapid growth in scale and resource-intensivity. Notably, some of
the models (e.g. TNLG v2 (530B)) we benchmark 1000$\times$ larger than ELMo
and BERT. These models can cost millions of dollars to train, requiring
extensive systems-level optimizations and dedicated large-scale compute
(Narayanan et al., 2021). These changes have also translated from research to
deployment: language models are directly exposed as commercial APIs or are
integrated into ubiquitous products (see Bommasani et al., 2022, §5) as part
of an emerging commercial ecosystem (Bommasani et al., 2021, §1.2).787878See
the emerging startup ecosystem for foundation models:
https://www.scalevp.com/blog/introducing-the-scale-generative-ai-index

##### Benchmarks in NLP.

Similar to language modeling, benchmarking has a long history in NLP. As Karen
Spärck Jones put it in her ACL Lifetime Achievement Award speech, “proper
evaluation is a complex and challenging business” (Spärck Jones, 2005). To
address this challenge, the practice of benchmarking rose to prominence as the
core methodology in the 1980’s and, especially, the 1990’s (see Liberman,
2010; Spärck Jones & Galliers, 1995). This transition was well-demonstrated by
initiatives such as the Message Understanding Conference (MUC; Grishman &
Sundheim, 1996) and the Text Retrieval Conference (TREC; Voorhees & Harman,
1998). And it coincided with a broader shift in the field towards statistical
and data-driven methods with large datasets (e.g. the Penn Treebank (Marcus et
al., 1999)) and new venues like the Conference on Empirical Methods for
Natural Language Processing (EMNLP, 1996).

More than a decade later, with the rise of deep learning in the 2010s
(Collobert & Weston, 2008; Turian et al., 2010; Collobert et al., 2011; Socher
et al., 2011a; b; Sutskever et al., 2011; Mikolov et al., 2013; Pennington et
al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015;
Vaswani et al., 2017), larger benchmarks such as SNLI (Bowman et al., 2015)
and SQuAD (Rajpurkar et al., 2016) were developed to provide both adequate
data for training systems in addition to evaluating systems. This parallels
concurrent developments in other areas of AI: most notably, the ImageNet
benchmark (Deng et al., 2009) that shaped modern computer vision. Like their
predecessors, these benchmarks assign each model a single score (e.g. the
SQuAD F1 score) to measure the accuracy for a single task.

As more general-purpose approaches to NLP grew, often displacing more bespoke
task-specific approaches, new benchmarks such as SentEval (Conneau & Kiela,
2018), DecaNLP (McCann et al., 2018), GLUE (Wang et al., 2019b), and SuperGLUE
(Wang et al., 2019a) co-evolved to evaluate their capabilities. In contrast to
the previous class of benchmarks, these benchmarks assign each model a vector
of scores to measure the accuracy for a suite of scenarios. In some cases,
these benchmarks also provide an aggregate score (e.g. the GLUE score, which
is the average of the accuracies for each of the constituent scenarios).

More recently, this theme of meta-benchmarks that assess model accuracy across
a range of tasks has continued (see Bommasani et al., 2021, §4.4.3): for
example, GEM (Gehrmann et al., 2021) provides a suite for natural language
generation tasks, XTREME (Hu et al., 2020) provides a suite for tasks spanning
numerous languages, and GEMv2 (Gehrmann et al., 2022a) provides a suite for
generation across languages. This approach is also the dominant approach to
language model evaluation,797979We note Efrat et al. (2022) as a very recent
counter-example to this trend that takes a much more minimalistic and succinct
unit-testing perspective. often with even broader collections: Brown et al.
(2020) popularized the approach in their work on GPT-3, where they evaluated
on 42 datasets. Indeed, this is the approach used in all the works that
introduced models we evaluate in this work. Efforts like the EleutherAI
Language Model Evaluation Harness (Gao et al., 2021b), HuggingFace’s Evaluate
library (von Werra et al., 2022), and Big-Bench (Srivastava et al., 2022) have
centralized and expanded these evaluations into systematic repositories.

Situated against this landscape, what differentiates our work is our holistic
approach, which manifests in both our benchmark design process and our
concrete benchmark. HELM is the byproduct of an explicit two-step process: we
taxonomize the space for language model evaluation, structured around use
cases (scenarios) and desiderata (metrics), and then systematically select
points in a way that reflects our priorities. This makes explicit the
aspiration, the concrete benchmark, and, consequently, what our benchmark
lacks that we should aspire to evaluate. More simply, our concrete benchmark
differs from both traditional benchmarks like ImageNet that assign a single
score (i.e. the ImageNet accuracy) and meta-benchmarks like GLUE that assign a
score vector (i.e. the accuracies on the GLUE datasets) to each model.
Instead, we assign a score matrix to each model: for each use case, we report
scores across several desiderata (e.g. accuracy, calibration, robustness,
fairness, efficiency).

Independent of the fact we measure holistically, one may wonder what the
relationship is between the scenarios we select and those evaluated in prior
works. To help understand this relationship, in Appendix F, we document the
scenarios that were evaluated for in past work (e.g. the scenarios evaluated
by Chowdhery et al. (2022) in the PaLM paper or by Gao et al. (2021b) in the
EleutherAI Language Model Evaluation Harness) as well as past results for the
models we evaluate on our scenarios (e.g. the HellaSwag accuracy reported by
Brown et al. (2020) in the GPT-3 paper). Further, to build on BIG-Bench
specifically, we highlight that our codebase integrates all BIG-Bench
scenarios, augmented with metrics beyond accuracy and the ability to evaluate
all models we support. We emphasize that at this time, no common standard
exists for language modeling evaluation, especially as the capabilities,
harms, and limitations of these models are still being understood through the
ongoing design of evaluations. We believe that establishing such a standard is
necessary for the ecosystem to mature, and that holistic approaches are
integral for building just standards.

##  10 What is missing

One of our three requirements for holistic evaluation is the recognition of
limitations: holistic evaluation should foreground where what is implemented
falls short of the ambition of evaluating models in their totality. Our
benchmark foregrounds the limitations of our current benchmark by design: our
benchmark is a subset of a pre-specified taxonomy. That is, the difference
between what is in the taxonomy and what is in the benchmark identifies what
we currently miss.

While it is useful to articulate what is missing, given the immense scale of
the use cases and desiderata we could have for language models, we believe it
is necessary to have clear priorities on how to navigate the space of what we
lack. These priorities are deeply subjective: many compelling arguments can be
presented to increase focus on any specific region of the design space for
language models and language model evaluation. Indeed, this is also made clear
through the diversity of different works happening in parallel throughout the
AI community to evaluate language models.

In this section, having conducted our holistic evaluation, we reflect on what
we believe should be prioritized based on this experience. That is, we
identify specific regions of the language model evaluation design space which
we hope the community will improve, either because we feel HELM currently only
scratches the surface or because we believe these concepts have been largely
neglected in the broader NLP and AI communities. To organize this, we consider
how HELM can be improved going forward across the five axes that we discuss in
this work: (i) scenarios, (ii) metrics, (ii) targeted evaluations, (iv)
models, and (v) adaptation.

###  10.1 Missing scenarios

Since we define scenarios in terms of tasks, domains, and languages, we begin
by considering what we miss for each of these. For tasks, we emphasize that we
deliberately chose to prioritize classical user-facing tasks, but other tasks
also can confer significant societal impact, even if they are not user-facing
(e.g. syntactic parsing and natural language inference). Beyond this, even
while our focus is oriented by societal impact in task selection, other tasks
may be more discriminative in identifying specific properties, limitations, or
capabilities of models (e.g. natural language inference often plays such a
diagnostic role, which may also align with our component skill on linguistic
capabilities. We also explicitly acknowledge certain tasks that we do not
currently implement as an oversight in terms of determining what was user-
facing, such as data-to-text generation which has many existing benchmarks
(e.g. Gardent et al., 2017; Novikova et al., 2017).

We especially highlight how the deployment of language models is giving rise
to entirely new tasks, beyond the standard purview of the NLP and AI research
communities. For example, several startups such as
Jasper.AI808080https://www.jasper.ai/ and Copy.AI818181https://www.copy.ai/
are deploying systems for copy-writing with language models as their
foundation.828282See https://www.scalevp.com/blog/introducing-the-scale-
generative-ai-index for more examples. Even further, we are seeing the rise of
new types of creative and generative applications like story and email
generation (see Lee et al., 2022b; Forthcoming). We believe it is incumbent on
the NLP and AI research communities to develop evaluations for these new use
cases, which not only may confer economic/commercial value but societal impact
going forward, as key instances where practical deployments can inform
research priorities.

For domains we highlight three priorities. First, for coverage of what, i.e.
the topic or genre of the text, many genres of strong practical economic and
societal consequence are not covered in HELM. As examples, we highlight
biomedical and clinical data, financial data, education data, and customer
service, pointing to how language technologies are increasingly
widespread838383For example, see the growing sub-communities in the NLP
research community for each of these areas, such as the FinNLP workshop for
financial technology and NLP at
https://sites.google.com/nlg.csie.ntu.edu.tw/finnlp-2022/. in these sectors
but we have no coverage of these anywhere in our benchmark. We do note we have
some coverage of legal data via LegalSupport, though the realism of the
entailment-based data for legal language technologies could stand to be
improved (cf. Guha et al., 2022). Second, for coverage of when, i.e. the time
period of the text, we highlight the contrast between the ever-changing nature
of language, world, and society with the relative rigidity of many current
language models in being able to update/edit knowledge (Lazaridou et al.,
2021). Early efforts to create such evaluations exist such as StreamingQA
(Liska et al., 2022). And symmetrically, while perhaps of less commercial
utility, we have seen language models being used for inquiry in the
computational social sciences and digital humanities, where evaluation of
performance on historic/ancient texts may be especially pertinent (see Bamman
& Burns, 2020; Yamshchikov et al., 2022). Third, for coverage of who, we note
that there are many standard demographic categories within the US where we
currently have no evaluation (e.g. age, socioeconomic status), and in fact the
metadata (i.e. speaker demographics) required to knowingly evaluate
performance for these subgroups often is unavailable (and may be in contention
with speaker privacy). Beyond this, we highlight two further nuances that
relate to speaker identity: (i) native vs. non-native speakers of English as
well as (ii) speaker demographics that are culturally situated in English-
speaking contexts beyond the US (e.g. caste in India as discussed by
Sambasivan et al. (2021) and Bhatt et al. (2022)), which are of specific
relevance for language sociolinguistically but are not standard US demographic
groups.

To build on the considerations of speaker identity, we further note that we
scoped our evaluation to English given the models we evaluate, but a clear
area for improvement is coverage of other languages, as many have called for
throughout the history of NLP (e.g. Bender, 2009; 2011; 2012; Joshi et al.,
2020). Currently, we believe we take important strides in coverage of English
varieties both in terms of varieties like African American English (mainly
through TwitterAAE but also through data augmentations following Ziems et al.
(2022)) and English spoken in different nations through ICE. But we emphasize
that a specific place to improve is to situate these evaluations in the
context of societally consequential use cases, whereas currently to ensure
some coverage we are relegated to measuring performance for these varieties in
language modeling (i.e. the setting where no labelled data is required), even
still requiring the extensive effort of many linguists to build ICE. We point
to a more general trend of how to improve not just the coverage of languages,
especially across typologically diverse languages, but to improve the cultural
sensitivity of language model and language technology evaluation (Hershcovich
et al., 2022).

###  10.2 Missing metrics

As we enumerate in Table 3, there are numerous desiderata that we could have
for AI systems and language technologies that we do not currently evaluate. In
large part, what we currently measure in HELM reflects what is feasible given
the the information and access we have to language models. Further, given
language models may be integrated in broader systems, we do not currently
taxonomize nor evaluate desiderata (e.g. physical safety when language models
serve as interfaces for robotic control and manipulation) of these broader
systems. For the desiderata we do taxonomize, beyond having to improve model
access to adequately measure them, we specifically highlight (i) user
experience, given models increasingly function as user interfaces (see Lee et
al., Forthcoming), (ii) linguistic plausibility, given models show language
capabilities that invite comparison with humans (see Linzen, 2020), and (iii)
provenance/credibility, given current modeling approaches often forsake
provenance in favor of other desiderata (see Metzler et al., 2021; Khattab et
al., 2021).

For the metrics we do measure, we highlight the following specific axes for
improvement. For both robustness and fairness when using perturbations,
appropriate estimation of when perturbations should be introduced (see Dhole
et al., 2021; Ziems et al., 2022) remains a challenge for constructing
synthetic data that is realistic. Further, when using contrast sets to measure
robustness equivariance and demographic metadata to measure performance
disparities, availability of these resources is the key challenge, especially
for more generative scenarios. For social bias, the validity of our measures
has largely not been verified, and the broader study of biases in model
generations when situated in realistic contexts (i.e. without assuming access
to specific metadata) remains quite open. Similarly for toxicity, we resort to
using the PerspectiveAPI in spite of its established flaws, and broader
protocols that would enable toxicity measurement that reflects the
perspectives of different social groups and individuals would be desirable
(Gordon et al., 2022). Finally, for measurement of training and inference
efficiency, improving the reliable disclosure of necessary information would
help ensure the accurate measurement of computational costs, environmental
emissions, and various runtimes.

###  10.3 Missing targeted evaluations

In terms of missing targeted evaluations, we first consider targeted
evaluations that we did not evaluate. For model capabilities, we note that
linguistic understanding, knowledge, and reasoning correspond to core
capabilities studied in the vast literature on human cognitive function.
Drawing inspiration from that literature, we note that planning is another
standard consideration that we do not explicitly study. While planning has
close connections with reasoning, and may also manifest in language contexts
through long-documents (where the coverage in HELM could stand to improve), we
emphasize that planning may also be better evaluated when considering
foundation models that ground language in other modalities (e.g. robotic
control). For model harms, we identify other forms of harm that we do not
currently evaluate. In terms of malicious use cases (akin to disinformation),
we note that language models could be used to automate spam generation and for
other forms of fraud, which are not currently widely studied in the NLP
community. Further, in terms of unintentional harms (akin to model biases), we
highlight that there are many other variants of harms, which differ from but
are related to bias, such as dehumanization (Mendelsohn et al., 2020),
denigration (Caines et al., 2018), and condescension (Wang & Potts, 2019) in
the growing canon of work on the harms of language technologies (Bender et
al., 2021; Weidinger et al., 2022; Rauh et al., 2022; Kirk et al., 2022)

In terms of improving the existing targeted evaluations, we highlight a
specific axis of improvement for each. For linguistic understanding, we
evaluate linguistic phenomena across syntax, semantics, and morphology, but do
not evaluate other levels of linguistic abstraction, highlighting pragmatic
and discourse as a specific area of focus. For knowledge, we evaluate domain,
commonsense, and world knowledge, but we can deepen the domain knowledge (e.g.
domain-specific knowledge bases beyond Wikidata) as well as expand to social
and cultural knowledge (e.g. SocialIQA; Sap et al., 2019b). For reasoning, we
evaluate many forms of reasoning, but we can broaden the domain-specific
reasoning beyond law (i.e. LegalSupport) as well as consider more language-
centric reasoning (e.g. multi-hop reasoning as in HotPotQA; Yang et al.,
2018). For copyright, we highlight the potential for broader evaluation of
privacy risk (e.g. personally identifying information), evaluation with
knowledge of the training (to better understand claims of memorization), and
harms of greater societal (e.g. plagiarism of student essays) or legal (e.g.
violation of copyright law) consequence. For disinformation, we highlight the
use of trained annotators as important for better characterizing true
disinformation risk, along with grounded user studies to observe the effect of
machine disinformation generation in advancing specific goals of persuasion
and deception influencing human behavior. Finally, for bias and toxicity, we
reiterate our general position of pushing evaluation towards more
contextually-situated measurement (see Rauh et al., 2022), which is why we
measured these as generative harms for all generative scenarios beyond
isolated evaluations.

###  10.4 Missing models

For models, we highlight three categories. First, there are models that we can
access that we do not evaluate, largely because they were released very close
to the release of this work (e.g. Galactica (Taylor et al., 2022) was released
on the same day as this work was released). Noteworthy examples, especially
given our findings on instruction-tuning, are Flan-T5 (Chung et al., 2022),
Tk-Instruct (Wang et al., 2022c), and BLOOMZ (Muennighoff et al., 2022). There
are newer versions of commercial APIs from AI21 Labs and Cohere which we have
yet to evaluate. We hope the current exclusion of these models will be merely
temporary, and that we will be able to reliably evaluate models that are
released openly.

Second, there are models that have been disclosed publicly but we do not have
access. Notably, at present, prominent models from DeepMind and Google (e.g.
Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022), LaMDA
(Thoppilan et al., 2022), PaLM (Chowdhery et al., 2022), and U-PaLM (Tay et
al., 2022b)) fall in this category. Consistent with the recommendations of
Liang et al. (2022), we believe research access to publicly benchmark and
document these models is necessary, even if the broader practices for model
release will differ across model providers. To this end, we recommend patterns
of developer-mediated access as potential middlegrounds to ensure these models
can be benchmarked transparently as a form of structured model access
(Shevlane, 2022).

Third, we recognize that there exist many language models, some which
potentially play consequential roles in society by underpinning high impact
products and services, that are entirely undisclosed. Given the growing
importance of language models, and their potential to undergird many different
language technologies without this being obvious to the public, we expect new
mechanisms for visibility into algorithm pipelines will be necessary (see
Bommasani et al., 2022). We raise this as an important open question for the
community: how do we ensure foundation models (including language models) are
transparently benchmarked when we do not know they exist and no existing
mechanism requires they be disclosed?

###  10.5 Missing adaptation

With the advances in language models and foundation models, we have witnessed
the rise of a sprawling class of different adaptation methods (see Bommasani
et al., 2021, §4.3). At present, these include a variety of gradient-free
prompting strategies (see Liu et al., 2022c) like chain-of-thoughts (Wei et
al., 2022c), parameter-efficient gradient-based methods (see He et al., 2022)
like adapters (Houlsby et al., 2019), and full gradient-based fine-tuning.
Beyond exploring any of these specific methods, we recommend work also explore
how interoperable these methods are across different scenarios, metrics, and
models so as to understand how best practices for adaptation should emerge. We
also emphasize these methods assume different levels of access to models and
may use different resources, which we could imagine tracking as means for
fairer comparison (Perez et al., 2021; Bommasani et al., 2021, §4.2). Finally,
we encourage work to explore model adaptation more expansively beyond specific
machine learning methods, such as new modes of interaction that empower humans
and broader forms of adaptation such as continual learning (see Bommasani et
al., 2021, §2.5, §4.3).

##  11 Limitations and future work

To understand how our work is limited, and therefore opportunities for future
work, we consider three categories: (i) our results, (ii) our benchmark
implementation, and (iii) our underlying benchmark design principles.848484We
note that there are other important limitations (e.g. what is missing) that we
have already discussed.

###  11.1 Limitations of results

For our results, we identify three key limitations: (i) the relevance for
practical use of language models, (i) the generalizability of the findings,
and (iii) the dependence on adaptation decisions.

##### Relevance for practical use.

In practice, language models are used not only for diverse scenarios but in
diverse contexts. In these contexts, language models are likely to be further
specialized (e.g. fine-tuned on much larger data than 5 examples from the
domain of interest). Consequently, our results should not be taken as
levelling the universal claim that models that perform well are always
desirable and models that perform poorly are always undesirable. For example,
GPT-J (6B) does not perform particularly well, but may be suitable in many
contexts due to its smaller size and the ease of fine-tuning (e.g. with fewer
hardware/compute constraints), so that the model can make use of more in-
distribution data. More broadly, we expect the totality of the results we
provide are not relevant for every practical use case: we anticipate
practitioners should first identify scenarios and metrics pertinent to their
use conditions, and then prioritize these scenarios/metrics in interpreting
the results of this benchmark.

##### Generalizability.

For our results to constitute generalizable findings, instances from the test
distribution should not be included in the (pre)training data for the model
(i.e. there should be no train-test contamination). However, as several works
have discussed (e.g. Brown et al., 2020; Dodge et al., 2021; Sanh et al.,
2021), given that the language models we consider are trained on massive,
multi-source, and incompletely-documented data (e.g. text scraped from the
Internet), it is difficult to directly determine if they are contaminated. We
document all known evidence from prior work of contamination in Appendix G,
though we acknowledge the extent to which models are contaminated and this
compromises the validity of our results largely remains unclear.

##### Adaptation.

We emphasize that our results, and qualitative trends drawn from the results,
depend on the choice and implementation of prompting as the adaptation
mechanism. In other words, we should not assume we will see the same trends if
models are fine-tuned or if prompts are explicitly optimized (Shin et al.,
2020; Zhou et al., 2022). This is compounded by the evidence that we show in
§8.2: prompting-analysis, along with several prior works (e.g. Bach et al.,
2022), that model behavior is very sensitive to prompt design. Further, due to
resource constraints, the sensitivity of the results to many lower-level
decisions (e.g. decoding hyperparameters) was not investigated and remains
unclear.

###  11.2 Limitations of HELM implementation

For our implementation, we see the main limitation as the lack of coverage of
what is missing. In fact, beyond what we discuss in §11: limitations, we
emphasize our holistic approach generalizes to the evaluation of any
foundation model: in future work, we can imagine specifying both taxonomies
and concrete benchmarks for forms of natural language beyond text (e.g. sign,
speech), and data modalities even beyond natural language (e.g. images, code,
videos, proteins) as in Tamkin et al. (2021; 2022). More generally, we
highlight our assumption of/evidence for the validity and reliability of our
scenarios and metrics.

##### Validity and reliability.

Many datasets can instantiate a scenario (i.e. agree in task and domain), but
differ enormously in the extend to which results reported on those datasets
are useful. In the parlance of measurement modeling (Loevinger, 1957; Messick,
1987; 1988; Jackman, 2008; Liao et al., 2021; Jacobs & Wallach, 2021), we
would like results we report to be valid (i.e. reflect the underlying
construct such as summarization of legal documents). While all the datasets we
use were introduced in works with some process for quality assurance, and the
datasets we introduce similarly have some process for quality assurance, we
note no unified standard has been set to ensure all datasets are sufficiently
valid. Consequently, the quality and useful of our benchmark is contingent on
this assumption: we encourage future work to interrogate the validity of our
datasets and to introduce protocols to help ensure the validity of future
datasets. Since benchmarks incentivize future work to build models that
improve on the benchmark, validity is especially critical in light of the oft-
cited Strathern’s Law (Strathern, 1997) — “When a measure becomes a target, it
ceases to be a good measure” (also see Goodhart, 1984; Linzen, 2020; Bowman &
Dahl, 2021).

Further, in addition to validity, measurement modeling and other approaches
for measure design emphasize the reliability of a measure (i.e. the measure
should not be overly sensitive to undesired sources of variation such as
properties of the specific annotators who labelled data). Akin to validity, we
lack uniform evidence for the reliability of our measurement. In particular,
we highlight the importance of significance testing for making meaningful
comparisons given we only run 3 random seeds (due to cost) for selecting in-
context examples and we evaluate on 1000 instances instead of the full
validation/test set for a given dataset. That is, both of these settings for
our evaluation may render some of our claims statistically insignificant; we
encourage future work to consider how to better address significance given the
scale of this evaluation.858585We do note that we perform paired bootstrap
tests for all model comparisons made in the human evaluations (§8.5: human-
evaluations).

###  11.3 Limitations of HELM design

Given the nature of our benchmark design, we highlight the important question
of aggregation (see Ethayarajh & Jurafsky, 2020; Ma et al., 2021). In
comparison to prior work, where a model would receive a single score (e.g.
SQuAD F1 accuracy) or a score vector (e.g. GLUE accuracies), we produce a more
complex collection of results per-model (i.e. a score matrix of scenarios
$\times$ metrics). We believe this is necessary to capture the complexity of
the artifacts we characterize (Liao et al., 2021): the generality of language
models and the plurality of desiderata we should require of such
systems.868686We this said, we note that using our holistic evaluation as an
oracle of sorts, one then may be able to identify minimal evaluations that
discriminate between different models. In other words, holistic, broad, and
expansive evaluations like HELM complement and possibly could inform succinct
unit-testing approaches like HANS (McCoy et al., 2019), CheckList (Ribeiro et
al., 2020) and LMentry (Efrat et al., 2022).

However, this complexity does come at a serious cost. For example, in
comparison to benchmarks like ImageNet or SQuAD, we cannot simply rank models
by accuracy to get a total order: we believe this correctly reflects the
trade-offs that different models impose over the evaluation space. In other
words, while it is possible for model A to be strictly better on every metric
for every scenario than model B (i.e. strict Pareto dominance), in almost
cases A is sometimes better and B is sometimes better when one is sufficiently
holistic/expansive. To designate A or B as better requires making a judgment
that (implicitly/explicitly) weighs the circumstances in which each is better
than the other.

Practically, by significantly increasing the volume of results we report for
each model, our benchmark could overload a consumer of the results, making it
difficult to interpret or act upon them. By providing structure (e.g.
taxonomizing scenarios and metrics, factorizing into core scenarios vs.
specific components), we hope to retain clarity as we provide nuance. Overall,
the detail of our benchmark exposes decision points for different stakeholders
to prefer one model over the other based on their values, preferences, and
circumstances (e.g. an organization deploying a model on mobile should assign
higher priority to the efficiency results).

We leave the question of aggregating model performance to a single number (or
a single number per scenario, or per metric) to future work. We do not believe
there exists a universal aggregation that satisfies all preferences, reflects
all values, or captures all circumstances appropriately. However, we do
believe single-number metrics, though reductive, are useful practical tools to
simplify decision-making. 878787By analogy, single-number metrics like GDP and
calories are very helpful in practice, albeit flawed and reductive in encoding
the true complexities of the economy and nutrition.

##  12 Conclusion

Language models have transformed AI, ushering in the paradigm of foundation
models. The reach of modern language models extends well beyond research, with
language models being rapidly productionized into consequential and ubiquitous
language technologies, which we expect to only increase in the near-term
future. We lack transparency on language models at present, which is
especially concerning given their rapid growth and burgeoning impact: as a
community, we do not understand language models in their totality. For this
reason, we have pushed for holistic evaluation in this effort, as we believe
holistic evaluation is a critical means for providing the necessary
transparency for language models.

Transparency begets trust and standards. Viewing benchmarks as models for
social change, given they orient the development of AI systems, our broader
objective is to transform foundation models from immature emerging
technologies to reliable tools that support human flourishing. With this
objective in mind, we recognize the history and trajectory of AI benchmarking
aligns with institutional privilege (Koch et al., 2021). Benchmarks set the
agenda and orient progress: we should aspire for holistic, pluralistic and
democratic benchmarks (Birhane et al., 2022). Given the understated but
significant power of benchmarks to drive change, which in turn indicates
benchmark design confers power, we foreground our objectives for HELM along
with its limitations. We hope the community will interrogate, adopt, and
improve HELM going forward to actualize the ambition of holistic evaluation.
In this way, we hope holistic evaluations for language models and for other
classes of foundation models will give rise to useful, responsible, and
societally beneficial technology.

##### Acknowledgements.

We thank Alex Tamkin, Colin Raffel, Dan Jurafsky, Deep Ganguli, Douwe Kiela,
Emily Dinan, Eric Horvitz, Girish Sastry, Iason Gabriel, James Manyika, Jason
Wei, Jie Tang, Judy Shen, Miles Brundage, Neil Band, Nelson Liu, Opher Lieber,
Pang Wei Koh, Stella Biderman, Steven Cao, Susan Zhang, Teven Le Scao, and Yi
Tay for their valuable feedback on the manuscript. We thank Steven Cao and
Nelson Liu for guidance in the selection of core scenarios that we describe in
§3: core-scenarios, and Kathy McKeown for her guidance on the summarization
section. We thank Carlos Guestrin, Daniel Zhang, John Hewitt, Kawin
Ethayarajh, Lauren Gillespie, Mina Lee, Rob Reich, Rohan Taori, Sandra Luksic,
Shelby Grossman, and Yann Dubois for helpful feedback and support throughout
this project. We thank the CRFM community for helpful feedback on the effort
overall.

##### Model providers.

We thank the following individuals at their respective organizations for the
access, support, and/or credits required to evaluate their models:

  * •

AI21 Labs. Opher Lieber, Barak Lenz, Dan Padnos, Yoav Shoham

  * •

Anthropic. Ben Mann, Jackson Kernion, Deep Ganguli, Jack Clark, Dario Amodei

  * •

Cohere. Lewis Stott, Ellie Evans, Bill McCartney, Aidan Gomez

  * •

Microsoft and the Turing Academic Program. Payal Bajaj, Barun Patra, Ahmed H.
Awadallah, Saurabh Tiwary, Eric Horvitz

  * •

OpenAI. Lama Ahmad, Miles Brundage

We thank CoreWeave for providing API access to GPT-J (6B) and GPT-NeoX (20B)
for initial prototyping.

##### Funding and major compute.

We thank Google through the Stanford HAI-Google collaboration for providing
funding for this work, and especially Sebastian Gehrmann for overall feedback
on this effort. In addition, this work was supported in part by the AI2050
program at Schmidt Futures (Grant G-22-63429). We thank Together Computer
(which is backed by compute from Stanford University, ETH Zurich, Open Science
Grid, University of Wisconsin-Madison, and Crusoe Cloud) for providing the
infrastructure for benchmarking all the open models. Rishi Bommasani was
supported by the NSF Graduate Research Fellowship Program under grant number
DGE-1655618.

##### Reflexivity.

This work was developed at the Center for Research on Foundation Models
(CRFM), a center at Stanford University borne out of the Stanford Institute
for Human-Centered Artificial Intelligence (Stanford HAI). It is accompanied
by a policy brief on how evaluations substantively improve
transparency.888888See https://hai.stanford.edu/foundation-model-issue-brief-
series.

This work was made possible by two unique positions that CRFM occupies. First,
to develop the framework, build the benchmark, and execute the evaluation,
CRFM serves as an interdisciplinary hub for coordinating and uniting the many
authors of this work at Stanford University. Second, to acquire the access and
resources required to evaluate all the models in this work, CRFM leverages its
relationships with the associated foundation model developers.

We further highlight that this work continues a trend documented by Koch et
al. (2021) that many works that introduce (influential) benchmarks come from
privileged, high-resourced, and powerful institutions, such as Stanford (e.g.
Socher et al., 2013; Bowman et al., 2015; Rajpurkar et al., 2016; Srivastava
et al., 2021). We actively encourage community-driven approaches to benchmark
design, as well as mechanisms to center and drive adoption for benchmarks
developed at other institutions. This is of specific importance given how
benchmarks embed values and priorities (Ethayarajh & Jurafsky, 2020; Birhane
et al., 2022), which should be interrogated, as they are often developed by a
small subset of the AI community but go on to orient the work of the broader
AI community. For this reason, we are very explicit about our design decisions
(e.g. prioritizing user-facing tasks in scenario selection) and directly
foreground the limitations of our work with respect to the breadth of
considerations in the AI community. By fully releasing the open-source tools
and raw model predictions, we invite everyone to contribute further scenarios,
metrics, and models as we evolve HELM to better reflect the values of the AI
community.

## References

  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G Robinson.  Roles for computing in social change.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pp. 252–260, 2020.
  * Abedjan et al. (2016) Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang.  Detecting data errors: Where are we and what needs to be done?  _Proceedings of the VLDB Endowment_ , 9(12):993–1004, 2016.
  * Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou.  Persistent anti-muslim bias in large language models.  _arXiv preprint arXiv:2101.05783_ , 2021.
  * Aggarwal & Zhai (2012) Charu C. Aggarwal and ChengXiang Zhai.  A survey of text classification algorithms.  In _Mining text data_ , pp. 163–222. Springer, 2012.
  * Alex et al. (2021) Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller.  Raft: A real-world few-shot text classification benchmark.  In J. Vanschoren and S. Yeung (eds.), _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1, 2021\.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/ca46c1b9512a7a8315fa3c5a946e8265-Paper-round2.pdf.
  * Anthony et al. (2020) Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  _arXiv preprint arXiv:2007.03051_ , 2020.
  * Antoniak & Mimno (2021) Maria Antoniak and David Mimno.  Bad seeds: Evaluating lexical methods for bias measurement.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1889–1904, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.148.  URL https://aclanthology.org/2021.acl-long.148.
  * Arora et al. (2022) Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush S Bhatia, Ines Chami, Frederic Sala, and Christopher R’e.  Ask me anything: A simple strategy for prompting language models.  _ArXiv_ , abs/2210.02441, 2022.
  * Askell et al. (2021) Amanda Askell, Yushi Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan.  A general language assistant as a laboratory for alignment.  _ArXiv_ , abs/2112.00861, 2021.
  * Bach et al. (2022) Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush.  PromptSource: An integrated development environment and repository for natural language prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pp. 93–104, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-demo.9.  URL https://aclanthology.org/2022.acl-demo.9.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, S. El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, S. Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, C. Olah, Benjamin Mann, and J. Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv_ , 2022.
  * Baker (1975a) James K. Baker.  The dragon system – an overview.  _IEEE Transactions on Acoustic Speech Signal Processing_ , 1975a.
  * Baker (1975b) James K. Baker.  _Stochastic Modeling for Automatic Speech Understanding_ , pp. 297–307.  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1975b.  ISBN 1558601244.
  * Bamman & Burns (2020) David Bamman and Patrick J. Burns.  Latin bert: A contextual language model for classical philology.  _ArXiv_ , abs/2009.10053, 2020.
  * Barocas & Selbst (2016) Solon Barocas and Andrew D. Selbst.  Big data’s disparate impact.  _104 California Law Review_ , 3:671–732, 2016.
  * Bartolo et al. (2020) Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp.  Beat the AI: Investigating adversarial human annotation for reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 8:662–678, 2020.  doi: 10.1162/tacl_a_00338.  URL https://aclanthology.org/2020.tacl-1.43.
  * Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn.  The pushshift reddit dataset.  In _ICWSM_ , 2020.
  * Belinkov & Bisk (2018) Yonatan Belinkov and Yonatan Bisk.  Synthetic and natural noise both break neural machine translation.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Bender (2009) Emily M. Bender.  Linguistically naïve != language independent: Why NLP needs linguistic typology.  In _Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?_ , pp. 26–32, Athens, Greece, March 2009. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W09-0106.
  * Bender (2011) Emily M. Bender.  On achieving and evaluating language-independence in nlp.  _Linguistic Issues in Language Technology_ , 6, 2011.
  * Bender (2012) Emily M. Bender.  100 things you always wanted to know about linguistics but were afraid to ask*.  In _Tutorial Abstracts at the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Montréal, Canada, June 2012. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/N12-4001.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 610–623, 2021.
  * Benjamin (2019) Ruha Benjamin.  _Race after Technology_.  Polity Press, 2019.
  * Benkler et al. (2018) Yochai Benkler, Robert Faris, and Hal Roberts.  3\. Epistemic Crisis.  In _Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics_. Oxford University Press, 11 2018.  ISBN 9780190923624.  doi: 10.1093/oso/9780190923624.003.0001.  URL https://doi.org/10.1093/oso/9780190923624.003.0001.
  * Beukeboom & Burgers (2019) Camiel J. Beukeboom and Christian Burgers.  How stereotypes are shared through language: a review and introduction of the aocial categories and stereotypes communication (SCSC) framework.  _Review of Communication Research_ , 7:1–37, 2019.  URL https://research.vu.nl/en/publications/how-stereotypes-are-shared-through-language-a-review-and-introduc.
  * Bhatt et al. (2022) Shaily Bhatt, Sunipa Dev, Partha P. Talukdar, Shachi Dave, and Vinodkumar Prabhakaran.  Re-contextualizing fairness in nlp: The case of india.  _ArXiv_ , abs/2209.12226, 2022.
  * Bhattamishra et al. (2020) Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal.  On the ability and limitations of transformers to recognize formal languages.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7096–7116, 2020.
  * Biderman et al. (2022) Stella Rose Biderman, Kieran Bicheno, and Leo Gao.  Datasheet for the pile.  _ArXiv_ , abs/2201.07311, 2022.
  * Biggio et al. (2013) Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pp. 387–402, 2013.
  * Birhane et al. (2022) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao.  The values encoded in machine learning research.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, pp. 173–184, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450393522.  doi: 10.1145/3531146.3533083.  URL https://doi.org/10.1145/3531146.3533083.
  * Black et al. (2022) Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, M. Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, J. Tow, Ben Wang, and Samuel Weinbach.  GPT-NeoX-20B: An open-source autoregressive language model.  _arXiv_ , 2022.
  * Blodgett & OConnor (2017) Su Lin Blodgett and Brendan OConnor.  Racial disparity in natural language processing: A case study of social media African-American English.  _arXiv preprint arXiv:1707.00061_ , 2017.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor.  Demographic dialectal variation in social media: A case study of African-American English.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1119–1130, 2016.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5454–5476, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.485.  URL https://www.aclweb.org/anthology/2020.acl-main.485.
  * Blodgett et al. (2021) Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach.  Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1004–1015, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.81.  URL https://aclanthology.org/2021.acl-long.81.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? Debiasing word embeddings.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 4349–4357, 2016.
  * Bommasani & Cardie (2020) Rishi Bommasani and Claire Cardie.  Intrinsic evaluation of summarization datasets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 8075–8096, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.649.  URL https://aclanthology.org/2020.emnlp-main.649.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4758–4781, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.431.  URL https://aclanthology.org/2020.acl-main.431.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bommasani et al. (2022) Rishi Bommasani, Kathleen A. Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang.  Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?  In _Advances in Neural Information Processing Systems_ , 2022.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 2787–2795, 2013.
  * Bordia & Bowman (2019) Shikha Bordia and Samuel R. Bowman.  Identifying and reducing gender bias in word-level language models.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pp. 7–15, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-3002.  URL https://www.aclweb.org/anthology/N19-3002.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 2206–2240. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/borgeaud22a.html.
  * Borkan et al. (2019a) Daniel Borkan, Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Limitations of pinned auc for measuring unintended bias.  _arXiv preprint arXiv:1903.02088_ , 2019a.
  * Borkan et al. (2019b) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _World Wide Web (WWW)_ , pp. 491–500, 2019b.
  * Bowman et al. (2015) Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2015\.
  * Bowman & Dahl (2021) Samuel R. Bowman and George Dahl.  What will it take to fix benchmarking in natural language understanding?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4843–4855, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.385.  URL https://www.aclweb.org/anthology/2021.naacl-main.385.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * BSR (2018) BSR.  _Human Rights Impact Assessment: Facebook in Myanmar_.  BSR, 2018.
  * Buchanan et al. (2021) Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova.  Truth, lies, and automation.  2021\.
  * Bulian et al. (2020) Jannis Bulian, Jordan Boyd-Graber, Markus Leippold, Massimiliano Ciaramita, and Thomas Diggelmann.  Climate-fever: A dataset for verification of real-world climate claims.  In _NeurIPS 2020 Workshop on Tackling Climate Change with Machine Learning_ , 2020.
  * Buolamwini & Gebru (2018) Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on Fairness, Accountability and Transparency_ , pp. 77–91, 2018.
  * Burk (2019) Dan L Burk.  Algorithmic fair use.  _U. Chi. L. Rev._ , 86:283, 2019.
  * Caines et al. (2018) Andrew Caines, Sergio Pastrana, Alice Hutchings, and Paula Buttery.  Aggressive language in an online hacking forum.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pp. 66–74, Brussels, Belgium, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5109.  URL https://aclanthology.org/W18-5109.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186, 2017.
  * Cao et al. (2020) Qingqing Cao, Aruna Balasubramanian, and Niranjan Balasubramanian.  Towards accurate and reliable energy measurement of nlp models.  _arXiv preprint arXiv:2010.05248_ , 2020.
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li.  Faithful to the original: Fact aware neural abstractive summarization.  In _thirty-second AAAI conference on artificial intelligence_ , 2018\.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 19)_, pp. 267–284, 2019.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pp. 2633–2650, 2021.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_ , 2022.
  * Chalabi & Flowers (2017) Mona Chalabi and Andrew Flowers.  Dear mona, what’s the most common name in america, 2017.
  * Chen et al. (2022) Lingjiao Chen, Zhihua Jin, Sabri Eyuboglu, Christopher Re, Matei Zaharia, and James Y. Zou.  HAPI: A large-scale longitudinal dataset of commercial ML API predictions.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=CZeIOfCjMf.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374, 2021.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer.  QuAC: Question answering in context.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2018\.
  * Chomsky (1957) Noam Chomsky.  Syntactic structures.  _The Hague, The Netherlands_ , 1957.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, A. Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, B. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, M. Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, S. Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, D. Luan, Hyeontaek Lim, Barret Zoph, A. Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, T. S. Pillai, Marie Pellat, Aitor Lewkowycz, E. Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, K. Meier-Hellstern, D. Eck, J. Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling language modeling with pathways.  _arXiv_ , 2022.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc Le, and Jason Wei.  Scaling instruction-finetuned language models.  _ArXiv_ , abs/2210.11416, 2022.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019.
  * Clark et al. (2020) Peter Clark, Oyvind Tafjord, and Kyle Richardson.  Transformers as soft reasoners over language.  In _IJCAI_ , 2020.
  * Cobbe et al. (2020) Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Coleman et al. (2017) Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia.  Dawnbench: An end-to-end deep learning benchmark and competition.  2017\.
  * Collobert & Weston (2008) Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _International Conference on Machine Learning (ICML)_ , pp. 160–167, 2008.
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  Natural language processing (almost) from scratch.  _Journal of Machine Learning Research (JMLR)_ , 12:2493–2537, 2011.
  * Conneau & Kiela (2018) Alexis Conneau and Douwe Kiela.  Senteval: An evaluation toolkit for universal sentence representations.  _arXiv preprint arXiv:1803.05449_ , 2018.
  * Conneau & Lample (2019) Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 7059–7069, 2019.
  * Council (2018) Human Rights Council.  Report of the independent international fact-finding mission on myanmar. united nations., 2018.  URL https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_64.pdf.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees.  Overview of the trec 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_ , 2020.
  * Crenshaw (1989) Kimberlé Crenshaw.  Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics.  _University of Chicago Legal Forum_ , Vol.1989, Article 8, 1989.  URL https://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=1052&context=uclf.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu.  Convolutional neural networks for soft-matching n-grams in ad-hoc search.  In _Proceedings of the eleventh ACM international conference on web search and data mining_ , pp. 126–134, 2018.
  * DeGroot & Fienberg (1983) Morris H. DeGroot and Stephen E. Fienberg.  The comparison and evaluation of forecasters.  _Journal of the Royal Statistical Society. Series D (The Statistician)_ , 32:12–22, 1983.
  * Dehaene (2020) Stanislas Dehaene.  _How We Learn: Why Brains are Better than Any Machine …for now_.  Penguin Random House, 2020.
  * Dehghani et al. (2018) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser.  Universal transformers.  _arXiv preprint arXiv:1807.03819_ , 2018.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition (CVPR)_ , pp. 248–255, 2009.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Association for Computational Linguistics (ACL)_ , pp. 4171–4186, 2019.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta.  Bold: Dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, pp. 862–872, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383097.  doi: 10.1145/3442188.3445924.  URL https://doi.org/10.1145/3442188.3445924.
  * Dhole et al. (2021) Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard H. Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel-Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, and et al.  NL-Augmenter: A framework for task-sensitive natural language augmentation.  _arXiv preprint arXiv:2112.02721_ , 2021.
  * Diaz et al. (2022) Mark Diaz, Razvan Amironesei, Laura Weidinger, and Iason Gabriel.  Accounting for offensive speech as a practice of resistance.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pp. 192–202, Seattle, Washington (Hybrid), July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.woah-1.18.  URL https://aclanthology.org/2022.woah-1.18.
  * DiResta et al. (2022) Renée DiResta, Shelby Grossman, and Alexandra Siegel.  In-house vs. outsourced trolls: How digital mercenaries shape state influence strategies.  _Political Communication_ , pp. 1–32, 2022.
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 1286–1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.98.  URL https://aclanthology.org/2021.emnlp-main.98.
  * Du et al. (2021) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, M. Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, K. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  GLaM: Efficient scaling of language models with mixture-of-experts.  _arXiv_ , 2021.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  GLM: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 320–335, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.26.  URL https://aclanthology.org/2022.acl-long.26.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5055–5070, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.454.  URL https://aclanthology.org/2020.acl-main.454.
  * Durmus et al. (2022) Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto.  Spurious correlations in reference-free evaluation of text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1443–1454, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.102.  URL https://aclanthology.org/2022.acl-long.102.
  * Dutta et al. (2020) Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush R. Varshney.  Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org, 2020.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel.  Fairness through awareness.  In _Innovations in Theoretical Computer Science (ITCS)_ , pp. 214–226, 2012.
  * Ebrahimi et al. (2020) Javid Ebrahimi, Dhruv Gelda, and Wei Zhang.  How can self-attention networks recognize dyck-n languages?  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4301–4306, 2020.
  * Efrat et al. (2022) Avia Efrat, Or Honovich, and Omer Levy.  Lmentry: A language model benchmark of elementary language tasks.  2022\.  doi: 10.48550/ARXIV.2211.02069.  URL https://arxiv.org/abs/2211.02069.
  * El-Yaniv & Wiener (2010) Ran El-Yaniv and Yair Wiener.  On the foundations of noise-free selective classification.  _Journal of Machine Learning Research (JMLR)_ , 11, 2010.
  * EMNLP (1996) EMNLP.  Conference on empirical methods in natural language processing.  1996\.  URL https://aclanthology.org/W96-0200.
  * Ethayarajh & Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4846–4853, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.393.  URL https://aclanthology.org/2020.emnlp-main.393.
  * Evans et al. (2022) Owain Evans, Stephanie Lin, and Jacob Hilton.  How do new models from OpenAI, DeepMind and Anthropic perform on TruthfulQA?  _AI Alignment Forum_ , 2022.  URL https://www.lesswrong.com/posts/yYkrbS5iAwdEQyynW/how-do-new-models-from-openai-deepmind-and-anthropic-perform.
  * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong.  QAFactEval: Improved QA-based factual consistency evaluation for summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2587–2601, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.187.  URL https://aclanthology.org/2022.naacl-main.187.
  * Fogal et al. (2018) Daniel Fogal, Daniel W. Harris, and Matt Moss.  _New Work on Speech Acts_.  Oxford University Press, 2018.
  * Franceschelli & Musolesi (2022) Giorgio Franceschelli and Mirco Musolesi.  Copyright in generative deep learning.  _Data & Policy_, 4, 2022.
  * Friedman & Nissenbaum (1996) Batya Friedman and Helen Nissenbaum.  Bias in computer systems.  _ACM Transactions on Information Systems_ , 14(3):330–347, July 1996.  ISSN 1046-8188.  doi: 10.1145/230538.230561.  URL https://doi.org/10.1145/230538.230561.
  * Gabriel et al. (2022) Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi.  Misinfo reaction frames: Reasoning about readers’ reactions to news headlines.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3108–3127, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.222.  URL https://aclanthology.org/2022.acl-long.222.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yushi Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom B. Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _ArXiv_ , abs/2209.07858, 2022.
  * Gao et al. (2021a) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800GB Dataset of Diverse Text for Language Modeling.  _arXiv preprint arXiv:2101.00027_ , 2021a.  URL https://arxiv.org/abs/2101.00027.
  * Gao et al. (2021b) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation.  _Version v0. 0.1. Sept_ , 2021b.
  * Gao et al. (2021c) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan.  Complement lexical retrieval model with semantic residual embeddings.  In _European Conference on Information Retrieval_ , pp. 146–160. Springer, 2021c.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini.  The WebNLG challenge: Generating text from RDF data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pp. 124–133, Santiago de Compostela, Spain, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/W17-3518.  URL https://aclanthology.org/W17-3518.
  * Gardner et al. (2019) Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min.  Question answering is a format; when is it useful?  _arXiv preprint arXiv:1909.11291_ , 2019.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou.  Evaluating NLP models via contrast sets.  _arXiv preprint arXiv:2004.02709_ , 2020.
  * Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ , (16):E3635–E3644, 2018.  doi: 10.1073/pnas.1720347115.  URL https://www.pnas.org/doi/abs/10.1073/pnas.1720347115.
  * Gauthier et al. (2020) Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy.  SyntaxGym: An online platform for targeted evaluation of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pp. 70–76, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-demos.10.  URL https://aclanthology.org/2020.acl-demos.10.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_ , 2020.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pp. 96–120, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.gem-1.10.  URL https://aclanthology.org/2021.gem-1.10.
  * Gehrmann et al. (2022a) Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna V. Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qinqin Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja vStajner, Sébastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Yi Xu, Yisi Sang, Yixin Liu, and Yufang Hou.  Gemv2: Multilingual nlg benchmarking in a single line of code.  _ArXiv_ , abs/2206.11249, 2022a.
  * Gehrmann et al. (2022b) Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text.  _ArXiv_ , abs/2202.06935, 2022b.
  * Geifman & El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv.  Selective classification for deep neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Gillotte (2019) Jessica L Gillotte.  Copyright infringement in ai-generated artworks.  _UC Davis L. Rev._ , 53:2655, 2019.
  * Goel et al. (2020) Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré.  Model patching: Closing the subgroup performance gap with data augmentation.  _arXiv preprint arXiv:2008.06775_ , 2020.
  * Goel et al. (2021) Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré.  Robustness gym: Unifying the NLP evaluation landscape.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_ , pp. 42–55, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-demos.6.  URL https://aclanthology.org/2021.naacl-demos.6.
  * Gokaslan & Cohen (2019) Aaron Gokaslan and Vanya Cohen.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * Goldfarb-Tarrant et al. (2021) Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez.  Intrinsic bias metrics do not correlate with application bias.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1926–1940, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.150.  URL https://aclanthology.org/2021.acl-long.150.
  * Goldman-Eisler (1958) Frieda Goldman-Eisler.  Speech production and the predictability of words in context.  _Quarterly Journal of Experimental Psychology_ , 10(2):96–106, 1958.  doi: 10.1080/17470215808416261.  URL https://doi.org/10.1080/17470215808416261.
  * Goldstein et al. (Forthcoming) Josh A. Goldstein, Micah Musser, Girish Sastry, Renée DiResta, Matthew Gentzel, and Katerina Sedova.  Generative language models and automated influence operations: Emerging threats and potential mitigations.  Forthcoming.
  * Golshan et al. (2017) Behzad Golshan, Alon Halevy, George Mihaila, and Wang-Chiew Tan.  Data integration: After the teenage years.  In _Proceedings of the 36th ACM SIGMOD-SIGACT-SIGAI symposium on principles of database systems_ , pp. 101–106, 2017.
  * Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Goodhart (1984) Charles AE Goodhart.  Problems of monetary management: the uk experience.  In _Monetary theory and practice_ , pp. 91–121. Springer, 1984\.
  * Gordon et al. (2022) Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein.  Jury learning: Integrating dissenting voices into machine learning models.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , CHI ’22, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450391573.  doi: 10.1145/3491102.3502004.  URL https://doi.org/10.1145/3491102.3502004.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett.  News summarization and evaluation in the era of gpt-3.  _ArXiv_ , abs/2209.12356, 2022.
  * Greenbaum (1991) Sidney Greenbaum.  ICE: The international corpus of english.  _Engl. today_ , 7(4):3–7, October 1991.
  * Greenbaum & Nelson (1996) Sidney Greenbaum and Gerald Nelson.  The international corpus of english (ice) project, 1996.
  * Greenwald et al. (1998) Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz.  Measuring individual differences in implicit cognition: the implicit association test.  _Journal of personality and social psychology_ , 74(6):1464, 1998.
  * Grishman & Sundheim (1996) Ralph Grishman and Beth Sundheim.  Message Understanding Conference- 6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_ , 1996.  URL https://aclanthology.org/C96-1079.
  * Gruppi et al. (2022) Maurício Gruppi, Benjamin D Horne, and Sibel Adalı.  Nela-gt-2021: A large multi-labelled news dataset for the study of misinformation in news articles.  _arXiv preprint arXiv:2203.05659_ , 2022.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 708–719, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1065.  URL https://www.aclweb.org/anthology/N18-1065.
  * Guha et al. (2022) Neel Guha, Daniel E. Ho, Julian Nyarko, and Christopher Ré.  Legalbench: Prototyping a collaborative benchmark for legal reasoning.  _arXiv_ , abs/2209.06120, 2022.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In _International Conference on Machine Learning (ICML)_ , pp. 1321–1330, 2017.
  * Hahn (2020) Michael Hahn.  Theoretical limitations of self-attention in neural sequence models.  _Transactions of the Association for Computational Linguistics_ , 8:156–171, 2020.
  * Hale (2001) John Hale.  A probabilistic Earley parser as a psycholinguistic model.  In _Second Meeting of the North American Chapter of the Association for Computational Linguistics_ , 2001.  URL https://aclanthology.org/N01-1021.
  * Halevy et al. (2021) Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard.  Mitigating racial biases in toxic language detection with an equity-based ensemble framework.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_ , EAAMO ’21, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450385534.  doi: 10.1145/3465416.3483299.  URL https://doi.org/10.1145/3465416.3483299.
  * Hammarström et al. (2021) Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank.  _Glottolog 4.4_.  Leipzig, 2021.  doi: 10.5281/zenodo.4761960.  URL https://glottolog.org/accessed2021-08-08.
  * Hao et al. (2018) Yiding Hao, William Merrill, Dana Angluin, Robert Frank, Noah Amsel, Andrew Benz, and Simon Mendelsohn.  Context-free transductions with neural stacks.  _EMNLP 2018_ , pp. 306, 2018.
  * Harman (2013) Gilbert Harman.  _Rationality_.  John Wiley & Sons, Ltd, 2013.
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=0RDcd5Axok.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  {DEBERTA}: {DECODING}-{enhanced} {bert} {with} {disentangled} {attention}.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=XPZIaotutsD.
  * Hede et al. (2021) Anushree Hede, Oshin Agarwal, Linda Lu, Diana C. Mutz, and Ani Nenkova.  From toxicity in online comments to incivility in American news: Proceed with caution.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 2620–2630, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.225.  URL https://aclanthology.org/2021.eacl-main.225.
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _Journal of Machine Learning Research_ , 21(248):1–43, 2020.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al.  Measuring coding challenge competence with apps.  _Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track_ , 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps.  _ArXiv_ , abs/2105.09938, 2021b.
  * Hendrycks et al. (2021c) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations (ICLR)_ , 2021c.
  * Hendrycks et al. (2021d) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _NeurIPS_ , 2021d.
  * Hermann et al. (2015a) Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, pp. 1693–1701, Cambridge, MA, USA, 2015a. MIT Press.
  * Hermann et al. (2015b) Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015b.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard.  Challenges and strategies in cross-cultural NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 6997–7013, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.482.  URL https://aclanthology.org/2022.acl-long.482.
  * Hewitt et al. (2020) John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D Manning.  Rnns can generate bounded hierarchical languages with optimal memory.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1978–2010, 2020.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre.  Training compute-optimal large language models.  _ArXiv_ , abs/2203.15556, 2022.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Horvitz (2022) Eric Horvitz.  On the horizon: Interactive and compositional deepfakes.  _Proceedings of the 2022 International Conference on Multimodal Interaction_ , 2022.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  _arXiv_ , 2019.
  * Hovy & Yang (2021) Dirk Hovy and Diyi Yang.  The importance of modeling social factors of language: Theory and practice.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 588–602, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.49.  URL https://aclanthology.org/2021.naacl-main.49.
  * Howard et al. (2017) Ayanna M. Howard, Cha Zhang, and Eric Horvitz.  Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems.  _2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO)_ , pp. 1–7, 2017.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _arXiv preprint arXiv:2003.11080_ , 2020.
  * Index (2020) U.S. Copyright Office Fair Use Index.  Dr. Seuss Enters., L.P. v. ComicMix LLC.  https://www.copyright.gov/fair-use/summaries/drseuss-comicmix-9thcir2020.pdf, 2020\.
  * Jackman (2008) Simon Jackman.  _Measurement_.  The Oxford Handbook of Political Methodology. Oxford Handbooks, 2008.  URL https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199286546.001.0001/oxfordhb-9780199286546-e-6.
  * Jacobs & Wallach (2021) Abigail Z. Jacobs and Hanna Wallach.  Measurement and fairness.  In _Proceedings of the 2021 Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, New York, NY, USA, 2021. Association for Computing Machinery.  URL https://arxiv.org/abs/1912.05511.
  * Järvelin & Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen.  Cumulated gain-based evaluation of ir techniques.  _ACM Trans. Inf. Syst._ , 20(4):422–446, 2002\.  URL http://doi.acm.org/10.1145/582415.582418.
  * Jelinek (1976) Frederick Jelinek.  Continuous speech recognition by statistical methods.  _Proceedings of the IEEE_ , 64:532–556, 1976.
  * Jelinek (1990) Frederick Jelinek.  _Self-Organized Language Modeling for Speech Recognition_ , pp. 450–506.  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1990.  ISBN 1558601244.
  * Jernite et al. (2022) Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Isaac Johnson, Gerard Dupont, Jesse Dodge, Kyle Lo, Zeerak Talat, Dragomir Radev, Aaron Gokaslan, Somaieh Nikpoor, Peter Henderson, Rishi Bommasani, and Margaret Mitchell.  Data governance in the age of large-scale data-driven language technology.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, pp. 2206–2222, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450393522.  doi: 10.1145/3531146.3534637.  URL https://doi.org/10.1145/3531146.3534637.
  * Jia & Liang (2017) Robin Jia and Percy Liang.  Adversarial examples for evaluating reading comprehension systems.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2017\.
  * Jo & Gebru (2020) Eun Seo Jo and Timnit Gebru.  Lessons from archives: Strategies for collecting sociocultural data in machine learning.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, pp. 306–316, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450369367.  doi: 10.1145/3351095.3372829.  URL https://doi.org/10.1145/3351095.3372829.
  * Joachims (1998) Thorsten Joachims.  Text categorization with support vector machines: Learning with many relevant features.  In _European conference on machine learning_ , pp. 137–142. Springer, 1998.
  * Jones et al. (2021) Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang.  Selective classification can magnify disparities across groups.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury.  The state and fate of linguistic diversity and inclusion in the nlp world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Seattle, Washington, July 2020. Association for Computational Linguistics.  URL https://arxiv.org/abs/2004.09095.
  * Jung et al. (2019) Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy.  Earlier isn’t always better: Sub-aspect analysis on corpus and system biases in summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3324–3335, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1327.  URL https://aclanthology.org/D19-1327.
  * Jurafsky & Martin (2000) Daniel Jurafsky and James H Martin.  _Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition_.  Prentice Hall Prentice Hall, 2000.
  * Kaack et al. (2021) Lynn Kaack, Priya Donti, Emma Strubell, George Kamiya, Felix Creutzig, and David Rolnick.  Aligning artificial intelligence with climate change mitigation.  2021\.
  * Kachru et al. (2009) Braj B Kachru, Yamuna Kachru, and Cecil L Nelson.  _The handbook of world Englishes_ , volume 48.  John Wiley & Sons, 2009.
  * Kandpal et al. (2022) Nikhil Kandpal, Eric Wallace, and Colin Raffel.  Deduplicating training data mitigates privacy risks in language models.  _arXiv preprint arXiv:2202.06539_ , 2022.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei.  Scaling laws for neural language models.  _ArXiv_ , abs/2001.08361, 2020.
  * Kasirzadeh & Gabriel (2022) Atoosa Kasirzadeh and Iason Gabriel.  In conversation with artificial intelligence: aligning language models with human values.  2022\.
  * Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Khani & Liang (2020) Fereshte Khani and Percy Liang.  Feature noise induces loss discrepancy across groups.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Khashabi et al. (2022) Daniel Khashabi, Yeganeh Kordi, and Hannaneh Hajishirzi.  Unifiedqa-v2: Stronger generalization via broader cross-format training.  _ArXiv_ , abs/2202.12359, 2022.
  * Khattab et al. (2021) Omar Khattab, Christopher Potts, and Matei Zaharia.  A moderate proposal for radically better AI-powered Web search.  Stanford HAI Blog, 2021.  URL https://hai.stanford.edu/news/moderate-proposal-radically-better-ai-powered-web-search.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4110–4124, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.324.  URL https://aclanthology.org/2021.naacl-main.324.
  * Kilcher (2022) Yannic Kilcher.  Gpt-4chan, 2022.  URL https://github.com/yk/gpt-4chan-public.
  * Kirk et al. (2022) Hannah Rose Kirk, Abeba Birhane, Bertie Vidgen, and Leon Derczynski.  Handling and presenting harmful text in nlp research.  2022\.
  * Kirkpatrick (2020) Andy Kirkpatrick.  _The Routledge handbook of world Englishes_.  Routledge, 2020.
  * Koch et al. (2021) Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster.  Reduced, reused and recycled: The life of a dataset in machine learning research.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=zNQBIBKJRkd.
  * Kočisky et al. (2017) Tomš Kočisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette.  The NarrativeQA reading comprehension challenge.  _arXiv preprint arXiv:1712.07040_ , 2017.
  * Koenecke et al. (2020) Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel.  Racial disparities in automated speech recognition.  _Proceedings of the National Academy of Sciences_ , 2020.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.  WILDS: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Konda et al. (2016) Pradap Konda, Sanjib Das, AnHai Doan, Adel Ardalan, Jeffrey R Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeff Naughton, Shishir Prasad, et al.  Magellan: toward building entity matching management systems over data science stacks.  _Proceedings of the VLDB Endowment_ , 9(13):1581–1584, 2016.
  * Kumar et al. (2019) Ananya Kumar, Percy Liang, and Tengyu Ma.  Verified uncertainty calibration.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher.  Ask me anything: Dynamic memory networks for natural language processing.  In _International conference on machine learning_ , pp. 1378–1387. PMLR, 2016.
  * Kusner et al. (2017) Matt J Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 4069–4079, 2017.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  In _Association for Computational Linguistics (ACL)_ , 2019.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst.  SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177, 02 2022.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00453.  URL https://doi.org/10.1162/tacl_a_00453.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_ , 2019.
  * Ladhak et al. (2022) Faisal Ladhak, Esin Durmus, He He, Claire Cardie, and Kathleen McKeown.  Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1410–1421, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.100.  URL https://aclanthology.org/2022.acl-long.100.
  * Lauscher et al. (2022) Anne Lauscher, Archie Crowley, and Dirk Hovy.  Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender, 2022.  URL https://arxiv.org/abs/2202.11923.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom.  Mind the gap: Assessing temporal generalization in neural language models.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 29348–29363. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/f5bf0ba0a17ef18f9607774722f5698c-Paper.pdf.
  * Le Scao & Rush (2021) Teven Le Scao and Alexander Rush.  How many data points is a prompt worth?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2627–2636, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.208.  URL https://aclanthology.org/2021.naacl-main.208.
  * Lee et al. (2022a) Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee.  Do language models plagiarize?  _arXiv preprint arXiv:2203.07618_ , 2022a.
  * Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  _arXiv preprint arXiv:2107.06499_ , 2021.
  * Lee et al. (2022b) Mina Lee, Percy Liang, and Qian Yang.  Coauthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities.  In _Conference on Human Factors in Computing Systems (CHI)_ , 2022b.
  * Lee et al. (Forthcoming) Mina Lee, Megha Srivastava, Amelia Hardy, Esin Durmus, Ashwin Paranjape, John Thickstun, Ines Gerard-Ursin, Faisal Ladhak, Frieda Rong, Rose E. Wang, Xiang Lisa Li, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael Bernstein, and Percy Liang.  On the ability of language models to interact with humans.  Forthcoming.
  * Lees et al. (2022) Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Scott Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman.  A new generation of perspective api: Efficient multilingual character-level transformers.  _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022.
  * Lemley & Casey (2020) Mark A Lemley and Bryan Casey.  Fair learning.  _Tex. L. Rev._ , 99:743, 2020.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Levy (2008) Roger Levy.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177, 2008.  ISSN 0010-0277.  doi: https://doi.org/10.1016/j.cognition.2007.05.006.  URL https://www.sciencedirect.com/science/article/pii/S0010027707001436.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Association for Computational Linguistics (ACL)_ , 2020a.
  * Lewis et al. (2020b) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7871–7880, Online, July 2020b. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Lewis et al. (2020c) Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, M. Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020c.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Association for Computational Linguistics (ACL)_ , 2021.
  * Li et al. (2020) Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, and Wang-Chiew Tan.  Deep entity matching with pre-trained language models.  _arXiv preprint arXiv:2004.00584_ , 2020.
  * Liang & Reich (2022) Percy Liang and Rob Reich.  Condemning the deployment of gpt-4chan, 2022.  URL https://docs.google.com/forms/d/e/1FAIpQLSdh3Pgh0sGrYtRihBu-GPN7FSQoODBLvF7dVAFLZk2iuMgoLw/viewform.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Kathleen A. Creel, and Rob Reich.  The time is now to develop community norms for the release of foundation models, 2022.  URL https://crfm.stanford.edu/2022/05/17/community-norms.html.
  * Liao et al. (2021) Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt.  Are we learning yet? a meta review of evaluation failures across machine learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=mPducS1MsEK.
  * Liberman (2010) Mark Liberman.  Obituary: Fred jelinek.  _Comput. Linguist._ , 36(4):595–599, dec 2010\.  ISSN 0891-2017.  doi: 10.1162/coli_a_00032.  URL https://doi.org/10.1162/coli_a_00032.
  * Lieber et al. (2021) Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham.  Jurassic-1: Technical details and evaluation.  _White Paper, AI21 Labs_ , 2021.  URL https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Lin et al. (2021a) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates.  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ , 14(4):1–325, 2021a.
  * Lin et al. (2021b) Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods, 2021b.  URL https://arxiv.org/abs/2109.07958.
  * Linzen (2020) Tal Linzen.  How can we accelerate progress towards human-like linguistic generalization?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5210–5217, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.465.  URL https://www.aclweb.org/anthology/2020.acl-main.465.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics (TACL)_ , 4, 2016.
  * Liska et al. (2022) Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou.  StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 13604–13622. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/liska22a.html.
  * Liu et al. (2022a) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _ArXiv_ , abs/2205.05638, 2022a.
  * Liu et al. (2022b) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pp. 100–114, Dublin, Ireland and Online, May 2022b. Association for Computational Linguistics.  doi: 10.18653/v1/2022.deelio-1.10.  URL https://aclanthology.org/2022.deelio-1.10.
  * Liu et al. (2022c) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Comput. Surv._ , sep 2022c.  ISSN 0360-0300.  doi: 10.1145/3560815.  URL https://doi.org/10.1145/3560815.  Just Accepted.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Loevinger (1957) Jane Loevinger.  Objective tests as instruments of psychological theory.  _Psychological Reports_ , 3(3):635–694, 1957\.  doi: 10.2466/pr0.1957.3.3.635.  URL https://doi.org/10.2466/pr0.1957.3.3.635.
  * Lounsburg (1954) Floyd G. Lounsburg.  Transitional probability, linguistic structure and systems of habitfamily hierarchies.  _Psycholinguistics: a survey of theory and research_ , 1954.
  * Luhn (1958) Henry P. Luhn.  The automatic creation of literature abstracts.  _IBM Journal of Research and Development_ , 2:159–165, 1958.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning.  Effective approaches to attention-based neural machine translation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1412–1421, 2015.
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Yu Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=TCarYAus7JL.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In _Association for Computational Linguistics (ACL)_ , 2011.
  * MacAvaney et al. (2021) Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian.  Simplified data wrangling with ir datasets.  In _SIGIR_ , 2021.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Mani (1999) Inderjeet Mani.  _Advances in Automatic Text Summarization_.  MIT Press, Cambridge, MA, USA, 1999.  ISBN 0262133598.
  * Manning et al. (2008) Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze.  Introduction to information retrieval.  2008\.
  * Marcus et al. (1999) Mitchell Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor.  _Treebank-3_ , 1999.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 1906–1919, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.173.  URL https://aclanthology.org/2020.acl-main.173.
  * McAuley et al. (2012) Julian McAuley, Jure Leskovec, and Dan Jurafsky.  Learning attitudes and attributes from multi-aspect reviews.  _2012 IEEE 12th International Conference on Data Mining_ , pp. 1020–1025, 2012.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_ , 2018.
  * McCoy et al. (2019) R Thomas McCoy, Ellie Pavlick, and Tal Linzen.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Association for Computational Linguistics (ACL)_ , 2019.
  * Mei et al. (2021) Yinan Mei, Shaoxu Song, Chenguang Fang, Haifeng Yang, Jingyun Fang, and Jiang Long.  Capturing semantics for imputation with pre-trained language models.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pp. 61–72. IEEE, 2021.
  * Mendelsohn et al. (2020) Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky.  A framework for the computational linguistic analysis of dehumanization.  _Frontiers in Artificial Intelligence_ , 3, 2020.  ISSN 2624-8212.  doi: 10.3389/frai.2020.00055.  URL https://www.frontiersin.org/articles/10.3389/frai.2020.00055.
  * Mercier & Sperber (2017) Hugo Mercier and Dan Sperber.  _The Enigma of Reason_.  Harvard University Press, 2017.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  _arXiv preprint arXiv:1609.07843_ , 2016.
  * Merity et al. (2018) Stephen Merity, Nitish Shirish Keskar, and Richard Socher.  An analysis of neural language modeling at multiple scales.  _ArXiv_ , abs/1803.08240, 2018.
  * Merrill (2021) William Merrill.  Formal language theory meets modern nlp.  _arXiv preprint arXiv:2102.10094_ , 2021.
  * Messick (1987) Samuel Messick.  Validity.  _ETS Research Report Series_ , 1987(2):i–208, 1987.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2330-8516.1987.tb00244.x.
  * Messick (1988) Samuel Messick.  The once and future issues of validity: Assessing the meaning and consequences of measurement.  _ETS Research Report Series_ , 1988.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2330-8516.1986.tb00185.x.
  * Metzler et al. (2021) Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork.  Rethinking search: Making domain experts out of dilettantes.  _SIGIR Forum_ , 55(1), jul 2021.  ISSN 0163-5840.  doi: 10.1145/3476415.3476428.  URL https://doi.org/10.1145/3476415.3476428.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2018\.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Miller et al. (2021) John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt.  Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Mitra & Craswell (2019) Bhaskar Mitra and Nick Craswell.  An updated duet model for passage re-ranking.  _arXiv preprint arXiv:1903.07666_ , 2019.
  * Morris et al. (2020) John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  _arXiv preprint arXiv:2005.05909_ , 2020.
  * Mrini et al. (2021) Khalil Mrini, Can Liu, and Markus Dreyer.  Rewards with negative examples for reinforced topic-focused abstractive summarization.  In _Proceedings of the Third Workshop on New Frontiers in Summarization_ , pp. 33–38, 2021.