# Catwalk: A Unified Language Model Evaluation Framework
for Many Datasets

Dirk Groeneveld♠ Anas Awadalla♢ Iz Beltagy♠ Akshita Bhagia♠
Ian Magnusson♠ Hao Peng♠ Oyvind Tafjord♠ Pete Walsh♠
Kyle Richardson♠ Jesse Dodge♠
♠Allen Institute for Artificial Intelligence
♢Paul G. Allen School of Computer Science & Engineering, University of
Washington
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The success of large language models has shifted the evaluation paradigms in
natural language processing (NLP). The community’s interest has drifted
towards comparing NLP models across _many_ tasks, domains, and datasets, often
at an extreme scale. This imposes new engineering challenges: efforts in
constructing datasets and models have been fragmented, and their formats and
interfaces are incompatible. As a result, it often takes extensive
(re)implementation efforts to make fair and controlled comparisons at scale.

Catwalk aims to address these issues. Catwalk provides a unified interface to
a broad range of existing NLP datasets and models, ranging from both canonical
supervised training and fine-tuning, to more modern paradigms like in-context
learning. Its carefully-designed abstractions allow for easy extensions to
many others. Catwalk substantially lowers the barriers to conducting
controlled experiments at scale. For example, we finetuned and evaluated over
64 models on over 86 datasets with a single command, _without writing any
code_. Maintained by the AllenNLP team at the Allen Institute for Artificial
Intelligence (AI2), Catwalk is an ongoing open-source effort:
https://github.com/allenai/catwalk.

Catwalk: A Unified Language Model Evaluation Framework
for Many Datasets

Dirk Groeneveld♠ Anas Awadalla♢ Iz Beltagy♠ Akshita Bhagia♠ Ian Magnusson♠ Hao
Peng♠ Oyvind Tafjord♠ Pete Walsh♠ Kyle Richardson♠ Jesse Dodge♠ ♠Allen
Institute for Artificial Intelligence ♢Paul G. Allen School of Computer
Science & Engineering, University of Washington<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

(a) Catwalk provides a unified interfaces to connect models and datasets. This
compares $n$ models on $m$ tasks with only $n+m$ custom implementations,
substantially reducing the implementation workload compared to current
practice where $nm$ implementations are needed.

(b) Illustration of some decisions a user of Catwalk makes to get a result.
For example, choosing GPT-2 Large, with Ranked Classification, on WinoGrande
gives an accuracy of $0.553$, while choosing T5-Base on RTE gives an accuracy
of $0.653$.

Figure 1: Catwalk and its workflow.

Large language models (LLMs) have led to a paradigm shift in NLP. Before, a
system would be built to tackle a specific task and trained and evaluated on a
dedicated dataset. Now, general-purpose pretrained language models are adapted
to a broad range of tasks through techniques like finetuning and in-context
learning. The new paradigm opens research opportunities to investigate models’
generalization and robustness across _many_ tasks and domains. This imposes
new engineering challenges. The fragmented efforts in producing models and
datasets have led to countless codebases and data formats that are hardly
compatible. Fairly comparing $n$ models on $m$ datasets often require $nm$
custom implementations, substantially raising the barriers to conducting
controlled experiments at scale.

Catwalk aims to reduce practitioners’ workload in projects involving making
fair comparisons among many models and datasets. At a high level, Catwalk can
be thought of as a nexus connecting $n$ models and $m$ datasets. Through
Catwalk’s unified interfaces, training and evaluating all models on all
datasets requires $n+m$ customizations instead of $mn$, many of which have
already been implemented. To include a new model or dataset into Catwalk, one
only needs to instantiate a few interfaces, and it can be evaluated on _all_
datasets/models that Catwalk supports. Figures 1(a) and 1(b) provide
illustrative examples. Whenever possible, Catwalk caches and reuses
intermediate experimental steps such as loaded datasets/models, model
predictions, as well as trained models.111Catwalk’s caching mechanism is
enabled by AI2-Tango (Groeneveld et al., 2023):
https://github.com/allenai/tango. This helps minimize redundant machine
workloads in, e.g., loading the data and training the models, and better
utilize the limited hardware resources. Catwalk’s caching mechanism is
especially useful in large-scale experiments that involve many models and/or
many tasks.

Catwalk is an ongoing effort. We aim to jump-start a virtuous cycle of dataset
and model authors adopting support for Catwalk’s unified interfaces. To this
end, we contribute:

  * •

6 model abstractions that support a variety of existing models.

  * •

Zero-shot and few-shot evaluations, as well as fine-tuning.

  * •

9 data formats that cover common use cases.

  * •

Support for 86 standalone datasets as well as curated benchmarks that contain
many datasets themselves, for a total of over 800 datasets, along with
evaluation metrics.

  * •

A case study that compares 30 widely-used models on 17 datasets.

We expect the above list to keep growing. Catwalk is open-source:
https://github.com/allenai/catwalk. It is maintained by the AllenNLP team at
the Allen Institute for Artificial Intelligence (AI2) and has been a useful
tool for their projects, including being the main evaluation framework used in
the OLMo (Open Language Model) project.222See more details at
https://allenai.org/olmo as well as https://github.com/allenai/ai2-olmo-eval
for details about our LLM evaluation efforts. We hope it can benefit the
community’s research too.

##  2 Structure and organization

The purpose of Catwalk is to answer the question "How well does model X do on
dataset Y?", for as many models and datasets as possible. The possible models
include GPT-style decoder-only models Radford et al. (2018), T5-style
encoder/decoder models Raffel et al. (2019), and BERT-style fine-tuned models
Devlin et al. (2018). Catwalk imposes no constraint on the types of models it
can support. The model does not have to be transformer-based or even be a
language model at all, though all currently implemented models are
transformer-based language models.

To achieve this flexibility, Catwalk transforms instances from a dataset into
“formats” that are more suited to model execution. Crucially, these formats
are the same across many datasets, so model implementations do not have to
concern themselves with the differences between datasets, and can be written
in a simple, generic style.

Model style | Enc-only | Dec-only | Enc/dec | Finetuning | Zero-shot | Few-shot
---|---|---|---|---|---|---
Ranked classification | ✗ | ✓ | ✓ | ✓ | ✓ | ✓
Bert-style | ✓ | ✗ | ✗ | ✓ | ✗ | ✗
Language models | ✗ | ✓ | ✗ | ✗ | ✓ | ✓
PromptSource | ✗ | ✓ | ✓ | ✓ | ✓ | ✓
MetaICL | ✗ | ✓ | ✓ | ✗ | ✓ | ✓
Eleuther-style | ✗ | ✓ | ✗ | ✗ | ✓ | ✓
Table 1: Model styles and supported features in Catwalk.

##  3 Models and Model Wrappers

Catwalk’s models combine an underlying model, a model wrapper, and a method to
process the outputs from the model. For example, a decoder-only language model
can be used to compute the probability of various answer options. This can be
done using T0-style human-readable prompts Sanh et al. (2021), or it can be
done using machine-readable prompts in a few-shot setting Gao et al. (2023).
This model could even be fine-tuned and used with a task-specific head Devlin
et al. (2018). These options have different characteristics and produce
different results. They are all encapsulated in Catwalk’s model wrapper
abstraction.

###  3.1 Model Wrappers

Model wrappers come in two general varieties, _native interfaces_ that provide
generic support for a wide class of models and _adapter interfaces_ that aim
to facilitate systematic comparisons with existing model types or evaluation
frameworks. We list the different model wrappers below current implemented in
Catwalk (with the first three being our native interfaces and the last three
being example adapter interfaces; see Table 1).

  * •

Ranked classification models. This model wrapper uses decoder-only and
encoder/decoder models. It calculates the probability of all answer options,
and chooses the most likely answer. It supports zero-shot and few-shot
evaluations, and it can be fine-tuned on a dataset, or a list of datasets.

  * •

BERT-style models. This model wrapper uses the native API from the Huggingface
library Wolf et al. (2019) to perform multiple-choice, question answering, and
classification tasks. This is the way to run encoder-only models like BERT
Devlin et al. (2018), RoBERTa Liu et al. (2019), or DeBERTa He et al. (2020).
It supports fine-tuning, but no few-shot evaluations.

  * •

Language models. This model wrapper is a generic class for decoder-only
generation models that supports few- and zero-shot evaluation. This is the
class most relevant to recent GPT-style LLM evaluation. In contrast to
Eleuther-style models Gao et al. (2023) _(see below)_ , it is designed to
support a wider range of generation metrics and token-based scoring strategies
beyond those hard-coded in existing toolkits. It also has special features for
returning per-instance predictions that allow for more fine-grained analysis
and debugging.

  * •

PromptSource models. This model wrapper also performs ranked classification,
but it uses the human-readable prompts from the PromptSource project Bach et
al. (2022). It supports zero-shot and few-shot evaluations.

  * •

MetaICL models. This model wrapper applies the prompt truncation strategy used
by Min et al. (2022b) to decoder-only ranked classification. It truncates each
of several in-context demonstrations separately and truncates again after
concatenation.

  * •

Eleuther-style models. This model wrapper uses decoder-only models in the
style of Gao et al. (2023). It reproduces those numbers exactly, though it
runs a little faster. Like Gao et al. (2023), it supports zero-shot and few-
shot evaluations. On many datasets, this is equivalent to the ranked
classification models.

###  3.2 Models

All of Catwalk’s existing model implementations load parameters and
configurations from Huggingface. They can be configured to load any model from
the Huggingface hub, but Catwalk gives a few of them names, tests them
regularly, and makes them available on the command line. The following is a
list of models that have names in Catwalk and that have been extensively
tested:

  * •

BERT {tiny, base, large}, {cased, uncased} Devlin et al. (2018).

  * •

BLOOM {560M, 1B1, 1B7, 3B, 7B1} Scao et al. (2022).

  * •

CT0-11b Scialom et al. (2022).

  * •

DEBERTA {v3-small, v3-base, v3-large, v2-xlarge, v2-xxlarge} He et al. (2020,
2023).

  * •

DistilBERT-base-cased-distilled-squad Sanh et al. (2019).

  * •

GPT-J-6B Wang and Komatsuzaki (2021).

  * •

GPT-Neo {125M, 1.3B, 2.7B} Gao et al. (2020).

  * •

GPT-NeoX-20B Black et al. (2022).

  * •

GPT2 {tiny, regular, medium, large, xl} Radford et al. (2018).

  * •

mT5 {small, base, large, xl} Xue et al. (2020).

  * •

OPT {125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B} Zhang et al. (2022).

  * •

RoBERTa {base, large} Liu et al. (2019).

  * •

T0 {regular, 3B, original_task_only, single_prompt, p, pp} Sanh et al. (2021).

  * •

T5 {very_small_random, small, small-lm-adapt, v1_1-small, base, base-lm-adapt,
v1_1-base, large, large-lm-adapt, v1_1-large, 3B, xl-lm-adapt, v1_1-xl, 11b,
xxl-lm-adapt, v1_1-xxl} Raffel et al. (2019).

  * •

Pythia {70M,160M,410M,1B,1.4B,2.7B,6.9B,12B} Biderman et al. (2023)

  * •

Falcon {1B,7B,40B} Almazrouei et al. (2023)

  * •

Llama(-2) {7B, 13B, 70B} Touvron et al. (2023a, b)

  * •

MPT {7B, 30B} MosaicML (2023)

##  4 Prompting

Catwalk supports a number of formats for dataset instances. By converting
dataset instances into a common format, we allow model implementations to only
depend on the format, and not on the vagaries of individual datasets.

Catwalk supports the following formats.

###  4.1 Huggingface dictionaries

This format makes few guarantees beyond being a Python dictionary. It is the
raw format of the data, and rarely useful directly. This is the format that
the Huggingface datasets library Lhoest et al. (2021) produces. It isn’t used
by any of the models directly, but all the other formats use this as their
starting point.  ⬇ {  "sentence1": "No Weapons of Mass Destruction Found
Yet.",  "sentence2": "Weapons of Mass Destruction Found.",  "label": 1,
"idx": 0 }

###  4.2 Huggingface multiple choice

This is a structured format containing a question, a list of answer choices,
and optionally the index of the correct answer. This corresponds exactly to
the expectation of Huggingface multiple choice models. It is used by the BERT-
style models.  ⬇ HFMCInstance(  id="Mercury_7220990",  question="Which factor
will most likely cause a " \+  "person to develop a fever?",  answer_choices=[
"a leg muscle relaxing after exercise",  "a bacterial population in the
bloodstream",  "several viral particles on the skin",  "carbohydrates being
digested in the stomach" ],  correct_answer_index=1 )

###  4.3 Huggingface question-answering

This is a structured format containing a context, a question, and a list of
possible correct answers. This corresponds exactly to the expectation of
Huggingface question-answering models. It is used by the BERT-style models.

⬇

HFQAInstance(

id="5733be284776f41900661182",

question="To whom did the Virgin Mary appear in 1858?",

context="… the Virgin Mary appeared to Saint " \+

"Bernadette Soubirous in 1858 …",

answers={

"text": ["Saint Bernadette Soubirous"],

"answer_start": [515]})

###  4.4 Huggingface classification

This is a structured format containing a context, as a string or a pair of
strings, and an optional label. The possible answer choices are not part of
the format but are part of the model that uses these instances. This
corresponds exactly to the expectation of Huggingface classification models.
It is used by the BERT-style models.  ⬇ HFClassificationInstance(  text=(  "No
Weapons of Mass Destruction Found Yet.",  "Weapons of Mass Destruction Found."
), label=1)

###  4.5 Eleuther

There are several formats specific to the Eleuther evaluation framework Gao et
al. (2023). They are useful to build other format converters on top of, but
are rarely useful on their own. Only the Eleuther-style models use these
formats directly.

###  4.6 T5

This format represents instances as strings starting with the dataset name
followed by named fields that make up the instance. It also includes a gold
answer that is expected of the model. This corresponds exactly to the way T5
Raffel et al. (2019) was trained. The format is only supported for a small
number of datasets.  ⬇ (  "rte sentence1: No Weapons of Mass Destruction found
yet sentence2: Weapons of Mass Destruction Found.", "not_entailment" )

###  4.7 Ranked classification

This format represents instances as a list of options, together with an
optional label indicating the correct option. Each option is a text pair,
called “context” and “continuation”. Models generally compute the probability
of the continuation given the context to determine the best answer, though
recent work suggests other possibilities Min et al. (2022a). The format is
used by the built-in ranked classification models, and it forms the basis for
the PromptSource format.  ⬇ RankClassificationInstance(  choices=[  (  "No
Weapons of Mass Destruction Found Yet.\n" \+  "Question: Weapons of Mass
Destruction Found. " \+  "True or False?\nAnswer:",  " True" ), (  "No Weapons
of Mass Destruction Found Yet.\n" \+  "Question: Weapons of Mass Destruction
Found. " \+  "True or False?\nAnswer:",  " False")  ],correct_choice=1)

###  4.8 PromptSource

This format makes the prompts from the PromptSource project Bach et al. (2022)
available in Catwalk. In the PromptSource project, every dataset has multiple
prompts. These are reflected in the format by having multiple versions of the
instance, one per prompt, in a dictionary. Every possible version looks
exactly like a ranked classification instance, and can be used by model
implementations accordingly. The PromptSource model implementation uses this
format.

⬇

{"does the claim": RankClassificationInstance(

choices=[

(

"Does the claim ’Weapons of Mass Destruction " \+

"Found.’ follow from the fact that ’No Weapons " \+

"of Mass Destruction Found Yet.’? Please " \+

"answer either yes or no.",

"yes"

), (

"Does the claim ’Weapons of Mass Destruction ’" \+

"Found.’ follow from the fact that ’No Weapons " \+

"of Mass Destruction Found Yet.’? Please " \+

"answer either yes or no.", "no"

)

],

correct_choice=1),

"imply": RankClassificationInstance(

choices=[

(

"Does ’No Weapons of Mass Destruction Found " \+

"Yet.’ imply that ’Weapons of Mass Destruction " \+

"Found.’? Please answer either yes or no.", "yes"

), (

"Does ’No Weapons of Mass Destruction Found " \+

"Yet.’ imply that ’Weapons of Mass Destruction " \+

"Found.’? Please answer either yes or no.",

"no"

)

],correct_choice=1)}

###  4.9 Perplexity

Catwalk also has a general-purpose task format for perplexity evaluation,
which was developed as part of the PALOMA benchmark Magnusson et al. (2023).
It follows best practices for perplexity analysis such as avoiding document
concatenation Gao et al. (2020), has advanced batching strategies (e.g.,
sorting by input length to avoid excessive padding) and supports both non-
overlapping and sliding window inference from Press et al. (2020) to handle
common issues related to document maximum length truncation.

##  5 Datasets

The datasets supported by Catwalk can be extended with just a few lines of
configs with prompts. Currently Catwalk contains the following datasets:

#### Perplexity

  * •

WikiText Merity (2019)

  * •

LAMBADA Paperno et al. (2016)

#### Classification

  * •

CoLA Warstadt et al. (2019)

  * •

MRPC Dolan and Brockett (2005)

  * •

QQP DataCanary et al. (2017)

  * •

SST-2 Socher et al. (2013)

  * •

BoolQ Clark et al. (2019)

  * •

MultiRC Khashabi et al. (2018)

  * •

WiC Pilehvar and Camacho-Collados (2019)

  * •

MC-TACO Ben Zhou and Roth (2019)

  * •

PubMedQA Jin et al. (2019)

  * •

Hendrycks Ethics Hendrycks et al. (2021a)

  * •

RAFT Alex et al. (2021)

#### Open-ended QA

  * •

SQuAD Rajpurkar et al. (2016)

  * •

SquadShifts Miller et al. (2020)

  * •

MRQA Fisch et al. (2019)

  * •

SQuAD2 Rajpurkar et al. (2018)

  * •

DROP Dua et al. (2019)

  * •

TriviaQA Joshi et al. (2017)

  * •

WebQuestions Berant et al. (2013)

  * •

TruthfulQA Lin et al. (2021)

  * •

Hendrycks Math Hendrycks et al. (2021c)

  * •

Arithmetic Brown et al. (2020)

  * •

Anagrams 1 and 2 Brown et al. (2020)

  * •

Cycle Letters Brown et al. (2020)

  * •

Random Insertion Brown et al. (2020)

  * •

Reversed Words Brown et al. (2020)

  * •

Natural Questions Kwiatkowski et al. (2019)

#### Multiple-choice QA

  * •

PIQA Bisk et al. (2020)

  * •

COPA Gordon et al. (2012)

  * •

WSC Levesque et al. (2011)

  * •

PROST Aroca-Ouellette et al. (2021)

  * •

SciQ Welbl et al. (2017)

  * •

QA4MRE Peñas et al. (2013)

  * •

ARC Easy and Challenge Clark et al. (2018)

  * •

LogiQA Liu et al. (2020)

  * •

HellaSwag Zellers et al. (2019)

  * •

OpenBookQA Mihaylov et al. (2018)

  * •

RACE Lai et al. (2017)

  * •

HEAD-QA Vilares and Gómez-Rodríguez (2019)

  * •

MathQA Amini et al. (2019)

  * •

WINOGRANDE Sakaguchi et al. (2019)

  * •

MuTual Cui et al. (2020)

  * •

MMLU Hendrycks et al. (2021b)

  * •

CaseHOLD Zheng et al. (2021)

  * •

SocialIQA Sap et al. (2019)

  * •

CSQA Talmor et al. (2018)

#### Summarization

  * •

SciTLDR Cachola et al. (2020)

  * •

XSUM Narayan et al. (2018)

  * •

EURLEX Aumiller et al. (2022)

#### NLI

  * •

RTE from GLUE Wang et al. (2018)

  * •

RTE from SuperGLUE Wang et al. (2019)

  * •

MNLI Williams et al. (2018)

  * •

QNLI Wang et al. (2018)

  * •

WNLI Wang et al. (2018)

  * •

CB de Marneffe et al. (2019)

  * •

Adverserial NLI Nie et al. (2020)

#### Mixtures of tasks

  * •

P3 Sanh et al. (2022)

  * •

MetaICL Min et al. (2022b)

Note that some of these datasets are collections that show up in Catwalk as
multiple subsets. For example, there are actually 11 RAFT datasets, each
taking a different angle on language model evaluation. In total, this adds up
to over 800 datasets.

Adding new datasets to Catwalk is easy. There are classes for including
datasets from the Huggingface hub, and format conversion functions can often
be written in a single line of code.

##  6 Evaluation

In Catwalk, the responsibility for calculating metrics lies with the model
implementations. Model implementations can calculate any metric that makes
sense in the context of the model. However, for a given dataset we often want
the same metrics independent of the model, so we can compare models against
each other. To facilitate this, Catwalk allows one to attach metrics to
datasets, which act as a suggestion to the models for which metrics to
compute. All implemented model styles except the Eleuther style compute the
suggested metrics.

By default, Catwalk computes the following metrics and can be easily extended
to include others.

  * •

For multiple-choice, classification, and entailment datasets: Accuracy and
relative improvement over the random baseline

  * •

For question-answering datasets: SQuAD metric as defined by Rajpurkar et al.
(2016).

  * •

For language modeling datasets: Perplexity per word, perplexity per byte, and
entropy.

##  7 Case Studies

We include some examples of the kinds of analyses Catwalk enables and in the
end describe some ongoing work. Table 3 in the Appendix gives an example of
the dense matrix of models and datasets that Catwalk can generate. There are
many research questions that can be posed and answered with a matrix of
results like this:

#### What strategy for adapting pretrained models to downstream tasks performs
best overall?

For instance we can compare finetuning an encoder-only model against zero-shot
evaluation on autoregressive models. We find that, macro averaged over 17
datasets, the best finetuned models had 62% greater relative improvement in
accuracy over a random baseline than the zero-shot models , although the
largest zero-shot model is an order of magnitude larger than the largest
finetuned model.

#### Which datasets are easy/hard?

_All_ zero-shot and finetuned models perform well on SciQ, with an average of
more than 200% relative accuracy improvement compared to the random-guess
baseline. Among the datasets in our experiments, QNLI is the most challenging
one for zero-shot ranked classification models, with all models barely
outperforming random guesses. Unsurprisingly, all finetuned models
substantially outperform the chance baseline on all datasets, and in all cases
they outperform zero-shot models by a wide margin.

#### What impact does model type or size have across datasets?

Among zero-shot models, we see that, while T5 variants are the only ones that
significantly outperform the chance baseline on MNLI, QQP, and RTE, they
_underperform_ others and even the random baseline on CoLA and COPA. This
could be attributed to the different choices T5 models make in terms of
learning objectives, pretraining data, etc., which requires further
investigation. Also, in both the zero-shot ranked classification and
finetuning settings, larger model sizes, in general, lead to better
performance _within the same model family_ , with few exceptions.

#### Is the ordering of models by performance consistent across datasets?

Table 2 in the Appendix shows that every dataset puts the models into roughly
the same order by accuracy, but correlations are often weak. The largest
correlations are observed between the ARC Easy, ARC Challenge, and OpenbookQA
datasets. All three of these are multiple-choice datasets targeting science
questions. On the other end of the spectrum, LogiQA and SciQ correlate the
least with other datasets, with coefficients below $0.5$. This might be
explained by LogiQA being the most difficult, and SciQ being the easiest
dataset among the 17 that were part of this study. In one case the models
cannot get much signal out of the dataset, and in the other they saturate. A
thorough investigation in this style would have to include many more models
and prompting methods.

These analyses only scratch the surface of the questions that Catwalk can help
answer. There are many open questions that bear investigation:

  * •

How well does perplexity predict performance on other datasets?

  * •

Do the zero-shot results hold in few-shot settings? How does the number of
few-shot examples affect the results?

  * •

How important is the format of the prompt?

  * •

Some large language models make intermediate checkpoints available for
analysis. At what point in model training do abilities emerge?

#### Ongoing work

As mentioned at the outset, Catwalk is currently be used as the main
evaluation framework for the OLMo project at AI2. As such, it has many
features that allow it to be a replacement for other evaluation frameworks,
such as Eleuther Harness Gao et al. (2023). For more details about Catwalk for
LLM evaluation, see https://github.com/allenai/ai2-olmo-eval.

##  8 Related Work

Holistic Evaluation of Language Models (HELM; Liang et al., 2022) also aims to
increase the density of coverage between models and evaluations and take a
complementary path to ours. HELM describes itself as a top-down approach; they
design a taxonomy of scenarios and metrics and use them to curate a specific
subset of combinations. Catwalk is bottom-up, and aims to lower the
engineering barriers for model and task creators to integrate their work into
a shared benchmark. For example, HELM chooses to only adapt models via a
standardized 5-shot prompting method, while Catwalk explicitly incorporates
different varieties of prompting and even fine-tuning.

Our work builds on the EleutherAI LM Evaluation Harness Gao et al. (2023). It
similarly addresses the task of building a unified evaluation framework and
focuses on decoder-only models and zero-/few-shot evaluations. By contrast,
Catwalk also includes encoder-only and encoder/decoder models and supports
fine-tuning.

##  9 Conclusion

We presented Catwalk, a system to evaluate many models on many datasets.
Catwalk’s APIs eliminate redundant implementation efforts and unify
incompatible approaches to language model evaluation. The approach enables
experimentation across datasets and language models at an unprecedented scale.
While a large number of models and datasets is already integrated, the Allen
Institute for Artificial Intelligence continues to provide hands-on
engineering help to grow this list, and invites further contributions as an
open-source effort.

## Limitations

Language model evaluation can only be as good as the datasets that are used.
Catwalk takes no position on this topic, and instead makes as many datasets as
possible available for study. As shown in Table 2, the choice of dataset can
have a big influence on which model appears “best”. Similarly, prompted models
are often sensitive to the prompt, and fine-tuned models are sensitive to
their training regime. We hope that Catwalk will enable the kind of research
that allows us to take strong, supported positions on these topics in the
future.

## Ethics Statement

We believe the impact of this framework to be largely beneficial, in that it
encourages the community to adopt reproducible research practices. A unified
interface for evaluation allows for easy comparison across various papers,
without the need for reimplementation and rerunning experiments that may take
up significant time and compute resources.

We also note that the datasets present in our framework may contain biases and
artifacts of their own, and a model performing well within our framework on
these datasets should not be considered as having “solved” the underlying
task.

## References

  * Alex et al. (2021) Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller. 2021.  RAFT: A real-world few-shot text classification benchmark.  _CoRR_ , abs/2109.14076.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  _CoRR_ , abs/1905.13319.
  * Aroca-Ouellette et al. (2021) Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021.  PROST: Physical reasoning about objects through space and time.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4597–4608, Online. Association for Computational Linguistics.
  * Aumiller et al. (2022) Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022.  Eur-lex-sum: A multi-and cross-lingual dataset for long-form summarization in the legal domain.  _arXiv preprint arXiv:2210.13448_.
  * Bach et al. (2022) Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022.  Promptsource: An integrated development environment and repository for natural language prompts.
  * Ben Zhou and Roth (2019) Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. 2019.  “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.  In _EMNLP_.
  * Berant et al. (2013) Jonathan Berant, Andrew K. Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cachola et al. (2020) Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S Weld. 2020.  Tldr: Extreme summarization of scientific documents.  _arXiv preprint arXiv:2004.15011_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457.
  * Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020.  Mutual: A dataset for multi-turn dialogue reasoning.  _Proceedings of ACL_.
  * DataCanary et al. (2017) hilfialkaff DataCanary, Lili Jiang, Meg Risdal, and Nikhil Dandekar. 2017.  Quora question pairs.
  * de Marneffe et al. (2019) Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _Proceedings of Sinn und Bedeutung 23_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _North American Chapter of the Association for Computational Linguistics_.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  _CoRR_ , abs/1910.09753.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Gordon et al. (2012) Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012.  SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 394–398, Montréal, Canada. Association for Computational Linguistics.
  * Groeneveld et al. (2023) Dirk Groeneveld, Pete Walsh, Akshita Bhagia, and Michael Schmitz. 2023.  Ai2 tango.
  * He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _Proceedings of ICLR_.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced BERT with disentangled attention.  _CoRR_ , abs/2006.03654.
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a.  Aligning ai with shared human values.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring massive multitask language understanding.  _Proceedings of ICLR_.
  * Hendrycks et al. (2021c) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021c.  Measuring mathematical problem solving with the math dataset.  _NeurIPS_.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2567–2577.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , Vancouver, Canada. Association for Computational Linguistics.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  _arXiv preprint arXiv:1704.04683_.
  * Levesque et al. (2011) Hector J. Levesque, Ernest Davis, and L. Morgenstern. 2011.  The winograd schema challenge.  In _International Conference on Principles of Knowledge Representation and Reasoning_.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario Šaško, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scao, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Nathan Raw, Sylvain Lesage, Anton Lozhkov, Matthew Carrigan, Théo Matussière, Leandro von Werra, Lysandre Debut, Stas Bekman, and Clément Delangue. 2021.  Datasets: A Community Library for Natural Language Processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184. Association for Computational Linguistics.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022.  Holistic evaluation of language models.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _CoRR_ , abs/2109.07958.
  * Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020\.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  _CoRR_ , abs/2007.08124.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Magnusson et al. (2023) Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneneh Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. 2023.  PaLOMa: A benchmark for evaluating language model fit.  _technical report_.
  * Merity (2019) Stephen Merity. 2019.  The wikitext long term dependency language modeling dataset.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Miller et al. (2020) John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020.  The effect of natural distribution shift on question answering models.  _CoRR_ , abs/2004.14444.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a.  Noisy channel language model prompting for few-shot text classification.  _ACL 2022_.
  * Min et al. (2022b) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b.  MetaICL: Learning to learn in context.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2791–2809, Seattle, United States. Association for Computational Linguistics.
  * MosaicML (2023) MosaicML. 2023.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  Accessed: 2023-05-05.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  _CoRR_ , abs/1606.06031.
  * Peñas et al. (2013) Anselmo Peñas, Eduard H. Hovy, Pamela Forner, Álvaro Rodrigo, Richard F. E. Sutcliffe, and Roser Morante. 2013.  Qa4mre 2011-2013: Overview of question answering for machine reading evaluation.  In _Conference and Labs of the Evaluation Forum_.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Press et al. (2020) Ofir Press, Noah A Smith, and Mike Lewis. 2020.  Shortformer: Better language modeling using shorter inputs.  _arXiv preprint arXiv:2012.15832_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _OpenAI_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _CoRR_ , abs/1806.03822.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  _CoRR_ , abs/1606.05250.
  * Sakaguchi et al. (2019) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  WINOGRANDE: an adversarial winograd schema challenge at scale.  _CoRR_ , abs/1907.10641.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.  _CoRR_ , abs/2110.08207.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019.  Socialiqa: Commonsense reasoning about social interactions.  _arXiv preprint arXiv:1904.09728_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, Franccois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurenccon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etxabe, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris C. Emezue, Christopher Klamm, Colin Leong, Daniel Alexander van Strien, David Ifeoluwa Adelani, Dragomir R. Radev, Eduardo G. Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady ElSahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Josephine L. Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro von Werra, Leon Weber, Long Phan, Loubna Ben Allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, Mar’ia Grandury, Mario vSavsko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad Ali Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto L’opez, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, S. Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal V. Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Févry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiang Tang, Zheng Xin Yong, Zhiqing Sun, Shaked Brody, Y Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre Franccois Lavall’ee, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aur’elie N’ev’eol, Charles Lovering, Daniel H Garrette, Deepak R. Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, S. Osher Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenvek Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ananda Santa Rosa Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Olusola Ajibade, Bharat Kumar Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David M. Lansky, Davis David, Douwe Kiela, Duong Anh Nguyen, Edward Tan, Emily Baylor, Ezinwanne Ozoani, Fatim T Mirza, Frankline Ononiwu, Habib Rezanejad, H.A. Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jan Passmore, Joshua Seltzer, Julio Bonis Sanz, Karen Fort, Lívia Macedo Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, M. K. K. Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nourhan Fahmy, Olanrewaju Modupe Samuel, Ran An, R. P. Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas L. Wang, Sourav Roy, Sylvain Viguier, Thanh-Cong Le, Tobi Oyebade, Trieu Nguyen Hai Le, Yoyo Yang, Zachary Kyle Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Kumar Singh, Benjamin Beilharz, Bo Wang, Caio Matheus Fonseca de Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel Le’on Perin’an, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully A. Burns, Helena U. Vrabec, Iman I.B. Bello, Isha Dash, Ji Soo Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthi Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, María Andrea Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Cullan, Michael Weinberg, M Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patricia Haller, R. Chandrasekhar, R. Eisenberg, Robert Martin, Rodrigo L. Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Pratap Bharati, T. A. Laud, Th’eo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yashasvi Bajaj, Y. Venkatraman, Yifan Xu, Ying Xu, Yun chao Xu, Zhee Xao Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _ArXiv_ , abs/2211.05100.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.  Fine-tuned language models are continual learners.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vilares and Gómez-Rodríguez (2019) David Vilares and Carlos Gómez-Rodríguez. 2019.  HEAD-QA: A healthcare dataset for complex reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 960–966, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _ArXiv_ , abs/1905.00537.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _BlackboxNLP@EMNLP_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  _ArXiv_ , abs/1707.06209.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _CoRR_ , abs/2010.11934.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence? _CoRR_ , abs/1905.07830.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zheng et al. (2021) Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021\.  When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings.  In _Proceedings of the eighteenth international conference on artificial intelligence and law_ , pages 159–168.

##  Appendix A Appendix

| ARC Challenge | ARC Easy | CoLA | COPA | HEAD-QA EN | HellaSwag | LogiQA | MNLI | MRPC | OpenBookQA | PIQA | QNLI | QQP | RTE | SciQ | SST | WINOGRANDE
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ARC Challenge | $1.000$ | $0.932$ | $0.606$ | $0.811$ | $0.800$ | $0.826$ | $0.587$ | $0.716$ | $0.642$ | $0.939$ | $0.614$ | $0.852$ | $0.696$ | $0.796$ | $0.543$ | $0.577$ | $0.747$
ARC Easy | $0.932$ | $1.000$ | $0.650$ | $0.896$ | $0.860$ | $0.862$ | $0.492$ | $0.649$ | $0.652$ | $0.914$ | $0.713$ | $0.744$ | $0.585$ | $0.693$ | $0.603$ | $0.502$ | $0.805$
CoLA | $0.606$ | $0.650$ | $1.000$ | $0.616$ | $0.440$ | $0.381$ | $0.618$ | $0.622$ | $0.864$ | $0.600$ | $0.244$ | $0.525$ | $0.433$ | $0.503$ | $0.086$ | $0.590$ | $0.475$
COPA | $0.811$ | $0.896$ | $0.616$ | $1.000$ | $0.727$ | $0.822$ | $0.442$ | $0.482$ | $0.552$ | $0.765$ | $0.641$ | $0.617$ | $0.475$ | $0.535$ | $0.454$ | $0.429$ | $0.784$
HEAD-QA EN | $0.800$ | $0.860$ | $0.440$ | $0.727$ | $1.000$ | $0.778$ | $0.295$ | $0.438$ | $0.501$ | $0.804$ | $0.811$ | $0.531$ | $0.336$ | $0.525$ | $0.691$ | $0.249$ | $0.751$
HellaSwag | $0.826$ | $0.862$ | $0.381$ | $0.822$ | $0.778$ | $1.000$ | $0.238$ | $0.532$ | $0.422$ | $0.772$ | $0.825$ | $0.666$ | $0.499$ | $0.608$ | $0.679$ | $0.398$ | $0.892$
LogiQA | $0.587$ | $0.492$ | $0.618$ | $0.442$ | $0.295$ | $0.238$ | $1.000$ | $0.570$ | $0.634$ | $0.515$ | $0.085$ | $0.482$ | $0.506$ | $0.516$ | $-0.040$ | $0.483$ | $0.204$
MNLI | $0.716$ | $0.649$ | $0.622$ | $0.482$ | $0.438$ | $0.532$ | $0.570$ | $1.000$ | $0.675$ | $0.773$ | $0.311$ | $0.835$ | $0.822$ | $0.788$ | $0.222$ | $0.712$ | $0.420$
MRPC | $0.642$ | $0.652$ | $0.864$ | $0.552$ | $0.501$ | $0.422$ | $0.634$ | $0.675$ | $1.000$ | $0.648$ | $0.240$ | $0.586$ | $0.540$ | $0.556$ | $0.137$ | $0.685$ | $0.455$
OpenBookQA | $0.939$ | $0.914$ | $0.600$ | $0.765$ | $0.804$ | $0.772$ | $0.515$ | $0.773$ | $0.648$ | $1.000$ | $0.581$ | $0.874$ | $0.769$ | $0.828$ | $0.520$ | $0.608$ | $0.726$
PIQA | $0.614$ | $0.713$ | $0.244$ | $0.641$ | $0.811$ | $0.825$ | $0.085$ | $0.311$ | $0.240$ | $0.581$ | $1.000$ | $0.417$ | $0.235$ | $0.308$ | $0.641$ | $0.160$ | $0.774$
QNLI | $0.852$ | $0.744$ | $0.525$ | $0.617$ | $0.531$ | $0.666$ | $0.482$ | $0.835$ | $0.586$ | $0.874$ | $0.417$ | $1.000$ | $0.897$ | $0.828$ | $0.381$ | $0.750$ | $0.562$
QQP | $0.696$ | $0.585$ | $0.433$ | $0.475$ | $0.336$ | $0.499$ | $0.506$ | $0.822$ | $0.540$ | $0.769$ | $0.235$ | $0.897$ | $1.000$ | $0.855$ | $0.251$ | $0.801$ | $0.413$
RTE | $0.796$ | $0.693$ | $0.503$ | $0.535$ | $0.525$ | $0.608$ | $0.516$ | $0.788$ | $0.556$ | $0.828$ | $0.308$ | $0.828$ | $0.855$ | $1.000$ | $0.403$ | $0.719$ | $0.523$
SciQ | $0.543$ | $0.603$ | $0.086$ | $0.454$ | $0.691$ | $0.679$ | $-0.040$ | $0.222$ | $0.137$ | $0.520$ | $0.641$ | $0.381$ | $0.251$ | $0.403$ | $1.000$ | $0.089$ | $0.583$
SST | $0.577$ | $0.502$ | $0.590$ | $0.429$ | $0.249$ | $0.398$ | $0.483$ | $0.712$ | $0.685$ | $0.608$ | $0.160$ | $0.750$ | $0.801$ | $0.719$ | $0.089$ | $1.000$ | $0.311$
WINOGRANDE | $0.747$ | $0.805$ | $0.475$ | $0.784$ | $0.751$ | $0.892$ | $0.204$ | $0.420$ | $0.455$ | $0.726$ | $0.774$ | $0.562$ | $0.413$ | $0.523$ | $0.583$ | $0.311$ | $1.000$
Table 2: Spearman correlations between models when ranked in order of
performance on 17 common datasets.

| Models | ARC Challenge | ARC Easy | CoLA | COPA | HEAD-QA | HellaSwag | LogiQA | MNLI | MRPC | OpenBookQA | PIQA | QNLI | QQP | RTE | SciQ | SST | WINOGRANDE
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot Ranked Classification | BLOOM 560M | $-4.1$ | $60.4$ | $9.9$ | $24.0$ | $47.0$ | $42.7$ | $6.3$ | $6.4$ | $36.8$ | $23.2$ | $30.6$ | $-0.5$ | $-24.8$ | $5.4$ | $190.0$ | $30.7$ | $2.4$
BLOOM 1B1 | $3.1$ | $79.3$ | $38.3$ | $32.0$ | $51.5$ | $63.7$ | $3.2$ | $6.6$ | $36.8$ | $24.8$ | $36.1$ | $-1.0$ | $-26.4$ | $5.4$ | $198.0$ | $23.6$ | $9.4$
BLOOM 1B7 | $5.5$ | $90.1$ | $38.3$ | $38.0$ | $64.5$ | $84.0$ | $4.5$ | $6.5$ | $36.8$ | $28.8$ | $38.1$ | $-1.0$ | $-26.4$ | $6.9$ | $198.8$ | $7.3$ | $14.0$
BLOOM 3B | $15.0$ | $109.6$ | $37.1$ | $40.0$ | $67.0$ | $108.8$ | $10.0$ | $6.7$ | $36.8$ | $40.8$ | $41.7$ | $-1.1$ | $-26.4$ | $12.6$ | $223.2$ | $2.1$ | $17.0$
BLOOM 7B1 | $27.6$ | $127.6$ | $32.3$ | $32.0$ | $77.6$ | $136.2$ | $4.5$ | $7.0$ | $36.8$ | $58.4$ | $47.2$ | $2.4$ | $-26.3$ | $8.3$ | $231.2$ | $-1.6$ | $28.7$
GPT-J 6B | $43.0$ | $140.1$ | $35.4$ | $54.0$ | $89.6$ | $163.6$ | $17.4$ | $12.2$ | $36.8$ | $68.0$ | $52.2$ | $2.9$ | $-23.5$ | $9.0$ | $238.4$ | $-1.4$ | $28.2$
GPT-2 | $-8.2$ | $52.7$ | $33.1$ | $34.0$ | $30.2$ | $23.7$ | $0.2$ | $1.2$ | $13.2$ | $8.8$ | $23.6$ | $0.3$ | $-25.4$ | $6.1$ | $167.6$ | $8.0$ | $3.2$
GPT-2 Medium | $-1.4$ | $70.7$ | $38.3$ | $30.0$ | $39.7$ | $57.2$ | $8.8$ | $5.5$ | $36.8$ | $20.8$ | $35.1$ | $-1.1$ | $-26.3$ | $5.4$ | $168.4$ | $11.9$ | $6.2$
GPT-2 Large | $-2.4$ | $83.2$ | $37.9$ | $36.0$ | $41.7$ | $77.4$ | $13.1$ | $7.7$ | $30.4$ | $18.4$ | $39.3$ | $-1.3$ | $-25.6$ | $5.4$ | $182.0$ | $-1.1$ | $10.7$
GPT-2 XL | $11.6$ | $103.7$ | $38.3$ | $40.0$ | $47.9$ | $99.2$ | $5.7$ | $9.6$ | $26.5$ | $39.2$ | $41.8$ | $2.9$ | $-25.6$ | $4.7$ | $193.2$ | $-1.8$ | $16.7$
OPT 125M | $-10.9$ | $54.5$ | $38.3$ | $26.0$ | $30.4$ | $25.2$ | $14.9$ | $3.5$ | $36.8$ | $8.0$ | $23.8$ | $-1.2$ | $-26.5$ | $-0.4$ | $186.8$ | $6.2$ | $0.6$
OPT 350M | $-6.8$ | $55.9$ | $38.3$ | $30.0$ | $34.4$ | $44.1$ | $13.1$ | $3.3$ | $36.8$ | $14.4$ | $29.9$ | $-0.9$ | $-26.4$ | $4.0$ | $178.8$ | $23.4$ | $5.4$
OPT 1.3B | $9.6$ | $96.3$ | $21.6$ | $46.0$ | $48.4$ | $113.7$ | $0.8$ | $7.5$ | $32.8$ | $32.0$ | $45.3$ | $2.8$ | $-21.7$ | $3.2$ | $198.0$ | $63.5$ | $19.2$
OPT 2.7B | $18.8$ | $117.2$ | $-30.2$ | $46.0$ | $56.8$ | $138.8$ | $5.7$ | $6.5$ | $35.8$ | $53.6$ | $49.6$ | $2.4$ | $-13.4$ | $9.0$ | $206.0$ | $3.4$ | $22.2$
OPT 6.7B | $33.8$ | $136.4$ | $-5.3$ | $58.0$ | $61.4$ | $168.4$ | $11.8$ | $-1.6$ | $24.5$ | $56.8$ | $54.2$ | $1.8$ | $-21.6$ | $10.5$ | $226.4$ | $52.8$ | $30.5$
T5 Small | $-14.3$ | $17.8$ | $31.0$ | $-4.0$ | $16.3$ | $6.2$ | $10.0$ | $7.8$ | $37.3$ | $11.2$ | $12.1$ | $-0.6$ | $-20.9$ | $-6.1$ | $143.2$ | $41.7$ | $-0.7$
T5 Base | $-11.3$ | $18.4$ | $-37.3$ | $-4.0$ | $30.0$ | $22.3$ | $12.4$ | $77.3$ | $-36.8$ | $27.2$ | $11.3$ | $0.9$ | $26.4$ | $30.7$ | $172.4$ | $14.7$ | $0.2$
T5 Large | $0.3$ | $22.2$ | $-38.3$ | $2.0$ | $34.8$ | $55.8$ | $1.4$ | $5.4$ | $-36.8$ | $30.4$ | $11.3$ | $1.1$ | $26.4$ | $10.5$ | $224.0$ | $0.5$ | $13.0$
T5 3B | $9.9$ | $74.6$ | $-21.2$ | $6.0$ | $43.1$ | $91.2$ | $4.5$ | $61.8$ | $35.3$ | $28.0$ | $35.9$ | $7.4$ | $35.3$ | $40.8$ | $255.2$ | $77.5$ | $1.3$
T5 11B | $27.6$ | $72.1$ | $-38.3$ | $18.0$ | $47.5$ | $109.0$ | $10.0$ | $43.6$ | $-33.8$ | $31.2$ | $43.7$ | $14.0$ | $26.2$ | $29.2$ | $186.8$ | $75.5$ | $5.1$
Finetuned MLMs | BERT Base Cased | $33.1$ | $116.2$ | $68.4$ | $38.0$ | $54.3$ | $53.8$ | $26.0$ | $152.0$ | $72.5$ | $107.2$ | $33.2$ | $80.4$ | $81.4$ | $36.5$ | $184.8$ | $86.2$ | $3.9$
BERT Large Cased | $38.9$ | $135.5$ | $70.3$ | $36.0$ | $57.0$ | $82.2$ | $29.6$ | $159.0$ | $73.5$ | $132.8$ | $40.4$ | $83.3$ | $82.0$ | $50.2$ | $198.0$ | $87.8$ | $28.5$
BERT Base Uncased | $34.8$ | $122.9$ | $67.8$ | $40.0$ | $57.0$ | $60.8$ | $33.3$ | $152.9$ | $73.0$ | $132.0$ | $30.7$ | $82.6$ | $82.0$ | $37.9$ | $182.0$ | $87.4$ | $4.3$
BERT Large Uncased | $38.6$ | $136.5$ | $68.9$ | $48.0$ | $54.3$ | $91.8$ | $37.0$ | $159.0$ | $74.5$ | $134.4$ | $18.3$ | $83.2$ | $82.5$ | $51.6$ | $194.0$ | $89.0$ | $3.9$
DeBERTa-v3 Small | $73.4$ | $163.8$ | $73.3$ | $58.0$ | $68.3$ | $198.2$ | $26.6$ | $163.4$ | $82.4$ | $140.0$ | $58.3$ | $83.7$ | $83.8$ | $54.5$ | $223.6$ | $89.0$ | $32.8$
DeBERTa-v3 Base | $118.4$ | $200.0$ | $75.1$ | $78.0$ | $87.1$ | $252.8$ | $51.2$ | $170.2$ | $78.9$ | $184.0$ | $64.0$ | $88.1$ | $84.7$ | $71.8$ | $238.0$ | $92.0$ | $65.1$
DeBERTa-v3 Large | $178.8$ | $246.5$ | $77.2$ | $96.0$ | $107.1$ | $276.1$ | $72.7$ | $172.9$ | $82.4$ | $232.8$ | $76.0$ | $90.3$ | $85.6$ | $83.4$ | $254.8$ | $92.0$ | $75.7$
DeBERTa-v2 XLarge | $34.1$ | $204.4$ | $71.4$ | $48.0$ | $67.4$ | $269.4$ | $1.4$ | $171.8$ | $77.9$ | $200.0$ | $60.0$ | $86.8$ | $83.9$ | $74.0$ | $233.6$ | $93.8$ | $73.3$
RoBERTa Base | $34.8$ | $115.8$ | $67.2$ | $44.0$ | $55.0$ | $151.1$ | $25.3$ | $162.3$ | $79.4$ | $113.6$ | $39.0$ | $83.4$ | $83.0$ | $59.6$ | $158.4$ | $90.1$ | $31.2$
RoBERTa Large | $75.8$ | $158.4$ | $74.1$ | $80.0$ | $43.7$ | $233.5$ | $31.5$ | $170.6$ | $80.4$ | $109.6$ | $25.4$ | $87.9$ | $83.2$ | $76.9$ | $196.0$ | $93.3$ | $54.1$
Table 3: Improvement relative in percentage compared to the random baseline on
17 common tasks