# BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Thomas Scialom
Sorbonne Université, CNRS, LIP6, F-75005
reciTAL, Paris, France
<EMAIL_ADDRESS>
Felix Hill
DeepMind
<EMAIL_ADDRESS>

###### Abstract

Natural language processing (NLP) systems are increasingly trained to generate
open-ended text rather than classifying between responses. This makes research
on evaluation _metrics_ for generated language – functions that score system
output given the context and/or human reference responses – of critical
importance. However, different metrics have different strengths and biases,
and reflect human intuitions better on some tasks than others. There is
currently no simple, unified way to compare, analyse or evaluate metrics
across a representative set of tasks. Here, we describe the Benchmark to
Evaluate Automatic Metrics (BEAMetrics), a resource to make research into new
metrics _itself easier to evaluate_. BEAMetrics users can quickly compare
existing and new metrics with human judgements across a diverse set of tasks,
quality dimensions (fluency vs. coherence vs. informativeness etc), and
languages. As generation experts might predict, BEAMetrics reveals stark task-
dependent differences between existing metrics, and consistently poor
performance on tasks with complex answer spaces or high reliance on general
knowledge. While this analysis highlights a critical issue facing current
research practice, BEAMetrics also contribute to its resolution by
facilitating research into better metrics – particularly those that can
account for the complex interaction between context and general knowledge
inherent to many modern NLP applications.111BEAMetrics is available under the
MIT License: https://github.com/ThomasScialom/BEAMetrics

##  1 Introduction

In the past, natural language generation (NLG) [65] was important only for a
subset Natural Language Processing (NLP) applications. Today, driven in part
by the success of large autoregressive language models [5], many NLP
applications involve models that generate running text from open-ended
vocabularies (see e.g. [10, 8, 79]). Beyond classic NLG applications like
machine translation and summarization, tasks that were previously framed as
classification problems – such as open-domain question answering – are also
now addressed by NLG systems. This trend is undoubtedly positive, since it
points towards more general, flexible and expressive language technology.
However, one side-effect is that NLP as a whole inherits the challenges
inherent to NLG; in particular the issue of how to effectively evaluate
systems that generate open-ended text.

Evaluating NLG systems is notoriously difficult [51, 71, 69]. The ‘ideal’
approach (matching the setting of a deployed system) is to show system outputs
to humans and have them rate their quality along different dimensions such as
’fluency‘ or ‘informativeness‘. However, doing so is expensive, time-consuming
and can be hard to replicate. Research therefore also relies critically on
automatic evaluation metrics: functions that score system outputs given either
human reference responses, the input context, or both.

Figure 1: The challenge faced by metrics evaluating modern NLG systems Example
output from a system built with CLIP [60] and GPT2 [59] on the OK-VQA [45]
task. The next generation of metrics must find a way to reasonably assign
credit to such responses given the available context and/or reference human
responses.

Automatic metrics are generally quick to apply and consistent making them
essential for enabling fast iteration when designing systems. Despite their
obvious utility, however, there is no known metric (or set of metrics) that
adequately reflects human intuitions across the growing spectrum of NLG-
relevant tasks [69]. Moreoever, as illustrated in Figure 1, many of the new
application areas for NLG models pose the greatest challenges for evaluation.
Given that NLG research depends critically on metrics, and NLP as a whole
relies increasingly on NLG, the need for new approaches to evaluation metrics
has never been greater.222This view is certainly shared, for instance, by
Ruder [67], who argues that _we need to rethink how we design our benchmarks
and evaluate our models so that they can still serve as useful indicators of
progress._

With this motivation, we propose _BEAMetrics_ , the Benchmark to Evaluate
Automatic Metrics. BEAMetrics is a resource for _evaluation evaluation_ \- a
simple and easy way to fairly compare and evaluate evaluation metrics. With
the code provided, users of BEAMetrics can quickly and easily compare the
predictions of new NLG metrics against a fixed set of human judgements across
different quality dimensions, NLG tasks and languages. It therefore removes
any need for the developers of new evaluation metrics to conduct their own
human evaluations, and in doing so provides a consistent basis for detailed
analysis and replicable comparisons. As such, we hope that BEAMetrics will
motivate a concerted focus among researchers to consider and improve
evaluation metrics, in much the same way that the ImageNet Challenge [13] and
GLUE [81] stimulated progress in image and language classification
respectively.

##  2 Related Work

###  2.1 Benchmarks

Benchmarks have always played a key role in both guiding and measuring the
progress of AI research. The ImageNet challenge [68], for instance, played an
important role in revealing the potential of Deep Learning [33]. Many
benchmarks today assess the generality of a system, by aggregating suites of
related tasks. A notable example of this is the recent Abstraction and
Reasoning Corpus (ARC) [9], psychometric intelligence test composed 600 unique
tasks, to measure the level of general intelligence in artificial systems.

Language is a highly general and multi-faceted domain, and benchmarks are
playing an increasingly important role in NLP: the General Language
Understanding Evaluation benchmark (GLUE) contributing to highlight the
potential of self-supervised learning for language problems [58, 14]. GLUE
encompasses 9 different Language _classification_ tasks including sentiment
analysis [78], sentence similarity [15, 7], Natural Language Inference [12,
83], Question Answering [62], and coreference [37]. While state-of-the-art
approaches have now achieve a human-level performance on GLUE [42, 38, 61],
Wang et al. [82] proposed a more challenging successor, SuperGLUE.

Most recently, Gehrmann et al. [22] proposed GEM, a benchmark specifically for
tasks requiring Natural Language _Generation_. GEM encompasses 4 tasks through
11 datasets: Summarization [72, 49, 19], Structure To Text [20, 40, 17, 57,
16, 54], Dialogue [63], and Simplification [25, 85, 1]. As noted in the
introduction, evaluation of generated language relies not only on tasks, but
also on automatic metrics (or humans). The developers of GEM provide trusted
implementations of known metrics such as BLEU, ROUGE, BERTscore
etc333https://github.com/GEM-benchmark/GEM-metrics. However, because there is
currently no way to assess the trustfulness of metrics, GEM also encourages
human evaluation during the workshops, 444https://gem-benchmark.com/workshop.

###  2.2 Evaluating Metrics

While human evaluation will always be valuable, practicalities mean that
effective automatic metrics can play a critical role in guiding and enabling
research on NLG problems. The limitations of current automatic metrics,
however, are well known. Callison-Burch et al. [6] showed that a BLEU
improvement is neither necessary nor sufficient for achieving an actual
improvement. Louis & Nenkova [43] showed similar findings regarding other
metrics like ROUGE. Choshen & Abend [11] demonstrated the inherent presence of
biases in reference-based evaluation.

Novikova et al. [51] make an impassioned appeal for ‘why we need new
evaluation metrics for NLG’. They investigated a wide range of metrics, and
reported that human judgement is only weakly reflected. Importantly, they also
showed that metric performance is very often task, and even dataset dependent.
This finding indicates the importance of evaluating and understanding the
performance of metrics on _multiple tasks_.

While metrics may be valuable for evaluating any NLG task, many of the best
known metrics, like BLEU score, were primarily designed for evaluating Machine
Translation (MT) systems. In a large and comprehensive analysis of metrics in
MT, however, [29] recommends deprecating BLEU as the MT evaluation standard.
They suggest using more recent metrics such as BERTScore [87] or COMET [64],
that are shown to reflect human judgement significantly better.

Given the impressive performance on MT for BERTScore, its adoption on other
tasks has now started, for instance in Summarization [39, 35, 53, 26, 86].
Unfortunately, BERTScore actually performs weakly on Summarization [18, 4].
This pattern reflects a general tendency for researchers to apply a metric
designed or proven for one NLG task to some other task, in apparent ignorance
of its limited scope. This in turn emphasises the need for an easy and
replicable way to quantify the performance of new or existing metrics across a
representative sample of NLG tasks. Such a resource would also encourage the
research community to develop metrics that generalize beyond only one task.

##  3 The BEAMetrics benchmark

Dataset | Abbreviation | Task | Languages | Dimensions | Nb Ex
---|---|---|---|---|---
WebNLG | Web | Data2text | EN | Cor, Flu, Gra | 2,007
Asset-Eval | Asv | Simplification | EN | Cor, Flu, Sim | 162
MUSS | MUS | Simplification | EN, FR, ES | Cor, Flu | 150
WMT2019 | WMT |  | Machine
---
Translation
| EN, DE, FI, GU,
---
KK, LT, RU, ZH
Cor | 480,332
Pascal50s | Pas | Captioning | EN | Cor | 4,000
Flickr8k | Fli | Captioning | EN | Cor | 5,664
RealSum | Rea | Summarization | EN | Cor | 2,500
SummEval | SumE | Summarization | EN | Cor, Rel, Coh, Flu | 1,600
MultiSummEval | mSu | Summarization |  | EN, DE, ES, FR,
---
RU, TR, ZH, ID
Cor, Rel | 2,160
Efficient QA | OpQA | QA | EN | Cor, Obv | 9,000
OkVQA | OkVQA | VQA | EN | Cor, Obv, Pos | 300
Table 1: Datasets included in the BEAMetrics benchmark, and their
characteristics. They are detailed on Section 3.3. The dimensions are defined
in 5.1; Cor stands for Correctness, Flu: Fluency, Gra: Grammar, Sim:
Simplicity, Rel: Relevance, Coh: Coherence, Obv: Obviousness, and Pos:
Possibility.

###  3.1 Design principles

In constructing a benchmark for metrics, we set out various desiderata and
priorities based on what has worked well in other benchmarks in the
literature. In particular, the benchmark should reflect the metric
generalisation across three main axes: tasks, languages, and evaluated
dimensions:

#### Multilingual

NLP research has mostly focused on English while 7000+ languages are spoken
around the world.555https://ruder.io/nlp-beyond-english/. Token level metrics
like ROUGE or BERTScore are not suited to all the language morphologies [36].
We want _BEAMetrics_ to measure the multilingual ability of a metric.

#### Multitask

As noted above, the panorama or NLG-relevant tasks in NLP is rapidly
expanding. The performance of metrics should be known on a representative
sample of these tasks. In this aspect, BEAMetrics draws on the impact of
general multitasks benchmarks like GLUE.

#### Evaluated Dimensions

In many cases, the quality of textual output can (and should) be evaluated
along different dimensions, as this better capturing the multi-faceted
function and nature of language. Overall, BEAMetrics contains evaluations for
9 such dimensions: adequacy, correctness, fluency, simplicity, grammar,
relevance, coherence, possibility, and obviousness. Each of these dimensions
are defined in the data-card of the corresponding dataset (see Section 3.2
about the data-cards).

Beyond these three axes, we also aimed for high diversity among the collected
data:

  * •

different dataset covering various domains, and with different
characteristics, e.g. long Vs short references;

  * •

diversity in the systems evaluated: state-of-the-art systems as well as older;
extractive and abstractive models;

  * •

different evaluation protocols, based on different scales (likert, ranking).

This diversity should add the robustness of conclusions drawn from BEAMetrics,
and help to stimulate the development of increasingly general metrics -
potentially sharing knowledge across ’tasks’.

Finally, the datasets we selected in BEAMetrics have all been peer reviewed
and publicly released.

Overall, across its 11 datasets, BEAMetrics is multi[task, lingual and
dimensional], as we show in Table 1. For each dataset, we provide a complete
data statement, as presented in the next Section.

###  3.2 Data Statement

To mitigate system bias, improve reproducibility and enable better science,
[3] and [21] recommend implementing a Data Statement. It consists of a
standardized process of documenting the datasets.

We therefore propose a template card that contains the important information
about a human evaluation dataset: the evaluation set(s) used to generate the
texts, the language(s), the task, the number of human references available for
each examples, the evaluated dimensions and their exact definition in the
original paper, the annotation protocol and the rating scale, information
about the annotators, any additional comments, and finally the citation of the
paper.

For each dataset that composed BEAMetrics, we make available the template card
completed.666https://github.com/ThomasScialom/BEAMetrics/tree/main/data/datacards
Note that a data card is directly integrated in the BEAMetrics code: it is
generated by filling our proposed template with the information that are
provided in the configuration file of the dataset. This makes the process both
standardized and mandatory, when a a new dataset is added to the suite.

###  3.3 Selected Datasets

WMT2019 (WMT) We consider 7 language pairs from the WMT19 metrics shared task
[44]: de-en, fi-en, gu-en, kk-en, lt-en, ru-en, and zh-en. We rely on the DaRR
corpus corresponding to an evaluation via Direct Assessment from 1 to 100,
followed by Relative Ranking over 480,332 outputs.

SummEval (SumE) [18] is one the largest studies in Summarization. It includes
the evaluation of 1,600 generated summaries on the CNN/DM corpus [48] from 16
models on 100 source documents ($16*100=1600$). The annotators rated the
summaries on four dimensions: Fluency, Correctness, Coherence and Relevance. A
notable aspect of this study is the use of expert annotators, which is
important given the difficulty of evaluating summaries, even for humans.

SumEval-multi (mSu) [31] is a recent multilingual extension of SummEval. It
contains the evaluation of 2,160 summaries from four multilingual datasets
including a total of 8 languages: French, Spanish, German, Russian, and
Turkish from MLSUM [72], Indonesian from Liputan6 [30], English from CNN/DM
[10], and Chinese from LCSTS [24]. Two dimensions are evaluated: the precision
and the recall for the outputs of 2 abstractive systems: i) Pointer Generator
[74] a model with copy mechanism and not pretraining, and ii) BERT-gen [70].

REALSumm (Rea) [4] is an evaluation on 2,500 system generated summaries, that
includes outputs for 14 abstractive and 11 extractive models. The annotation
protocol is based on the lightweight-pyramid [76], a cost effective adaption
of the pyramid [50] method that allows to use Mechanical Turk. In lightweight-
pyramid, the time consuming merge of duplicate Semantic Content Units (SCU)
from different reference summaries is replaced by a simple SCU sampling.

Asset-eval (Asv)) [2] is a corpus of a total of 9,000 ratings that was
released along with the ASSET corpus. Mechanical Turkers rated on a Likert
Scale the Fluency, Correctness, and Simplicity of the system-generated
simplifications.

MUSS-eval (MUS) [46] is a human evaluation of 150 text simplification outputs
in three languages (English, French, and Spanish). The annotators were
volunteers that rated each sample over three dimensions: Simplicity, Adequacy,
and Fluency.

WebNLG-eval (Web): [77] consists of a set of 2,000 English descriptions of
structured tables generated by 10 different systems and annotated on their
Fluency, Correctness and Grammar. This corpus was provided along with the
WebNLG Data-To-Text generation corpus, which consists of tables and
corresponding descriptions in 16 DBPedia categories (e.g., Airport, Astronaut,
etc.).

Flickr8k (Fli): [23] is an image captioning evaluation, where three human-
expert annotators rated the relevance of 5,822 generated captions regarding
their source image, from 1 to 4.

PASCAL50S (Pas): [80] is an image captioning evaluation over 4k examples. It
evaluation is a comparison between two generated captions: the annotators were
asked to judge which caption is more appropriated given 50 references of the
given image.

NeurIPS Question Answering (OpQA) [47] corresponds to the human evaluation
conducted for the Efficient QA track at the NeurIPS 2020
competition.777https://efficientqa.github.io/ The Top 5 QA systems were
manually evaluated over 1,800 questions. The raters marked each of the 9,000
generated answer ($1,800*5$) as either ‘definitely correct’, ‘plausibly
correct’, or ‘definitely incorrect’. We therefore consider the two following
dimensions: i) _Correctness_ : True if the answer is definitely or plausibly
correct, and False if it is incorrect. ii) _Obviousness_ : True if the answer
is definitely correct, and False otherwise.

OKVQA-Eval (OkVQA) OKVQA [45] is a VQA dataset with questions about images
that require Outside Knowledge to be integrated with the information in the
image, see for instance two examples in Figure 1. We chose to conduct a human
evaluation on OKVQA responses and to include them in BEAMetrics, since this
task emphasises an important challenge that metrics of the future may need to
address. The information required to resolve an OKVQA question is not entirely
contained within the context image, so that answering questions requires
’outside knowledge’. Consequently, a complete evaluation of model output must
also require outside knowledge (See 5.3 for discussion of this point).

To get reasonable model responses on OKVQA for humans to assess, we generated
answers using an NLG system that combines two well-known large-scale data-
intensive models: CLIP [60] and GPT-2 [59]. The model integrates CLIP ouput
into the GPT-2 input space via a non-convex combination of word
embedddings.888The model was developed by Jamie Kiros [28], and is available
at
https://colab.research.google.com/drive/1fokumWeasHTo0KXpfeZ6Z0OgLOQ2SUso?usp=sharing.

The annotators were given an image-question pair, and the generated answer,
and were asked to evaluate three different dimensions:

  1. 1.

Correctness: Is the answer definitely factually correct (use Google if
necessary)?

  2. 2.

Possibility: In your opinion, is the answer possible in some possible
situation?

  3. 3.

Obviousness: If the image was shown to 100 people and the question was asked,
how many people do you think would give the answer? This question were
inspired by the TV Show Family Fortune
999https://en.wikipedia.org/wiki/Family_Fortunes and can be interpreted as
representing the mass probability among the possible answers.

In term of inter-rater agreement, we obtain a Krippendorff alpha [32] of 0.67
for Correctness, 0.74 for Obviousness and 0.48 for Possibility. The
significantly higher alpha for Obviousness than Possibility indicates that
annotators agree more on the mass probability among the possible answers
allocated by other humans, rather than the Possibility of the given answer.

Additional details are given in the Appendix, including among others the full
annotation protocol, a screenshot of the annotation tool, the average time to
annotate an answer.

##  4 Metrics

Numerous metrics have been proposed to evaluate NLG systems. Several surveys
have made interesting taxonomies [69, 29] to regroup metrics regarding some
fundamental distinctions, such as being parametric (i.e. supervised) Vs rule-
based, acting at the sequence level Vs at a token level, or the token
representation being neural based Vs n-gram based.

In this paper, our focus is to measure the ability of the metrics. Therefore,
beyond there internal characteristics, we group the metrics regarding only the
inputs they used to evaluate a text:

  1. 1.

Reference-based: metrics that leverage the reference(s) to evaluate the text.
The source can also be used additionally;

  2. 2.

Reference-less: metrics that use only the source, without requiring any
reference. Note that this setup corresponds to the namely Quality Estimation
(QE) in MT.

As baselines, we consider n-gram statistics, standard metrics like BLEU,
ROUGE, or METEOR, and recent neural metrics like BERTScore, Nubia and BLEURT.

BLEU [52] measures the overlap of n-grams between the evaluated text and its
reference(s).

ROUGE [41] stands for Recall-Oriented Understudy for Gisting Evaluation.
Similarly to BLEU, it is based on the count of overlapping n-grams, but it
recall oriented.

METEOR [34] was proposed to fix some of the problems in BLEU. While BLEU seeks
correlation at the corpus level, METEOR provide scores at the sentence level.

BERTScore [87] leverages the contextualised representation of BERT to compute
the similarity between the tokens.

BLEURT [75] is a learned evaluation metric based on BERT, then fine-tuned on
WMT human annotations to emulate the annotators.

Nubia [27] stands for NeUral Based Interchangeability Assessor. It is composed
of three modules: first, a module that obtains different neural
representations of the evaluated text, then an aggregator of the
representations, and finally a calibration module.

GPT2 Perplexity [59] We report the perplexity of the evaluated text using
GPT2.

Statistics We also report the correlations of simple heuristics:

  * •

Length: The number of n-grams in the evaluated text;

  * •

Repetitions: the number of n-grams that are repeated multiple times,
normalised by the length;

  * •

Abstractness: the number of n-grams not present in the source text, normalised
by the length.

All the metrics are integrated in the BEAMetrics framework using either the
official implementation (e.g. for Nubia), a trustable version (e.g. SacreBLEU
[55], the HuggingFace implementation [84].

##  5 Results and Discussion

| Dataset: | WMT | Web | Asv | MUS | Pas | Fli | mSu | Rea | SumE | OpQA | OkVQA | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | MT | DTG | Sim | Sim | ImCa | ImCa | Sum | Sum | Sum | QA | VQA | All
Max | ROUGE-1 | 16.0 | 63.6 | 61.8 | 41.6 | 52.3 | 48.7 | 50.7 | 47.4 | 16.7 | 35.5 | 19.3 | 14.2
ROUGE-L | 16.8 | 60.9 | 59.4 | 40.9 | 52.0 | 49.4 | 52.9 | 42.6 | 14.2 | 35.4 | 19.8 | 13.9
BLEU | 15.5 | 61.3 | 47.6 | 32.7 | 50.3 | 52.0 | 48.0 | 37.6 | 11.7 | 10.8 | 15.5 | 12.0
METEOR | 16.2 | 63.7 | 65.8 | 40.6 | 56.0 | 56.5 | 46.3 | 53.7 | 17.3 | 33.7 | 5.1 | 14.2
BERTScore P | 20.0 | 60.6 | 69.9 | 37.9 | 49.6 | 48.2 | 61.6 | 29.3 | 9.1 | 11.5 | 4.8 | 12.6
BERTScore R | 20.0 | 72.9 | 73.3 | 36.3 | 53.5 | 41.1 | 60.4 | 45.4 | 14.7 | 13.1 | 15.9 | 14.0
BERTScore F1 | 20.5 | 62.1 | 73.4 | 37.5 | 52.6 | 47.4 | 62.8 | 39.3 | 13.1 | 12.4 | 9.9 | 13.5
BLEURT | 22.8 | 68.4 | 79.9 | 37.7 | 57.3 | 60.6 | 46.3 | 34.1 | 9.4 | 22.6 | 18.4 | 14.3
Nubia | 22.1 | 78.7 | 62.2 | 43.5 | 52.9 | 58.6 | 37.7 | 12.5 | 6.0 | 33.5 | 13.8 | 13.2
1 | ROUGE-1 | 16.0 | 69.7 | 47.9 | 41.6 | 43.4 | 37.1 | 50.7 | 47.4 | 17.9 | 35.5 | 19.5 | 13.3
ROUGE-L | 16.8 | 61.2 | 43.0 | 40.9 | 41.4 | 38.2 | 52.9 | 42.6 | 15.7 | 35.4 | 19.5 | 12.7
BLEU | 15.5 | 53.6 | 29.9 | 32.7 | 29.5 | 32.2 | 48.0 | 37.6 | 7.0 | 10.8 | 19.0 | 9.9
METEOR | 16.2 | 67.9 | 52.2 | 40.6 | 42.9 | 41.6 | 46.3 | 53.7 | 16.2 | 33.7 | 5.7 | 13.0
BERTScore P | 20.0 | 59.2 | 45.8 | 37.9 | 37.3 | 36.7 | 61.6 | 29.3 | 9.1 | 11.5 | 3.6 | 11.0
BERTScore R | 20.0 | 70.8 | 66.3 | 36.3 | 45.9 | 25.2 | 60.4 | 45.4 | 14.1 | 13.1 | 7.4 | 12.7
BERTScore F1 | 20.5 | 60.8 | 61.4 | 37.5 | 43.9 | 33.5 | 62.8 | 39.3 | 12.4 | 12.4 | 6.2 | 12.2
BLEURT | 22.8 | 77.1 | 68.1 | 37.7 | 51.6 | 53.2 | 46.3 | 34.1 | 9.8 | 22.6 | 15.2 | 13.7
Nubia | 22.1 | 78.7 | 62.2 | 43.5 | 52.9 | 58.6 | 37.7 | 12.5 | 6.0 | 33.5 | 13.8 | 13.2
0 | Abstr-1 | 1.4 | - | -14.4 | -58.8 | - | - | - | 8.9 | -19.3 | -8.7 | 5.9 | -2.7
Abstr-3 | 1.3 | - | -32.2 | -55.7 | - | - | - | -2.8 | -35.2 | 13.4 | 7.9 | -3.2
Length | 1.7 | 19.4 | 19.4 | 16.6 | 15.1 | -16.2 | -4.4 | 29.4 | 8.1 | 14.9 | -9.4 | 3.0
Repet-1 | 1.7 | 18.5 | 19.4 | 16.6 | 14.6 | -16.2 | -4.4 | 29.4 | 8.1 | 14.9 | -9.4 | 2.9
Repet-3 | 1.7 | 16.8 | 19.4 | 16.6 | 15.0 | -16.2 | -4.3 | 29.4 | 8.1 | 12.8 | -9.8 | 2.8
-GPT2 Perpl. | 3.5 | 24.7 | 23.1 | 10.5 | 14.4 | -2.9 | 3.0 | 12.6 | -3.1 | 1.1 | 8.6 | 3.0
Table 2: Correctness dimension: Pearson coefficient between automatic metrics
and human judgement for Correctness on the 10 human evaluation datasets. The
top bloc corresponds to coefficients computed when all the human references
were available. The second bloc corresponds to coefficients computed given a
single human reference. The third bloc corresponds to coefficients computed
given no human reference.

###  5.1 Results Display

As discussed in Section 3.1, several different dimensions are represented in
BEAMetrics. Averaging all the results into a single number can limit the
interest of the benchmark, and could be an incentive for “leaderboard chasing”
approaches. On the other hand, the large number of datasets and evaluated
dimensions make the final results difficult to interpret at a glance, if not
clearly structured. For this reason, we propose to group the results into two
distinct tables: First, Correctness: the correctness of the predication given
the context. This is arguably the principal dimension: is of of the utmost
importance for a text to be factually consistent with its context. This also
explains why each evaluation set described in Section 3.3 has included this
dimension in their evaluation. In the second Table, we regroup the other
dimensions:

  * •

Fluency: The quality of individual sentences.

  * •

Coherence: The text should be well-structured and well-organized, not just be
a heap of related information.

  * •

Simplicity: In Simplification, is the text easy to read, and composed of
simple words?

  * •

Relevance: In Summarization, the selection of important content from the
source.

  * •

Possibility: In QA, is the answer possible?

  * •

Obviousness: In QA, how expected is the answer, in the context?

We can consider the correlations among the different tasks for Correctness, as
consistent all together, and hence also report the average score. Conversely,
we don’t report any average for the second Table, as it contains correlations
over unrelated dimensions.

Finally, in each tables we will report the correlations in three separate
blocs, that correspond to three different scenarios: i) The top bloc contains
reference-based metrics that have access to _all_ the references available in
the corpus. It indicates the metric potential given a large number of
references. ii) The middle bloc, contains reference-based metrics that have
access to _only one_ reference. It indicates the metric performance in a more
likely scenario, given that most of modern datasets provide only a single
reference. iii) At the bottom bloc, the reference-less metrics results. Those
metrics are more challenging, but enable to evaluate a text without requiring
a gold-reference.

| Dataset: | Web | Asv | Asv | MUS | mSu | SumE | SumE | SumE | OpQA | OkVQA | OkVQA
---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | DTG | Sim | Sim | Sim | Sum | Sum | Sum | Sum | QA | VQA | VQA
Dim: | Flu | Flu | Sim | Flu | Rel | Rel | Coh | Flu | Obv | Pos | Obv
Max | ROUGE-1 | 51.0 | 42.0 | 42.4 | 26.1 | 50.9 | 32.3 | 18.1 | 13.6 | 41.7 | 14.3 | 29.1
ROUGE-L | 52.9 | 40.9 | 41.0 | 25.4 | 52.8 | 24.8 | 15.5 | 11.0 | 41.6 | 14.3 | 29.5
BLEU | 54.2 | 28.9 | 29.5 | 22.4 | 49.6 | 29.0 | 25.0 | 13.9 | 12.0 | 7.2 | 20.6
METEOR | 50.3 | 41.3 | 38.5 | 26.6 | 54.9 | 36.0 | 16.8 | 11.4 | 40.1 | 0.2 | 3.6
BERTScore P | 60.6 | 57.0 | 57.1 | 21.1 | 52.6 | 24.7 | 31.0 | 16.3 | 10.5 | 14.1 | 18.2
BERTScore R | 64.7 | 56.9 | 50.1 | 21.2 | 65.0 | 39.9 | 33.8 | 13.8 | 11.1 | 13.2 | 24.3
BERTScore F1 | 60.3 | 58.0 | 54.7 | 21.4 | 60.3 | 34.0 | 34.7 | 16.7 | 10.9 | 14.6 | 22.5
BLEURT | 56.1 | 64.6 | 57.8 | 31.8 | 43.5 | 27.3 | 14.2 | 17.2 | 21.2 | 30.9 | 27.6
Nubia | 50.4 | 43.6 | 39.3 | 24.3 | 29.4 | 14.4 | 7.2 | 7.5 | 38.2 | 8.9 | 20.2
1 | ROUGE-1 | 55.4 | 33.7 | 31.2 | 26.1 | 50.9 | 33.3 | 18.8 | 13.6 | 41.7 | 13.8 | 28.9
ROUGE-L | 52.0 | 31.8 | 28.5 | 25.4 | 52.8 | 26.8 | 18.2 | 12.1 | 41.6 | 13.8 | 28.9
BLEU | 43.8 | 25.6 | 23.5 | 22.4 | 49.6 | 20.8 | 13.0 | 6.9 | 12.0 | 9.7 | 23.5
METEOR | 53.8 | 35.3 | 31.7 | 26.6 | 54.9 | 30.4 | 15.3 | 11.8 | 40.1 | 1.0 | 4.3
BERTScore P | 58.6 | 39.2 | 41.2 | 21.1 | 52.6 | 26.5 | 27.6 | 13.9 | 10.5 | 13.9 | 15.9
BERTScore R | 61.4 | 49.5 | 45.1 | 21.2 | 65.0 | 36.5 | 29.4 | 12.8 | 11.1 | 9.3 | 14.6
BERTScore F1 | 58.5 | 48.5 | 46.8 | 21.4 | 60.3 | 34.0 | 30.7 | 14.5 | 10.9 | 13.5 | 17.3
BLEURT | 64.0 | 55.3 | 48.7 | 31.8 | 43.5 | 28.1 | 14.4 | 14.8 | 21.2 | 26.7 | 23.6
Nubia | 50.4 | 43.6 | 39.3 | 24.3 | 29.4 | 14.4 | 7.2 | 7.5 | 38.2 | 8.9 | 20.2
0 | Abstr-1 | - | -10.2 | -7.5 | -31.1 | - | -24.5 | -27.2 | -13.3 | -5.4 | 11.3 | 7.4
Abstr-3 | - | -22.7 | -14.3 | -26.4 | - | -31.0 | -30.5 | -26.8 | 9.6 | 9.0 | -2.3
Length | -1.7 | 2.5 | -0.8 | 6.9 | 10.5 | 26.6 | 8.6 | -2.9 | 11.0 | -19.0 | -16.4
Repet-1 | -2.8 | 2.5 | -0.8 | 6.9 | 10.5 | 26.6 | 8.6 | -2.9 | 11.0 | -19.0 | -16.4
Repet-3 | -5.0 | 2.5 | -0.8 | 6.9 | 10.6 | 26.6 | 8.6 | -2.9 | 10.2 | -19.7 | -16.6
-GPT2 Perpl. | 16.2 | 22.9 | 20.4 | 12.1 | 1.9 | 19.8 | 15.7 | 8.9 | 0.2 | -4.9 | 4.6
Table 3: Non Correctness dimensions: Pearson coefficient between automatic
metrics and human judgement for the dimensions other than Correctness. The top
bloc corresponds to coefficients computed when all the human references were
available. The second bloc corresponds to coefficients computed given a single
human reference. The third bloc corresponds to coefficients computed given no
human reference.

###  5.2 Results

We report the Pearson Coefficients for the Correctness and the Other
dimensions, respectively in Tables 2 and 3. Note that we also report the
Kendall Tau correlations and the p-values in the Appendix.

As expected, we observe that BLEU performs worst than the neural metrics (i.e.
BERTScore, Nubia and BLEURT). More surprising, the other n-gram based metrics,
namely ROUGE and METEOR are actually competitive with neural metrics: on
Correctness, BLEU obtains in average the worst performance (12.0), BLEURT
performs the best (14.3), but only slightly above ROUGE1 and METEOR (14.2). We
hypothesise that the poor performance of BLEU can be explained by its overused
trough time, a conclusion in line with a recent study [29]. This result also
gives a new light and perspective about the performances of recent neural
metrics compared to more standard ones.

An other unexpected result is the strong performance of BLEURT compared to the
other neural metrics. BLEURT is a supervised metric trained to predict the
annotator score of a given segments in MT. While we can expect BLEURT to
compare favorably on MT, its relatively good performance beyond MT is more
surprising. We note that few works outside MT have reported BLEURT, as opposed
to BERTScore. It will be worth exploring the reasons behind this good
performance in future works.

Finally, the average correlations are relatively low, which emphasize the need
to develop better metrics. We hope that BEAMetrics will contribute in this
direction, by providing off-the-shelf tools for researchers to evaluate
evaluation metrics.

###  5.3 The Future of Metrics

#### Question answering and NLG

To the best of our knowledge, ours is the first treatment to consider in depth
the evaluation challenges posed by systems that can answer questions (QA) with
open-ended running text. Indeed, QA is not a domain included in GEM, so a
fortiori has not been a major concern for those interested in the evaluation
of generated language. This is in part because QA was in the past normally
framed as a classification task e.g. in SQuAD [62]. Such a constraint does not
yield particularly flexible or expressive systems, however, and the advent of
better generative language models has recently placed both open-domain [66,
56] and visual [8, 79] QA tasks squarely within the scope of NLG.

As illustrated in Figure 1, question-answering in its most general form poses
arguably the greatest evaluation challenge of any NLG domain. For some
questions, there may be a single correct answer (a factoid), in which case a
metric must simply account for superficial variation in how that answer may be
expressed. However, for many other questions, there can be a complex space of
possible answers, characterised by gradual (rather than discrete) variation of
along quality dimensions _and_ multiple distinct answer ‘modes’ (semantically
distinct answers that are equally correct). This complexity points to the need
for knowledge-driven metrics that are themselves complex (potentially non-
linear) functions learned themselves from large and diverse human data.
Perhaps unsurprisingly, then, on the QA and VQA BEAMetrics tasks we observe
very low correspondence with human intuitions across all current automatic
metrics. Among those, even the most expressive learned metric, BLEURT, was
only trained on a single, very different, task (MT). By including two
question-answering tasks in BEAMetrics, we emphasize the importance of
developing smarter, more expressive metrics to support research on truly open-
ended, flexible NLG systems.

Figure 2: The performance of metrics drops substantially for tasks with a more
complex answer space and/or that require greater general knowledge. Pearson
correlation with human ratings, on the Correctness dimension for the tasks in
English. Green line: an average of n-gram based metrics (i.e. ROUGE, BLEU,
METEOR); Red line, average of neural based metrics (i.e. BERTScore, BLEURT,
Nubia).

#### A problematic trend

This consideration of what makes QA a difficult setting to evaluate NLG
systems also points to a general axis of variation among the tasks included in
BEAMetrics. For classical NLG settings like MT, the space of responses for a
given stimulus is relatively unimodal. In addition, to evaluate a translation,
one may require comparatively little general knowledge beyond what information
is contained in the stimulus and reference responses. A task like abstractive
summarization increases the difficulty along both of these axes. For
summarization, there may be two equally good summaries that differ somewhat in
meaning (i.e. the answer space is _less_ unimodal). Further, good abstractive
summarization arguably requires a degree more general knowledge than in the
case for MT. While image captioning may require somewhat less general
knowledge, the answer space is clearly not unimodal - various semantically
distinct captions can be appropriate for a given image. Question answering can
often exhibit the highest degree of difficulty along both of these axes. As
shown in Figure 2, the performance of current evaluation metrics (both n-gram
based and neural) degrades quite consistently as one moves along these axes.
Importantly, this phenomenon seems amplified for neural metrics, which
actually perform worse than n-gram ones on Summarization, Open-ended QA and
OkVQA.

This analysis suggests that developing good metrics for MT – where most
research on metrics to date has focused – may be a categorically different
challenge than developing those for other types of NLG systems. Research on
metrics for QA must be expressive enough to model a complex distribution of
potential answers, and may have to focus to a greater extent than current
approaches on both the integration of general knowledge and the consideration
of both context. Of course, it is possible that the task of perfectly
evaluating an open-ended NLG system, is almost as challenging as performing
the task itself. A metric does not have to be perfect to be valuable, however;
but it does need to reach a certain level of quality to provide a useful
signal for guiding research.

###  5.4 Limitations and Maintenance Strategy

While systems performance improve, and the quality of their outputs increases,
it becomes more challenging for metrics to reflect human preference [73].
Therefore, it is important to keep evaluating the metrics not only on outdated
systems, but also on recent state-of-the-art systems. Therefore, we plan to
progressively integrate new human evaluations in BEAMetrics, as the filed will
evolve.

In this work we have presented BEAMetrics, a Benchmark to Evaluate Automatic
Metrics. We hope it will serve the community, both as vehicle for better
understanding the strengths and limitations of current metrics across a broad
spectrum of tasks, as well as a tool to drive research into future metrics
that better address the challenges of evaluating flexible generative models of
language. BEAMetrics aims to place these evaluation challenges at the
forefront of the minds of more researchers, encouraging a comparable level of
focus on creative solutions for evaluation, as there is currently on
developing new systems. After all, before we can make things better, we need
robust, reliable, consistent and efficient ways of quantifying what better
actually means.

## References

  * Alva-Manchego et al. [2020a] Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., and Specia, L.  ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4668–4679, Online, July 2020a. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.acl-main.424.
  * Alva-Manchego et al. [2020b] Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., and Specia, L.  Asset: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations.  In _ACL 2020-58th Annual Meeting of the Association for Computational Linguistics_ , 2020b.
  * Bender & Friedman [2018] Bender, E. M. and Friedman, B.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604, 2018.  doi: 10.1162/tacl_a_00041.  URL https://aclanthology.org/Q18-1041.
  * Bhandari et al. [2020] Bhandari, M., Narayan Gour, P., Ashfaq, A., Liu, P., and Neubig, G.  Re-evaluating evaluation in text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020.
  * Brown et al. [2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Callison-Burch et al. [2006] Callison-Burch, C., Osborne, M., and Koehn, P.  Re-evaluating the role of bleu in machine translation research.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , 2006.
  * Cer et al. [2017] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_ , 2017.
  * Cho et al. [2021] Cho, J., Lei, J., Tan, H., and Bansal, M.  Unifying vision-and-language tasks via text generation.  _arXiv preprint arXiv:2102.02779_ , 2021.
  * Chollet [2019] Chollet, F.  On the measure of intelligence.  _arXiv preprint arXiv:1911.01547_ , 2019.
  * Chopra et al. [2016] Chopra, S., Auli, M., and Rush, A. M.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 93–98, 2016.
  * Choshen & Abend [2018] Choshen, L. and Abend, O.  Inherent biases in reference based evaluation for grammatical error correction and text simplification.  _arXiv preprint arXiv:1804.11254_ , 2018.
  * Dagan et al. [2005] Dagan, I., Glickman, O., and Magnini, B.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_ , pp. 177–190. Springer, 2005.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Devlin et al. [2018] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dolan & Brockett [2005] Dolan, W. B. and Brockett, C.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Dušek & Jurčíček [2019] Dušek, O. and Jurčíček, F.  Neural Generation for Czech: Data and Baselines.  In _Proceedings of the 12th International Conference on Natural Language Generation (INLG 2019)_ , pp. 563–574, Tokyo, Japan, October 2019.  URL https://www.aclweb.org/anthology/W19-8670/.
  * Dušek et al. [2019] Dušek, O., Howcroft, D. M., and Rieser, V.  Semantic Noise Matters for Neural Natural Language Generation.  In _Proceedings of the 12th International Conference on Natural Language Generation (INLG 2019)_ , pp. 421–426, Tokyo, Japan, 2019.  URL https://www.aclweb.org/anthology/W19-8652/.
  * Fabbri et al. [2021] Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., and Radev, D.  Summeval: Re-evaluating summarization evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409, 2021.
  * Faisal Ladhak & McKeown [2020] Faisal Ladhak, Esin Durmus, C. C. and McKeown, K.  Wikilingua: A new benchmark dataset for multilingual abstractive summarization.  In _Findings of EMNLP, 2020_ , 2020.
  * Gardent et al. [2017] Gardent, C., Shimorina, A., Narayan, S., and Perez-Beltrachini, L.  Creating training corpora for nlg micro-planners.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 179–188. Association for Computational Linguistics, 2017.  doi: 10.18653/v1/P17-1017.  URL http://www.aclweb.org/anthology/P17-1017.
  * Gebru et al. [2018] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J., Wallach, H., Daumé, III, H., and Crawford, K.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_ , 2018.
  * Gehrmann et al. [2021] Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., Du, W., Durmus, E., Dušek, O., Emezue, C. C., Gangal, V., Garbacea, C., Hashimoto, T., Hou, Y., Jernite, Y., Jhamtani, H., Ji, Y., Jolly, S., Kale, M., Kumar, D., Ladhak, F., Madaan, A., Maddela, M., Mahajan, K., Mahamood, S., Majumder, B. P., Martins, P. H., McMillan-Major, A., Mille, S., van Miltenburg, E., Nadeem, M., Narayan, S., Nikolaev, V., Niyongabo Rubungo, A., Osei, S., Parikh, A., Perez-Beltrachini, L., Rao, N. R., Raunak, V., Rodriguez, J. D., Santhanam, S., Sedoc, J., Sellam, T., Shaikh, S., Shimorina, A., Sobrevilla Cabezudo, M. A., Strobelt, H., Subramani, N., Xu, W., Yang, D., Yerukola, A., and Zhou, J.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pp. 96–120, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.gem-1.10.  URL https://aclanthology.org/2021.gem-1.10.
  * Hodosh et al. [2013] Hodosh, M., Young, P., and Hockenmaier, J.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ , 47:853–899, 2013.
  * Hu et al. [2015] Hu, B., Chen, Q., and Zhu, F.  LCSTS: A large scale Chinese short text summarization dataset.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 1967–1972, Lisbon, Portugal, September 2015\. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1229.  URL https://aclanthology.org/D15-1229.
  * Jiang et al. [2020] Jiang, C., Maddela, M., Lan, W., Zhong, Y., and Xu, W.  Neural CRF model for sentence alignment in text simplification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7943–7960, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.709.  URL https://www.aclweb.org/anthology/2020.acl-main.709.
  * Ju et al. [2020] Ju, J., Liu, M., Gao, L., and Pan, S.  Scisummpip: An unsupervised scientific paper summarization pipeline.  _arXiv preprint arXiv:2010.09190_ , 2020.
  * Kane et al. [2020] Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., and Coulibali, M.  Nubia: Neural based interchangeability assessor for text generation.  _arXiv preprint arXiv:2004.14667_ , 2020.
  * Kiros [2021] Kiros, J.  Multimodal few-shot learning by convex combination of token embeddings.  2021\.  URL https://colab.research.google.com/drive/1fokumWeasHTo0KXpfeZ6Z0OgLOQ2SUso?usp=sharing.
  * Kocmi et al. [2021] Kocmi, T., Federmann, C., Grundkiewicz, R., Junczys-Dowmunt, M., Matsushita, H., and Menezes, A.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  _arXiv preprint arXiv:2107.10821_ , 2021.
  * Koto et al. [2020] Koto, F., Lau, J. H., and Baldwin, T.  Liputan6: A large-scale Indonesian dataset for text summarization.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pp. 598–608, Suzhou, China, December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.aacl-main.60.
  * Koto et al. [2021] Koto, F., Lau, J. H., and Baldwin, T.  Evaluating the efficacy of summarization evaluation across languages.  _arXiv preprint arXiv:2106.01478_ , 2021.
  * Krippendorff [2011] Krippendorff, K.  Computing krippendorff’s alpha-reliability.  2011\.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25:1097–1105, 2012.
  * Lavie & Agarwal [2007] Lavie, A. and Agarwal, A.  Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments.  In _Proceedings of the second workshop on statistical machine translation_ , pp. 228–231, 2007.
  * Leblond et al. [2021] Leblond, R., Alayrac, J.-B., Sifre, L., Pislar, M., Lespiau, J.-B., Antonoglou, I., Simonyan, K., and Vinyals, O.  Machine translation decoding beyond beam search.  _arXiv preprint arXiv:2104.05336_ , 2021.
  * Lee et al. [2020] Lee, D., Shin, M. C., Whang, T., Cho, S., Ko, B., Lee, D., Kim, E., and Jo, J.  Reference and document aware semantic evaluation methods for Korean language summarization.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 5604–5616, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.491.  URL https://aclanthology.org/2020.coling-main.491.
  * Levesque et al. [2012] Levesque, H., Davis, E., and Morgenstern, L.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_ , 2012.
  * Lewis et al. [2019] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ , 2019.
  * Li et al. [2019] Li, S., Lei, D., Qin, P., and Wang, W. Y.  Deep reinforcement learning with distributional semantic rewards for abstractive summarization.  _arXiv preprint arXiv:1909.00141_ , 2019.
  * Lin et al. [2020] Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., and Ren, X.  CommonGen: A constrained text generation challenge for generative commonsense reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1823–1840, Online, November 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.165.
  * Lin [2004] Lin, C.-Y.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Liu et al. [2019] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Louis & Nenkova [2013] Louis, A. and Nenkova, A.  Automatically assessing machine summary content without a gold standard.  _Computational Linguistics_ , 39(2):267–300, 2013\.
  * Ma et al. [2019] Ma, Q., Wei, J., Bojar, O., and Graham, Y.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pp. 62–90, Florence, Italy, August 2019\. Association for Computational Linguistics.  doi: 10.18653/v1/W19-5302.  URL https://aclanthology.org/W19-5302.
  * Marino et al. [2019] Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3195–3204, 2019.
  * Martin et al. [2020] Martin, L., Fan, A., de la Clergerie, É., Bordes, A., and Sagot, B.  Muss: Multilingual unsupervised sentence simplification by mining paraphrases.  _arXiv preprint arXiv:2005.00352_ , 2020.
  * Min et al. [2021] Min, S., Boyd-Graber, J., Alberti, C., Chen, D., Choi, E., Collins, M., Guu, K., Hajishirzi, H., Lee, K., Palomaki, J., et al.  Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.  _arXiv preprint arXiv:2101.00133_ , 2021.
  * Nallapati et al. [2016] Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_ , 2016.
  * Narayan et al. [2018] Narayan, S., Cohen, S. B., and Lapata, M.  Don’t give me the details, just the summary!  _Topic-aware Convolutional Neural Networks for Extreme Summarization. In_ , 2018.
  * Nenkova & Passonneau [2004] Nenkova, A. and Passonneau, R.  Evaluating content selection in summarization: The pyramid method.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pp. 145–152, Boston, Massachusetts, USA, May 2 - May 7 2004. Association for Computational Linguistics.  URL https://aclanthology.org/N04-1019.
  * Novikova et al. [2017] Novikova, J., Dušek, O., Curry, A. C., and Rieser, V.  Why we need new evaluation metrics for nlg.  _arXiv preprint arXiv:1707.06875_ , 2017.
  * Papineni et al. [2002] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Paraschiv & Cercel [2020] Paraschiv, A. and Cercel, D.-C.  Upb at germeval-2020 task 3: Assessing summaries for german texts using bertscore and sentence-bert.  In _SwissText/KONVENS_ , 2020.
  * Parikh et al. [2020] Parikh, A. P., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., and Das, D.  Totto: A controlled table-to-text generation dataset.  _arXiv preprint arXiv:2004.14373_ , 2020.
  * Post [2018] Post, M.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pp. 186–191, Belgium, Brussels, October 2018. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W18-6319.
  * Prager [2006] Prager, J. M.  Open-domain question-answering.  _Found. Trends Inf. Retr._ , 1(2):91–231, 2006\.
  * Radev et al. [2020] Radev, D., Zhang, R., Rau, A., Sivaprasad, A., Hsieh, C., Rajani, N. F., Tang, X., Vyas, A., Verma, N., Krishna, P., Liu, Y., Irwanto, N., Pan, J., Rahman, F., Zaidi, A., Mutuma, M., Tarabar, Y., Gupta, A., Yu, T., Tan, Y. C., Lin, X. V., Xiong, C., and Socher, R.  Dart: Open-domain structured data record to text generation.  _arXiv preprint arXiv:2007.02871_ , 2020.
  * Radford et al. [2018] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. [2021] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ , 2021.
  * Raffel et al. [2019] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Rajpurkar et al. [2016] Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_ , 2016.
  * Rastogi et al. [2019] Rastogi, A., Zang, X., Sunkara, S., Gupta, R., and Khaitan, P.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  _arXiv preprint arXiv:1909.05855_ , 2019.
  * Rei et al. [2020] Rei, R., Stewart, C., Farinha, A. C., and Lavie, A.  Comet: A neural framework for mt evaluation.  _arXiv preprint arXiv:2009.09025_ , 2020.
  * Reiter & Dale [1997] Reiter, E. and Dale, R.  Building applied natural language generation systems.  _Natural Language Engineering_ , 3(1):57–87, 1997\.
  * Roberts et al. [2020] Roberts, A., Raffel, C., and Shazeer, N.  How much knowledge can you pack into the parameters of a language model?  _arXiv preprint arXiv:2002.08910_ , 2020.
  * Ruder [2021] Ruder, S.  Challenges and Opportunities in NLP Benchmarking.  http://ruder.io/nlp-benchmarking, 2021.
  * Russakovsky et al. [2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Sai et al. [2020] Sai, A. B., Mohankumar, A. K., and Khapra, M. M.  A survey of evaluation metrics used for nlg systems.  _arXiv preprint arXiv:2008.12009_ , 2020.
  * Scialom et al. [2020a] Scialom, T., Bordes, P., Dray, P.-A., Staiano, J., and Gallinari, P.  Bert can see out of the box: On the cross-modal transferability of text representations.  _arXiv e-prints_ , pp. arXiv–2002, 2020a.
  * Scialom et al. [2020b] Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J.  Discriminative adversarial search for abstractive summarization.  In III, H. D. and Singh, A. (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 8555–8564. PMLR, 13–18 Jul 2020b.  URL https://proceedings.mlr.press/v119/scialom20a.html.
  * Scialom et al. [2020c] Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020c.
  * Scialom et al. [2021] Scialom, T., Martin, L., Staiano, J., de la Clergerie, É. V., and Sagot, B.  Rethinking automatic evaluation in sentence simplification.  _arXiv preprint arXiv:2104.07560_ , 2021.
  * See et al. [2017] See, A., Liu, P. J., and Manning, C. D.  Get to the point: Summarization with pointer-generator networks.  In _ACL (1)_ , 2017.
  * Sellam et al. [2020] Sellam, T., Das, D., and Parikh, A. P.  Bleurt: Learning robust metrics for text generation.  _arXiv preprint arXiv:2004.04696_ , 2020.
  * Shapira et al. [2019] Shapira, O., Gabay, D., Gao, Y., Ronen, H., Pasunuru, R., Bansal, M., Amsterdamer, Y., and Dagan, I.  Crowdsourcing lightweight pyramids for manual summary evaluation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 682–687, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1072.  URL https://aclanthology.org/N19-1072.
  * Shimorina et al. [2018] Shimorina, A., Gardent, C., Narayan, S., and Perez-Beltrachini, L.  _WebNLG challenge: Human evaluation results_.  PhD thesis, Loria & Inria Grand Est, 2018.
  * Socher et al. [2013] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Tsimpoukelli et al. [2021] Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S., Vinyals, O., and Hill, F.  Multimodal few-shot learning with frozen language models.  _arXiv preprint arXiv:2106.13884_ , 2021.
  * Vedantam et al. [2015] Vedantam, R., Lawrence Zitnick, C., and Parikh, D.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4566–4575, 2015.
  * Wang et al. [2018] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. [2019] Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_ , 2019.
  * Williams et al. [2017] Williams, A., Nangia, N., and Bowman, S. R.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_ , 2017.
  * Wolf et al. [2019] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xu et al. [2016] Xu, W., Napoles, C., Pavlick, E., Chen, Q., and Callison-Burch, C.  Optimizing statistical machine translation for text simplification.  _Transactions of the Association for Computational Linguistics_ , 4:401–415, 2016.  URL https://cocoxu.github.io/publications/tacl2016-smt-simplification.pdf.
  * Zhang et al. [2021] Zhang, S., Celikyilmaz, A., Gao, J., and Bansal, M.  Emailsum: Abstractive email thread summarization.  _arXiv preprint arXiv:2107.14691_ , 2021.
  * Zhang et al. [2019] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.

## Checklist

The checklist follows the references. Please read the checklist guidelines
carefully for information on how to answer these questions. For each question,
change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly
encouraged to include a justification to your answer, either by referencing
the appropriate section of your paper or providing a brief inline description.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] , in the discussion, see
Section 5.3.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
, both in the supplementary material and an URL on the first page.

    2. (b)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? We report the p-values for the significance for
all our experiments in the Appendix.

  3. 3.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] , all the
dataset creators are cited, and a specific data card is also release.

    2. (b)

Did you include any new assets either in the supplemental material or as a
URL? [Yes]

    3. (c)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] , we restricted the selection to publicly
available data

  4. 4.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [Yes]  in the supplementary material.

    2. (b)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [Yes] , in the supplementary
material as well.

##  Appendix A Appendix

###  A.1 Example of a Data Card

All the Data Cards are publicly available, see for instance the SummEval card.

###  A.2 OkVQA annotation

Figure 3: Print Screen of the annotation tool used for our human evaluation on
No-VQA.

Three professional English Speakers were hired to evaluate the 300 answers.
They spend respectively 2h00min, 1h35min and 1h50min for the evaluation, and
were compensated with vouchers. They followed the protocol reported in Table 4

    Evaluation Protocol:

    You are shown an image, a question and its answer.
    Your task is to evaluate the answer regarding three specific
    dimensions.

    First, look at the image, read the question and then the
    answer.

    Then, answer the following questions using the rating scale:

    Question 1: In your opinion, is the answer possible?
    Does it make sense in some possible situation?
        (a) possible, in some conceivable situation
        (b) impossible, or makes no sense

    Question 2: If the image was shown to 100 people and the
    question was asked, how many people do you think would give
    the answer?
        (write a number between 1 and 100)

    Question 3: Can you say that the answer is definitely
    factually correct (use Google if necessary)?
        a) definitely correct
        b) can’t say (matter of opinion, depends on the
        situation, the question does not have a ’right’ answer)
        c) definitely incorrect

---
Table 4: Human annotation protocol of the OkVQA dataset.

The protocol was followed by three examples, before the evaluation to start.
We report in Figure 3 a print screen of the interface.

###  A.3 Kendall Tau Correlations & Confidence Scores

In this Section, we report the main results similar to Tables 2 and 3 in
Section 5, but this time using Kendall Tau, instead of Pearson coefficient.
Moreover, we also report the p-values for all the tables including the two
based on Pearson, in the main paper.

| Dataset: | WMT | Web | Asv | MUS | Pas | Fli | mSu | Rea | SumE | OpQA | OkVQA | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | MT | DTG | Sim | Sim | ImCa | ImCa | Sum | Sum | Sum | QA | VQA | All
Max | ROUGE-1 | 12.6 | 44.3 | 43.6 | 33.3 | 52.2 | 32.0 | 31.6 | 32.0 | 10.5 | 32.7 | 11.2 | 10.5
ROUGE-2 | 12.0 | 46.7 | 37.0 | 31.8 | 52.0 | 32.4 | 31.4 | 32.8 | 10.1 | 21.7 | 6.2 | 9.8
ROUGE-L | 13.1 | 44.0 | 41.2 | 34.0 | 52.0 | 31.6 | 31.8 | 29.8 | 9.1 | 32.7 | 11.4 | 10.3
BLEU | 12.3 | 48.8 | 32.6 | 32.6 | 50.2 | 33.7 | 26.3 | 27.7 | 8.0 | 10.8 | 11.2 | 9.2
METEOR | 12.7 | 46.2 | 46.3 | 31.0 | 56.0 | 38.1 | 26.4 | 37.4 | 12.5 | 32.9 | -1.4 | 10.6
BERTScore P | 15.8 | 50.6 | 51.0 | 29.6 | 49.6 | 32.6 | 39.9 | 19.2 | 5.0 | 9.3 | 3.0 | 9.5
BERTScore R | 15.9 | 55.7 | 54.3 | 29.4 | 53.4 | 28.7 | 40.3 | 31.3 | 10.4 | 10.7 | 7.8 | 10.6
BERTScore F1 | 16.2 | 54.4 | 55.0 | 30.4 | 52.6 | 32.8 | 41.5 | 26.4 | 8.5 | 10.3 | 5.4 | 10.4
BLEURT | 18.5 | 50.7 | 61.9 | 28.9 | 57.3 | 44.4 | 28.6 | 21.6 | 7.6 | 17.6 | 11.3 | 10.9
Nubia | 18.0 | 60.6 | 43.8 | 34.9 | 52.8 | 38.9 | 23.2 | 10.0 | 4.2 | 23.9 | 7.1 | 9.9
1 | ROUGE-1 | 12.6 | 49.2 | 32.0 | 33.3 | 43.4 | 24.4 | 31.6 | 32.0 | 12.2 | 32.7 | 15.0 | 9.9
ROUGE-2 | 12.0 | 45.0 | 27.9 | 31.8 | 27.4 | 23.1 | 31.4 | 32.8 | 11.1 | 21.7 | 9.2 | 8.5
ROUGE-L | 13.1 | 43.8 | 28.7 | 34.0 | 41.3 | 24.1 | 31.8 | 29.8 | 11.1 | 32.7 | 15.0 | 9.5
BLEU | 12.3 | 44.8 | 23.0 | 32.6 | 29.4 | 19.5 | 26.3 | 27.7 | 5.5 | 10.8 | 12.6 | 7.6
METEOR | 12.7 | 49.1 | 35.7 | 31.0 | 42.8 | 27.0 | 26.4 | 37.4 | 11.8 | 32.9 | 5.5 | 9.8
BERTScore P | 15.8 | 46.6 | 29.7 | 29.6 | 37.3 | 24.6 | 39.9 | 19.2 | 5.4 | 9.3 | 2.3 | 8.1
BERTScore R | 15.9 | 52.7 | 47.6 | 29.4 | 45.8 | 17.8 | 40.3 | 31.3 | 10.4 | 10.7 | -0.1 | 9.4
BERTScore F1 | 16.2 | 51.0 | 41.4 | 30.4 | 43.9 | 23.1 | 41.5 | 26.4 | 8.3 | 10.3 | 2.4 | 9.2
BLEURT | 18.5 | 59.6 | 48.3 | 28.9 | 51.6 | 38.1 | 28.6 | 21.6 | 7.8 | 17.6 | 7.4 | 10.2
Nubia | 18.0 | 60.6 | 43.8 | 34.9 | 52.8 | 38.9 | 23.2 | 10.0 | 4.2 | 23.9 | 7.1 | 9.9
0 | Abstr-1 | 1.4 | - | -8.1 | -38.3 | - | - | - | 1.2 | -12.6 | 2.6 | 5.5 | -1.5
Abstr-2 | 1.6 | - | -21.1 | -40.4 | - | - | - | -2.0 | -22.7 | 12.1 | -1.5 | -2.3
Abstr-3 | 1.6 | - | -21.5 | -41.8 | - | - | - | -3.3 | -24.7 | 13.7 | -6.0 | -2.6
Length | 1.3 | 15.1 | 18.3 | 9.6 | 15.1 | -8.7 | -1.5 | 20.3 | 7.5 | 12.4 | -7.6 | 2.6
Repet-1 | 1.2 | 14.8 | 17.3 | 9.2 | 14.6 | -8.7 | -1.5 | 20.1 | 7.2 | 12.4 | -7.8 | 2.5
Repet-2 | 1.2 | 14.9 | 18.1 | 9.7 | 15.0 | -8.7 | -1.5 | 20.1 | 7.3 | 12.4 | -7.7 | 2.5
Repet-3 | 1.3 | 15.0 | 18.1 | 9.8 | 15.0 | -8.7 | -1.4 | 20.2 | 7.4 | 13.7 | -7.7 | 2.6
-GPT2 Perpl. | 8.6 | 22.8 | 26.2 | 10.8 | 14.4 | -0.9 | -0.6 | 9.3 | -2.2 | 0.2 | 0.2 | 2.8
Table 5: Correctness dimensions: Kendall tau between automatic metrics and human judgement for Correctness. The top bloc corresponds to coefficients computed when all the human references were available. The second bloc corresponds to coefficients computed given a single human reference. The third bloc corresponds to coefficients computed given no human reference. | Dataset: | Web | Asv | Asv | MUS | mSu | SumE | SumE | SumE | OpQA | OkVQA | OkVQA
---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | DTG | Sim | Sim | Sim | Sum | Sum | Sum | Sum | QA | VQA | VQA
Max | ROUGE-1 | 33.5 | 30.5 | 32.2 | 17.9 | 32.5 | 20.9 | 12.3 | 6.2 | 37.2 | 8.0 | 13.5
ROUGE-2 | 36.4 | 24.8 | 27.4 | 16.7 | 32.9 | 16.7 | 9.7 | 4.8 | 26.1 | 6.7 | 10.5
ROUGE-L | 36.8 | 28.6 | 30.0 | 17.8 | 32.4 | 16.5 | 11.3 | 6.1 | 37.1 | 8.0 | 13.6
BLEU | 41.6 | 19.7 | 21.1 | 16.1 | 27.9 | 20.8 | 16.7 | 8.8 | 9.6 | 3.5 | 8.7
METEOR | 34.6 | 26.9 | 26.2 | 14.6 | 31.4 | 22.3 | 11.6 | 7.0 | 38.1 | -3.1 | -1.4
BERTScore P | 49.9 | 43.3 | 44.1 | 17.8 | 35.6 | 19.2 | 20.9 | 9.6 | 8.0 | 12.4 | 11.5
BERTScore R | 47.6 | 41.1 | 37.2 | 16.8 | 44.1 | 29.0 | 24.7 | 9.7 | 8.7 | 9.5 | 12.2
BERTScore F1 | 49.4 | 43.2 | 41.6 | 17.9 | 41.1 | 25.6 | 24.6 | 10.9 | 8.5 | 12.0 | 13.0
BLEURT | 39.4 | 48.9 | 45.2 | 17.4 | 26.6 | 17.6 | 9.8 | 12.5 | 16.1 | 26.1 | 17.3
Nubia | 36.0 | 29.9 | 27.6 | 19.2 | 19.4 | 10.2 | 5.1 | 4.0 | 24.8 | 5.4 | 9.1
1 | ROUGE-1 | 36.7 | 22.4 | 21.1 | 17.9 | 32.5 | 23.5 | 12.0 | 7.4 | 37.2 | 12.4 | 18.5
ROUGE-2 | 34.1 | 18.9 | 19.5 | 16.7 | 32.9 | 18.3 | 9.5 | 6.6 | 26.1 | 4.7 | 7.7
ROUGE-L | 35.4 | 21.4 | 20.2 | 17.8 | 32.4 | 19.3 | 12.0 | 8.6 | 37.1 | 12.4 | 18.5
BLEU | 34.9 | 19.3 | 19.6 | 16.1 | 27.9 | 18.1 | 8.8 | 3.4 | 9.6 | 5.9 | 10.3
METEOR | 36.9 | 23.6 | 22.9 | 14.6 | 31.4 | 21.6 | 10.4 | 8.0 | 38.1 | 1.3 | 5.1
BERTScore P | 44.1 | 26.2 | 29.1 | 17.8 | 35.6 | 19.7 | 18.3 | 8.3 | 8.0 | 11.6 | 10.7
BERTScore R | 42.9 | 35.2 | 32.9 | 16.8 | 44.1 | 27.5 | 20.9 | 8.5 | 8.7 | 4.7 | 5.6
BERTScore F1 | 44.2 | 34.1 | 33.5 | 17.9 | 41.1 | 25.7 | 21.4 | 9.3 | 8.5 | 9.8 | 9.9
BLEURT | 46.4 | 38.5 | 36.2 | 17.4 | 26.6 | 19.7 | 10.3 | 11.8 | 16.1 | 22.0 | 12.5
Nubia | 36.0 | 29.9 | 27.6 | 19.2 | 19.4 | 10.2 | 5.1 | 4.0 | 24.8 | 5.4 | 9.1
0 | Abstr-1 | - | -7.0 | -1.2 | -15.8 | - | -17.9 | -19.0 | -10.1 | 2.6 | 7.9 | 7.7
Abstr-2 | - | -13.9 | -8.5 | -18.9 | - | -21.5 | -22.3 | -16.2 | 8.8 | -2.2 | -2.7
Abstr-3 | - | -13.9 | -9.9 | -20.3 | - | -21.7 | -22.0 | -17.7 | 10.1 | -11.3 | -11.1
Length | -2.6 | 4.7 | 3.6 | -2.6 | 7.1 | 18.2 | 6.4 | -0.6 | 8.9 | -16.1 | -11.7
Repet-1 | -3.1 | 4.1 | 3.0 | -2.5 | 7.0 | 17.9 | 6.1 | -0.8 | 8.9 | -16.1 | -11.8
Repet-2 | -3.0 | 4.7 | 3.5 | -2.8 | 7.0 | 17.9 | 6.0 | -0.7 | 8.9 | -15.9 | -11.6
Repet-3 | -2.9 | 4.6 | 3.5 | -2.6 | 7.0 | 18.0 | 6.2 | -0.6 | 10.1 | -16.0 | -11.7
-GPT2 Perpl. | 10.3 | 24.1 | 21.6 | 5.8 | 1.1 | 13.4 | 12.5 | 8.5 | -3.1 | 1.1 | -1.6
Table 6: Non Correctness dimensions: Kendall tau between automatic metrics and human judgement for the dimensions other than Correctness. The top bloc corresponds to coefficients computed when all the human references were available. The second bloc corresponds to coefficients computed given a single human reference. The third bloc corresponds to coefficients computed given no human reference. |  | WMT | Web | Asv | MUS | Pas | Fli | mSu | Rea | SumE | OpQA | OkVQA | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | MT | DTG | Sim | Sim | ImCa | ImCa | Sum | Sum | Sum | QA | VQA | All
Max | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.2 | 1e-5
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
BLEU | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
METEOR | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.7 | 1e-5
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.4 | 1e-5
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.2 | 1e-5
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-2 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.1 | 1e-5
1 | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5
BLEU | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-2 | 1e-5
METEOR | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.2 | 1e-5
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 0.5 | 1e-5
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.9 | 1e-5
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.5 | 1e-5
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.1 | 1e-5
0 | Abstr-1 | 0.1 | - | 0.1 | 1e-5 | - | - | - | 0.3 | 1e-5 | 0.0 | 0.2 | 1e-5
Abstr-2 | 0.1 | - | 1e-3 | 1e-5 | - | - | - | 0.1 | 1e-5 | 1e-5 | 0.7 | 1e-5
Abstr-3 | 1e-5 | - | 1e-3 | 1e-5 | - | - | - | 0.0 | 1e-5 | 1e-5 | 0.1 | 1e-5
Length | 0.1 | 1e-5 | 1e-3 | 0.1 | 1e-5 | 1e-5 | 0.3 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
Repet-1 | 0.2 | 1e-5 | 1e-2 | 0.1 | 1e-5 | 1e-5 | 0.3 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
Repet-2 | 0.1 | 1e-5 | 1e-3 | 0.1 | 1e-5 | 1e-5 | 0.3 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
Repet-3 | 0.1 | 1e-5 | 1e-3 | 0.1 | 1e-5 | 1e-5 | 0.3 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
-GPT2 Perpl. | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.3 | 0.7 | 1e-5 | 0.2 | 0.8 | 0.9 | 0.1
Table 7: Confidence scores - Correctness dimensions: Kendall tau p-values between automatic metrics and human judgement for Correctness. The top bloc corresponds to coefficients computed when all the human references were available. The second bloc corresponds to coefficients computed given a single human reference. The third bloc corresponds to coefficients computed given no human reference. |  | Web | Asv | Asv | MUS | mSu | SumE | SumE | SumE | OpQA | OkVQA | OkVQA
---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | DTG | Sim | Sim | Sim | Sum | Sum | Sum | Sum | QA | VQA | VQA
Max | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 0.1 | 1e-2
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.2 | 0.0
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 0.1 | 1e-2
BLEU | 1e-5 | 1e-3 | 1e-3 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.4 | 0.0
METEOR | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.5 | 0.7
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-2
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-2
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-3
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-2 | 0.0 | 1e-5 | 0.2 | 0.0
1 | ROUGE-1 | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-3
ROUGE-2 | 1e-5 | 1e-3 | 1e-3 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.3 | 0.1
ROUGE-L | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-3
BLEU | 1e-5 | 1e-3 | 1e-3 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.2 | 0.0
METEOR | 1e-5 | 1e-5 | 1e-3 | 0.0 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.8 | 0.2
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-2
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.3 | 0.1
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 0.0
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-2 | 0.0 | 1e-5 | 0.2 | 0.0
0 | Abstr-1 | - | 0.2 | 0.8 | 0.0 | - | 1e-5 | 1e-5 | 1e-5 | 0.0 | 0.0 | 0.0
Abstr-2 | - | 1e-2 | 0.1 | 1e-2 | - | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.6 | 0.4
Abstr-3 | - | 1e-2 | 0.0 | 1e-2 | - | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-2
Length | 0.1 | 0.3 | 0.5 | 0.6 | 1e-5 | 1e-5 | 1e-3 | 0.7 | 1e-5 | 1e-3 | 1e-2
Repet-1 | 0.0 | 0.4 | 0.5 | 0.7 | 1e-5 | 1e-5 | 1e-3 | 0.6 | 1e-5 | 1e-3 | 1e-2
Repet-2 | 0.0 | 0.3 | 0.5 | 0.6 | 1e-5 | 1e-5 | 1e-3 | 0.7 | 1e-5 | 1e-3 | 1e-2
Repet-3 | 0.0 | 0.3 | 0.5 | 0.6 | 1e-5 | 1e-5 | 1e-3 | 0.7 | 1e-5 | 1e-3 | 1e-2
-GPT2 Perpl. | 1e-5 | 1e-5 | 1e-3 | 0.3 | 0.4 | 1e-5 | 1e-5 | 1e-3 | 1e-2 | 0.8 | 0.6
Table 8: Confidence scores - Non Correctness dimensions: Kendall tau p-values between automatic metrics and human judgement for the dimensions other than Correctness. The top bloc corresponds to coefficients computed when all the human references were available. The second bloc corresponds to coefficients computed given a single human reference. The third bloc corresponds to coefficients computed given no human reference. |  | WMT | Web | Asv | MUS | Pas | Fli | mSu | Rea | SumE | OpQA | OkVQA | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | MT | DTG | Sim | Sim | ImCa | ImCa | Sum | Sum | Sum | QA | VQA | All
Max | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5
BLEU | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5
METEOR | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.3 | 1e-5
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.4 | 1e-5
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-2 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.0 | 1e-5
1 | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5
BLEU | 1e-5 | 1e-5 | 1e-3 | 1e-3 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-3 | 1e-5
METEOR | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.3 | 1e-5
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.5 | 1e-5
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.2 | 1e-5
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.2 | 1e-5
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-2 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 0.0 | 1e-5
0 | Abstr-1 | 0.2 | - | 0.0 | 1e-5 | - | - | - | 1e-5 | 1e-5 | 1e-5 | 0.3 | 1e-5
Abstr-2 | 0.1 | - | 1e-3 | 1e-5 | - | - | - | 0.7 | 1e-5 | 1e-5 | 0.0 | 1e-5
Abstr-3 | 0.2 | - | 1e-3 | 1e-5 | - | - | - | 0.1 | 1e-5 | 1e-5 | 0.1 | 1e-5
Length | 0.1 | 1e-5 | 0.0 | 0.0 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-2 | 1e-5 | 0.1 | 1e-5
Repet-1 | 0.1 | 1e-5 | 0.0 | 0.0 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-2 | 1e-5 | 0.1 | 1e-5
Repet-2 | 0.1 | 1e-5 | 0.0 | 0.0 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-2 | 1e-5 | 0.1 | 1e-5
Repet-3 | 0.1 | 1e-5 | 0.0 | 0.0 | 1e-5 | 1e-5 | 0.0 | 1e-5 | 1e-2 | 1e-5 | 0.0 | 1e-5
-GPT2 Perpl. | 1e-5 | 1e-5 | 1e-2 | 0.2 | 1e-5 | 0.0 | 0.1 | 1e-5 | 0.2 | 0.3 | 0.1 | 1e-5
Table 9: Confidence scores - Correctness dimensions: Pearson coefficients p-values between automatic metrics and human judgement for Correctness. The top bloc corresponds to coefficients computed when all the human references were available. The second bloc corresponds to coefficients computed given a single human reference. The third bloc corresponds to coefficients computed given no human reference. |  | Web | Asv | Asv | MUS | mSu | SumE | SumE | SumE | OpQA | OkVQA | OkVQA
---|---|---|---|---|---|---|---|---|---|---|---|---
#Ref | Task: | DTG | Sim | Sim | Sim | Sum | Sum | Sum | Sum | QA | VQA | VQA
Dim: | Flu | Flu | Sim | Flu | Rel | Rel | Coh | Flu | Obv | Pos | Obv
Max | ROUGE-1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-2 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.2 | 1e-2
ROUGE-L | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.0 | 1e-5
BLEU | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.2 | 1e-3
METEOR | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.9 | 0.5
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-2
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-3
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-3
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-2 | 1e-2 | 1e-5 | 0.1 | 1e-3
1 | ROUGE-1 | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
ROUGE-2 | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 0.4 | 0.0
ROUGE-L | 1e-5 | 1e-3 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-5
BLEU | 1e-5 | 1e-2 | 1e-2 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 0.0 | 1e-3
METEOR | 1e-5 | 1e-5 | 1e-3 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.8 | 0.4
BERTScore P | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-2
BERTScore R | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.1 | 0.0
BERTScore F1 | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.0 | 1e-2
BLEURT | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 1e-3
Nubia | 1e-5 | 1e-5 | 1e-5 | 1e-2 | 1e-5 | 1e-5 | 1e-2 | 1e-2 | 1e-5 | 0.1 | 1e-3
0 | Abstr-1 | - | 0.2 | 0.3 | 1e-3 | - | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 0.0 | 0.2
Abstr-2 | - | 0.0 | 0.1 | 1e-2 | - | 1e-5 | 1e-5 | 1e-5 | 1e-3 | 1e-3 | 0.1
Abstr-3 | - | 1e-2 | 0.0 | 1e-2 | - | 1e-5 | 1e-5 | 1e-5 | 1e-5 | 0.1 | 0.6
Length | 0.4 | 0.7 | 0.9 | 0.4 | 1e-5 | 1e-5 | 1e-3 | 0.2 | 1e-5 | 1e-3 | 1e-2
Repet-1 | 0.2 | 0.7 | 0.9 | 0.4 | 1e-5 | 1e-5 | 1e-3 | 0.2 | 1e-5 | 1e-3 | 1e-2
Repet-2 | 0.0 | 0.7 | 0.9 | 0.4 | 1e-5 | 1e-5 | 1e-3 | 0.2 | 1e-5 | 1e-3 | 1e-2
Repet-3 | 0.0 | 0.7 | 0.9 | 0.4 | 1e-5 | 1e-5 | 1e-3 | 0.2 | 1e-5 | 1e-3 | 1e-2
-GPT2 Perpl. | 1e-5 | 1e-2 | 1e-2 | 0.1 | 0.3 | 1e-5 | 1e-5 | 1e-3 | 0.8 | 0.4 | 0.4
Table 10: Confidence scores - Non Correctness dimensions: Pearson coefficients
p-values between automatic metrics and human judgement for the dimensions
other than Correctness. The top bloc corresponds to coefficients computed when
all the human references were available. The second bloc corresponds to
coefficients computed given a single human reference. The third bloc
corresponds to coefficients computed given no human reference.