# Bidimensional Leaderboards:
Generate and Evaluate Language Hand in Hand

First Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
&Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Jungo Kasai Keisuke Sakaguchi Ronan Le Bras Lavinia Dunagan
Jacob Morrison Alexander R. Fabbri Yejin Choi Noah A. Smith
Paul G. Allen School of Computer Science & Engineering, University of
Washington
Allen Institute for AI Department of Linguistics, University of Washington
Salesforce Research
<EMAIL_ADDRESS>Work was done during an internship at AI2.

###### Abstract

Natural language processing researchers have identified limitations of
evaluation methodology for generation tasks, with new questions raised about
the validity of automatic metrics and of crowdworker judgments. Meanwhile,
efforts to improve _generation models_ tend to depend on simple n-gram overlap
metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics
should each more directly benefit and inform the other. We therefore propose a
generalization of leaderboards, bidimensional leaderboards (Billboards), that
simultaneously tracks progress in language generation models and metrics for
their evaluation. Unlike conventional unidimensional leaderboards that sort
submitted systems by predetermined metrics, a Billboard accepts both
generators and evaluation metrics as competing entries. A Billboard
automatically creates an ensemble metric that selects and linearly combines a
few metrics based on a global analysis across generators. Further, metrics are
ranked based on their correlation with human judgments. We release four
Billboards for machine translation, summarization, and image
captioning.111https://nlp.cs.washington.edu/billboard/. We demonstrate that a
linear ensemble of a few diverse metrics sometimes substantially outperforms
existing metrics in isolation. Our mixed-effects model analysis shows that
most automatic metrics, especially the reference-based ones, overrate machine
over human generation, demonstrating the importance of updating metrics as
generation models become stronger (and perhaps more similar to humans) in the
future.

##  1 Introduction

Figure 1: Bidimensional leaderboard (Billboard). When a generator developer
submits output text (output.txt), Billboard computes all metric scores. When a
metric developer submits an executable program (e.g., metric.py), Billboard
computes correlation with the human judgments, updates the ensemble metric
(§2.2), and measures how much the metric overrates machines (§2.3).

Recent modeling advances have led to improved natural language generation in
applications such as machine translation and summarization (Ng et al., 2019;
Raffel et al., 2020; Brown et al., 2020, _inter alia_). This progress is
typically measured with automatic scores, such as BLEU Papineni et al. (2002)
and ROUGE Lin (2004), executed by modeling researchers themselves. These
metrics allow for fast, inexpensive development cycles. They were adopted
based on reported correlations with human judgments at the time the metrics
were introduced, but it has since been established that the correspondence can
collapse when models of different types are compared Callison-Burch et al.
(2006) or models become increasingly powerful Ma et al. (2019); Edunov et al.
(2020).

Meanwhile, many evaluation metrics that improve correlation with human
judgments have been proposed (Clark et al., 2019; Zhang et al., 2020b; Sellam
et al., 2020; Hessel et al., 2021, _inter alia_), but this progress has yet to
be broadly adopted by the community of researchers focused on advancing
models. Indeed, consistent with prior meta-evaluations Marie et al. (2021), we
found that 68% of the machine translation papers from NAACL and ACL 2021
evaluated their models solely by BLEU, and only 5% measured the performance
using recent metrics with contextual representations such as COMET Rei et al.
(2020). Similarly, automatic evaluation in 66% of the summarization papers was
done only in terms of ROUGE.222We examined all papers whose title contains
“machine translation” and “summarization.” See Appendix A for details. We
believe this separation between generation modeling and automatic evaluation
represents a missed opportunity for each subcommunity to more rapidly benefit
from the advances of the other.

We therefore propose an abstraction of conventional leaderboards,
bidimensional leaderboards (Billboards), that simultaneously facilitates
progress in natural language generation and its evaluation (Fig. 1). A
Billboard accepts two types of submissions related to a given task and
dataset: generators and metrics. Unlike conventional leaderboards, model
ranking is not tied to a predetermined set of metrics; the generators are
ranked based on the metric that currently correlates best with human
judgments. Metric submissions are ranked by their correlations to human
judgments, and each is stored as an executable program, which will then be
used to evaluate future generation submissions. Our Billboard includes a
sparse regression that selects and linearly combines three existing metrics,
revealing complementary strengths. All leaderboard scores are readily
reproducible, allowing research on generation models and automatic metrics to
benefit from each other.

We release four Billboard interfaces
(https://nlp.cs.washington.edu/billboard/) spanning three generation tasks:
the WMT20 EN-DE and WMT20 ZH-EN machine translation tasks Barrault et al.
(2020), the CNNDM summarization task Hermann et al. (2015), and the MSCOCO
image captioning task Lin et al. (2014).

#### Key Findings

Using the collective analyses of Billboards, our main findings are as follows.

  * •

A simple linear combination of a few (diverse) metrics can sometimes improve
correlation. This finding quantifies complementary effects of different
metrics and encourages metric developers to seek out aspects of generated text
quality not yet measured by existing metrics.

  * •

Using linear mixed-effects models, we find that most automatic metrics,
especially conventional, reference-based ones such as BLEU and ROUGE,
_overrate_ machines over humans in all tasks. This result provides further
support for the claim that the metrics should be continually evaluated and
updated as our generation models become stronger (and perhaps, closer to
humans).

  * •

When only one reference is available per instance, COMET-QE (a strong
referenceless metric with crosslingual contextual representations; Rei et al.,
2020) achieves higher correlation with human judgments than all reference-
based metrics. This raises a concern about the current standard evaluation
practice in machine translation and summarization that uses reference-based
metrics with a single reference per instance.

  * •

Our findings confirm many others who report that recent metrics achieve
substantially higher correlation with human judgments than popular metrics
like BLEU and ROUGE in Billboards. We believe these older metrics continue to
be used mainly because modeling researchers value consistency and
accessibility of evaluation practice over long periods of time. Billboards
provide a way to maintain long-term comparability of system output while also
drawing better conclusions about system quality, using advances in evaluation.
All generators continue to be evaluated with new metrics on Billboards.

##  2 Bidimensional Leaderboards

We propose Billboards to simultaneously drive progress in natural language
generation and its evaluation, which are often disconnected in current
research. We first describe the general framework (§2.1) and the automatic
analyses they provide (§2.2-2.3). We then discuss our design choices (§2.4)
and the rubric-based, human judgment data necessary to initialize Billboards
(§2.5).

###  2.1 Billboard Framework

The leaderboard paradigm has driven research on state-of-the-art model
performance on many tasks in various fields (e.g., ImageNet, Russakovsky et
al., 2015; SQuAD, Rajpurkar et al., 2016). As applications and tasks become
more diverse, however, the conventional leaderboard paradigm presents a
serious challenge: the assumption becomes too strong that predetermined,
automatic metrics can reliably score the system performance over time. In
particular, scores from automatic metrics often diverge from human judgments
in language generation tasks, especially when models become increasingly
powerful Ma et al. (2019).

Much recent work proposed new evaluation metrics that improve correlations
with human judgments in certain generation tasks (Clark et al., 2019; Zhang et
al., 2020b; Sellam et al., 2020; Hessel et al., 2021, _inter alia_), but most
developers of generation models are not benefiting from them (See Appendix A
for our analysis of papers from NAACL/ACL 2021). From the perspective of
generation model developers, it is not clear which of these many metrics in
the literature is most reliable in which generation task or dataset, resulting
in community-wide overuse of long-standing metrics like BLEU and ROUGE.
Developers of evaluation metrics, on the other hand, are missing the
opportunity to apply their metrics to new generation models and compare them
with the existing ones. We propose Billboards that bridge this gap between
generation modeling and evaluation development.

#### Generators, Metrics, and Scores

A Billboard for a language generation task consists of sets of generators and
evaluation metrics:
$\mathcal{G}=\\{G_{i}\\}_{i=1}^{I},\mathcal{M}=\\{M_{j}\\}_{j=1}^{J}$. Each
generator $G_{i}$ takes as input $X_{k}$ (e.g., source text in machine
translation) and generates text: $Y_{i,k}=G_{i}(X_{k})$. A metric $M_{j}$
assigns a score to each generated text given the generation input and the
corresponding set of references $\mathcal{R}_{k}$:
$s_{i,j,k}=M_{j}(Y_{i,k},\mathcal{R}_{k},X_{k})$. The last two arguments to
the function are optional; some metrics do not require references (i.e.,
referenceless or quality estimation metrics) or the generation input (e.g.,
BLEU). We then compute the aggregate score $s_{i,j}$ by averaging $s_{i,j,k}$
over $K$ test examples.

#### Rankings

In contrast to standard leaderboards, Billboards have a dynamic set of
evaluation metrics, and generators are not ranked by a predefined metric. We
first rank the metrics by measuring their correlations to human judgments as
commonly done in the generation evaluation literature Zhang et al. (2020b);
Sellam et al. (2020). Let $h_{i,k}$ be a human score for $Y_{i,k}$ (i.e.,
output from generator $G_{i}$ on input $X_{k}$). We compute the instance-level
Pearson correlation for every metric $M_{j}$ between $h_{i,k}$ and $s_{i,j,k}$
($M_{j}$ score for $Y_{i,k}$). All metrics are ranked by their correlations.
We then use the top metric $M_{j^{*}}$ to rank the generators in the
descending order of $s_{i,j^{*}}$. We defer our discussions on alternative
design choices (§2.4) and human evaluations (§2.5). We note, however, that the
overall framework of Billboards still holds regardless of these decisions.

###  2.2 Ensemble of Metrics

So far, we have assumed that metrics are used individually in isolation, but
Billboards provide a unique opportunity to examine metrics collectively.
Different metrics can capture different aspects of generation quality; even if
a metric is not sufficiently informative in isolation, it might reflect an
important aspect of text quality that the existing metrics overlook. Here we
consider a straightforward and interpretable ensemble of metrics using a
regression model with $\ell_{1}$ regularization Tibshirani (1994). Let the
ensemble’s score be

$\displaystyle\hat{h}_{i,k}=\sum_{j=1}^{J}w_{j}\cdot s_{i,j,k},$

where $w_{j}$ is a scalar coefficient associated with the $j$th metric and the
intercept term is suppressed. We optimize the vector of coefficients
$\mathbf{w}$ with the pairs of output text and a human score
$\\{Y_{i,k},h_{i,k}\\}_{k=1}^{K}$ from the test data:

$\displaystyle\mathbf{w^{*}}=\operatorname*{arg\,min}_{\mathbf{w}}\sum_{k=1}^{K}\left(h_{i,k}-\hat{h}_{i,k}\right)^{2}+\lambda\lVert\mathbf{w}\rVert_{1}$

The $\ell_{1}$ regularization produces sparse coefficients and improves
interpretability by removing highly correlated metrics. Moreover, it avoids
the need for practitioners to run many metrics to obtain an ensemble score
when used outside our Billboards. Our goal for the ensemble is to provide a
useful signal to the research community, rather than to achieve the best
possible correlation with human judges at a given time; we tune $\lambda$ to
get three non-zero coefficients. Every metric is standardized by its mean and
standard deviation on the test data.

Similar to the individual metrics, we rank this ensemble metric by its
correlation to the human judgments. To make fair comparisons, we simulate
situations where the ensemble is applied to a newly submitted generator that
has no human evaluations. Specifically, we perform cross validation that holds
out the human judgments for each generator $G_{i}$ and runs regression on the
rest; we then apply these $I$ regression models to the corresponding held-out
data and calculate the overall correlation. We will see that the ensemble
metric outperforms all individual metrics in some cases, suggesting that
different metrics can capture different aspects.

#### Reproduciblity

The ensemble metric is updated every time a new metric is submitted (Fig. 1).
For reproducibility, we keep track of every past ensemble metric with a
signature that indicates its coefficients, $\lambda$, and input metrics in the
backend. Similar to SacreBLEU Post (2018), model developers can report the
signature for easy replication of their scores from the ensemble
metric.333E.g., ensemble.wmt20-zh-en+refs.AB+version.1. Further, all
generation outputs are saved on the leaderboards, so model developers can
download outputs from all past models and compare in any way.

###  2.3 Mixed-Effects Model Analysis

Recent work Kasai et al. (2022) observed that automatic metrics tend to
overrate machine-generated text over human one on the MSCOCO image captioning
task Chen et al. (2015). This problem is particularly severe in conventional
metrics that are based on n-gram overlap such as BLEU and CIDEr Vedantam et
al. (2015). This raises a significant concern about the continuous use of
these conventional metrics in generation tasks as models become increasingly
powerful (and more similar to humans); those metrics unintentionally
discourage researchers from developing human-like, strong generation models.
To quantify this undesirable property, we propose a linear mixed-effects model
that compares the two groups of machine- and human-generated text. The
underlying model assumes that $s_{i,j,k}$, the score from metric $M_{j}$ for
generator $G_{i}$ and test example $k$, can be expressed as (the intercept
term is suppressed for brevity):

$\displaystyle s_{i,j,k}\\!=\\!\beta_{0}^{j}\mathbbm{1}\\{G_{i}\>\text{is
machine}\\}\\!+\\!\beta_{1}^{j}h_{i,k}\\!+\\!\gamma_{k}\\!+\\!\epsilon_{i,j,k}$

where $\gamma_{k}$ is the random effect for example $k$, and
$\epsilon_{i,j,k}$ is Gaussian noise. Intuitively, $\beta_{0}^{j}$ measures
how much metric $M_{j}$ overrates machine generation over human one, compared
against the human judgment $h_{i,k}$. $\beta_{0}^{j}=0$ means being neutral,
and indeed we will find that $\beta_{0}^{j}$ is significantly positive in most
cases (§4). We standardize all metric scores over the test samples to compare
the size of $\beta_{0}^{j}$. We apply the _lme4_ package Bates et al. (2015).

Figure 2:  Comparisons and meta-evaluations of crowdworker and rubric-based,
expert evaluations for WMT20 ZH-EN and CNNDM summarization. Every dot
represents one test instance that is evaluated by the same numbers of experts
and crowdworkers (one for WMT20 ZH-EN and three for CNNDM) for fair
comparisons. We randomly sampled instances with diverging evaluations in two
areas  and conducted binary meta-evaluations (good  or bad quality ). Meta-
evaluations agree more with the expert evaluations:  > in the upper left
squares and  > in the lower right squares. We suspect that the highlighted
text might have caused the disagreement.

###  2.4 Design Choices and Discussion

In our current setup, we make several design choices for metrics and their
rankings:

  * •

M.1 Metrics are expected to positively correlate with the generation output
quality.

  * •

M.2 By default, metrics are ranked based on their instance-level Pearson
correlations with human judgments. We also compute and present their system-
level Kendall rank correlations.

  * •

M.3 When available, reference-based metrics use multiple references per
instance.

M.1 implies that we need to take the negative of metric scores that are
intended to negatively correlate (e.g., TER, Snover et al., 2006). This
normalization is also done in WMT metric competitions (Callison-Burch et al.,
2007, 2008, _inter alia_).

While instance-level correlations are commonly used to evaluate and compare
automatic metrics for various language generation tasks (Sellam et al., 2020;
Fabbri et al., 2021; Hessel et al., 2021, _inter alia_), there are several
alternatives to M.2. For example, Pearson, Spearman’s rank, or Kendall rank
correlations can be used on a system (i.e., generator) level Callison-Burch et
al. (2007); Macháček and Bojar (2014); Mathur et al. (2020b). However, such
system-level correlations would substantially reduce data points to compare
automatic scores, resulting in many ties in the ranking. Spearman’s and
Kendall rank correlations become brittle when multiple generators are similar
in overall output quality; penalizing a metric for swapping two similar
generators is misleading Macháček and Bojar (2014). Moreover, if a metric can
perform well on an instance level, it can be used to augment human judgments
by, for example, flagging likely wrong ratings Mathur et al. (2020b). Thus, we
encourage researchers to develop metrics that correlate well with human
judgments on an instance level. Prior work also points out other problems in
ranking metrics like outlier effects where outlier systems have a
disproportionately large effect on the overall correlation Mathur et al.
(2020a, b). We therefore assume M.2 in the current version of Billboards, but
this can be modified in a future version.

M.3 is supported by our experimental results in §4 that multiple references
substantially improve reference-based metrics, and a single reference is often
insufficient to outperform strong referenceless metrics. Some metrics have
specifications for multiple references (e.g., BLEU, CIDEr). In the other
cases, we evaluate outputs against every reference and take the maximum score,
following prior work on image captioning evaluation Zhang et al. (2020b);
Hessel et al. (2021).444Intuitively, the maximum score measures the distance
to the closest out of equally valid generations.

###  2.5 Human Evaluation

Human evaluations are required to initialize Billboards; they are used to rank
metrics, train the metric ensembling model, and assess how much each metric
overrates machines. Recent work, however, points out problems when evaluations
are done by crowdworkers even when extensive quality controls are performed
Gillick and Liu (2010); Toral et al. (2018); Freitag et al. (2021); Clark et
al. (2021). Freitag et al. (2021) show that rubric-based machine translation
evaluations by professional translators led to substantially different
generator rankings from the crowdsource evaluations in WMT 2020 Barrault et
al. (2020), where WMT participants or Amazon Mechanical Turkers directly
assess each translation’s adequacy by a single score (direct assessment).
These crowdworker evaluations depend highly on individual annotators’
discretion and understanding of the annotation scheme Freitag et al. (2021);
Clark et al. (2021), making it difficult to decompose, interpret, and validate
Kasai et al. (2022). Moreover, these direct assessment scores make it
difficult to interpret evaluation results for downstream applications where
some aspects are particularly important (e.g., accessibility for people with
visual impairments in image captioning, Gleason et al., 2020; gender bias in
machine translation, Stanovsky et al., 2019).

Motivated by this line of work, we perform meta-evaluations to compare
crowdsourced and rubric-based expert evaluations. Fig. 2 plots overall scores
for test examples from WMT20 ZH-EN Barrault et al. (2020); Freitag et al.
(2021) and CNNDM summarization Fabbri et al. (2021). Each instance is
evaluated by averaging the same number of crowdworkers and expert scores for
fair comparisons. We see that substantially many instances fall into
disagreement: crowdworkers give much higher scores than experts (lower right
square) or the reverse (upper left square). We sample and shuffle 20/25
examples from either type and ask a meta-evaluator to make a binary decision
(good  or bad quality ).555The meta-evaluations were done by a bilingual
speaker (WMT20 ZH-EN) and the first author of this paper (CNNDM). Meta-
evaluations agree more with the expert evaluations (e.g., 22 and 0  in the
upper left and lower right squares for CNNDM, respectively). In the examples
on the left, crowdworkers fail to properly assess a valid translation with
different structure than the reference (posted a video to celebrate vs.
congratulated via video) or a summary that combines information from different
parts of the article. The examples on the right illustrate that crowdworkers
can be fooled by inaccurate yet fluent generations (does not know the reason
vs. does not know if Sanchez decided). Given this result, we decide to
initialize our Billboards with rubric-based expert evaluations for all
generation tasks. We still encourage future work to explore ways to improve
crowdsourced evaluations for scalability.

##  3 Experiments

Having established the framework, we set up Billboards for three natural
language generation tasks: machine translation, summarization, and image
captioning. To maximize the performance of reference-based metrics, we use as
many references as possible for each task. See §4 for an analysis on the
effect of varying numbers of references.

###  3.1 Tasks

#### Machine Translation

We experiment with two language pairs from the WMT 2020 news translation task
Barrault et al. (2020): Chinese$\rightarrow$English (WMT20 ZH-EN) and
English$\rightarrow$German (WMT20 EN-DE). We use outputs from all submitted
translation systems.666https://www.statmt.org/wmt20/translation-task.html.
These two language pairs have expert, rubric-based scores (MQM) from Freitag
et al. (2021) for a subset of 10 submitted systems, including the top-
performing systems and human translations. Each output sentence is evaluated
by three professional translators. Following Freitag et al. (2021), the three
scores are averaged to get an instance-level score.

We use all human translations available as a reference set for reference-based
metrics. Concretely, every test instance in WMT20 ZH-EN has two translations
provided by different human translation services: Human-A and Human-B Barrault
et al. (2020). In addition to Human-A and Human-B, WMT20 EN-DE provides a
translation that is created by linguists who are asked to paraphrase Human-A
and Human-B as much as possible (Human-P, Freitag et al., 2020). These
paraphrased translations are shown to increase correlations with human
judgments by mitigating the translationese effect and diversifying the
reference when the generation quality is measured by reference-based metrics
Freitag et al. (2020).

|  |  |  | Single Metrics | Ensemble of Metrics
---|---|---|---|---|---
Dataset | $|\mathcal{G}|$ | $|\mathcal{M}|$ | Top Gen. | Top Metric | Corr. | Linear Combination | Corr.
WMT20 ZH-EN | 19 | 15 | Huoshan | COMET | 0.55 |  $1.72\cdot$COMET-QE+$1.48\cdot$COMET+$1.21\cdot$BLEURT | 0.61
WMT20 EN-DE | 17 | 11 | Tohoku | COMET | 0.49 |  $1.19\cdot$COMET+$0.36\cdot$COMET-QE+$0.02\cdot$Prism-ref | 0.51
CNNDM | 26 | 15 | Lead-3 | COMET | 0.41 |  $2.85\cdot$COMET+$0.26\cdot$COMET-QE+$0.01\cdot$BERTScore | 0.29
MSCOCO | 4 | 15 | VinVL-large | RefCLIP-S | 0.45 |  $2.08\cdot$RefCLIP-S+$1.51\cdot$RefOnlyC+$0.82\cdot$CIDEr | 0.45
Table 1: Summary of Billboards as of Jan. 10, 2022. Huoshan: Wu et al.
(2020a); Tohoku: Kiyono et al. (2020); VinVL-large: Zhang et al. (2021);
COMET, COMET-QE: Rei et al. (2020); BLEURT: Sellam et al. (2020); Prism-ref:
Thompson and Post (2020); BERTScore: Zhang et al. (2020b); RefCLIP-S: Hessel
et al. (2021); RefOnlyC: Kasai et al. (2022). COMET-QE is a referenceless
metric. BLEURT is specifically trained to evaluate into-English translations.
RefCLIP-S uses image features unlike most metrics for image captioning.
RefOnlyC removes image features from RefCLIP-S and only uses reference text
features from CLIP Radford et al. (2021).

Along with all submitted generators in WMT20 ZH-EN and WMT20 EN-DE, we train
three transformer baselines with the fairseq library Ott et al. (2019) and
place them in our Billboards: transformer-base, transformer-large, and
transformer-large-ensemble with similar hyperparameters (e.g., 6-layer encoder
and decoder) to the ones trained on the WMT16 EN-DE data in Vaswani et al.
(2017).777Data and models are available at
https://github.com/jungokasai/billboard/tree/master/baselines. These baselines
allow researchers to compare their translation models without resource-
intensive techniques such as backtranslation Sennrich et al. (2016a), model
ensembling, and deep encoders Kasai et al. (2021a). These techniques are all
used in top-performing systems of WMT20 Wu et al. (2020a); Kiyono et al.
(2020) but might be infeasible in many research settings. See Appendix B for a
list of all hyperparameters for the baselines.

#### Summarization

We use the CNN/DailyMail corpus (CNNDM, Hermann et al., 2015; Nallapati et
al., 2016). We use the standard train/dev./test split and 24 models from
Fabbri et al. (2021). 100 test articles are annotated with 10 summaries
written by humans Kryscinski et al. (2019). For those 100 articles, rubric-
based, expert evaluations for 18 generators, including human-written
highlights, are provided by Fabbri et al. (2021).888Some of the outputs are
lowercased and/or tokenized. In these cases, we apply the NLTK detokenizer
Bird et al. (2009) and/or the Stanford CoreNLP truecaser Manning et al.
(2014). We encourage, however, future model developers to provide clean,
untokenized output to improve the reproducibility and transparency of
evaluation results Post (2018); Kasai et al. (2022). Each output summary is
evaluated by three experts along four dimensions: coherence (collective
quality of all summary sentences), consistency (factual alignment with the
article, penalizing for hallucinations), fluency (quality of the individual
sentences), and relevance (selection of important content). An instance-level
score is computed by averaging scores over all these categories and the three
experts. Note that this aggregation method can be modified, depending on the
downstream task of interest Kasai et al. (2022). All 10 human-written
summaries are used as the reference set for reference-based metrics.999Prior
work used a concatenation of author-written highlights as a reference, but
here we do not add it to the reference set. This is because these highlights
are sometimes noisy (e.g., containing URLs) or lack coherence Fabbri et al.
(2021).

#### Image Captioning

We use the MSCOCO dataset Lin et al. (2014) that consists of everyday-scene
photos sampled from Flickr. Every image is annotated with five captions
written by crowdworkers Chen et al. (2015). We apply the standard Karpathy
split Karpathy and Fei-Fei (2015). For each of 500 test images, rubric-based
evaluations (THumB 1.0) are available for five systems, including one caption
from a crowdworker Kasai et al. (2022). Similar to machine translation and
summarization, we use all five crowdworker captions as a reference set for
reference-based metrics.

###  3.2 Mixed-Effects Models

Our mixed-effects model analyzes how much every automatic metric overrates
machines over humans (§2.3). This means that we need to free up one human
generation per instance to measure its scores in the reference-based metrics.
For machine translation, we score Human-B using the reference set of Human-A
(WMT20 ZH-EN) or Human-A and Human-P (WMT20 EN-DE). For CNNDM, we use
concatenated highlights as human-generated summaries and use the 10 human-
written summaries from Kryscinski et al. (2019) as the reference. We follow
Kasai et al. (2022) for MSCOCO and score their randomly-selected Human caption
using the other four as the reference. As the distinction between the
reference and human generation (e.g., Human-A vs. Human B on WMT20 ZH-EN) is
arbitrary, we found that swapping the roles would still lead to similar
results (See Appendix E).

##  4 Results and Analysis

Here we discuss the current results and make several key observations about
the state of language generation evaluation. Table 1 summarizes the four
Billboards. It is particularly noteworthy that COMET, a metric designed for
machine translation, achieves the best correlation on the CNNDM summarization
task as well. COMET evaluates the similarity between the crosslingual
representations from XLM-RoBERTa Conneau et al. (2020) for input text and its
translation candidate. But these crosslingual representations can, of course,
be used monolingually for English summarization. This illustrates an
additional benefit of Billboards that centralize different generation tasks
and find surprising task transferability of learning-based metrics. See
Appendices B and C for lists of all participating generators and metrics.

#### Ensemble Metric

The rightmost section of Table 1 shows the chosen metrics and their
coefficients in the ensemble (§2.2). On the machine translation tasks, the
ensemble metric outperforms the top individual metric.101010We found a major
reason for the anomaly in CNNDM; an outlier generator that does not use the
standard CNNDM training data (the GPT-2 zero-shot model; Ziegler et al., 2019)
has a disproportionately large effect on the regression models. The ensemble
metric outperformed the top individual metric of COMET when the zero-shot
model was removed.  In particular, we see a substantial gain of $0.06$ points
in WMT20 ZH-EN. The referenceless metric of COMET-QE is selected both for
WMT20 ZH-EN and WMT20 EN-DE, suggesting complementary effects of diverse
metrics. To further test this hypothesis, we perform ablations that drop one
out of the three metrics at a time (Table 2). We see that only dropping COMET-
QE would result in a decrease in the correlation score. This implies that the
referenceless metric provides important information that the others do not.

Removed Metric | – | COMET | COMET-QE | BLEURT
---|---|---|---|---
Correlation | 0.61 | 0.61 | 0.57 | 0.61
Table 2: Ensemble ablation studies on WMT20 ZH-EN. Only removing COMET-QE
leads to a correlation drop. See Appendix D for the other datasets.

#### Mixed-Effects Models

Seen in Table 3 are the results from our analysis that measures how much
metrics overrate machines over humans (§2.3). We see that the fixed-effect
coefficient $\beta_{0}$ is significantly positive in most cases. Referenceless
metrics tend to have smaller coefficients. This can be due to the more diverse
nature of human text than machine-generated text; reference-based metrics give
a low score to human text that differs from the references even if it is of
high quality. The conventional n-gram overlap-based metrics (BLEU, ROUGE, and
CIDEr) have particularly large coefficients. These results suggest that the
evaluation practice should be regularly updated as our generation models
become stronger (and perhaps, more similar to human generation) in the future.
Note that unlike the other tasks, “human-generated text” for CNNDM
summarization is an automatic concatenation of author highlights, which
contains substantial noise Fabbri et al. (2021). This might explain the
neutral and negative coefficients.

ZH-EN | COMET | COMET-QE | BLEURT | BLEU
---|---|---|---|---
$0.27_{\pm 0.02}$ | $0.13_{\pm 0.01}$ | $0.32_{\pm 0.02}$ | $0.62_{\pm 0.02}$
EN-DE | COMET | COMET-QE | Prism-ref | BLEU
$0.08_{\pm 0.03}$ | $-0.17_{\pm 0.02}$ | $0.44_{\pm 0.02}$ | $0.33_{\pm 0.03}$
CNNDM | COMET | COMET-QE | BERTScore | ROUGE-L
$-0.17_{\pm 0.12}$ | $0.02_{\pm 0.11}$ | $-0.04_{\pm 0.12}$ | $0.33_{\pm 0.13}$
COCO | RefCLIP-S | RefOnlyC | CIDEr | CLIP-S
$0.09_{\pm 0.06}$ | $0.24_{\pm 0.06}$ | $0.43_{\pm 0.06}$ | ${-0.04_{\pm 0.05}}$
Table 3: $\beta_{0}$ fixed-effect coefficients from the linear mixed-effects
models, quantifying how much automatic metrics overrate machines over humans,
relative to human raters. $\beta_{0}\\!=\\!0$ is neutral, and statistical
significance is indicated by red (positive) or blue text (negative). The
subscripts indicate 90% confidence intervals. Three metrics that correlate
best with the human judgments are shown as well as one popular metric. COMET-
QE and CLIP-S are referenceless. See §E for the other metrics. Figure 3:
Correlations with varying numbers of references. In all cases, one reference
is not sufficient to outperform the referenceless COMET-QE metric. The default
ROUGE assumes English input.

#### Effects of the Number of References

Fig. 3 plots correlations over varying numbers of references. COMET was the
top-performing reference-based metric regardless of the number of references,
but we observe that it underperforms the refererenceless metric when only one
reference is given. Model performance in machine translation and summarization
is commonly measured by applying reference-based metrics against one reference
per instance in the research community. Our finding thus raises a further
concern about the current evaluation practice. Finally, we see that popular
choices of BLEU and ROUGE metrics have much lower correlations than the recent
metrics over various numbers of references, in line with the recent studies
(Mathur et al., 2020a, _inter alia_).

##  5 Related and Future Work

#### Related Benchmarks

WMT organizes the metric competition track in parallel with the translation
task every year (Mathur et al., 2020b; Barrault et al., 2020, _inter alia_).
Participants submit automatic scores for the translation outputs from the
parallel translation task. Unfortunately, most of these new metrics are not
used by subsequent machine translation work, perhaps because they are tested
solely against the concurrent translation submissions and it is up to model
developers to execute or even implement new metrics. The GEM workshop Gehrmann
et al. (2021) conducts extensive analysis of models and evaluation methods
over a wide set of generation tasks. Billboards ease the burden through
standard leaderboard experience where generator developers only need to upload
generation outputs for the test split. Billboards also offer automatic
ensembling of metrics and quantify the diversity that a new metric adds. The
human-in-the-loop Genie leaderboard Khashabi et al. (2021) centralizes
crowdsourced evaluations for generation tasks. The current Billboard setup is
based on rubric-based, expert evaluation data from previous work, but future
work can explore ways to improve crowdsourced evaluations and use them to
update Billboards.

#### From Bidimensional to Multidimensional

Billboards lend themselves to a natural extension: multidimensional
leaderboards. In particular, generation models have more aspects than
generation quality, such as training and inference efficiency, sample
efficiency, and robustness. These aspects are often ignored in the current
leaderboard paradigm but are important to better serving practitioners’ needs
Schwartz et al. (2019); Ethayarajh and Jurafsky (2020); Mishra and Arunkumar
(2021). There are ongoing modeling and benchmarking efforts especially for
efficient machine translation (Heafield et al., 2020; Peng et al., 2021; Kasai
et al., 2021b, _inter alia_). We leave this extension to future work and
specifically target the gap between generation modeling and evaluation.

##  6 Conclusion

We introduced Billboards, a simple yet powerful generalization of leaderboards
that bridges the gap between generation modeling and evaluation research. We
established and released four Billboards on machine translation,
summarization, and image captioning tasks. We demonstrated that their built-in
analysis of metric ensembling and mixed-effects modeling revealed key insights
into the current state of natural language generation and its evaluation
methods. Billboards allow for a standard leaderboard experience both on the
modeling and evaluation sides. We invite submissions from researchers through
our website.

## Acknowledgements

We thank Kyunghyun Cho, Elizabeth Clark, Jesse Dodge, Saadia Gabriel, Michal
Guerquin, Daniel Khashabi, Swaroop Mishra, Jamie Morgenstern, Phoebe Mulcaire,
Hao Peng, Sofia Serrano, the ARK group at UW, the Mosaic team at the Allen
Institute for AI, and the anonymous reviewers for their helpful feedback on
this work. We also thank Xinyan Yu for WMT Chinese-English meta-evaluations
and Muyun Yang and Shujian Huang for sharing with us the CWMT training data.
This work was supported in part by the DARPA MCS program through NIWC Pacific
(N66001-19-2-4031) and Google Cloud Compute.

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: semantic propositional image caption evaluation.  In _Proc. of ECCV_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proc. of CVPR_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proc. of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_.
  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020.  Findings of the 2020 conference on machine translation (WMT20).  In _Proc. of WMT_.
  * Bates et al. (2015) Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015.  Fitting linear mixed-effects models using lme4.  _Journal of Statistical Software_.
  * Bawden et al. (2020) Rachel Bawden, Giorgio Maria Di Nunzio, Cristian Grozea, Inigo Jauregi Unanue, Antonio Jimeno Yepes, Nancy Mah, David Martinez, Aurélie Névéol, Mariana Neves, Maite Oronoz, Olatz Perez-de Viñaspre, Massimo Piccardi, Roland Roller, Amy Siu, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Dina Wiemann, and Lana Yeganova. 2020.  Findings of the WMT 2020 biomedical translation shared task: Basque, Italian and Russian as new additional languages.  In _Proc. of WMT_.
  * Bird et al. (2009) Steven Bird, Evan Klein, and Edward Loper. 2009.  _Natural Language Processing with Python_.  Cambridge University Press.
  * Böhm et al. (2019) Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019.  Better rewards yield better summaries: Learning to summarise without references.  In _Proc. of EMNLP_.
  * Bouscarrat et al. (2019) Léo Bouscarrat, Antoine Bonnefoy, Thomas Peel, and Cécile Pereira. 2019\.  STRASS: A light and effective method for extractive summarization based on sentence embeddings.  In _Proc. of ACL_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Proc. of NeurIPS_.
  * Callison-Burch et al. (2007) Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007.  (meta-) evaluation of machine translation.  In _Proc. of WMT_.
  * Callison-Burch et al. (2008) Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008.  Further meta-evaluation of machine translation.  In _Proc. of WMT_.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.  Re-evaluating the role of Bleu in machine translation research.  In _Proc. of EACL_.
  * Chen et al. (2020) Tanfang Chen, Weiwei Wang, Wenyang Wei, Xing Shi, Xiangang Li, Jieping Ye, and Kevin Knight. 2020.  DiDi’s machine translation system for WMT2020.  In _Proc. of WMT_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015.  Microsoft COCO captions: Data collection and evaluation server.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proc. of ACL_.
  * Clark et al. (2021) Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021.  All that’s ‘human’ is not gold: Evaluating human evaluation of generated text.  In _Proc. of ACL_.
  * Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019.  Sentence mover’s similarity: Automatic evaluation for multi-sentence texts.  In _Proc. of ACL_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proc. of ACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. of NAACL_.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Proc. of NeurIPS_.
  * Dong et al. (2018) Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018.  BanditSum: Extractive summarization as a contextual bandit.  In _Proc. of EMNLP_.
  * Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc’Aurelio Ranzato, and Michael Auli. 2020.  On the evaluation of machine translation systems trained with back-translation.  In _Proc. of ACL_.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proc. of EMNLP_.
  * Eyal et al. (2019) Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.  Question answering as an automatic evaluation metric for news article summarization.  In _Proc. of NAACL_.
  * Fabbri et al. (2021) Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating summarization evaluation.  _TACL_.
  * Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _TACL_.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proc. of EMNLP_.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proc. of GEM_.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proc. of EMNLP_.
  * Germann (2020) Ulrich Germann. 2020.  The University of Edinburgh’s submission to the German-to-English and English-to-German tracks in the WMT 2020 news translation and zero-shot translation robustness tasks.  In _Proc. of WMT_.
  * Gillick and Liu (2010) Dan Gillick and Yang Liu. 2010.  Non-expert evaluation of summarization systems is risky.  In _Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_.
  * Gleason et al. (2020) Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M. Kitani, and Jeffrey P. Bigham. 2020.  Twitter a11y: A browser extension to make twitter images accessible.  In _Proc. of CHI_.
  * Guo et al. (2018) Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018.  Soft layer-specific multi-task summarization with entailment and question generation.  In _Proc. of ACL_.
  * Gwinnup and Anderson (2020) Jeremy Gwinnup and Tim Anderson. 2020.  The AFRL WMT20 news translation systems.  In _Proc. of WMT_.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mengnan Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018.  Achieving human parity on automatic Chinese to English news translation.
  * Heafield et al. (2020) Kenneth Heafield, Hiroaki Hayashi, Yusuke Oda, Ioannis Konstas, Andrew Finch, Graham Neubig, Xian Li, and Alexandra Birch. 2020.  Findings of the fourth workshop on neural generation and translation.  In _Proc. of WNGT_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Proc. of NeurIPS_.
  * Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021.  CLIPScore: A reference-free evaluation metric for image captioning.  In _Proc. of EMNLP_.
  * Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018\.  A unified model for extractive and abstractive summarization using inconsistency loss.  In _Proc. of ACL_.
  * Jiang and Bansal (2018) Yichen Jiang and Mohit Bansal. 2018.  Closed-book training to improve summarization encoder memory.  In _Proc. of EMNLP_.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions.  In _Proc. of CVPR_.
  * Kasai et al. (2021a) Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2021a.  Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation.  In _Proc. of ICLR_.
  * Kasai et al. (2021b) Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, and Noah A. Smith. 2021b.  Finetuning pretrained transformers into RNNs.  In _Proc. of EMNLP_.
  * Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022.  Transparent human evaluation for image captioning.  In _Proc. of NAACL_.
  * Khashabi et al. (2021) Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2021.  GENIE: A leaderboard for human-in-the-loop evaluation of text generation.
  * Kiyono et al. (2020) Shun Kiyono, Takumi Ito, Ryuto Konno, Makoto Morishita, and Jun Suzuki. 2020.  Tohoku-AIP-NTT at WMT 2020 news translation task.  In _Proc. of WMT_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proc. of ACL Demo and Poster Sessions_.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proc. of EMNLP_.
  * Kryściński et al. (2018) Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018\.  Improving abstraction in text summarization.  In _Proc. of EMNLP_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proc. of ACL_.
  * Li et al. (2020) Zuchao Li, Hai Zhao, Rui Wang, Kehai Chen, Masao Utiyama, and Eiichiro Sumita. 2020\.  SJTU-NICT’s supervised and unsupervised neural machine translation systems for the WMT20 news translation task.  In _Proc. of WMT_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Proc. of Text Summarization Branches Out_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: common objects in context.  In _Proc. of ECCV_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proc. of EMNLP_.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proc. of WMT_.
  * Macháček and Bojar (2014) Matouš Macháček and Ondřej Bojar. 2014.  Results of the WMT14 metrics shared task.  In _Proc. of WMT_.
  * Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Proc. of ACL System Demonstrations_.
  * Marie et al. (2021) Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021.  Scientific credibility of machine translation research: A meta-evaluation of 769 papers.  In _Proc. of ACL_.
  * Mathur et al. (2020a) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a.  Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proc. of ACL_.
  * Mathur et al. (2020b) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b.  Results of the WMT20 metrics shared task.  In _Proc. of WMT_.
  * Meng et al. (2020) Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xianfeng Zeng, Qinsong Zeng, Peng Li, Ming Chen, Jie Zhou, Sifan Liu, and Hao Zhou. 2020.  WeChat neural machine translation systems for WMT20.  In _Proc. of WMT_.
  * Mishra and Arunkumar (2021) Swaroop Mishra and Anjana Arunkumar. 2021.  How robust are model rankings : A leaderboard customization approach for equitable evaluation.  In _Proc. of AAAI_.
  * Molchanov (2020) Alexander Molchanov. 2020.  PROMT systems for WMT 2020 shared news translation task.  In _Proc. of WMT_.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proc. of CoNLL_.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Ranking sentences for extractive summarization with reinforcement learning.  In _Proc. of NAACL_.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook FAIR’s WMT19 news translation task submission.  In _Proc. of WMT_.
  * Oravecz et al. (2020) Csaba Oravecz, Katina Bontcheva, László Tihanyi, David Kolovratnik, Bhavani Bhaskar, Adrien Lardilleux, Szymon Klocek, and Andreas Eisele. 2020.  eTranslation’s submissions to the WMT 2020 news translation task.  In _Proc. of WMT_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proc. of NAACL Demonstrations_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proc. of ACL_.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  In _Proc. of NAACL_.
  * Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021.  Random feature attention.  In _Proc. of ICLR_.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proc. of WMT_.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proc. of WMT_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proc. of WMT_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JLMR_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proc. of EMNLP_.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proc. of EMNLP_.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015.  ImageNet large scale visual recognition challenge.  _IJCV_.
  * Schwartz et al. (2019) Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019.  Green AI.  _CACM_.
  * Scialom et al. (2019) Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019\.  Answers unite! unsupervised metrics for reinforced summarization models.  In _Proc. of EMNLP_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proc. of ACL_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proc. of ACL_.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving neural machine translation models with monolingual data.  In _Proc. of ACL_.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _Proc. of ACL_.
  * Sharma et al. (2019) Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019.  An entity-driven framework for abstractive summarization.  In _Proc. of EMNLP_.
  * Shi et al. (2020) Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei Dang, Xue Zhengshan, and Jie Hao. 2020.  OPPO’s machine translation systems for WMT20.  In _Proc. of WMT_.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006\.  A study of translation edit rate with targeted human annotation.  In _Proc. of AMTA_.
  * Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019.  Evaluating gender bias in machine translation.  In _Proc. of ACL_.
  * Thompson and Post (2020) Brian Thompson and Matt Post. 2020.  Automatic machine translation evaluation in many languages via zero-shot paraphrasing.  In _Proc. of EMNLP_.
  * Tibshirani (1994) Robert Tibshirani. 1994.  Regression shrinkage and selection via the Lasso.  _Journal of the Royal Statistical Society, Series B_.
  * Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018.  Attaining the unattainable? reassessing claims of human parity in neural machine translation.  In _Proc. of WMT_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proc. of NeurIPS_.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  CIDEr: Consensus-based image description evaluation.  In _Proc. of CVPR_.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proc. of ACL_.
  * Wang et al. (2016) Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016.  CharacTer: Translation edit rate on character level.  In _Proc. of WMT_.
  * Wei et al. (2020) Daimeng Wei, Hengchao Shang, Zhanglin Wu, Zhengzhe Yu, Liangyou Li, Jiaxin Guo, Minghan Wang, Hao Yang, Lizhi Lei, Ying Qin, and Shiliang Sun. 2020.  HW-TSC’s participation in the WMT 2020 news translation shared task.  In _Proc. of WMT_.
  * Wu et al. (2020a) Liwei Wu, Xiao Pan, Zehui Lin, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2020a.  The Volctrans machine translation system for WMT20.  In _Proc. of WMT_.
  * Wu et al. (2020b) Shuangzhi Wu, Xing Wang, Longyue Wang, Fangxu Liu, Jun Xie, Zhaopeng Tu, Shuming Shi, and Mu Li. 2020b.  Tencent neural machine translation systems for the WMT20 news translation task.  In _Proc. of WMT_.
  * Wu and Hu (2018) Yuxiang Wu and Baotian Hu. 2018.  Learning to extract coherent summary via deep reinforcement learning.  In _Proc. of AAAI_.
  * Xu and Durrett (2019) Jiacheng Xu and Greg Durrett. 2019.  Neural extractive text summarization with syntactic compression.  In _Proc. of EMNLP_.
  * Yu et al. (2020) Lei Yu, Laurent Sartran, Po-Sen Huang, Wojciech Stokowiec, Domenic Donato, Srivatsan Srinivasan, Alek Andreev, Wang Ling, Sona Mokra, Agustin Dal Lago, Yotam Doron, Susannah Young, Phil Blunsom, and Chris Dyer. 2020.  The DeepMind Chinese–English document translation system at WMT2020.  In _Proc. of WMT_.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.  PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization.  In _Proc. of ICML_.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  VinVL: Making visual representations matter in vision-language models.  In _Proc. of CVPR_.
  * Zhang et al. (2020b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b.  BERTScore: Evaluating text generation with BERT.  In _Proc. of ICLR_.
  * Zhang et al. (2018) Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018.  Neural latent extractive document summarization.  In _Proc. of EMNLP_.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and VQA.  In _Proc. of AAAI_.
  * Zhou et al. (2018) Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018\.  Neural document summarization by jointly learning to score and select sentences.  In _Proc. of ACL_.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.

##  Appendix A Case Studies of Evaluation Practice

Fig. 4 depicts breakdowns of evaluation metrics used in the papers on machine
translation and summarization from NAACL and ACL 2021. We examined all papers
whose title contains “machine translation” and “summarization.” We see the
clear gap between generation modeling and evaluation research; most
researchers do not take advantage of recent metrics that correlate better with
human judgments.

Figure 4: Breakdowns of evaluation metrics used in the papers on machine
translation and summarization from NAACL and ACL 2021. We examined all papers
whose title contains “machine translation” and “summarization” and disregarded
papers primarily on evaluation metrics. “QA” metrics use a QA system to
evaluate summaries (e.g., Eyal et al., 2019). “Specialized” indicates
specialized evaluation in a particular dimension, rather than the overall
generation quality, such as document-level evaluations on contrastive sets
Voita et al. (2019).

##  Appendix B Participating Generators

Here we list the generators submitted in the initial Billboards.

###  B.1 WMT20 ZH-EN

Hyperparameter | Value
---|---
label smoothing | 0.1
# max tokens | 4096
dropout rate | 0.1
encoder embedding dim | 512
encoder ffn dim | 2048
# encoder attn heads | 8
decoder embedding dim | 512
decoder ffn dim | 2048
# decoder attn heads | 8
max source positions | 1024
max target positions | 1024
Adam lrate | $5\times 10^{-4}$
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.98
lr-scheduler | inverse square
warm-up lr | $1\times 10^{-7}$
# warmup updates | 4000
# max updates | 600K
# GPUs | 8
length penalty | 0.6
Table 4: Transformer-base fairseq hyperparameters and setting. Hyperparameter | Value
---|---
label smoothing | 0.1
# max tokens | 4096
dropout rate | 0.1
encoder embedding dim | 1024
encoder ffn dim | 4096
# encoder attn heads | 16
decoder embedding dim | 1024
decoder ffn dim | 4096
# decoder attn heads | 16
max source positions | 1024
max target positions | 1024
Adam lrate | $5\times 10^{-4}$
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.98
lr-scheduler | inverse square
warm-up lr | $1\times 10^{-7}$
# warmup updates | 4000
# max updates | 600K
# GPUs | 8
length penalty | 0.6
Table 5: Transformer-large and transformer-large-ensemble fairseq
hyperparameters and setting. Transformer-large-ensemble ensembles four
transformer-large models with different random initializations.

We use all 16 submissions for the WMT20 ZH-EN task Barrault et al.
(2020)111111https://www.statmt.org/wmt20/results.html. as well as our own
three transformer baselines that were implemented in fairseq Ott et al.
(2019). Our baselines allow researchers to compare their translation models
without resource-intensive techniques such as backtranslation Sennrich et al.
(2016a), model ensembling, and deep encoders Kasai et al. (2021a). Tables 4
and 5 list the hyperprameters. We generally follow the setting from Vaswani et
al. (2017). We use newstest-2019 as the dev. set and the official training
data.121212http://www.statmt.org/wmt20/translation-task.html. We apply Moses
tokenization Koehn et al. (2007) and BPE with 32K operations Sennrich et al.
(2016b) to English text. We tokenize Chinese text with the Jieba
package,131313https://github.com/fxsjy/jieba. following Hassan et al. (2018).
Separately from English, BPE with 32K operations is then applied to Chinese.
The decoder input and output embeddings are tied. Moses detokenization is
applied to get the final outputs in the last step. We make the three models
and preprocessed train/dev. data publicly
available.141414https://github.com/jungokasai/billboard/tree/master/baselines.
Table 6 lists all generators and their automatic evaluation scores from the
top-performing metric (ensemble in this case).

Generator | Description | Score
---|---|---
Huoshan Translate | Wu et al. (2020a) | 78.85
THUNLP | Not available | 78.81
Huawei TSC | Wei et al. (2020) | 78.79
DeepMind | Yu et al. (2020) | 78.76
WeChat AI | Meng et al. (2020) | 78.75
Tencent Translation | Wu et al. (2020b) | 78.74
DiDi NLP | Chen et al. (2020) | 78.66
OPPO | Shi et al. (2020) | 78.59
Online-B | Not available | 78.36
SJTU-NICT | Li et al. (2020) | 78.27
trans-large-ensemble | §B.1 | 77.35
trans-large | §B.1 | 76.98
Online-A | Not available | 76.86
trans-base | §B.1 | 76.79
dong-nmt | Not available | 76.74
Online-G | Not available | 76.44
zlabs-nlp | Not available | 75.79
Online-Z | Not available | 75.05
WMT Biomed Baseline | Bawden et al. (2020) | 73.89
Table 6: WMT20 ZH-EN generators and reference papers. The score column
indicates the score from the metric that currently correlates best with the
human judgments (ensemble).

###  B.2 WMT20 EN-DE

Similar to WMT20 ZH-EN, we use all 14 submissions for the WMT20 EN-DE task
along with our three transformer baselines. The same hyperparameters are
chosen as in WMT20 ZH-EN (Tables 4 and 5). We preprocess both English and
German text by the Moses tokenizer and joint BPE with 32K operations. All
embeddings are shared. We apply the Moses detokenizer to get the final
outputs. Table 7 shows the generators and their automatic evaluation scores
from the top-performing metric (ensemble).

Generator | Description | Score
---|---|---
Tohoku-AIP-NTT | Kiyono et al. (2020) | 90.50
Tencent Translate | Wu et al. (2020b) | 90.43
OPPO | Shi et al. (2020) | 90.42
eTranslation | Oravecz et al. (2020) | 90.39
Online-B | Not available | 90.38
Huoshan Translate | Wu et al. (2020a) | 90.32
AFRL Gwinnup and Anderson (2020) | 90.16
Online-A | Not available | 90.12
UEDIN | Germann (2020) | 89.98
PROMT NMT | Molchanov (2020) | 89.66
trans-large | §B.2 | 89.60
trans-large-ensemble | §B.2 | 89.59
trans-base | §B.2 | 89.35
Online-Z | Not available | 89.26
Online-G | Not available | 88.98
zlabs-nlp | Not available | 88.65
WMT Biomed Baseline | Bawden et al. (2020) | 88.23
Table 7: WMT20 EN-DE generators and reference papers. The score column
indicates the score from the metric that currently correlates best with the
human judgments (ensemble).

###  B.3 CNNDM Summarization

We submit all 26 models from Fabbri et al.
(2021).151515https://github.com/Yale-LILY/SummEval. Table 8 shows all models
and their automatic evaluation scores from the top-performing metric (COMET).

Generator | Description | Score
---|---|---
Lead-3 | First 3 sentences | -0.011
T5 | Raffel et al. (2020) | -0.030
BART | Lewis et al. (2020) | -0.032
Pegasus-dynamic-mix | Zhang et al. (2020a) | -0.044
RNES | Wu and Hu (2018) | -0.049
Unified-ext-abs | Hsu et al. (2018) | -0.056
Pegasus-huge-news | Zhang et al. (2020a) | -0.056
REFRESH | Narayan et al. (2018) | -0.067
ROUGESal Pasunuru and Bansal (2018) | -0.073
Human-H | Highlights | -0.075
NEUSUM | Zhou et al. (2018) | -0.083
BanditSum | Dong et al. (2018) | -0.083
LATENT | Zhang et al. (2018) | -0.099
Closed-book-decoder | Jiang and Bansal (2018) | -0.112
Multi-task-Ent-QG | Guo et al. (2018) | -0.117
Pointer-Generator | See et al. (2017) | -0.144
UniLM | Dong et al. (2019) | -0.151
Bottom-Up | Gehrmann et al. (2018) | -0.160
JEC Xu and Durrett (2019) | -0.167
Fast-abs-rl | Chen and Bansal (2018) | -0.189
NeuralTD | Böhm et al. (2019) | -0.215
Improve-abs | Kryściński et al. (2018) | -0.329
BertSum-abs | Liu and Lapata (2019) | -0.341
STRASS | Bouscarrat et al. (2019) | -0.405
GPT-2-zero-shot | Ziegler et al. (2019) | -0.441
SENECA | Sharma et al. (2019) | -0.735
Table 8: CNNDM summarization generators and reference papers. They are from
Fabbri et al. (2021), but we apply detokenization Bird et al. (2009) and/or
truecasing Manning et al. (2014) to standardize the model outputs for better,
reproducible evaluations. The score column indicates the score from the metric
that currently correlates best with the human judgments (COMET).

###  B.4 MSCOCO Image Captioning

We submit the four strong models from the literature Kasai et al.
(2022).161616https://github.com/jungokasai/THumB/tree/master/mscoco. They
share similar pipeline structure but vary in model architecture, (pre)training
data, model size, and (pre)training objective. Table 9 shows the models with
their papers and automatic scores from the top-performing metric (RefCLIP-S).

Generator | Description | Score
---|---|---
VinVL-large171717Model with CIDEr optmization, https://github.com/microsoft/Oscar/blob/master/VinVL_MODEL_ZOO.md#Image-Captioning-on-COCO. | Zhang et al. (2021) | 83.78
VinVL-base181818Model with CIDEr optmization. | Zhang et al. (2021) | 83.45
Unified-VLP | Zhou et al. (2020) | 82.59
Up-Down191919Model with cross-entropy optimization, https://vision-explorer.allenai.org/image_captioning. | Anderson et al. (2018) | 80.63
Table 9: MSCOCO image captioning generators and reference papers. The score
column indicates the score from the metric that currently correlates best with
the human judgments (RefCLIP-S).

##  Appendix C Participating Metrics

Table 10 discusses details and configurations of the automatic metrics that we
implement in our initial Billboards.

Metric | Description | Refs. | Src. | Cont.
---|---|---|---|---
BLEU202020SacreBLEU implementation of sentence-level BLEU-4; https://github.com/mjpost/sacreBLEU/blob/v1.2.12/sacrebleu.py#L999. | Papineni et al. (2002) | ✓ | ✗ | ✗
ROUGE-3212121https://pypi.org/project/rouge-score/. | Lin (2004) | ✓ | ✗ | ✗
ROUGE-L | Lin (2004) | ✓ | ✗ | ✗
METEOR | Banerjee and Lavie (2005) | ✓ | ✗ | ✗
TER222222https://github.com/mjpost/sacrebleu. | Snover et al. (2006) | ✓ | ✗ | ✗
METEOR232323https://www.nltk.org/_modules/nltk/translate/meteor_score.html. | Banerjee and Lavie (2005) | ✓ | ✗ | ✗
chrF242424https://github.com/m-popovic/chrF. | Popović (2015) | ✓ | ✗ | ✗
CIDEr252525https://github.com/salaniz/pycocoevalcap. | Vedantam et al. (2015) | ✓ | ✗ | ✗
SPICE | Anderson et al. (2016) | ✓ | ✗ | ✗
CharacTER262626https://github.com/rwth-i6/CharacTER. | Wang et al. (2016) | ✓ | ✗ | ✗
chrF++ | Popović (2017) | ✓ | ✗ | ✗
SummaQA272727https://github.com/ThomasScialom/summa-qa. | Scialom et al. (2019) | ✗ | ✓ | ✓
BERTScore | Zhang et al. (2020b) | ✓ | ✗ | ✓
BLEURT282828https://huggingface.co/metrics/bleurt. | Sellam et al. (2020) | ✓ | ✗ | ✓
COMET292929https://github.com/Unbabel/COMET/. | Rei et al. (2020) | ✓ | ✓ | ✓
COMET-QE | Rei et al. (2020) | ✗ | ✓ | ✓
Prism-ref303030https://github.com/thompsonb/prism. | Thompson and Post (2020) | ✓ | ✗ | ✓
Prism-src | Thompson and Post (2020) | ✗ | ✓ | ✓
CLIP-S313131https://github.com/salaniz/pycocoevalcap. | Hessel et al. (2021) | ✗ | ✓ | ✓
RefCLIP-S | Hessel et al. (2021) | ✓ | ✓ | ✓
RefOnlyC | Kasai et al. (2022) | ✓ | ✗ | ✓
Table 10: Automatic metrics and their reference papers. The refs., src., and
cont. columns indicate whether they use references, input source features, and
pretrained contextual representations (e.g., BERT; Devlin et al., 2019),
respectively.

##  Appendix D Additional Ensemble Metric Ablations

Seen in Table 11 are ablation studies for the ensemble metrics where one of
the three selected metrics is removed at a time. Dropping one metric often has
no impact on the correlation score, suggesting that these metrics are highly
redundant and capture similar aspects of the output quality. Billboards
encourage researchers to explore ways to diversify automatic evaluations by
updating the ensemble metric every time a new metric is submitted.

ZH-EN | – | COMET | COMET-QE | BLEURT
---|---|---|---|---
0.61 | 0.61 | 0.57 | 0.61
EN-DE | – | COMET | COMET-QE | Prism-ref
0.51 | 0.52 | 0.52 | 0.52
CNNDM | – | COMET | COMET-QE | BERTScore
0.29 | 0.23 | 0.31 | 0.31
COCO | – | RefCLIP-S | RefOnlyC | CIDEr
0.45 | 0.44 | 0.42 | 0.43
Table 11: Correlations from ensemble ablation studies. One of the three
selected metrics is removed at a time, and a new Lasso regression model is
trained on the remaining metrics. The bigger the correlation drop is, the
bigger the contribution is from the removed metric. COMET-QE is a
referenceless metric.

##  Appendix E Additional Mixed-Effects Analysis

Table 12 presents fixed-effect coefficients that measure how much each
automatic metric overrates machines over humans (§2.3). With some exceptions
in CNNDM summarization, almost all automatic metrics underrate human
generations (significantly positive coefficients). Table 13 swaps the roles of
human-generated text, but we still see similar patterns: almost all metrics
overrate machines over humans, but the problem is mitigated in COMET-QE, a
referenceless, quality estimation metric. This confirms that our findings hold
independently of the design choice.

ZH-EN | COMET-QE | Ensemble | COMET | BLEURT | BERTScore | CharacTER | MoverScore | METEOR
---|---|---|---|---|---|---|---|---
$0.13_{\pm 0.01}$ | $0.26_{\pm 0.01}$ | $0.27_{\pm 0.02}$ | $0.32_{\pm 0.02}$ | $0.52_{\pm 0.02}$ | $0.56_{\pm 0.02}$ | $0.57_{\pm 0.02}$ | $0.57_{\pm 0.02}$
Prism-ref | chrF | TER | chrF++ | ROUGE-3 | BLEU | ROUGE-L | Prism-src
$0.58_{\pm 0.02}$ | $0.58_{\pm 0.02}$ | $0.59_{\pm 0.02}$ | $0.60_{\pm 0.02}$ | $0.61_{\pm 0.02}$ | $0.62_{\pm 0.02}$ | $0.64_{\pm 0.02}$ | $1.13_{\pm 0.02}$
EN-DE | COMET-QE | Ensemble | COMET | MoverScore | chrF | chrF++ | BLEU | CharacTER
$-0.17_{\pm 0.02}$ | $0.03_{\pm 0.02}$ | $0.08_{\pm 0.02}$ | $0.22_{\pm 0.03}$ | $0.29_{\pm 0.02}$ | $0.32_{\pm 0.02}$ | $0.33_{\pm 0.03}$ | $0.33_{\pm 0.03}$
BERTScore | Prism-ref | TER | Prism-src |  |  |  |
$0.43_{\pm 0.02}$ | $0.44_{\pm 0.02}$ | $0.49_{\pm 0.03}$ | $1.46_{\pm 0.03}$ |  |  |  |
CNNDM | TER | COMET | Ensemble | BERTScore | MoverScore | COMET-QE | CharacTER | BLEURT
$-0.58_{\pm 0.14}$ | $-0.17_{\pm 0.12}$ | $-0.16_{\pm 0.12}$ | $-0.04_{\pm 0.12}$ | $-0.03_{\pm 0.11}$ | $0.02_{\pm 0.11}$ | $0.14_{\pm 0.15}$ | $0.25_{\pm 0.12}$
SummaQA | ROUGE-L | BLEU | Prism-ref | chrF | chrF++ | ROUGE-3 | METEOR
$0.27_{\pm 0.10}$ | $0.33_{\pm 0.13}$ | $0.37_{\pm 0.11}$ | $0.38_{\pm 0.12}$ | $0.43_{\pm 0.13}$ | $0.45_{\pm 0.13}$ | $0.49_{\pm 0.11}$ | $0.53_{\pm 0.12}$
COCO | CLIP-S | RefCLIP-S | CharacTER | chrF | ROUGE-3 | chrF++ | RefOnlyC | Ensemble
$-0.04_{\pm 0.05}$ | $0.09_{\pm 0.06}$ | $0.13_{\pm 0.07}$ | $0.18_{\pm 0.07}$ | $0.22_{\pm 0.07}$ | $0.23_{\pm 0.07}$ | $0.24_{\pm 0.06}$ | $0.24_{\pm 0.06}$
SPICE | METEOR | BLEU | CIDEr | ROUGE-L | BERTScore | TER | MoverScore
$0.25_{\pm 0.07}$ | $0.32_{\pm 0.07}$ | $0.39_{\pm 0.07}$ | $0.43_{\pm 0.06}$ | $0.44_{\pm 0.07}$ | $0.45_{\pm 0.06}$ | $0.45_{\pm 0.07}$ | $0.51_{\pm 0.05}$
Table 12: Fixed-effect coefficients $\beta_{0}$ from the linear mixed-effects analysis that measures how much automatic metrics overrate machine text over human, as compared to human raters (§2.3). $\beta_{0}=0$ is neutral, and statistical significance is indicated by red (positive) or blue text (negative). The subscripts indicate 90% confidence intervals. COMET-QE, Prism-src, SummaQA and CLIP-S are referenceless metrics. In both WMT20 ZH-EN and WMT20 EN-DE, Human-B is evaluated as human-generated translations. Human-A (WMT20 ZH-EN) and Human-A and Human-P (WMT20 EN-DE) are used as the reference set for reference-based metrics. ZH-EN | COMET-QE | Ensemble | COMET | BLEURT | TER | BERTScore | ROUGE-3 | Prism-ref
---|---|---|---|---|---|---|---|---
$0.03_{\pm 0.01}$ | $0.07_{\pm 0.01}$ | $0.08_{\pm 0.02}$ | $0.09_{\pm 0.02}$ | $0.23_{\pm 0.02}$ | $0.24_{\pm 0.02}$ | $0.24_{\pm 0.02}$ | $0.25_{\pm 0.02}$
CharacTER | ROUGE-L | chrF | MoverScore | METEOR | chrFpp | BLEU | Prism-src
$0.25_{\pm 0.02}$ | $0.26_{\pm 0.02}$ | $0.27_{\pm 0.02}$ | $0.27_{\pm 0.02}$ | $0.29_{\pm 0.02}$ | $0.29_{\pm 0.02}$ | $0.30_{\pm 0.02}$ | $0.79_{\pm 0.02}$
EN-DE | COMET-QE | Ensemble | COMET | MoverScore | Prism-ref | chrF | BERTScore | CharacTER
$-0.09_{\pm 0.02}$ | $-0.07_{\pm 0.02}$ | -$0.06_{\pm 0.03}$ | $0.02_{\pm 0.02}$ | $0.18_{\pm 0.02}$ | $0.20_{\pm 0.02}$ | $0.21_{\pm 0.02}$ | $0.22_{\pm 0.02}$
chrF++ | BLEU | TER | Prism-src |  |  |  |
$0.22_{\pm 0.02}$ | $0.23_{\pm 0.02}$ | $0.32_{\pm 0.02}$ | $1.38_{\pm 0.03}$ |  |  |  |
Table 13: Fixed-effect coefficients $\beta_{0}$ from the linear mixed-effects
analysis that measures how much automatic metrics overrate machine text over
human, as compared to human raters (§2.3). The roles of human translations are
swapped: Human-A is evaluated, and Human-B (WMT20 ZH-EN) and Human-B and
Human-P (WMT20 EN-DE) are used as the reference. We still see similar patterns
to Table 12: almost all automatic metrics overrate machines over humans, but
the problem is less severe in the referenceless metric of COMET-QE.

##  Appendix F Crowdworker vs. Rubric-based Expert Evaluations

Seen in Table 14 are examples where crowdworker evaluators Barrault et al.
(2020) and professional translators Freitag et al. (2021) disagree:
crowdworkers give lower scores to the human-generated translations than the
machine-generated ones. The first case requires document-level context to
properly evaluate. Document-level context and diversity in high-quality human
translations can mislead crowdworkers.

|  WMT20 ZH-EN |
---|---|---
Source |  希望兴安省继续为白俄罗斯企业提供便利条件。 |  凭的是相机而动的时势驾驭。
Huoshan |  It is hoped that Xing’an Province will continue to provide convenient conditions for Belarusian enterprises. |  It is based on the current situation of the camera.
Human-A |  He hoped that Hung Yen Province would continue to provide convenient conditions for Belarusian enterprises. |  This relies on the ability to seize opportunities.
Human-B |  He hoped that this could continue in the future. |  It is based on the observation of various situations at different times.
Table 14: Examples where crowdsource evaluators Barrault et al. (2020) and
professional translators Freitag et al. (2021) disagree: crowdworkers give
lower scores to the human-generated translations than the machine-generated
ones. The first case requires document-level context to properly evaluate. 兴安省
is Hung Yen Province in Vietnam in this context, but there is entity
ambiguity. (Xing’an Province that existed in the Republic of China.) The
second one illustrates the diversity of human generations that misleads
crowdworkers.