# Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes
Risk Decoding: A Case Study for COMET

Chantal Amrhein1 Rico Sennrich1,2
1Department of Computational Linguistics, University of Zurich
2School of Informatics, University of Edinburgh
<EMAIL_ADDRESS>
###### Abstract

Neural metrics have achieved impressive correlation with human judgements in
the evaluation of machine translation systems, but before we can safely
optimise towards such metrics, we should be aware of (and ideally eliminate)
biases towards bad translations that receive high scores. Our experiments show
that sample-based Minimum Bayes Risk decoding can be used to explore and
quantify such weaknesses. When applying this strategy to COMET for
en$\rightarrow$de and de$\rightarrow$en, we find that COMET models are not
sensitive enough to discrepancies in numbers and named entities. We further
show that these biases cannot be fully removed by simply training on
additional synthetic data.

##  1 Introduction

Recently, neural machine translation evaluation metrics have reached better
correlation scores with human evaluators than surface-level metrics like BLEU
(Papineni et al., 2002). In particular, COMET (Rei et al., 2020a) has shown
significant potential as a leading evaluation metric both in shared tasks
(Mathur et al., 2020; Freitag et al., 2021d) and other studies on machine
translation evaluation metrics (Kocmi et al., 2021). The main benefits of such
neural metrics are that they do not rely on surface-level similarity to a
reference translation and that some of them operate in a multilingual
representation space which allows to also directly compare translations to the
original source sentence.

A recent evaluation as part of the WMT 2021 metrics shared task (Freitag et
al., 2021d) suggests that neural metrics are also less susceptible to many
weaknesses of earlier non-neural metrics, e.g. an antonym in the translation
hurting the BLEU score exactly the same amount as a synonym. However, it is
still unclear whether or not these metrics also introduce new biases that are
harder to detect since they are essentially ‘black box’ metrics that do not
explain why a certain score is attributed to a translation. Failing to
identify these biases in neural metrics could lead the community to optimise
towards metric ‘blind spots’, either directly through methods such as Minimum
Risk Training Shen et al. (2016), or more slowly by basing modelling choices
on metric scores. It is therefore worthwhile to find new means to uncover
weaknesses of neural machine translation metrics.

In this paper, we show that sampling-based Minimum Bayes Risk (MBR) decoding -
where a pool of samples are compared against each other using a machine
translation evaluation metric as a utility function - can render blind spots
of these metrics more observable. When applying COMET as the utility function,
we find many examples where this metric chooses a translation hypothesis that
contains different numbers or named entities than the source and reference
(see examples in Table 1). Through a targeted sensitivity analysis, we
identify that these are indeed weaknesses of COMET and we show that it can be
hard to remove them from the model.

Our contributions are the following:

  * •

We propose to use sample-based MBR decoding to explore and measure weaknesses
of neural machine translation evaluation metrics.

  * •

We find that COMET is not sensitive enough to number differences and
mistranslations of named entities when translating from en$\rightarrow$de and
de$\rightarrow$en.

  * •

We show that simply retraining COMET on synthetic data is not enough to
eliminate these blind spots.

src | Schon drei Jahre nach der Gründung verließ Green die Band 1970.
---|---
ref | Green left the band three years after it was formed, in 1970.
MBRchrF++ | Already three years after the foundation, Green left the band in 1970.
MBRCOMET | Three years after the creation, Green left the band in 1980.
src | […] Mahmoud Guemama’s Death - Algeria Loses a Patriot […], Says President Tebboune.
ref | […] Mahmoud Guemamas Tod - Algerien verliert einen Patrioten […], sagt Präsident Tebboune.
MBRchrF++ | […] Mahmoud Guemamas Tod - Algerien verliert einen Patriot […], sagt Präsident Tebboune.
MBRCOMET | […] Mahmud Guemamas Tod - Algerien verliert einen Patriot […], sagt Präsident Tebboene.
Table 1: Examples of MBR decoding outputs with chrF++ and COMET as utility
metrics. The outputs chosen with COMET indicate less sensitivity towards
discrepancies in numbers and named entities.

##  2 Related Works

How to best evaluate machine translation models has been a long-standing
question in the research community. Ideally, we could employ humans to judge
the quality of different models but this is time-consuming, costly and
requires trained professionals. Various automatic machine translation metrics
have been proposed over the years that typically compare a machine translation
output to a reference sentence according to surface-level similarity (Papineni
et al., 2002; Popović, 2015) or on a shallow semantic level (Banerjee and
Lavie, 2005).

With the rise of contextual embeddings and large multilingual Transformer
language models, metrics that map translations and references into the same
latent space and compare the cosine similarity between them (Lo, 2020) or use
them as inputs to predict a score (Sellam et al., 2020; Rei et al., 2020a)
have become popular. Such neural metrics have been shown to agree more with
human evaluation than previously popular automatic metrics such as BLEU
(Papineni et al., 2002) or chrF (Popović, 2015).

However, these neural metrics can also introduce new biases that we are not
yet aware of. In this paper, we aim to find a way to identify such weaknesses
via Minimum Bayes Risk (MBR) decoding. While MBR decoding was a frequently
used decoding strategy in the days of statistical machine translation (Goel
and Byrne, 2000; Kumar and Byrne, 2004; Tromble et al., 2008), it has only
recently gained traction in the context of neural machine translation. Eikema
and Aziz (2020) argue that MBR decoding using samples as hypotheses results in
an unbiased candidate pool in contrast to beam search outputs which maximise
the probability under the model. Indeed, if the machine translation model
generating the samples is strong enough, humans prefer MBR-decoded hypotheses
selected with BLEURT (Sellam et al., 2020) as the utility function over beam
search outputs (Freitag et al., 2021c).

Müller and Sennrich (2021) further show that MBR outputs can inherit biases
from the utility function that is used, for example, the length bias (Nakov et
al., 2012) with BLEU as the utility function. Consequently, it stands to
reason that MBR decoding can also be used to uncover new biases of metrics
that are used as utility functions, as we will show in this work. We first
describe how MBR decoding works in Section 3 and the setup of our experiments
in Section 4. Then, we show how we explore weaknesses of COMET in Section 5
and formulate a new approach based on MBR decoding to quantify a metric’s
sensitivity to different error types in Section 6.

##  3 Minimum Bayes Risk Decoding

Traditionally, maximum a posteriori (MAP) decoding is used in the context of
neural machine translation. The goal is to find the translation hypothesis
$h_{i}$ among all possible hypotheses $H$ that is most probable under the
translation model given the source sentence $x$ and the model parameters
$\theta$:

$y^{*}=\operatorname*{argmax}_{h_{i}\in H}p_{model}(h_{i}|x,\theta)$ (1)

In practice, it is not feasible to consider every possible hypothesis. Beam
search offers a popular and effective approximation.

In contrast, MBR decoding aims to find a translation that minimises the
expected cost (risk) of choosing a candidate translation $h_{i}$, assuming
that we have some loss function $L$ to compare a prediction $h_{i}$ to a true
translation $h_{j}$, and access to the probability distribution $P$:

$y^{*}=\operatorname*{argmin}_{h_{i}\in H}\sum_{h_{j}\in
H}P(h_{j}|x)L(h_{i},h_{j})$ (2)

Since we do not have access to the true probability distribution $P$, and
cannot exhaustively sum over all possible translations $H$, we have to make
approximations based on the translation model distribution and a subset of
hypotheses. Eikema and Aziz (2020) suggest drawing ancestral samples from the
translation model as a set of unbiased hypotheses, and we follow this
sampling-based MBR approach. Since the functions we use to compare
translations are similarity functions where higher values are better, we will
call them utility functions $u$, and search for the translation $h_{i}$ that
maximises the expected utility:

$y^{*}=\operatorname*{argmax}_{h_{i}\in C}\frac{1}{|S|}\sum_{h_{j}\in
S}u(h_{i},h_{j})$ (3)

The set $C$ of translations that are considered as candidates, and the set $S$
of translations that act as ‘support hypotheses’ to calculate the utility,
could be created separately. In this work, we follow Eikema and Aziz (2020)
and let our translation model produce a set of 100 ancestral samples that are
used both as candidates and support ($C=S$).

Any automatic machine translation evaluation metric can be used as the utility
function $u$. Eikema and Aziz (2021) find that BEER (Stanojević and Sima’an,
2014) works best among a range of non-neural metrics. More recently, Freitag
et al. (2021c) compare several metrics as utility functions in a human
evaluation of MBR-decoded outputs where the neural metric BLEURT (Sellam et
al., 2020) clearly outperforms non-neural metrics. In this paper, we explore
the use of another neural evaluation metric as the utility function, namely
COMET. Since the reference-based COMET model takes the source, a translation
hypothesis and a reference (approximated in MBR decoding with another
hypothesis) as input, our formulation of MBR decoding now takes into account
the source sentence $x$:

$y^{*}=\operatorname*{argmax}_{h_{i}\in C}\frac{1}{|S|}\sum_{h_{j}\in
S}u(x,h_{i},h_{j})$ (4)

For an efficiency-related discussion of our implementation, please refer to
Section 4.3.

##  4 Experiment Setup

###  4.1 Translation Model

To be able to generate samples, we train two Transformer Base machine
translation models (Vaswani et al., 2017) using the
nematus111github.com/EdinburghNLP/nematus (Sennrich et al., 2017) framework,
one from de$\rightarrow$en and one from en$\rightarrow$de. We follow Eikema
and Aziz (2021) and use all available parallel data from the WMT 2018 news
shared task (Bojar et al., 2018) except for Paracrawl as training data. This
amounts to 5.9 million sentence pairs. After deduplication, we are left with
approximately 5.6 million sentence pairs for training.

Both models are trained for 250k updates and we choose the best checkpoint
based on the BLEU score as evaluated on newstest2017 using
SacreBLEU222BLEU+case.mixed+lang.X-X+numrefs.1+smooth.exp+tok.13a+version.1.4.2
(Post, 2018). We compute a joint subword vocabulary of size 32k with byte pair
encoding (Sennrich et al., 2016) using the SentencePiece implementation (Kudo
and Richardson, 2018). During training and decoding, the maximum sequence
length is set to 200 tokens.

Our models are built with 6 encoder layers, 6 decoder layers, 8 attention
heads with an embedding and hidden state dimension of 512 and a feed-forward
network dimension of 2048. For regularisation, we use a dropout rate of 0.1
for BPE-dropout (Provilkov et al., 2020) during training, for the embeddings,
for the residual connections, in the feed-forward sub-layers and for the
attention weights. We train with tied encoder and decoder input embeddings as
well as tied decoder input and output embeddings (Press and Wolf, 2017) and
apply exponential smoothing of model parameters (decay $10^{-4}$) Junczys-
Dowmunt et al. (2018). Following previous work on MBR decoding (Eikema and
Aziz, 2020), we train without label smoothing.

For optimisation, we use Adam (Kingma and Ba, 2015) with standard
hyperparameters and a learning rate of $10^{-4}$. We follow the Transformer
learning schedule described in (Vaswani et al., 2017) with a linear warm-up
over 4,000 steps. Our token batch size is set to 16,348 and we train on 4
NVIDIA Tesla V100 GPUs.

###  4.2 COMET Models

We experiment with two COMET models that were trained towards two different
regression objectives:

  * •

wmt20-comet-da (Rei et al., 2020b), developed for the WMT 2020 metrics shared
task (Mathur et al., 2020) and trained to predict Direct Assessment (DA)
(Graham et al., 2017) scores.

  * •

wmt21-comet-mqm (Rei et al., 2021), developed for the WMT 2021 metrics shared
task (Freitag et al., 2021d) and trained to predict MQM scores (Freitag et
al., 2021a) based on the Multidimensional Quality Metrics (MQM) methodology
(Uszkoreit and Lommel, 2013).

###  4.3 MBR Decoding Implementations

For non-neural evaluation metrics, we use the MBR decoding
implementation333https://github.com/Roxot/mbr-nmt provided by Eikema and Aziz
(2021). We decide to use only unique samples both as candidates and as the
support such that no hypothesis is assigned a higher average MBR score simply
because it perfectly matches one or multiple other hypotheses in the support.
In our experiments, we use chrF++ (Popović, 2017) and BLEU as non-neural
metrics. For the latter metric, the implementation internally makes use of
SacreBLEU (Post, 2018).

For our experiments with COMET, we adapt the official COMET
implementation444https://github.com/Unbabel/COMET and implement an option for
MBR decoding. Since COMET first creates a pooled sentence representation of
the source and each of the two hypotheses before constructing a single vector
from these representations and passing it through a regression layer, it is
crucial that the implementation does not naively call COMET on every
hypothesis pair. Instead, we encode the source sentence and every hypothesis
with XLM-R (Conneau et al., 2020) only once and then score all combinations of
hypothesis pairs in parallel.

###  4.4 Evaluation Data

We decide to use the test sets from the WMT 2021 news shared task (Akhbardeh
et al., 2021) as our evaluation data. This dataset brings three major benefits
for our analysis:

  * •

In the de$\leftrightarrow$en directions, it provides at least two references
for every source sentence. This allows us to compare how much MBR scores
differ between two equivalent human translation alternatives as a reference
point.

  * •

This dataset was not part of the training data of the wmt20-comet-da and
wmt21-comet-mqm COMET models which avoids the risk that the models have seen
scores for similarly erroneous translations of these source sentences before.

  * •

During the creation of the test set, special attention was paid to selecting
only sentences originally written in the source language. These source
sentences were then translated to the target language by professional human
translators. Consequently, we can ignore the distorting effects that
‘translationese’ could otherwise have on our evaluation.

There are 1000 sentence triplets (source, two human translations) for
de$\rightarrow$en where we use translation A as our reference and translation
B as an alternative translation and 1002 sentence triplets for
en$\rightarrow$de where we use translation C as our reference and translation
D as an alternative translation.

| Numbers |  | Named Entities
---|---|---|---
| de-en | en-de |  | de-en | en-de
reference |  06.78 |  |  06.54 |  |  | n/a |  | n/a |
alternative |  05.20 | -01.58 |  04.34 | -02.20 |  | 20.61 |  | 29.86 |
beam search |  04.10 | -02.68 |  04.30 | -02.24 |  | 26.05 | +05.44 | 37.12 | +07.26
MBR chrF++ |  08.78 | +02.00 |  06.61 | +00.07 |  | 31.84 | +11.23 | 41.45 | +11.59
MBR bleu |  06.13 | -00.65 |  08.64 | +02.10 |  | 31.01 | +10.40 | 42.93 | +13.07
MBR wmt20-comet-da |  09.66 | +02.88 | 10.90 | +04.36 |  | 32.88 | +12.27 | 49.90 | +20.04
MBR wmt21-comet-mqm | 17.74 | +10.96 | 23.26 | +16.72 |  | 40.03 | +19.42 | 52.13 | +22.27
Table 2: Results of the automatic evaluation. Error rates (%) for numbers and
named entities and error rate changes compared to the reference for numbers
and alternative translation for named entities.

##  5 Exploration of MBR-Decoded Outputs

We employ sampling-based MBR decoding as a strategy to identify weaknesses in
evaluation metrics that are used as utility functions. We believe that – in
addition to general errors – we may also find other errors in such a setup
that can stem from two sources:

First, since samples are often of lower quality than hypotheses produced with
beam search, neural metrics may behave unexpectedly when faced with errors
that occur less frequently in beam search based machine translation outputs on
which they were trained. Second, in MBR decoding, we compare a candidate
translation hypothesis to a pseudo-reference (another hypothesis) instead of
an actual reference. This is also something neural metrics were neither
trained on nor designed to do.

We are most interested in general errors and errors of the first type since
the second type is only relevant for MBR decoding itself. Therefore, we
propose additional experiments in Section 6 to distinguish between these two
sources for the errors we identify below. Note that errors of the second type
may become more important to investigate as MBR decoding becomes more
prevalent.

In our experiments, we first manually compare MBR-decoded outputs that were
chosen with two different evaluation metrics as the utility function: chrF++
and COMET. For COMET, we notice several cases where the chosen hypothesis
contains numbers and named entities that do not match with the source and the
reference, even though the majority of samples in the support set contain the
correct numbers and named entities. Two examples are shown in Table 1. This
stands in contrast to the corrupted reference analysis performed as part of
the WMT 2021 metrics shared task (Freitag et al., 2021d) where COMET mostly
preferred the correct alternative human translation to one with swapped
numbers when comparing to the reference.

To test if these findings apply at scale, we run an automatic evaluation. For
numbers, we use regular expressions to identify numbers in the MBR-decoded
outputs. We then compute error rates compared to the source averaged over a)
how many numbers occurring in the source also occur in the translation and b)
how many numbers occurring in the translation also occur in the source. We
decide to compare to the source to be able to compute the overlaps for the
reference and the alternative human translation as well. The results can be
seen in the left part of Table 2.

For named entities, we use spaCy555English: en_core_web_lg, German:
de_core_news_lg (Honnibal et al., 2020) to identify named entities of type
‘person’. Here, we compute the error rates against the reference rather than
the source (as was done for numbers) since the named entity recognition (NER)
models are not identical for English and German. Specifically, we average over
a) how many named entities occurring in the reference also occur in the
translation and b) how many named entities occurring in the translation also
occur in the reference. The results can be seen in the right part of Table 2.

These simple automatic ‘gold’ annotations produce false positives666For
example, translating “3 pm” in the source to ”15:00” is a valid translation,
but would be counted as a mistake with the automatic number matching., which
explains why neither the reference nor the alternative reference (for named
entities) achieves an error rate of 0%. Nevertheless, these results indicate
that MBR decoding with the COMET metrics chooses more erroneous translations
with respect to these criteria than with the two non-neural metrics or
compared to beam search decoding. Interestingly, the wmt21-comet-mqm model
performs considerably worse than the wmt20-comet-da model in this analysis.

This automatic evaluation has strengthened the findings in our manual
exploration that wrong number and named entity translations are recurring
problems. To better quantify how sensitive COMET models are towards these
error types, we propose to perform an MBR-based sensitivity analysis in the
next section.

| Samples as Support | References as Support
---|---|---
| de-en | en-de | de-en | en-de
| wmt20-comet-da | \+ syn | wmt20-comet-da | \+ syn | wmt20-comet-da | \+ syn | wmt20-comet-da | \+ syn
numsub | 0.028 | 0.032 | 0.016 | 0.020 | 0.044 | 0.057 | 0.030 | 0.039
numadd | 0.050 | 0.061 | 0.029 | 0.042 | 0.091 | 0.119 | 0.064 | 0.088
NEdel | 0.047 | 0.057 | 0.048 | 0.043 | 0.083 | 0.109 | 0.082 | 0.085
numdel | 0.052 | 0.052 | 0.043 | 0.049 | 0.089 | 0.096 | 0.071 | 0.086
NEadd | 0.057 | 0.070 | 0.056 | 0.049 | 0.102 | 0.135 | 0.113 | 0.110
numwhole | 0.066 | 0.084 | 0.045 | 0.057 | 0.115 | 0.154 | 0.085 | 0.112
NEsub | 0.060 | 0.067 | 0.067 | 0.055 | 0.119 | 0.148 | 0.117 | 0.114
altern. | 0.111 | 0.108 | 0.106 | 0.112 | - | - | - | -
noundel | 0.218 | 0.203 | 0.116 | 0.120 | 0.322 | 0.332 | 0.222 | 0.219
nounadd | 0.257 | 0.232 | 0.162 | 0.156 | 0.392 | 0.390 | 0.265 | 0.268
nounsub | 0.272 | 0.245 | 0.182 | 0.172 | 0.417 | 0.408 | 0.299 | 0.282
copy | 0.597 | 0.779 | 1.450 | 1.513 | 0.553 | 0.731 | 1.367 | 1.630
hallucin. | 1.277 | 1.081 | 1.561 | 1.402 | 1.907 | 1.720 | 2.057 | 1.893
Table 3: Effects of randomly adding, substituting or deleting a digit in a
number or a letter in a noun or named entity (NE). Average absolute difference
to MBR score for reference (left) and 1-best beam search output (right) when
using wmt20-comet-da as the utility function. Reference points are marked in
bold. Changes when retraining COMET on added synthetic data are shown in
corresponding “+ syn” columns (retrain-comet-da). Green indicates the MBR
score difference to reference (left) or 1-best beam search output (right) has
increased, while red indicates the MBR score difference has decreased.

##  6 MBR-Based Sensitivity Analysis

As described in Section 3, in MBR decoding, every candidate translation is
assigned a score that represents the average similarity to every hypothesis in
the support computed with a utility function. Consequently, if the support is
kept constant and a targeted change is made to a candidate translation, the
absolute difference in the MBR score indicates how sensitive the utility
function was towards this change. We term this kind of analysis an “MBR-based
sensitivity analysis”.

To measure COMET’s sensitivity towards changes in numbers and named entities,
we create a candidate pool that consists of the reference translation and
several changed variants. Note that the support still contains the same 100
samples that were used to find the MBR-decoded outputs described in Section 5.
In particular, we make the following targeted changes to the reference to
measure the sensitivity towards each change:

  * •

numadd: one digit is added to a number at a random position.

  * •

numdel: one digit is removed from a number at a random position.

  * •

numsub: one digit is substituted with another digit in a number at a random
position.

  * •

numwhole: one entire number is substituted with another number.

  * •

NEadd: one letter is added to a named entity at a random position.

  * •

NEdel: one letter is removed from a named entity at a random position.

  * •

NEsub: one letter is substituted with another letter in a named entity at a
random position.

As a control, we also apply the same types of changes to random nouns in the
reference:

  * •

nounadd: one letter is added to a random noun at a random position.

  * •

noundel: one letter is removed from a random noun at a random position.

  * •

nounsub: one letter is substituted with another letter in a random noun at a
random position.

Additionally, our candidate pool also contains the following hypotheses to be
used as reference points:

  * •

altern.: the second human reference provided as part of the WMT 2021 news
shared task simulating an alternative translation.

  * •

copy: the original, unchanged source sentence simulating a model that simply
copied the source to the decoder side.

  * •

hallucin.: a sentence that is completely unrelated to the source.

We use the same tools to identify numbers and named entities as in Section 5
to create these perturbations in the reference. For each new candidate, we
compute the absolute difference to the MBR score of the reference. We then
average those differences across sentences for each perturbation type. The
results for this experiment with the wmt20-comet-da model can be seen in the
left part of Table 3 in the non-coloured columns.

The reference point candidates, i.e. alternative translation, copied sentence
and hallucination, behave as expected. The average MBR score difference to the
hallucinated sentence is by far the largest, followed by the copied source
sentence. The case of the alternative reference is interesting as it divides
the targeted changes to the reference into two groups. For the _random nouns_
, all targeted changes lead to greater differences to the reference (measured
in MBR score) than for the alternative reference. On the other hand, all
targeted changes to _numbers_ or _named entities_ result in a much smaller
difference in MBR score compared to the alternative reference. This shows that
COMET is not as sensitive to such discrepancies as to other changes.

It is a valid concern that the generally lower quality of samples that are
used as support might affect these results. The hypothesis here is that if we
were to compare to high-quality translations, COMET may be more sensitive
towards number and named entity differences as there would be fewer other
discrepancies between the candidates and the support. To test if this is the
case, we repeat the sensitivity analysis but now use the two alternative
references as the support instead of the 100 samples that were used before.
The candidates are now formed by applying the same perturbations as before to
the 1-best beam search output instead of the reference. The results for this
experiment are shown on the right-hand side of Table 3 in the non-coloured
columns. Note that we cannot compare to an alternative translation for the
beam search output in this setup.

The differences to the MBR score of the unperturbed beam search output are
generally larger in this setup, which indicates that COMET is indeed more
sensitive to errors when used as intended, i.e. with high-quality reference
translations. However, we can still see that the perturbations made to random
nouns result in much larger differences than perturbations made to numbers or
named entities. This indicates that the problem cannot be attributed to MBR
decoding and low-quality pseudo-references alone.

We focus here on the wmt20-comet-da model since this is currently the
recommended model to use by the
authors.777https://github.com/Unbabel/COMET/blob/master/METRICS.md Running the
same analysis with the wmt21-comet-mqm model, results in the same ordering
(see Table 9 in Appendix D) but with much less variance in the scores between
all error types. The same finding concerning the small variance in scores is
highlighted by the authors of COMET in their GitHub repository.

##  7 COMET Retraining

One possible explanation for the low sensitivity of COMET to perturbations of
numbers and named entities is that these errors are too rare in the WMT
outputs used to train COMET. We decide to retrain COMET on the original
training data plus added synthetic data for which we perform the same
perturbations as described in Section 6. The idea is that the newly trained
model has more sensitivity towards named entity or number mismatches between
the translation and its reference and/or source.

For the retrain-comet-da model, we use the data from the WMT metrics shared
tasks collected in the years 2017 to 2019 (Bojar et al., 2017; Ma et al.,
2018, 2019) as training data. For every de$\rightarrow$en or en$\rightarrow$de
system output that contains a number or a named entity, we randomly apply one
of the perturbations described in Section 6 (except for the perturbations of
random nouns). To encourage COMET to punish such synthetically inserted
mismatches, we modify the scores of the original examples by subtracting -0.20
from the Direct Assessment (DA) score. The resulting $\sim$61k synthetic
training examples are then added to the $\sim$640k original examples which
means that roughly 10% of the data are synthetic.

For the retrain-comet-mqm model, we use the same data as described above but
additionally use the DA data from the 2020 WMT metrics shared task (Mathur et
al., 2020) for which we create the synthetic training examples in the same
way. This results in $\sim$770k original training examples and $\sim$66k
synthetic training examples. We first finetune XLM-R on this DA data and in a
second step, we continue finetuning on an MQM score-annotated dataset (Freitag
et al., 2021b). Following Rei et al. (2021), we exclude en$\rightarrow$de and
zh$\rightarrow$en examples from the 2020 DA dataset because the MQM dataset
includes the same examples for these language pairs.

We follow the hyperparameter suggestions in Rei et al. (2020b) and Rei et al.
(2021) for retraining the COMET models but we do not perform model averaging.
The models are trained for two epochs (with an additional epoch of finetuning
on the MQM data for the retrain-comet-mqm model). The hyperparameters are
listed in full in Appendix A.

The model that we finetune towards MQM scores produces generally larger scores
than the official wmt21-comet-mqm model which makes direct comparison less
straightforward. The results for this retrain-comet-mqm model can be found in
Appendix D but here we focus on the retrain-comet-da model. We ensure that the
retrained model still performs as well as the original model on the WMT 2020
metrics shared task Mathur et al. (2020). The average difference in system-
level Pearson correlation to the original comet model is +0.0016 across all
language pairs. The full results can be found in Appendix B in Table 7.

The results of the sensitivity analysis for retrain-comet-da can be seen in
Table 3 in the coloured columns. In most cases, our retrained COMET model (+
sys) does show slightly larger MBR score differences for numbers and named
entities than the original model. However, the differences to the random noun
permutations are still much larger and the differences for named entities in
en$\rightarrow$de even became smaller.

When repeating the automatic analysis from Section 5 with our retrained
models, we see similar effects as for the sensitivity analysis. There are
small differences in the error rates compared to the original COMET models but
our retrained models do not consistently choose better or worse samples in
terms of number and named entity matches. Moreover, the error rate differences
to the original COMET models are smaller than the differences to other utility
functions which indicates that the new models do not differ that much from the
original ones. The full results can be seen in Appendix C.

From this experiment, we conclude that removing such blind spots from COMET -
once identified - might need more effort than simply training on additional
synthetic data. We leave a more extensive analysis of biases in the training
data (e.g. unpunished number mismatches) and further experiments on weakness-
targeted training for future work.

##  8 Conclusion

Identifying weaknesses of neural machine translation evaluation metrics
becomes more important as these essentially ‘black box’ evaluation tools
become more popular and are optimised towards during model development. We
show that MBR decoding can be used to explore biases of such metrics. Through
a case study, we show that COMET is relatively insensitive to mistranslated
numbers and named entities. This can be seen both in the MBR-decoded output
which contains a higher number of these errors compared to beam search (or MBR
with other utility functions) and in an MBR-based sensitivity analysis which
compares the differences in MBR scores that arise when such errors are
introduced to a candidate translation. We also show that this insensitivity is
not simply the result of insufficient training data containing such errors:
retraining COMET with additional synthetic data did not fully alleviate this
weakness.

While errors related to number and named entity translation were very salient
in our exploration, we do not claim that this case study is exhaustive. In our
manual analysis, we also see anecdotal evidence of polarity errors and
nonsensical German compounds. We hope our findings motivate further research
into identifying and mitigating biases of neural machine translation metrics –
we envision that actively searching for biases in neural metrics, for example
by using them as utility functions in MBR, could become an important step
during metric development.

## Acknowledgements

We thank Janis Goldzycher, Mathias Müller and Tannon Kew for their valuable
inputs and the interesting discussions throughout the development of this
work.

## References

  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bojar et al. (2018) Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018.  Findings of the 2018 conference on machine translation (WMT18).  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
  * Bojar et al. (2017) Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017.  Results of the WMT17 metrics shared task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Eikema and Aziz (2020) Bryan Eikema and Wilker Aziz. 2020.  Is MAP decoding all you need? the inadequacy of the mode in neural machine translation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4506–4520, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Eikema and Aziz (2021) Bryan Eikema and Wilker Aziz. 2021.  Sampling-based minimum bayes risk decoding for neural machine translation.
  * Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.
  * Freitag et al. (2021b) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021b.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2021c) Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2021c.  Minimum bayes risk decoding with neural metrics of translation quality.
  * Freitag et al. (2021d) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021d.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Goel and Byrne (2000) Vaibhava Goel and William J Byrne. 2000.  Minimum bayes-risk automatic speech recognition.  _Computer Speech & Language_, 14(2):115–135.
  * Graham et al. (2017) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017.  Can machine translation systems be evaluated by the crowd alone.  _Natural Language Engineering_ , 23(1):3–30.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Kumar and Byrne (2004) Shankar Kumar and William Byrne. 2004.  Minimum Bayes-risk decoding for statistical machine translation.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Lo (2020) Chi-kiu Lo. 2020.  Extended study on using pretrained language models and YiSi-1 for machine translation evaluation.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 895–902, Online. Association for Computational Linguistics.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 671–688, Belgium, Brussels. Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Müller and Sennrich (2021) Mathias Müller and Rico Sennrich. 2021.  Understanding the properties of minimum Bayes risk decoding in neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 259–272, Online. Association for Computational Linguistics.
  * Nakov et al. (2012) Preslav Nakov, Francisco Guzman, and Stephan Vogel. 2012.  Optimizing for sentence-level BLEU+1 yields short translations.  In _Proceedings of COLING 2012_ , pages 1979–1994, Mumbai, India. The COLING 2012 Organizing Committee.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Press and Wolf (2017) Ofir Press and Lior Wolf. 2017.  Using the output embedding to improve language models.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 157–163, Valencia, Spain. Association for Computational Linguistics.
  * Provilkov et al. (2020) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020.  BPE-dropout: Simple and effective subword regularization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1882–1892, Online. Association for Computational Linguistics.
  * Rei et al. (2021) Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021\.  Are references really needed? unbabel-IST 2021 submission for the metrics shared task.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1030–1040, Online. Association for Computational Linguistics.
  * Rei et al. (2020a) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rei et al. (2020b) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b.  Unbabel’s participation in the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 911–920, Online. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Sennrich et al. (2017) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nădejde. 2017.  Nematus: a toolkit for neural machine translation.  In _Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 65–68, Valencia, Spain. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Minimum risk training for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1683–1692, Berlin, Germany. Association for Computational Linguistics.
  * Stanojević and Sima’an (2014) Miloš Stanojević and Khalil Sima’an. 2014.  Fitting sentence level translation evaluation with many dense features.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 202–206, Doha, Qatar. Association for Computational Linguistics.
  * Tromble et al. (2008) Roy Tromble, Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008.  Lattice Minimum Bayes-Risk decoding for statistical machine translation.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 620–629, Honolulu, Hawaii. Association for Computational Linguistics.
  * Uszkoreit and Lommel (2013) Hans Uszkoreit and Arle Lommel. 2013.  Multidimensional quality metrics: A new unified paradigm for human and machine translation quality assessment.  _Localization World, London_ , pages 12–14.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.

##  Appendix A Hyperparameters for COMET Retraining

We list all hyperparameters used for training retrain-comet-da and for
training and finetuning retrain-comet-mqm in Tables 4, 5 and 6. Each model was
trained on 1 NVIDIA Tesla V100 GPU.

Hyperparameter | Value
---|---
nr_frozen_epochs | 1
keep_embeddings_frozen | True
optimizer | Adam
encoder_learning_rate | 1.0e-05
learning_rate | 3.0e-05
layerwise_decay | 0.95
encoder_model | XLM-RoBERTa
pretrained_model | xlm-roberta-large
pool | avg
layer | mix
dropout | 0.1
batch_size | 2
accumulate_grad_batches | 8
hidden_sizes | 3072, 1536
load_weights_from_checkpoint | null
min_epochs | 2
max_epochs | 2
Table 4: Hyperparameters used to retrain wmt20-comet-da. Hyperparameter | Value
---|---
nr_frozen_epochs | 0.3
keep_embeddings_frozen | True
optimizer | AdamW
encoder_learning_rate | 1.0e-05
learning_rate | 3.1e-05
layerwise_decay | 0.95
encoder_model | XLM-RoBERTa
pretrained_model | xlm-roberta-large
pool | avg
layer | mix
dropout | 0.15
batch_size | 4
accumulate_grad_batches | 4
hidden_sizes | 3072, 1024
load_weights_from_checkpoint | null
min_epochs | 2
max_epochs | 2
Table 5: Hyperparameters used to retrain wmt21-comet-mqm on DA training data. Hyperparameter | Value
---|---
nr_frozen_epochs | 1
keep_embeddings_frozen | True
optimizer | AdamW
encoder_learning_rate | 1.0e-05
learning_rate | 1.0e-05
layerwise_decay | 0.95
encoder_model | XLM-RoBERTa
pretrained_model | xlm-roberta-large
pool | avg
layer | mix
dropout | 0.15
batch_size | 4
accumulate_grad_batches | 4
hidden_sizes | 3072, 1024
load_weights_from_checkpoint | last ckpt DA model
min_epochs | 1
max_epochs | 1
Table 6: Hyperparameters used to finetune wmt21-comet-mqm on MQM training
data.

| wmt20-comet-da | retrain-comet-da | Difference
---|---|---|---
en-cs | 0.978 | 0.981 |  $+$ 0.003
en-de | 0.972 | 0.971 |  $-$ 0.001
en-ja | 0.974 | 0.987 |  $+$ 0.013
en-pl | 0.981 | 0.983 |  $+$ 0.002
en-ru | 0.925 | 0.863 |  $-$ 0.062
en-ta | 0.944 | 0.948 |  $+$ 0.004
en-zh | 0.007 | 0.026 |  $+$ 0.019
en-iu | 0.860 | 0.861 |  $+$ 0.001
cs-en | 0.783 | 0.799 |  $+$ 0.016
de-en | 0.998 | 0.996 |  $-$ 0.002
ja-en | 0.964 | 0.966 |  $+$ 0.002
pl-en | 0.591 | 0.570 |  $-$ 0.021
ru-en | 0.923 | 0.924 |  $+$ 0.001
ta-en | 0.880 | 0.888 |  $+$ 0.008
zh-en | 0.952 | 0.952 |  $+$ 0.000
iu-en | 0.852 | 0.878 |  $+$ 0.026
km-en | 0.971 | 0.981 |  $+$ 0.010
ps-en | 0.941 | 0.951 |  $+$ 0.010
Table 7: Pearson correlation of to-and-from-English system-level COMET scores
with DA human assessments. Results with wmt20-comet-da corresponding “COMET”
in Tables 5 and 6 in Mathur et al. (2020).

##  Appendix B Correlation with Human Evaluators

We use our retrained retrain-comet-da model to score all systems that are part
of the WMT 2020 metrics shared task evaluation (Mathur et al., 2020).888We run
the run_ref_metrics.sh script provided at
https://drive.google.com/drive/folders/1n_alr6WFQZfw4dcAmyxow4V8FC67XD8p Then,
we use the official evaluation script999https://github.com/WMT-Metrics-
task/wmt20-metrics from the WMT 2020 shared task to compute the system-level
Pearson correlation for our retrained model. The results can be seen in Table
7. We also ensure that evaluation setup results in the same scores as in the
WMT 2020 publication (Mathur et al., 2020) when we use wmt20-comet-da instead
of retrain-comet-da to score the systems. For most language pairs, the two
models reach an almost identical correlation with human assessments.

##  Appendix C Automatic Analysis Results After Retraining

We repeat the automatic analysis for number and named entity error rates from
Section 5. The results can be seen in Table 8. Error rates that are reduced
are marked in green and error rates that increased after retraining are marked
in red. Retraining does not show clear benefits. For further discussion of the
results, see Section 7.

| Numbers | Named Entities
---|---|---
| de-en | en-de | de-en | en-de
reference |  06.78 |  |  06.54 |  | n/a |  | n/a |
alternative |  05.20 | -01.58 |  04.34 | -02.20 |  | 20.61 | 29.86 |
beam search |  04.10 | -02.68 |  04.30 | -02.24 | 26.05 | +05.44 | 37.12 | +07.26
MBR chrF++ |  08.78 | +02.00 |  06.61 | +00.07 | 31.84 | +11.23 | 41.45 | +11.59
MBR bleu |  06.13 | -00.65 |  08.64 | +02.10 | 31.01 | +10.40 | 42.93 | +13.07
MBR wmt20-comet-da |  09.66 | +02.88 | 10.90 | +04.36 | 32.88 | +12.27 | 49.90 | +20.04
MBR retrain-comet-da | 09.61 | +02.83 | 11.67 | +05.13 | 35.14 | +14.53 | 51.24 | +21.38
MBR wmt21-comet-mqm | 17.74 | +10.96 | 23.26 | +16.72 | 40.03 | +19.42 | 52.13 | +22.27
MBR retrain-comet-mqm | 15.77 | +08.99 | 22.20 | +15.66 | 38.33 | +17.72 | 52.82 | +22.96
Table 8: Results of the automatic evaluation. Error rates (%) for numbers and named entities and error rate changes compared to the reference for numbers and alternative translation for named entities. Error rates that have decreased after retraining COMET are marked in green, while error rates that have increased are marked in red. | Samples as Support | References as Support
---|---|---
| de-en | en-de | de-en | en-de
| wmt21-comet-mqm | \+ syn | wmt21-comet-mqm | \+ syn | wmt21-comet-mqm | \+ syn | wmt21-comet-mqm | \+ syn
numsub | 0.0006 | 0.0318 | 0.0005 | 0.0239 | 0.0011 | 0.0534 | 0.0009 | 0.0438
numadd | 0.0011 | 0.0616 | 0.0008 | 0.0505 | 0.0022 | 0.1084 | 0.0019 | 0.0946
NEdel | 0.0011 | 0.0399 | 0.0009 | 0.0353 | 0.0022 | 0.0809 | 0.0015 | 0.0640
numdel | 0.0012 | 0.0445 | 0.0009 | 0.0405 | 0.0022 | 0.0803 | 0.0017 | 0.0719
NEadd | 0.0014 | 0.0538 | 0.0013 | 0.0487 | 0.0028 | 0.1035 | 0.0023 | 0.0855
numwhole | 0.0016 | 0.0755 | 0.0011 | 0.0618 | 0.0029 | 0.1330 | 0.0024 | 0.1120
NEsub | 0.0014 | 0.0519 | 0.0012 | 0.0478 | 0.0031 | 0.1055 | 0.0024 | 0.0883
altern. | 0.0020 | 0.0820 | 0.0018 | 0.0977 | - | - | - | -
noundel | 0.0043 | 0.1469 | 0.0025 | 0.1082 | 0.0065 | 0.2284 | 0.0046 | 0.1910
nounadd | 0.0052 | 0.1690 | 0.0035 | 0.1334 | 0.0080 | 0.2697 | 0.0059 | 0.2212
nounsub | 0.0057 | 0.1761 | 0.0040 | 0.1449 | 0.0086 | 0.2807 | 0.0066 | 0.2437
copy | 0.0046 | 0.2075 | 0.0099 | 0.4912 | 0.0065 | 0.2782 | 0.0111 | 0.5001
hallucin. | 0.0190 | 0.6660 | 0.0211 | 0.7303 | 0.0310 | 1.0518 | 0.0305 | 1.0545
Table 9: Effects of randomly adding, substituting or deleting a digit in a
number or a letter in a noun or named entity (NE). Average absolute difference
to MBR score for reference (left) and 1-best beam search output (right) when
using wmt21-comet-mqm as the utility function. Reference points are marked in
bold. Changes when retraining COMET on added synthetic data are shown in
corresponding “+ syn” columns (retrain-comet-mqm). Green indicates the MBR
score difference to reference (left) and 1-best beam search output (right) has
increased, while red indicates the MBR score difference has decreased.

##  Appendix D Results with wmt21-comet-mqm

Table 9 shows the result of the sensitivity analysis with the wmt21-comet-mqm
and the retrain-comet-mqm models. These models show the same ordering of error
types as the models trained purely on Direct Assessment discussed in Section 6
and Section 7. For the retrained model, we see that perturbations to random
nouns still cause much larger differences in MBR score compared to
perturbations to numbers and named entities. However, we note that the
differences between individual scores are much smaller with wmt21-comet-mqm
than with our retrained retrain-comet-mqm model and with the other models.
This makes it harder to identify the effects of training on additional
synthetic data. We hypothesise that this difference in the scale and variance
of the scores could arise because we did not perform model averaging before
finetuning on the MQM data.