# Is Context Helpful for Chat Translation Evaluation?

Sweta Agrawal1, Amin Farajian2, Patrick Fernandes1,4,5
Ricardo Rei1,2,3, André F.T. Martins1,2,4
1Instituto de Telecomunicações, 2Unbabel, 3INESC-ID
4Instituto Superior Técnico & Universidade de Lisboa (Lisbon ELLIS Unit)
5Carnegie Mellon University
<EMAIL_ADDRESS>

Work partially developed during internship at Unbabel.

###### Abstract

Despite the recent success of automatic metrics for assessing translation
quality, their application in evaluating the quality of machine-translated
chats has been limited. Unlike more structured texts like news, chat
conversations are often unstructured, short, and heavily reliant on contextual
information. This poses questions about the reliability of existing sentence-
level metrics in this domain as well as the role of context in assessing the
translation quality. Motivated by this, we conduct a meta-evaluation of
existing sentence-level automatic metrics, primarily designed for structured
domains such as news, to assess the quality of machine-translated chats. We
find that reference-free metrics lag behind reference-based ones, especially
when evaluating translation quality in out-of-English settings. We then
investigate how incorporating conversational contextual information in these
metrics affects their performance. Our findings show that augmenting neural
learned metrics with contextual information helps improve correlation with
human judgments in the reference-free scenario and when evaluating
translations in out-of-English settings. Finally, we propose a new evaluation
metric, Context-MQM, that utilizes bilingual context with a large language
model (LLM) and further validate that adding context helps even for LLM-based
evaluation metrics.

##  1 Introduction

Automatically estimating the quality of machine or human-generated
translations has received a lot of attention over the past two decades from
the NLP community Han et al. (2021), specifically via shared tasks organized
by WMT since 2014-present Macháček and Bojar (2014); Stanojević et al. (2015);
Bojar et al. (2016, 2017); Ma et al. (2018, 2019); Mathur et al. (2020);
Freitag et al. (2021, 2022, 2023). A variety of evaluation metrics have been
developed for this purpose, encompassing lexical matching approaches such as
BLEU Papineni et al. (2002), chrF Popović (2015), and METEOR Lavie and Agarwal
(2007); embedding-based methods like BERTScore Zhang et al. (2019) and Word
Mover Distance Zhao et al. (2019); learned metrics like Comet Rei et al.
(2020a), and BLEURT Sellam et al. (2020); and metrics that employ prompting
techniques with large language models (LLMs) like Gemba-MQM Kocmi and
Federmann (2023a) or AutoMQM Fernandes et al. (2023a).

Among these metrics, neural learned metrics have gained widespread acceptance
Freitag et al. (2022) as they are directly trained to predict sentence-level
translation quality assessment scores Kreutzer et al. (2015); Rei et al.
(2020a); Sellam et al. (2020), word-level error annotations collected by
professional linguists Guerreiro et al. (2023) or post-editing efforts as
measured by HTER Snover et al. (2006); Fonseca et al. (2019); Specia et al.
(2021). However, the reliance on human-written reference translations and
judgments collected predominantly from structured domains like news or
Wikipedia as training data raises questions about their adaptability and
reliability in detecting errors in other domains Zouhar et al. (2024), for
example in evaluating the quality of translations in more informal settings.

Unlike news articles which involve carefully authored and well-formatted text,
which current translation systems are well equipped for, chat conversations
are often synchronous and short, and involve formal language, colloquial
expressions, and slang that may not have direct equivalents in the target
language Gonçalves et al. (2022). An example of such a conversation is
presented in Figure 1. Although chat messages are generally brief and easier
to translate, errors introduced by machine translation (MT) systems might go
unnoticed by end users if not detected properly, potentially leading to
miscommunication, conversation breakdowns, or even more serious implications
on high-risk domains (e.g. patient-physician chats) Yamashita et al. (2009);
Robertson and Díaz (2022).

Figure 1: An example bilingual conversation from the MAIA corpus Martins et
al. (2020): the agent and the customer only see the texts in English and
Portuguese, respectively. The errors (both MT and user-generated) are bold-
faced.

Moreover, conversational texts rely heavily on context, meaning that the
interpretation of a text is largely influenced by the surrounding contextual
information. Hence, MT systems for such domains are often trained with
contextual information and this has been shown to improve translation quality,
lexical inconsistency, and coherence of the generated outputs Farinha et al.
(2022); Fernandes et al. (2021). However, it is unclear to what extent context
plays a role in estimating translation quality for machine-translated
conversations.

In this work, we first systematically analyze the nature and the frequency of
the errors in real bilingual chat translations from customer support and
contrast them with the structured news domain (§ 2). We find that translation
errors are 21% less frequent in chat relative to the news domain and that the
nature of the errors introduced by MT systems in the two domains is also
different. This underscores the importance of understanding and evaluating
existing automatic metrics for chat translation. Due to the infrequent error
occurrences at the sentence level, MT systems for chat translation tasks might
receive higher scores, necessitating a more nuanced evaluation of the metrics.

Motivated by the above, we present a meta-evaluation of existing sentence-
level automatic metrics, primarily tested on news translation tasks, in their
ability to gauge the quality of machine-translated chats (§ 3). We evaluate
these metrics across two scenarios: on all translation pairs as well as
imperfect translations as judged by humans. We then study the impact of
augmenting a subset of these learned neural metrics with different types of
conversational contextual information (§ 4). Our findings are summarized
below:

  * •

Comet-22, a reference-based metric, achieves the highest overall correlation
with human judgments.

  * •

Reference-based Comet-22 does not benefit from the added contextual
information, whereas, reference-free Comet-20-QE has better correlations with
human judgments as the context window increases when evaluating translations
in out-of-English settings. This is useful as reference-free metrics allow
translation quality to be assessed on the fly.

  * •

Adding context helps assess translation quality better for short and ambiguous
sentences. Using correct and complete context from both speakers is essential
for improving chat quality estimation via neural learned metrics.

Finally, we present Context-MQM, an LLM-based evaluation metric that utilizes
context for chat translation quality estimation (§ 5.3). Our preliminary
experiments with Context-MQM show that adding bilingual context to the
evaluation prompt helps improve the quality assessment of machine-translated
chats on imperfect translations.

##  2 Errors in Chat vs. News: A Case Study

To better understand how the error types differ in these domains, we present
an analysis of the nature and the frequency of errors using Multidimensional
Quality Metrics (MQM) annotations (conversation: 7120, news: 4800 translation
pairs) collected for English-German as a part of the WMT22 Metrics shared task
Freitag et al. (2022).

Figure 2: Conversational texts tend to be much shorter relative to news texts
in the WMT22 English-German dataset.

##### Errors are less frequent.

We calculate the percentage of translations with a perfect MQM score from the
news and conversational subset of the WMT22 dataset: only 46.4% of news
translations have perfect MQM scores, whereas this percentage is 57.8% for the
conversational domain. This suggests that errors are less frequent in the
conversational domain compared to the news, likely due to the relatively short
(see Figure 2) and probably less complex text dominant in conversations.
However, it is important to note that these errors do not occur in isolation
and have the potential to escalate into larger issues, leading to significant
breakdowns in communication. Moreover, as the errors are less frequent, the
true quality of the model-generated translations could be misrepresented if
they are not accurately identified by quality estimation systems.

##### Most frequent error types across domains are different.

Errors related to fluency, such as issues with spelling, consistency, and
register, occur more frequently in conversations compared to accuracy-based
errors like mistranslation, which are more common in the news domain (Figure
3). This underscores the idea that the nature of observed errors is also
influenced by the specific domain context.

Figure 3: Counts of MQM error categories normalized by the number of annotated
instances for each domain: frequent errors differ in the two domains.

The variation in error types and their frequencies across the two domains
underscores the need for a systematic study of the suitability of current
automatic metrics within the chat domain.

##  3 Meta Evaluation of Automatic Metrics

We now turn towards investigating to what extent automatic sentence-level MT
evaluation metrics frequently used in the literature capture the translation
quality of conversational data. We detail the setup used for our meta-
evaluation below:

LP | Sender | Count | Length | CUA | % Perfect MQM
---|---|---|---|---|---
# (Instances) | # (Chats) | # (Systems) | Source | MT | Ref | Weak | Moderate | Good | Excellent
En-De | Agent | 2715 | 30 | 5 | 48.82 | 57.34 | 57.20 | 89 | 90 | 120 | 2219 | 73.6
De-En | Customer | 2720 |  |  | 38.14 | 34.46 | 35.49 | 113 | 88 | 166 | 2067 | 65.4
En-Fr | Agent | 1868 | 23 | 2 | 36.26 | 42.81 | 44.21 | 100 | 118 | 129 | 1274 | 46.7
Fr-En | Customer | 1020 |  |  | 35.72 | 32.36 | 33.19 | 45 | 37 | 41 | 748 | 65.7
En-Pt | Agent | 1318 | 28 | 2 | 43.21 | 46.34 | 45.96 | 77 | 81 | 103 | 872 | 50.1
Pt-En | Customer | 1016 |  |  | 30.44 | 31.06 | 31.35 | 40 | 68 | 66 | 624 | 54.5

Table 1: Statistics extracted from the human annotations of the WMT 22 Chat
Shared Task: Errors are less frequent in the chat domain with 46.7% to 73.6%
translations receiving perfect MQM scores.

###  3.1 Dataset

We use the MQM annotations collected from the WMT 2022 Chat Shared Task
Farinha et al. (2022). The dataset consists of genuine bilingual customer
support conversations which were translated by participants’ submitted
automatic MT systems. The outputs from the submitted MT systems were then
evaluated via an MQM-based human evaluation that is tailored to assess
customer support content. The evaluation was conducted by Unbabel’s in-house
professional linguists and translators using the whole conversational context.

We convert token-level MQM spans into a sentence-level score via the following
formula:

$\mathrm{MQM}=-(C_{\mathrm{Min}}+5\times C_{\mathrm{Maj}}+10\times
C_{\mathrm{Cri}})$ (1)

where, $C_{\mathrm{Min}}$, $C_{\mathrm{Maj}}$ and $C_{\mathrm{Cri}}$ denote
the number of minor, major, and critical errors respectively Lommel et al.
(2014); Farinha et al. (2022).

Table 1 shows several statistics extracted from the data.111Note that the
Agent communicates always in English and the Customer in non-English
languages. Across different language pairs, the percentage of instances with
no errors, i.e., % Perfect MQM ranges from 46.7 to 73.6%, which further
confirms our initial analysis that errors are less frequent in translated
chats. Besides the % perfect MQM, following Farinha et al. (2022), we also
present Customer Utility Analysis (CUA) which buckets MQM scores in four
regions: Weak (negative - 39 MQM); Moderate (40 - 59 MQM); Good (60 - 79 MQM)
and Excellent (80 - 100 MQM). From the table, it is apparent that, overall,
translating agent directions tends to be easier compared to translating
customer directions as indicated by the higher counts in the “good $+$
excellent” categories, a finding highlighted by Farinha et al. (2022).

###  3.2 Metrics

We benchmark several metrics spanning lexical, embedding-based, and neural
metrics frequently used for translation evaluation:

##### BLEU

Papineni et al. (2002) estimates the translation quality based on n-gram
overlap between the hypotheses and references. We compute sentence-level BLEU
Chen and Cherry (2014) using the Sacrebleu Post (2018)
library.222https://github.com/mjpost/sacrebleu/:
nrefs:1$|$case:mixed$|$eff:no$|$tok:13a$|$smooth:exp$|$version:2.4.0

##### chrF

Popović (2015) evaluates the similarity by computing an F1 score between the
overlapping character n-grams in the hypotheses and references.

##### BERTScore

Zhang et al. (2019) uses the cosine similarity between the pre-trained
contextualized embeddings to measure the similarity between the hypotheses and
references.

##### BLEURT

Sellam et al. (2020) utilizes pre-trained transformer models to estimate a
quality score that reflects the semantic similarity between the hypothesis and
the reference. BLEURT is a neural regression metric trained on an existing
collection of human judgments.

##### Comet-22

Rei et al. (2022a) is an XLM-R-based Conneau et al. (2020) regression model
trained on direct assessments from WMT17 to WMT20 and provides scores ranging
from 0 to 1. Unlike BLEURT which only utilizes the reference and the
hypothesis texts, COMET uses the triplet (source, hypothesis, reference) to
return a score that reflects the quality of the hypothesis relative to both
source and reference.

##### Comet-20-QE

Rei et al. (2020b) is also an XLM-R-based regression metric trained to predict
the direct assessment scores using representations extracted from the (source,
hypothesis) pair.

##### CometKiwi-22-QE

Rei et al. (2022b) is a reference-free model built on top of the InfoXLM-R
model Chi et al. (2021) and is trained to predict direct assessments from
WMT17-20 and the MLQE-PE corpus Fomicheva et al. (2022). It generates scores
ranging from 0 to 1.

##### xComet-XL

Guerreiro et al. (2023), with 3.5B parameters, is an explainable learned
metric trained to predict both sentence-level quality scores and MQM-like
error spans from the (source, hypothesis, reference) triplet. Furthermore, the
metric can also be used for quality estimation by using only the source and
the hypothesis as input, referred to as xComet-QE-XL.

##### MetricX-23-XL

Juraska et al. (2023) is a regression-based metric built on top of mT5 Xue et
al. (2021) and is trained to regress the true MQM score to predict an error
score in the range [0, 25]. The input to the model is the concatenated
hypothesis and reference translations.

##### MetricX-23-QE-XL

Juraska et al. (2023) is a reference-free version of MetricX-23-XL that
instead takes as input the source and the hypothesis.

Following the WMT QE shared task evaluation Blain et al. (2023), we report
Spearman-rank correlation Zar (2005) to assess how well the automatic metric
scores correlate with human judgments.333While the QE shared task uses
Spearman rank, the Metrics shared task employs both Pearson and Kendall Tau.
In this work, we opted for Spearman since it serves as a compromise between
Pearson and Kendall Deutsch et al. (2023).

| Metric | Average | En-De | En-Fr | En-Pt
---|---|---|---|---|---
| All | Imperfect | All | Imperfect | All | Imperfect | All | Imperfect
Ref-based | chrF | 0.531 | 0.404 | 0.430 | 0.253 | 0.666 | 0.510 | 0.496 | 0.448
BLEU | 0.500 | 0.319 | 0.363 | 0.151 | 0.642 | 0.426 | 0.494 | 0.381
BERTScore | 0.545 | 0.403 | 0.439 | 0.250 | 0.680 | 0.501 | 0.516 | 0.457
BLEURT | 0.372 | 0.511 | 0.398 | 0.411 | 0.326 | 0.663 | 0.393 | 0.458
Comet-22 | 0.633 | 0.551 | 0.578 | 0.445 | 0.721 | 0.634 | 0.602 | 0.573
xComet-XL | 0.402 | 0.525 | 0.402 | 0.432 | 0.381 | 0.649 | 0.422 | 0.494
MetricX-23-XL | 0.622 | 0.543 | 0.535 | 0.404 | 0.723 | 0.692 | 0.609 | 0.535
Ref-Free | Comet-20-QE | 0.379 | 0.316 | 0.410 | 0.294 | 0.376 | 0.339 | 0.351 | 0.315
CometKiwi-22-QE | 0.300 | 0.416 | 0.366 | 0.416 | 0.203 | 0.508 | 0.331 | 0.325
xComet-QE-XL | 0.276 | 0.363 | 0.349 | 0.376 | 0.160 | 0.383 | 0.318 | 0.329
MetricX-23-QE-XL | 0.447 | 0.402 | 0.438 | 0.371 | 0.437 | 0.479 | 0.466 | 0.356

Table 2: Comet-22 achieves the highest correlation on average across all Agent
language pairs with MetricX-23-XL as a close competitor in the Ref-Based
setup. MetricX-23-QE-XL outperforms Comet alternatives in the ref-Free
setting.

| Metric | Average | De-En | Fr-En | Pt-En
---|---|---|---|---|---
| All | Imperfect | All | Imperfect | All | Imperfect | All | Imperfect
Ref-Based | chrF | 0.427 | 0.188 | 0.400 | 0.201 | 0.411 | 0.157 | 0.469 | 0.205
BLEU | 0.396 | 0.166 | 0.390 | 0.154 | 0.373 | 0.147 | 0.425 | 0.198
BERTScore | 0.484 | 0.280 | 0.445 | 0.239 | 0.467 | 0.332 | 0.539 | 0.269
BLEURT | 0.559 | 0.451 | 0.540 | 0.445 | 0.520 | 0.464 | 0.617 | 0.444
Comet-22 | 0.610 | 0.517 | 0.580 | 0.438 | 0.588 | 0.535 | 0.661 | 0.578
xCOMET-XL | 0.454 | 0.566 | 0.357 | 0.514 | 0.479 | 0.588 | 0.527 | 0.594
MetricX-23-XL | 0.608 | 0.551 | 0.589 | 0.511 | 0.583 | 0.544 | 0.651 | 0.598
Ref-Free | Comet-20-QE | 0.516 | 0.415 | 0.554 | 0.381 | 0.471 | 0.429 | 0.523 | 0.435
CometKiwi-22-QE | 0.438 | 0.443 | 0.385 | 0.463 | 0.456 | 0.406 | 0.473 | 0.461
xCOMET-QE-XL | 0.447 | 0.493 | 0.388 | 0.492 | 0.462 | 0.479 | 0.492 | 0.508
MetricX-23-QE-XL | 0.395 | 0.497 | 0.383 | 0.490 | 0.382 | 0.431 | 0.420 | 0.569

Table 3: On average, on the Customer language pairs, Comet-22 and
MetricX-23-XL achieve the highest correlation scores and neural learned
metrics consistently outperform lexical metrics.

##  4 Context-Aware Translation Evaluation

Let Agent (A) and Customer (C) represent the two participants in a bilingual
chat, where A and C communicate in languages $l_{a}$ and $l_{c}$ respectively.
Given a text $x$ generated by A or C, the goal is to predict the quality of
its translation, $\hat{y}$. We extend existing sentence-level automatic
metrics to utilize conversational context as detailed below:

##### Incorporating Context

Following Vernikos et al. (2022), for a neural learned metric like Comet, we
obtain the contextual representations of the current source, $x$, and the
reference ($y$)/hypothesis ($\hat{y}$) sentences by prepending up to $k$
sentences of context preceding to it. However, only the representations of the
current instance are pooled before passing them to the regressor module that
generates the quality score. This allows the learned metric to be informed by
the surrounding context when evaluating the translation quality of the current
instance. This simple approach was used to evaluate document-level translation
quality and was shown to be more effective over sentence-level counterparts
Vernikos et al. (2022), whereas, in our work, we study and extend its
applicability to contextualized sentence-level chat quality estimation. We
emphasize that our goal is not to evaluate translation quality at the
conversation level but rather to utilize context to inform the sentence-level
assessment, which renders document-level metrics that aggregate scores over
sentences like SliDe Raunak et al. (2023) not applicable to our use case.

##### Choice of Context

We explore the usage of two types of contextual information for translation
quality estimation: within and across participants. The current text, assuming
it is generated by Customer C, can be preceded by the context from previous
turns by the same participant (within), or the context from both participants
(across). For example, for the source text generated by Customer C at time
$t=7$ in Figure 1, the two preceding contextual sentences ($k=2$) for both
settings are shown:

> Context (within):
> $t=2$
> Original: como faço para cancelar ?
> Translation: How do I cancel?
> $t=5$
> Original: esse link não abre
> Translation: This link does not open.
>
>
> Context (across):
> $t=5$
> Original: esse link não abre
> Translation: This link does not open.
> $t=6$
> Original: No worries, I can cancel it here in my ned.
> Translation: Não se preocupe, eu posso cancelar aqui no meu blog.

Adding context in the within setting is straightforward, we prepend up to $k$
sentences of source and translation context to the current source ($x$) and
the hypothesis ($\hat{y}$)/reference ($y$) from the same participant separated
by a tag, $<$sep$>$. For the across setting, to have the context in the same
language on each side, we also prepend the translated context to the source
($x$) and the source context to the hypothesis ($\hat{y}$)/reference ($y$)
from the other participant. Using our proposed extensions of the sentence-
level metrics, we study how they impact the evaluation of translation quality.

##  5 Results

We first present the results of the meta-evaluation of existing automatic
sentence-level MT metrics. Then we show the impact of adding context to a
subset of these metrics with ablations on how context impacts translation
quality evaluation. Finally, we present a preliminary study on utilizing
context with LLM-based MT evaluation.

###  5.1 Meta Evaluation

Tables 2 and 3 show the correlation of human judgments with automatic metrics
for all the Agent and the Customer language pairs respectively. “Imperfect”
translations are instances marked with an MQM score of $<$0 by humans.

Overall, in the reference-based setup, when considering all the instances in
the corpus (“All”), Comet-22 achieves the highest Spearman Correlation on both
settings (Agent and Customer), outperforming all other metrics, with
MetricX-23-XL as a close competitor. For reference-free evaluation,
MetricX-23-QE-XL and Comet-20-QE achieve the highest correlation with human
judgments when evaluating translations out of English and into English
respectively. However, there is a big gap between the best-performing metric
for the reference-free and reference-based evaluation via learned metrics on
average across all Agent ($\delta$(Comet-22,MetricX-23-QE-XL): $0.186$) and
Customer ($\delta$(Comet-22,Comet-20-QE): $0.094$) language pairs in the chat
domain.

When considering the imperfect translations, there is no clear winner: most
metrics suffer a drop in correlation for this subset compared to “All”
translations except xComet-XL and xComet-QE-XL. xComet-XL consistently
achieves better correlations on this subset than “ALL” data. We hypothesize
that this could be due to an over-prediction of errors via the metric for
chats.

(a) Into-English (Customer)

(b) Out-of-English (Agent)

Figure 4:  Impact of varying context window and context type (across/within)
on average correlation across Agent and Customer settings: adding complete
context (across) helps improve metrics performance in out-of-English
reference-free settings (Agent) but is detrimental for into-English (Customer)
evaluation.

Figure 5: Context helps the most in improving the translation quality
estimation of shorter (source character length $\leq$ 20) and potentially
ambiguous sentences (averaged over “all” Agent language pairs).

Neural reference-based metrics consistently outperform lexical metrics in most
settings, specifically when evaluating translations into English. However,
when assessing the translation quality in out-of-English language pairs
(Agent), reference-based lexical metrics achieve better correlations with
human judgments than neural learned metrics, suggesting room for improvement
for reference-free evaluation for assessing translations in languages other
than English.

###  5.2 Context-Aware Translation Evaluation

We consider the context-aware extensions of two metrics: reference-based
Comet-22 and reference-free Comet-20-QE. For each of these metrics, we study
the impact of adding contextual information as detailed in § 4 in both within
or across participants settings. For reference-free Comet-20-QE, we
additionally consider the setup where we use the machine-translated hypothesis
instead of the reference as context. This is to mimic the real-word chat
scenario where references are generally unavailable. We hypothesize that noisy
context can still provide useful information in estimating the quality of the
current (source, translation) pair.

####  5.2.1 Main Results

Figure 4 shows results of adding up to last nine sentences as context to the
above configurations averaged across customer (“Average Customer”) and agent
(“Average Agent”) language pairs.

##### Context is not helpful when references are available.

Reference-based Comet-22 on average across all language pairs does not benefit
(Agent) or hurts (Customer) correlation with the added context information.
This could be because most of the necessary information to resolve any
ambiguity is ideally already included in the reference. Adding more context
could potentially introduce ambiguity or inconsistency that is not present in
the reference text, hurting the evaluation process.

##### Adding context is detrimental when evaluating translation into English.

In all settings for customer directions, adding context almost always hurts
the metric’s correlation with human judgments. Even in the reference-free
scenario, translation evaluation into English does not benefit from the added
context. We attribute this to the limited contextual phenomena observed in
English compared to other languages (PT, FR, DE).

##### Adding context improves correlation for Comet-20-QE in reference-free
out-of-English settings.

Reference-free Comet-20-QE significantly benefits from the added context on
average across all “Agent” settings. The correlation increases as the context
increases. Specifically, using the complete contextual information from both
participants (across) is key to getting the most out of the added contextual
information. Shorter segments ($\leq 20$ characters) benefit the most from the
added context as depicted in Figure 5. The above trend holds when using either
reference-based or hypothesis-based contexts, which is promising.

####  5.2.2 Ablation Analysis

We use the across participant setting with a context window of 2 for
Comet-20-QE, as this setup led to the most improvement in correlation. In
addition, we use the hypothesis as the context instead of the reference for
all the analysis, mimicking the real-world scenario where references are
unavailable. With this setup, we first study whether adding context improves
correlation for specific error types and severity levels. We then evaluate the
impact of adding noise to the context with the proposition that unrelated or
partial context should hurt the metric’s performance.

Figure 6: Adding context improves correlation with human judgments across most
error types (except Date and Time Format) and all severity levels.

##### Breakdown by error types and severity

We bucket sentences by the MQM error typology and filter error types with at
least 20 instances. Figure 6 shows the correlation with human judgments for
each error type bucket and across severity levels. Including contextual
information improves correlation with human judgments on several error types
as well as all error severity levels. However, adding context hurts Spelling,
Date, and Time Format errors, possibly because adding context might introduce
noise and not provide any useful information to identify such errors which are
solely based on linguistic or formatting rules.

##### Impact of Noisy or Partial Context

To validate that complete and correct context is necessary for meaningful
improvement in metrics’ performance, we inject two types of noise into the
context: Swap, where we use unrelated context from a different instance; and
Drop where we drop one of the two (source, translation) pairs (“pair”) or
unpaired sentences from the preceding context (“random”). We additionally
consider injecting these noises into either the source, the translation, or
both.

| Noise to? | Avg-Agent
---|---|---
No Context | - | 0.379
Context | - | 0.420
Swap | Source | 0.364
| Translation | 0.296
| Both | 0.299
Drop (Random) | Source | 0.402
| Translation | 0.326
| Both | 0.346
Drop (Pair) | Source | 0.389
| Translation | 0.399
| Both | 0.325

Table 4: Corrupting context hurts correlation.

Table 4 shows that adding either kind of noise to the context leads to a drop
in correlation relative to the “Context” baseline, often even performing worse
than using any contextual information (“No Context”). Unrelated context
(“Swap”) has a more adverse impact on the metric’s performance compared to
changing the context via dropping partial information. Furthermore, dropping
paired contextual sentences results in a larger drop in correlation than
dropping unrelated source-translation instances. This further solidifies our
argument that complete and related context is key to utilizing context for
chat translation quality estimation.

Figure 7: Adding context improves correlation on contextually ambiguous
sentences (character length $\leq$ 20) for English-German.

##### Impact on Contextually Ambiguous Sentences

Given that shorter sentences benefit the most from the added context (Figure
5), we further investigate whether this is indeed due to the increased
ambiguity in texts. We use MuDa Fernandes et al. (2023b) to identify
translation pairs with specific discourse phenomena (formality, pronouns, verb
form, lexical consistency) for English-German. This enables us to mark
instances that potentially require context for disambiguation. Figure 7 shows
that the sentences with marked discourse phenomena benefit the most from the
added context. On the other hand, adding context to sentences without marked
discourse phenomena hurts correlation. We leave the detailed exploration of
when to rely on context for quality estimation to future work.

System: You are an annotator for the quality of machine translation. Your task
is to identify errors and assess the quality of the translation.
---
The categories of errors are: accuracy (addition, mistranslation, omission,
untranslated text), fluency (character encoding, grammar, inconsistency,
punctuation, register, spelling), style (awkward), terminology (inappropriate
for context, inconsistent use), non-translation, other, source error or no-
error.
Each error is classified as one of three categories: critical, major, and
minor. Critical errors inhibit comprehension of the text. Major errors disrupt
the flow, but what the text is trying to say is still understandable. Minor
errors are technically errors, but do not disrupt the flow or hinder
comprehension.
{$k$ Few Shot Incontext Examples}
User: Context: “‘{context}”’
{sender} source (source_lang): “‘{source_seg}”’
{target_lang} translation: “‘{target_seg}”’
Based on the conversation context between the agent and the customer, the
current source by "sender" in {source_lang} and its machine translation in
{target_lang} surrounded with triple backticks, identify error types in the
translation and classify them.
Figure 8:  Contextual Prompt for Chat Quality Estimation.

###  5.3 LLM-based Contextual Quality Estimation

Motivated by the recent surge of interest in using LLMs like GPT OpenAI et al.
(2023) for text evaluation, specifically in MT Kocmi and Federmann (2023b);
Fernandes et al. (2023a); Lu et al. (2023); Kocmi and Federmann (2023a), we
also explore their potential in assessing the translation quality of machine
translated chats. To better elicit the reasoning and in-context learning
capabilities of these LLMs, we prompt GPT-4444gpt-4-0613, accessed on February
26, 2024. to identify and categorize errors in machine-generated translations
instead of asking for an overall score for translation quality like direct
assessment.555Our initial experiments with open-sourced LLMs suggested limited
ability of the models to do well on this task.

We present Context-MQM, a context-informed LLM-based quality estimation metric
for chat translation evaluation. We adopt the MQM-style prompting techniques
to elicit the reasoning capabilities of the LLM Fernandes et al. (2023a);
Kocmi and Federmann (2023a) and modify it to utilize contextual information:
following Section 5.2, we include the past $k=8$ bilingual source sentences as
context and one in-domain in-context example as shown in Figure 8. We perform
our evaluation on a subset of 1000 English-German sentences sampled uniformly
from the dataset and contrast our metric with the evaluation prompting
technique that does not utilize any contextual information, LLM-MQM (No
Context).666We did not conduct a full-scale evaluation due to the high cost of
accessing the GPT-4 API. The cost for running the experiments on English-
German was $\sim$ $300.

The results are presented in Table 5: adding context positively impacts
correlation for LLM-based evaluation of machine-translated chats. Context-MQM
improves correlation with human judgments, outperforming both non-contextual
LLM-MQM (All: +0.013, Imperfect: 0.048) as well as Comet-22 (All: +0.091,
Imperfect: +0.107). The improvement is larger on the imperfect translations,
suggesting that context helps identify errors better on these (source,
translation) pairs. These initial results show the potential of utilizing LLMs
for evaluating chat translation quality with contextual information.
Additionally, exploring alternative prompting strategies to integrate context
across diverse language pairs and LLMs merits further investigation in future
research endeavors.

| All | Imperfect
---|---|---
LLM-MQM (No Context) | 0.642 | 0.512
Context-MQM | 0.655 | 0.560
Comet-22 | 0.564 | 0.453

Table 5: Context-MQM outperforms Comet-22 and non-contextual LLM-MQM on
English-German Chat Quality Estimation.

##  6 Related Work

##### Automatic MT Metrics

Designing automatic metrics to assess translation quality has been an active
area of research over the past decade. Metrics shared tasks organized at WMT
have significantly facilitated research where recent metrics like BLEURT
Sellam et al. (2020) or COMET Rei et al. (2020b) based on neural architectures
and trained with human assessments are shown to consistently outperform
lexical metrics. However, these metrics are primarily evaluated by assessing
the quality of isolated translations, primarily for news-like data. Recent
work, however, has focused on developing document-level evaluation metrics
acknowledging that sentences often do not occur in isolation in the wild and
the correctness of translation is dependent on the context Voita et al.
(2019). Document-level metrics like SliDe Raunak et al. (2023) or BlonDe Jiang
et al. (2022) use discourse information to assess the translation quality at
the paragraph level. In this work, however, our goal is still to assess the
translation quality of individual sentences but inform the quality evaluation
with the available contextual information.

##### Chat Translation Quality Estimation

Li et al. (2022) introduce the erroneous chat translation detection task and
propose an error detection model that classifies a given translation in a
bilingual two-utterance chat as either correct or erroneous. However, their
approach requires training a detection model tailored explicitly to chat-
related data, whereas, in this work we benchmark existing metrics for general
quality estimation and study the impact of conversation context on quality
estimation via existing metrics.

Menezes et al. (2023) propose a new framework for identifying contextual
errors in conversational datasets. They expand the MQM categories to account
for errors introduced due to contextual triggers. They further show that these
errors are indeed critical and that current metrics fall short in detecting
them. Our evaluation instead targets estimating chat translation quality
regardless of the specific errors and is aimed at providing a more general
view of the ability of existing metrics to assess the quality of machine-
translated chats.

##### Contextual Machine Translation

In many scenarios, translation requires leveraging information beyond the
sentence level to resolve inter-sentence dependencies and improve translation
quality. Incorporating context to generate high-quality translations has been
explored for conversation and news documents, with approaches ranging from
simply concatenating the context to the original input Tiedemann and Scherrer
(2017) to more complex options Jean et al. (2017); Maruf et al. (2018, 2019).
However, despite having access to context, contextual MT models still struggle
to effectively use it Fernandes et al. (2021), and most MT metrics fail to
capture this due to a lack of utilization of context in the evaluation metrics
themselves. Hence, we instead use context to assess the translation quality of
a given (source, target) pair and show that it benefits the evaluation of non-
English translations.

##  7 Conclusion and Discussion

Estimating translation quality in diverse domains is crucial to ensure that
the metrics employed in MT evaluation accurately reflect the MT system’s
quality across various types of content. We show that the nature and the type
of errors in the conversational context are different from the generally
evaluated news domain. Hence, designing robust metrics that can capture these
errors is very important. Our work presents a step in that direction by
systematically benchmarking existing automatic MT metrics on machine-
translated chats. Given the highly contextual nature of the chat domain, we
extend and evaluate context-based reference-free and reference-based metrics
and show that context can be extremely helpful in judging translation quality,
especially in a reference-free setup. Our analysis sheds some light on how and
when the context can be helpful. However, there remain several open questions
and directions for future work.

##### Improving Detection of MT errors

As illustrated by our results, although Comet-22 achieves high correlations
with human judgments overall, there is a drop in correlation for imperfect
segments, suggesting a need for designing a metric that can do well at
estimating quality for both perfect and imperfect translations.

##### Better Reference-free Evaluation

Our findings show that reference-free learned metrics lag behind reference-
based ones in evaluating the translation quality of bilingual chats. This
presents opportunities to develop effective evaluation methods in the absence
of reference translations.

##### Optimizing Context Utilization

We implemented a very simple approach to utilize context from both
participants in both learned metrics as well as for LLM-based MT evaluation.
However, it remains to be investigated how context can be utilized in other
ways and when the metric should rely on the contextual information.

## Acknowledgments

We thank Ben Peters, António Farinhas, and Duarte Alves for their useful and
constructive comments. This work was supported by the Portuguese Recovery and
Resilience Plan through project C645008882-00000055 (Center for Responsible
AI), by the EU’s Horizon Europe Research and Innovation Actions (UTTER,
contract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763), and by
Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020.

## References

  * Blain et al. (2023) Frederic Blain, Chrysoula Zerva, Ricardo Ribeiro, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins. 2023.  Findings of the WMT 2023 shared task on quality estimation.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 629–653, Singapore. Association for Computational Linguistics.
  * Bojar et al. (2017) Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017.  Results of the WMT17 metrics shared task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics.
  * Bojar et al. (2016) Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016.  Results of the WMT16 metrics shared task.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 199–231, Berlin, Germany. Association for Computational Linguistics.
  * Chen and Cherry (2014) Boxing Chen and Colin Cherry. 2014.  A systematic comparison of smoothing techniques for sentence-level BLEU.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 362–367, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021.  InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3576–3588, Online. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Deutsch et al. (2023) Daniel Deutsch, George Foster, and Markus Freitag. 2023.  Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12914–12929, Singapore. Association for Computational Linguistics.
  * Farinha et al. (2022) Ana C Farinha, M. Amin Farajian, Marianna Buchicchio, Patrick Fernandes, José G. C. de Souza, Helena Moniz, and André F. T. Martins. 2022.  Findings of the WMT 2022 shared task on chat translation.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 724–743, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Fernandes et al. (2023a) Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023a.  The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 1066–1083, Singapore. Association for Computational Linguistics.
  * Fernandes et al. (2023b) Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023b.  When does translation require context? a data-driven, multilingual exploration.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 606–626, Toronto, Canada. Association for Computational Linguistics.
  * Fernandes et al. (2021) Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021.  Measuring and increasing context usage in context-aware machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6467–6478, Online. Association for Computational Linguistics.
  * Fomicheva et al. (2022) Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022.  MLQE-PE: A multilingual quality estimation and post-editing dataset.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4963–4974, Marseille, France. European Language Resources Association.
  * Fonseca et al. (2019) Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019.  Findings of the WMT 2019 shared tasks on quality estimation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 1–10, Florence, Italy. Association for Computational Linguistics.
  * Freitag et al. (2023) Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023.  Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 578–628, Singapore. Association for Computational Linguistics.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Freitag et al. (2021) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Gonçalves et al. (2022) Madalena Gonçalves, Marianna Buchicchio, Craig Stewart, Helena Moniz, and Alon Lavie. 2022.  Agent and user-generated content and its impact on customer support MT.  In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_ , pages 201–210, Ghent, Belgium. European Association for Machine Translation.
  * Guerreiro et al. (2023) Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2023.  xcomet: Transparent machine translation evaluation through fine-grained error detection.  _arXiv preprint arXiv:2310.10482_.
  * Han et al. (2021) Lifeng Han, Alan Smeaton, and Gareth Jones. 2021.  Translation quality assessment: A brief survey on manual and automatic methods.  In _Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age_ , pages 15–33, online. Association for Computational Linguistics.
  * Jean et al. (2017) Sébastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017.  Does neural machine translation benefit from larger context? _ArXiv_ , abs/1704.05135.
  * Jiang et al. (2022) Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022.  BlonDe: An automatic evaluation metric for document-level machine translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1550–1565, Seattle, United States. Association for Computational Linguistics.
  * Juraska et al. (2023) Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023.  MetricX-23: The Google submission to the WMT 2023 metrics shared task.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 756–767, Singapore. Association for Computational Linguistics.
  * Kocmi and Federmann (2023a) Tom Kocmi and Christian Federmann. 2023a.  GEMBA-MQM: Detecting translation quality error spans with GPT-4.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 768–775, Singapore. Association for Computational Linguistics.
  * Kocmi and Federmann (2023b) Tom Kocmi and Christian Federmann. 2023b.  Large language models are state-of-the-art evaluators of translation quality.  In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_ , pages 193–203, Tampere, Finland. European Association for Machine Translation.
  * Kreutzer et al. (2015) Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015.  QUality estimation from ScraTCH (QUETCH): Deep learning for word-level translation quality estimation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 316–322, Lisbon, Portugal. Association for Computational Linguistics.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 228–231, Prague, Czech Republic. Association for Computational Linguistics.
  * Li et al. (2022) Yunmeng Li, Jun Suzuki, Makoto Morishita, Kaori Abe, Ryoko Tokuhisa, Ana Brassard, and Kentaro Inui. 2022.  Chat translation error detection for assisting cross-lingual communications.  In _Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems_ , pages 88–95, Online. Association for Computational Linguistics.
  * Lommel et al. (2014) Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014.  Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics.  _Tradumàtica_ , (12):0455–463.
  * Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023.  Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt.  _arXiv preprint arXiv:2303.13809_.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 671–688, Belgium, Brussels. Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Macháček and Bojar (2014) Matouš Macháček and Ondřej Bojar. 2014.  Results of the WMT14 metrics shared task.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Martins et al. (2020) André F. T. Martins, Joao Graca, Paulo Dimas, Helena Moniz, and Graham Neubig. 2020.  Project MAIA: Multilingual AI agent assistant.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 495–496, Lisboa, Portugal. European Association for Machine Translation.
  * Maruf et al. (2018) Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2018.  Contextual neural model for translating bilingual multi-speaker conversations.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 101–112, Brussels, Belgium. Association for Computational Linguistics.
  * Maruf et al. (2019) Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019.  Selective attention for context-aware neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3092–3102, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Menezes et al. (2023) Miguel Menezes, M. Amin Farajian, Helena Moniz, and João Varelas Graça. 2023.  A context-aware annotation framework for customer support live chat machine translation.  In _Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track_ , pages 286–297, Macau SAR, China. Asia-Pacific Association for Machine Translation.
  * OpenAI et al. (2023) OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023.  Gpt-4 technical report.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Popović (2015) Maja Popović. 2015.  chrf: character n-gram f-score for automatic mt evaluation.  In _Proceedings of the tenth workshop on statistical machine translation_ , pages 392–395.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Raunak et al. (2023) Vikas Raunak, Tom Kocmi, and Matt Post. 2023.  Evaluating metrics for document-context evaluation in machine translation.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 812–814, Singapore. Association for Computational Linguistics.
  * Rei et al. (2022a) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rei et al. (2020a) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rei et al. (2020b) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b.  Unbabel’s participation in the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 911–920, Online. Association for Computational Linguistics.
  * Rei et al. (2022b) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b.  CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Robertson and Díaz (2022) Samantha Robertson and Mark Díaz. 2022.  Understanding and being understood: User strategies for identifying and recovering from mistranslations in machine translation-mediated chat.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 2223–2238.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_ , pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  * Specia et al. (2021) Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. 2021.  Findings of the WMT 2021 shared task on quality estimation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 684–725, Online. Association for Computational Linguistics.
  * Stanojević et al. (2015) Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015.  Results of the WMT15 metrics shared task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 256–273, Lisbon, Portugal. Association for Computational Linguistics.
  * Tiedemann and Scherrer (2017) Jörg Tiedemann and Yves Scherrer. 2017.  Neural machine translation with extended context.  In _DiscoMT@EMNLP_.
  * Vernikos et al. (2022) Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022.  Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 118–128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1198–1212, Florence, Italy. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yamashita et al. (2009) Naomi Yamashita, Rieko Inaba, Hideaki Kuzuoka, and Toru Ishida. 2009.  Difficulties in establishing common ground in multiparty groups using machine translation.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 679–688.
  * Zar (2005) Jerrold H Zar. 2005.  Spearman rank correlation.  _Encyclopedia of biostatistics_ , 7.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.
  * Zouhar et al. (2024) Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024.  Fine-tuned machine translation metrics struggle in unseen domains.