# Prompting PaLM for Translation: Assessing Strategies and Performance

David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar,
George Foster
Google Research
{vilar, freitag, colincherry, jmluo, vratnakar<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) that have been trained on multilingual but not
parallel text exhibit a remarkable ability to translate between languages. We
probe this ability in an in-depth study of the pathways language model (PaLM),
which has demonstrated the strongest machine translation (MT) performance
among similarly-trained LLMs to date. We investigate various strategies for
choosing translation examples for few-shot prompting, concluding that example
quality is the most important factor. Using optimized prompts, we revisit
previous assessments of PaLM’s MT capabilities with more recent test sets,
modern MT metrics, and human evaluation, and find that its performance, while
impressive, still lags that of state-of-the-art supervised systems. We
conclude by providing an analysis of PaLM’s MT output which reveals some
interesting properties and prospects for future work.

##  1 Introduction

Large language models (LLMs) trained to predict the next token from a lengthy
context have demonstrated impressive machine translation capabilities, despite
being trained on corpora that are overwhelmingly English, with no
intentionally-included parallel text. In this paper, we carry out an in-depth
investigation into the translation capabilities of LLMs, testing different
prompting strategies and carefully assessing the resulting performance. We
study the recently-introduced PaLM model Chowdhery et al. (2022), which has
achieved the strongest MT results among LLMs trained exclusively on non-
parallel multilingual corpora.

$-40$$-30$$-20$$-10$$0$$10$$20$$30$$40$110100bleurt differenceFrequency (log
scale) Figure 1:  Histogram of the sentence-level bleurt difference between
two different 5-shot PaLM runs using the random prompt selection method from
the original paper on a corpus of 1000 sentences. Each bar corresponds to a
difference range of 1 bleurt point. A majority of sentences (516) show a
difference of more than 1 bleurt point, demonstrating that the choice of
prompt can strongly affect translation quality.

To ensure a fair assessment of PaLM’s MT capability, we begin with an
exploration of example selection methods for use with fixed prompt templates.
We vary both the pool from which examples are chosen and the method for
choosing them, comparing standard random selection to $k$-nearest-neighbour
($k\text{NN}$) selection that customizes prompts for specific inputs. Figure 1
highlights the importance of example selection by showing that two randomly-
selected sets of examples can result in significantly different distributions
of sentence-level bleurt scores.

Although Chowdhery et al. (2022) report interesting results on low-resource
and non-English language pairs, their most striking findings concern high-
resource pairs. Accordingly, we limit our investigation to French, German, and
Chinese translation to and from English. We evaluate sentence-level
translation quality using recommended practices for high-quality MT,
specifically:  (i) we use recent WMT test sets to guard against train/test
data leakage, and to facilitate comparison with state-of-the-art (SOTA) MT
systems;  (ii) we use a SOTA automatic metric (bleurt) instead of bleu which
has been demonstrated to be suboptimal for high-quality translations Kocmi et
al. (2021); Freitag et al. (2021b); and  (iii) we conduct an expert-based
human evaluation with detailed categories to characterize the error patterns
of the automatically generated translations.

Our contributions are as follows:

  * •

We carry out the first systematic study of LLM prompting for MT, exploring
both the example candidate pool and the selection strategy. We find that the
_quality_ of examples matters more than the domain from which they are drawn
or their lexico-semantic proximity to the current input.

  * •

We evaluate the translation capability of LLMs with the procedure currently
recommended by the MT community. This involves using the latest test sets, and
measuring performance with SOTA MT metrics and an additional expert-based
human evaluation. We find that, although impressive, the sentence-level
translation capacity of LLMs still lags behind SOTA MT.

##  2 Related Work

Inspired by the findings of Radford et al. (2019); Brown et al. (2020),
prompting strategies for LLMs have become a topic of intense interest,
generating work across a broad spectrum of methods and applications Liu et al.
(2021). A basic distinction can be made between _hard_ (explicit text)
prompting such as we use, and _soft_ prompting that seeks to learn embeddings
Lester et al. (2021), activations Li and Liang (2021); Hambardzumyan et al.
(2021), or attention weights Liu et al. (2022a) that condition the model to
perform a desired task. The latter approach is more expressive and more
efficient at inference time, but performance can be sensitive to
initialization Hou et al. (2022), and some techniques require modifications to
the model.

Hard prompts have the advantage of being easy to interpret and modify. Work in
this area includes tools to facilitate development of handcrafted prompts
Strobelt et al. (2022); Bach et al. (2022); algorithms to find optimal prompts
through gradient-guided search Shin et al. (2020) or exhaustive search through
labels Schick and Schütze (2021) or both labels and templates Gao et al.
(2021); as well as studies on the effect of example order Kumar and Talukdar
(2021); Lu et al. (2022). Hard prompts have also been used to analyze model
capabilities Garg et al. (2022); Li et al. (2022a), the role of data Singh et
al. (2022), and the nature of prompting itself Min et al. (2022).

With few exceptions, e.g. Li et al. (2022b); Liu et al. (2022b); Valvoda et
al. (2022), prompting approaches condition on the task rather than the
specific input. Our $k\text{NN}$ approach for conditioning on the input was
pioneered by Liu et al. (2022b), who used RoBERTa embeddings to identify
relevant GPT-3 prompts for sentiment, table-to-text, and QA tasks. In contrast
to our results, they found that $k\text{NN}$ works better than a random-
selection baseline, and that the advantage grows as the size of the (domain-
controlled) example pool increases.

There has been relatively little work on prompting LLMs for MT. The GPT-3 and
PaLM papers Brown et al. (2020); Chowdhery et al. (2022) adopt similar
approaches, comparing 0, 1, and $n$-shot111Where $n$ is 64 for GPT-3 and 5 for
PaLM. random selection of independent sentence pairs from WMT training
corpora, and testing on WMT test sets traditionally used in ML (older French,
German, Romanian tasks, to and from English), augmented in PaLM with
French$\rightarrow$German and Kazakh. For both models, performance increases
with number of shots, and $n$-shot bleu scores are found to be competitive
with previous unsupervised SOTA, and in some settings—particularly into
English—supervised SOTA as well.

In further MT work, Reynolds and McDonell (2021) experiment with prompt
templates for GPT-3, and find that 0-shot prompts with carefully-chosen
templates can outperform $n$-shot prompts with sub-optimal templates. Garcia
and Firat (2022) explore using prompts with mT5 Xue et al. (2021) to control
output attributes such as formality, and also examine the effect of using
prompt-like natural-language tags during fine-tuning. Patel et al. (2022)
propose autoregressive prompting (concatenating only the first predicted word
to a prompt and output prefix at each step) for bidirectional models trained
with masking objectives, and demonstrate that mT5 can match GPT-3 performance
on low-resource languages with this technique.

##  3 Prompting for Machine Translation

For a general task, prompting an LLM to generate a desired output $y$ from an
input $x$ can involve many steps Liu et al. (2021), including template
generation, slot filling, answer search, and answer mapping. In MT, the answer
search and mapping processes are simplified because the answers generated by
the LLM can be used directly; we simplify further by using a fixed template.
What we explore in depth is the slot filling portion; in particular, we test a
variety of methods to select few-shot examples for the prompt.

In initial experiments we determined that for few-shot prompting the exact
form of the template is unimportant, and we therefore adopted simple templates
of the form (for $n$-shot prompting):

[source]: [ $X_{1}$] [target]: [ $Y_{1}$] ... [source]: [ $X_{n}$] [target]: [
$Y_{n}$] [source]: [ $X$] [target]:

where [source] and [target] are instantiated with the names in English of the
source and target languages, e.g. English and German. Each slot pair
$(X_{i},Y_{i})$ is filled with a translation example for these languages, and
the final slot $X$ is filled with the current source text. Our algorithm for
$n$-shot translation from a source text $x$ to a target text $y$ is:

  1. 1.

Choose translation example pairs $(x_{1},y_{1})$ … $(x_{n},y_{n})$. In
general, these can depend on $x$.

  2. 2.

Plug the example pairs and $x$ into the template. Condition PaLM on the
resulting string.

  3. 3.

Perform a greedy search,222We found that using a sampling temperature other
than 0 tended to degrade translation quality. stopping when the model outputs
a newline.

  4. 4.

Output the predicted suffix verbatim as $y$.

Example selection operates in two phases: first choose a pool containing
parallel text, then choose examples from the pool. Choosing the pool lets us
control global attributes of examples such as domain and average quality. Our
baseline method for choosing examples is to select them randomly from the
pool. We also experiment with selecting examples that are “closest” to the
source text, on the hypothesis that such examples will help guide the model to
produce similar translations.

To find relevant examples, we use $k$-nearest neighbor ($k\text{NN}$) search
on the source side of our parallel pool, inspired by Khandelwal et al. (2021).
We carry out the search itself using the method of Guo et al.
(2020)333Available at https://github.com/google-research/google-
research/tree/master/scann., and investigate two possible representations of
the sentences, with associated distance measures:

Bag-of-words (BOW)

Each sentence is represented by a (sparse) vector of counts associated with
each word in the vocabulary. As the associated distance measure we use cosine
distance. This representation focuses just on the surface form of the words,
and thus favors lexical similarity between the examples.

Roberta

Another possibility is to embed the sentences in a continuous multidimensional
space and use Euclidean distance for $k\text{NN}$ retrieval. In our case we
used Roberta Liu et al. (2019), a multilingual transformer-based model. Note
that it would be conceivable to use PaLM itself as embedding model, which
would provide a representation (and associated similarity measure) closer to
the application that we are targeting. However, due to the high computational
cost and large amounts of data (for some experiments we embed the totality of
the WMT training data) we decided to use a smaller model. As Roberta is a
trained neural model, we expect its embeddings to better represent the
semantics of the sentences, and thus retrieve prompts that are relevant to
their subject matter.

##  4 Data

We experiment with translation into and out of English for Chinese, French and
German. After English (78.0%), German (3.5%) and French (3.3%) are the two
largest languages in PaLM’s 780B token training corpus; Chinese (0.4%) is the
15th largest, and it also represents an inherently more difficult translation
task. To facilitate comparisons with recent SOTA systems, and to minimize the
chance of overlap with PaLM’s training corpus, we test on news data from the
WMT 2021 evaluation campaign Akhbardeh et al. (2021). Since French was not
included in WMT21, we use data from WMT14; apart from being older, these test
sets are not purely source-original Freitag et al. (2019) like the more recent
ones. Table 1 summarizes our test data.

LP | Year | #sents | Ref
---|---|---|---
en $\rightarrow$ de | 2021 | 1002 | C
de $\rightarrow$ en | 2021 | 1000 | B
en $\rightarrow$ zh | 2021 | 1002 | A
zh $\rightarrow$ en | 2021 | 1948 | A
en $\rightarrow$ fr | 2014 | 3003 | N/A
fr $\rightarrow$ en | 2014 | 3003 | N/A
Table 1:  Test set information, including the newstest dataset year and, when applicable, the reference we use for scoring.  LP | Pool | Size
---|---|---
en $\rightarrow$ X | X $\rightarrow$ en
de $\leftrightarrow$ en | WMT-full | 96M
WMT-dev | 11 732 | 13 060
high-end | 152 para.
zh $\leftrightarrow$ en | WMT Full | 55M
WMT-dev | 7 481 | 5 916
high-end | 170 para.
fr $\leftrightarrow$ en | WMT Full | 40M
WMT-dev | 2 886 | 2 957
high-end |  98 para.
Table 2:  Size of the different prompt pools, measured in sentences for the WMT sets and in paragraphs for the high-end pool.  Genre | Proportion
---|---
en $\leftrightarrow$ de | en $\leftrightarrow$ fr | en $\leftrightarrow$ zh
biography | 31% | 20% | –
business | – | – | 15%
commentary | 25% | 10% | 16%
culture | – | 44% | 14%
fashion | 16% | – | –
food | – | 8% | –
news | 4% | 18% | 43%
obituary | 24% | – | 13%
Table 3: Genre distributions for the high-end pool.

We use three distinct pools for prompt selection: the full WMT training corpus
for each language pair, a pool consisting of corresponding WMT development
sets (WMT-dev), and a manually-curated “high-end ” pool. Sizes of the pools
are shown in Table 2. The WMT training data is largest and offers the highest
probability of close $k\text{NN}$ matches, but it is crawled text drawn from
sources of varying quality. For the WMT-dev pool, we download all the older
development sets available for the WMT evaluation campaigns, but selected only
the target-original part of the corpora,444The separation into source- and
target-original parts can easily be done using sacrebleu to download the data.
in order to encourage PaLM to produce more natural text. For German
$\leftrightarrow$ English and Chinese $\leftrightarrow$ English we include all
the newstest20XX test sets from 2010 until 2020. English $\leftrightarrow$
French is again a special case, as the language pair was discontinued after
2015. We used the development data from 2010 to 2013 but in order to increase
the pool of available sentences, we also include the newsdiscussion2015 set.
In general, the WMT-dev pools consist of text that has been manually
translated for each evaluation campaign; they are of higher quality than the
training data, and a closer domain match to our test sets.

The high-end material is selected from websites containing bilingual articles
that we judged to be professionally edited, with native or near-native quality
in both languages. As shown in Table 3, the articles are drawn from various
domains, with the news domain of the test sets comprising less than 50% of
each pool. We treat these articles as symmetrical, and use them as prompt
sources in both translation directions. Due to the non-literal nature of the
translations, there is often no 1-1 correspondence between sentence pairs, so
we extract aligned paragraphs for prompting. More detailed information about
the high-end pool is provided in Appendix A.

##  5 Experiments

For compatibility with Chowdhery et al. (2022), we ran all experiments at the
sentence level, translating each test sentence individually and in isolation
from its context. This deprives PaLM of the ability to exploit the longer
contexts it was exposed to during training, but it matches the operating mode
of our baselines (including SOTA baselines), and facilitates
evaluation.555Evaluation of document-level translations is complicated by
potentially non 1-1 sentence correspondences, resulting in long translation
units that are truncated by bleurt and can be difficult for humans to rate
reliably. We leave an exploration of potential gains from conditioning on
longer histories to future work.

In preliminary experiments, we varied the number of shots from 0 to 10, and
found clear performance gains as we increased the number of shots, with
diminishing returns after 5 sentence pairs. Accordingly we report all results
on the WMT pools in the 5-shot setting, where each shot is a single sentence
pair, matching the configuration in Chowdhery et al. (2022). For the high-end
pool, lacking 1-1 sentence alignments, we use 1-shot examples, where each shot
is a single paragraph pair. This provides roughly the same quantity of text as
5-shot with sentences, although it creates a stylistic mismatch with our
single-sentence test setup.

When randomly selecting examples, we observed that there is little variability
in automatic scores when selecting different samples666Note that this holds
for document level scores. The effect on single sentences can still be very
important, cf. Figure 1. (see Appendix B). For the results reported in this
section, we let PaLM produce translations with 5 different seeds and we
selected the run with the median bleurt score.

Following recent recommendations Kocmi et al. (2021); Freitag et al. (2021a)
we favour neural metrics (bleurt in our case) over bleu, although we also
report bleu scores for completeness. We use a cased version of bleurt Sellam
et al. (2020) that is based on RemBERT Chung et al. (2020). Furthermore,
instead of training bleurt on the unbounded z-normalized scores, we manually
scale the training data to a 0-1 range and clip the outliers to get more
interpretable scores. These changes did not impact the quality of bleurt. We
use bleu as implemented in sacrebleu Post (2018), with zh tokenization for
English-Chinese, and 13a tokenization for all other languages.777sacrebleu
signature: nrefs:1|case:mixed|eff:no| tok:TOK|smooth:exp|version:2.1.0, where
TOK is 13a or zh.

To perform human evaluation, we hired professional translators (7 for
En$\to$De, 5 for De$\to$En, 4 for Zh$\to$En, and 4 for En$\to$Zh) and measure
translation quality with a document-context version of mqm Lommel et al.
(2014) which mimics the setup proposed in Freitag et al. (2021a). This
includes using the same error categories, severity levels and error weighting
schema. As suggested in the study, we weight each major error with $5$ and
each minor error with $1$, except for minor punctuation errors which get a
score of $0.1$. The final segment-level score is an average over scores from
all annotators. We only used one annotator per document, so this average is
simply the total score for the segment from its single annotator.888One
difference from Freitag et al. (2021a) is that we do not impose a limit of 5
errors per sentence. We refer the reader to Freitag et al. (2021a) for the
details on error categories and annotator instructions.

Due to technical limitations on how long an evaluation session can be, we
limited the mqm evaluation to the first 12 segments per document.

###  5.1 Selection strategies and pools

We warm up by comparing example selection strategies on the two WMT pools,
using automatic metrics to evaluate quality on English $\leftrightarrow$
German. Results are shown in Table 4. The main observation is that the choice
of pool is much more important than the selection method: the results for WMT-
dev are notably higher than those for WMT-full across all settings. When
comparing $k\text{NN}$ selection methods, Roberta is more effective than BOW,
but it does not provide a consistent advantage over random selection.

We conjecture that the quality of an example is more important than its
proximity to the current source sentence. The larger size of the full WMT pool
means that the $k\text{NN}$ approaches will in general be able to find
examples that are closer to each source sentence than those from the dev pool,
but any resulting gain is offset by the greater risk that an example from the
full pool will be a poor translation (since we match only on the source side).
Interestingly, had we relied only on bleu results, we would have concluded
that the choice of pool is unimportant, and that random selection consistently
outperforms $k\text{NN}$.

LP | Pool | Selection | bleurt |  | bleu
---|---|---|---|---|---
en $\rightarrow$ de | full | random | 71.8 |  | 32.9
$k\text{NN}$ BOW | 71.7 |  | 32.4
$k\text{NN}$ Roberta | 73.0 |  | 32.5
dev | random | 74.8 |  | 32.8
$k\text{NN}$ Roberta | 74.8 |  | 32.3
de $\rightarrow$ en | full | random | 74.8 |  | 38.4
$k\text{NN}$ BOW | 72.7 |  | 36.9
$k\text{NN}$ Roberta | 73.8 |  | 35.4
dev | random | 75.9 |  | 38.0
$k\text{NN}$ Roberta | 75.8 |  | 37.2
Table 4:  Comparison of example selection strategies on the WMT-full and WMT-
dev pools. Values for random selection are averaged over 5 runs.

###  5.2 Results on all language pairs

Table 5 contains our main results, for German $\leftrightarrow$ English,
Chinese $\leftrightarrow$ English, and French $\leftrightarrow$ English. For
each language pair, we ran PaLM with random selection on all three pools and
with $k\text{NN}$ Roberta on the WMT-full pool. We compared these systems to
output from the best performing system in the 2021 WMT evaluation campaign for
German and Chinese, and for off-the-shelf Google Translate for all six
language pairs. We evaluate with bleu and bleurt as in the previous section,
augmented with human mqm assessments for German and Chinese. French is a
special case, as its evaluation set is eight years old, and it is difficult to
ensure that any of the MT systems we evaluate have not been exposed to it
during training. We include it mostly for the purposes of comparison to
Chowdhery et al. (2022), and do not provide SOTA results or perform human
evaluation.

Comparing PaLM results for German and Chinese, the pattern from the previous
section holds up: random selection from the WMT-dev pool outperforms selection
from the full pool. mqm scores correlate well with bleurt for these results.
Despite domain and style mismatch, results for the high-end pool are very
similar to those for WMT-dev—closer than any results on the full pool—adding
support to the hypothesis that example quality is the main determinant of
PaLM’s output quality.

The French results reverse the general pattern. For this language pair, random
selection from the WMT-full pool does best, although the results for all
methods are fairly similar, with a difference of approximately 0.5 bleurt
between the best and worst. One potential explanation is the age and quality
of newstest2014. The test set creation has dramatically improved since 2014
and thus following the recommendations of the MT community and testing on the
newest test sets would also make sense for French$\leftrightarrow$English to
get more conclusive results for these language pairs. Unfortunately,
French$\leftrightarrow$English hasn’t been part of WMT since 2014.

Turning to a comparison between PaLM and conventional MT systems, the
specialized SOTA systems have a substantial advantage of between 1 and 3
bleurt points over the best PaLM results, a gap that is reflected in their
much lower mqm scores. The difference is narrower for the general-purpose
Google Translate system: less than 1 bleurt except for
Chinese$\rightarrow$English (1.8), with French$\rightarrow$English at parity.
PaLM’s performance relative to the best MT system for each language pair is
generally better when translating into English, where it is lower by 1.0, 2.3,
and 0.0 bleurt for German, Chinese, and French, compared to drops of 2.1, 2.5,
and 0.6 in the reverse direction.

The mqm results show some interesting characteristics of translations produced
by PaLM. In all the language pairs evaluated, fluency mqm scores from PaLM are
generally of a similar magnitude as SOTA systems.999The mqm evaluation for
zh$\rightarrow$en is not finished at the time of writing. We will update the
paper with its results when it finishes. Accuracy mqm scores are generally
lower for PaLM though, and this gap is dominated by “Accuracy/Omission” errors
with “Major” severity (followed by “Major” severity errors in some other
“Accuracy/*” sub-types, but there is no consistent pattern for these sub-types
across language pairs). Another interesting fact is that in some of the
language pairs, “Style/Awkward” errors (which are mostly “Minor severity”
errors) are slightly lower in the best-performing PaLM systems, compared to
the SOTA systems. We show a selection of mqm error count comparisons between
PaLM WMT-dev random and SOTA systems in Table 6. More detailed mqm scores and
error counts are provided in Appendix C.

LP | System |  mqm $\downarrow$ |  bleurt $\uparrow$ |  |  bleu $\uparrow$
---|---|---|---|---|---
en $\rightarrow$ de | WMT21 Facebook Submission Tran et al. (2021) |  1.18† | 76.9 |  | 42.0
Google Trans. |  | 1.59 | 75.7 |  | 39.8
PaLM | WMT-full random | 1.90 | 73.7 |  | 32.9
WMT-full $k\text{NN}$ | 1.93 | 73.0 |  | 32.5
WMT-dev random | 1.58 | 74.8 |  | 32.8
high-end random | 1.67 | 74.7 |  | 32.9
de $\rightarrow$ en | WMT21 Facebook Submission Tran et al. (2021) |  1.31† | 76.9 |  | 41.9
Google Trans. |  | 1.71 | 76.4 |  | 40.9
PaLM | WMT-full random | 2.38 | 74.7 |  | 38.3
WMT-full $k\text{NN}$ | 3.03 | 73.8 |  | 35.4
WMT-dev random | 1.92 | 75.9 |  | 38.0
high-end random | 1.89 | 75.8 |  | 38.8
(a) German$\rightarrow$English (nt2021). All mqm results labelled with
$\dagger$ are significantly better than all other systems based on PERM-BOTH
pair-wise significance testing Deutsch et al. (2021) with $p=0.05$.

LP | System |  mqm $\downarrow$ |  bleurt $\uparrow$ |  |  bleu $\uparrow$
---|---|---|---|---|---
en $\rightarrow$ zh | WMT21 WeChat Submission Zeng et al. (2021) |  2.47† | 66.6 |  | 36.9
Google Trans. |  | 3.23 | 65.0 |  | 36.2
PaLM | WMT-full random | 4.35 | 62.2 |  | 28.6
WMT-full $k\text{NN}$ | 5.06 | 60.7 |  | 28.5
WMT-dev random | 3.24 | 64.1 |  | 29.2
high-end random | 3.70 | 63.9 |  | 29.6
zh $\rightarrow$ en | WMT21 Borderline Submission Wang et al. (2021) |  3.03† | 70.0 |  | 33.4
Google Trans. |  | 3.21 | 69.5 |  | 32.2
PaLM | WMT-full random | 3.99 | 67.2 |  | 25.8
WMT-full $k\text{NN}$ | 4.12 | 65.8 |  | 23.8
WMT-dev random | 3.59 | 67.5 |  | 25.3
high-end random | 3.97 | 67.7 |  | 25.1
(b) Chinese$\rightarrow$English (nt2021). All mqm results labelled with
$\dagger$ are significantly better than all other systems based on PERM-BOTH
pair-wise significance testing Deutsch et al. (2021) with p=0.05.

LP | System |  bleurt $\uparrow$ |  |  bleu $\uparrow$
---|---|---|---|---
en $\rightarrow$ fr | Google Trans. |  | 76.5 |  | 45.7
PaLM | WMT-full random | 75.9 |  | 42.3
WMT-full $k\text{NN}$ | 75.3 |  | 41.8
WMT-dev random | 75.4 |  | 41.9
high-end random | 75.2 |  | 38.6
fr $\rightarrow$ en | Google Trans. |  | 77.7 |  | 43.2
PaLM | WMT-full random | 77.7 |  | 42.7
WMT-full $k\text{NN}$ | 77.3 |  | 41.2
WMT-dev random | 77.2 |  | 42.1
high-end random | 77.6 |  | 40.4
(c) French$\rightarrow$English (nt2014).

Table 5: Translation results for all language pairs. Values for random selection are the bleurt median of 5 runs. LP | Severity | Category | PaLM | SOTA
---|---|---|---|---
de $\rightarrow$ en | Major | Accuracy/Omission | 51 | 19
en $\rightarrow$ de | Major | Accuracy/Omission | 26 | 7
en $\rightarrow$ zh | Major | Accuracy/Omission | 80 | 46
de $\rightarrow$ en | Minor | Style/Awkward | 73 | 81
en $\rightarrow$ de | Minor | Style/Awkward | 166 | 144
en $\rightarrow$ zh | Minor | Style/Awkward | 115 | 142
Table 6: Selected mqm error count comparisons between PaLM WMT-dev random and
SOTA.

###  5.3 Comparison to previous results

The only results in our experiments that are directly comparable to the few-
shot results from Chowdhery et al. (2022) are the WMT-full bleu scores in
table 5(c) (WMT14 French test-set). Our result for French$\rightarrow$English
matches theirs exactly, but our score for English$\rightarrow$French is lower
by 1.7 (42.3 versus 44.0). We do not have access to their output, but we
attribute this discrepancy to their use of the sacrebleu intl tokenizer; when
we evaluate our output using this version, we obtain matching scores.

Our general finding that PaLM’s into-English performance is better than the
reverse direction matches the conclusion from Chowdhery et al. (2022), while
our comparison with recent SOTA systems on current test sets contrasts with
their results indicating that PaLM can rival supervised performance in older
settings.

##  6 Analysis

In this section we delve further into various aspects of PaLM’s MT
performance.

###  6.1 $k\text{NN}$ versus random prompts

To understand the performance difference between $k\text{NN}$ Roberta and
randomly-selected examples, we performed a qualitative analysis, choosing
sentences with the largest bleurt difference between the two systems (similar
to the boundaries of the histogram in Figure 1).

In Table 7(a) we see an example where the $k\text{NN}$ system correctly
retrieves relevant translation examples in the football domain, guiding PaLM
to produce a better translation for “mastered the tasks of the championship
and the cup” than that of the random selection system. Figure 7(b) shows an
example where random selection outperforms $k\text{NN}$. In this case,
$k\text{NN}$ retrieves 5 relevant source sentences from the music domain, but
all of them have alignment errors, and as a result PaLM generates hallucinated
output.

In general, random selection is also prone to landing on alignment errors, but
as each prompt is selected independently, the odds that _all_ examples will be
errors are low. An informal analysis of $k\text{NN}$ examples seems to
indicate that if one non-parallel prompt is selected, the others also tend to
be of poor quality, perhaps due to corpus alignment errors that are
concentrated in particular documents or topics. Since $k\text{NN}$ matches
only on the source side, it is not robust to noise of this sort.

| Source |  "Wir haben die Pflichtaufgaben mit Meisterschaft und Pokal einfach hervorragend gemeistert.
---|---|---
| Reference |  “Quite simply, we have excellently mastered the necessary tasks for the Championship and the Cup.
$k\text{NN}$ Roberta | Hyp |  "We have simply mastered the tasks of the championship and the cup excellently.
Prompt 1 |  German: Mit einer verstärkten Mannschaft holte die Mannschaft das Double aus Meisterschaft und Pokal. English: The decision paid off as the team achieved a league and cup double.
Prompt 2 |  German: Darüber hinaus haben wir uns wichtige Meisterschaftspunkte im Kampf um den Vizetitel gesichert.“ English: We have furthermore secured some important championship points in the fight about the vice champion’s title.”
Prompt 3 |  German: Bring deine Mannschaft durch alle Spiele der Europameisterschaft und gewinne den Pokal! English: Take your team all the way through the Euro Cup stages and lift the trophy!
Prompt 4 |  German: So konnte er die französische Meisterschaft, den nationalen Pokal sowie den Supercup gewinnen. English: He helped the club to win the national championship and the Supercup.
Prompt 5 |  German: Roter Stern gewinnt in jener Saison das Double von Meisterschaft und Pokal. English: Red Star won their sixth double in this season.
Random | Hyp |  "We have the compulsory tasks with mastery and cup simply superbly mastered.
Prompt 1 |  German: Im November letzten Jahres war ein einzelner Steinadler und auch Bussarde im Blick der Kamera. English: In November last year a solitary golden eagle and buzzards too appeared in web camera view.
Prompt 2 |  German: Teile: Modesto-14th Street, Stanislaus, California, Kalifornien-Luft-Qualitätsindex English: Share: Modesto-14th Street, Stanislaus, California, California Air Quality Index
Prompt 3 |  German: So viel kostet ein Hotel in Chester English: How much does a hotel in Chester cost?
Prompt 4 |  German: "… alle Mitarbeiter, die alles tun, um hilfsbereit zu sein und sehr freundlich zu sein; köstliche Margaritas; Kolibris und Granatäpfel im Garten (sowie eine sehr freundliche Katze); Ein echtes Gefühl von Zuhause. " Aktionsangebot English: "… all staff, who go out of their way to be helpful and are extremely welcoming; delicious margaritas; hummingbirds and pomegranates in the garden (as well as a very friendly cat); a real home-from-home feeling. "
Prompt 5 |  German: Gansevoort Land zum Verkauf English: Gansevoort Land for Sale
(a) Example where $k\text{NN}$ outperforms random selection.

| Source |  Frei von Drogen veröffentlichte Green mit der Peter Green Splinter Group einige Alben, bis sich die Band 2004 auflöste.
---|---|---
| Reference |  Free of drugs, Green and the Peter Green Splinter Group released various albums before the band split up in 2004.
$k\text{NN}$ Roberta | Hyp |  The band released their debut album, The Last of the Great Pretenders, in 2003.
Prompt 1 |  German: Ab 1990 war er Sänger der Gruppe Talisman, mit der er sieben Studioalben veröffentlichte, bis sich die Band 2007 auflöste. English: From 1998 until his departure in 2007, he was the lead singer of the group Lonestar, which recorded seven studio albums on BNA Records during his tenure as lead vocalist.
Prompt 2 |  German: 2001 veröffentlichte die Band unter dem Namen Glass die rockige Single Out Of Nowhere, verpasste die Charts und löste sich im Anschluss auf. English: Around this time he wrote and presented the ITV Network productions The Rock that Doesn’t Roll and The Rock That Rolled Away.
Prompt 3 |  German: Mit ihrer Band Ex Cops veröffentlichte sie zwei Alben, bevor sich die Band 2015 auflöste. English: Their new band released two EPs before signing to Apparition Records in 2011.
Prompt 4 |  German: In seiner Jugend gründete David Haering die Punk-Band Side Effect, mit der er drei Alben und eine EP veröffentlichte. English: Peter Hajba and Alexander Brandon used OpenMPT to compose the soundtracks for Bejeweled 2, Bejeweled 3 and other PopCap games.
Prompt 5 |  German: Nach der Veröffentlichung des Live-Albums Beast from the East 1988 trennten sich die Wege der Musiker, als Don Dokken die Band auflöste. English: In 1988, after the Monsters of Rock Tour and a further platinum album, Don Dokken decided to break up the band and they went on their separate ways.
Random | Hyp |  Free from drugs, Green released several albums with the Peter Green Splinter Group, until the band broke up in 2004.
Prompt 1 |  German: 250 gr/m2: eine Reihe merino intermedia, vielseitigkeit und schutz garantiert. English: 250 gr/m2: Range merino intermediate, versatility and guaranteed protection.
Prompt 2 |  German: 127 Moo.3, Choeng Thale, Thalang, Phuket, Strand Bang Tao, Thailand (Karte anzeigen) English: 127 Moo.3, Choeng Thale, Thalang, Phuket, Bang Tao Beach (Phuket), Thailand (Show map)
Prompt 3 |  German: Ich bin stolz, sagen zu können, dass Ihr Produkt mir die Größe verliehen hat, von der ich jahrelang geträumt habe. English: I am proud to say that your product has given me the size that I sought for all these years.
Prompt 4 |  German: In der NBA Saison 2003–2004 schien es endlich besser für Garnett und die Timberwolves zu laufen. English: The 2002–03 season seemed to look up for the Timberwolves.
Prompt 5 |  German: APC-Fhrancis Oliver Lopez extern natürliche Links English: APC-Fhrancis Oliver Lopez External Passing Juice
(b) Example where random selection outperforms $k\text{NN}$.

Table 7: Example prompts for random and $k\text{NN}$ Roberta selection for the
German$\rightarrow$English translation direction.

###  6.2 Fixed versus random prompts

LP | Selection | bleurt
---|---|---
min | avg | max
en $\rightarrow$ de | fixed |  | 74.7 |
random | 74.5 | 74.7 | 75.0
de $\rightarrow$ en | fixed |  | 76.3 |
random | 75.6 | 75.8 | 75.9
en $\rightarrow$ zh | fixed |  | 64.7 |
random | 63.7 | 63.9 | 64.0
zh $\rightarrow$ en | fixed |  | 67.0 |
random | 67.3 | 67.5 | 67.7
en $\rightarrow$ fr | fixed |  | 75.5 |
random | 75.2 | 75.2 | 75.3
fr $\rightarrow$ en | fixed |  | 77.9 |
random | 77.4 | 77.6 | 77.6
Table 8:  Fixed (maximum-likelihood) prompts vs random prompts. All prompts
are drawn from the high-end pool, and performance is measured on the standard
test sets (WMT21 for German and Chinese, WMT14 for French). The scores for
random selection are the minimum, average, and maximum over 5 random draws.

The results from section 5.2 indicate that random selection from small, high-
quality prompt pools can work better than trying to customize prompts for
specific inputs. In this section we investigate the effect of using a _single_
high-quality prompt for all inputs, chosen using a maximum-likelihood
criterion. For convenience, we carried out experiments on the high-end pool
with 1-shot paragraph prompts. For each prompt in the pool, we computed the
probability of a set of held-out high-end paragraphs when PaLM was conditioned
on that prompt. We select the prompt that resulted in the highest probability
for each language pair.

Table 8 compares this method to random selection from the high-end pool. For
all language pairs except Chinese$\rightarrow$English, the fixed prompt does
as well or better than the average performance over 5 random runs where a
different prompt is selected for each input during each run. In
Chinese$\rightarrow$English, the prompt that ranked 5th according to the
probability criterion also outperformed the random average, suggesting
problems with our held-out set for that language pair.

We conclude that using a single high-quality prompt can be a safer strategy
than choosing a fresh randomly-selected prompt for each input. Model
probability appears to be a reasonable criterion for judging quality, but we
look forward to refining this heuristic in future work.

###  6.3 Overlap of test and training data

One major change with respect to Chowdhery et al. (2022) is our use of more
recent WMT test sets, which are unlikely to overlap with PaLM’s training data.
Note that throughout this section, when we refer to overlap, we refer to
target-side overlap only; we do not expect any substantial parallel data to
appear in our large LM training data, and therefore do not expect any parallel
overlap. Chowdhery et al. (2022) established a strategy for working with
train-test overlap, where one detects overlap based on high-order $n$-gram
match, and then measures the impact of overlap by reporting bleu both on the
original test set and on a clean version with any overlapping test points
removed. They showed that clean scores could be as much as 3 bleu points lower
than original scores. They also showed that this gap did not shrink as model
size shrinks, providing evidence that the gap was not due to memorization.

Year | LP | % Clean
---|---|---
2014 | fr $\rightarrow$ en | 69.2
en $\rightarrow$ fr | 93.6
2016 | de $\rightarrow$ en | 80.3
en $\rightarrow$ de | 97.3
2021 | en $\rightarrow$ de | 99.6
en $\rightarrow$ zh | 99.7
de $\rightarrow$ en | 97.9
zh $\rightarrow$ en | 98.1
Table 9: The size of clean (lacking 15-gram target-side overlap with PaLM
training data) versions of test sets for various WMT years and language pairs

We perform a similar analysis here, measuring 15-gram overlap as tokenized by
the mBERT tokenizer Devlin et al. (2019).101010We selected the mBERT
tokenizer, as opposed to the PaLM’s sentence-piece tokenizer, because it
decouples the measurement of overlap from the model under test. For test
sequences with fewer than 15 tokens, we consider them overlapping if the
complete sequence is found as a subsequence of a training example. We report
the degree of overlap by showing the percentage of original test examples that
survive in the clean test set after removing overlap in Table 9. We confirm
that the older French$\rightarrow$English and German$\rightarrow$English sets
have substantial overlap with PaLM’s training data, while the newer test sets,
whether into or out of English, have much smaller overlapping portions.

Data | %Clean | Method | bleu | bleurt
---|---|---|---|---
Orig | Clean |  $\neg$Clean | Orig | Clean |  $\neg$Clean
de $\rightarrow$ en 2016 | 80.3 | Google Trans. | 47.6 | 45.5 | 55.3 | 78.4 | 77.7 | 81.3
WMT-full Random | 46.1 | 43.5 | 54.9 | 77.3 | 76.3 | 81.5
Diff | 1.5 | 2.0 | 0.4 | 1.1 | 1.4 | -0.2
fr $\rightarrow$ en 2014 | 69.2 | Google Trans. | 43.1 | 42.1 | 44.8 | 77.7 | 76.8 | 79.6
WMT-dev Random | 43.0 | 41.3 | 45.4 | 77.7 | 76.9 | 79.5
Diff | 0.1 | 0.8 | -0.6 | 0.0 | -0.1 | 0.1
Table 10: Comparison between Google Translate and 5-shot PaLM using three test
sets: Orig. (original), Clean (overlapping examples removed) and $\neg$Clean
(including only overlapping examples). We use Random instead of WMT-dev Random
for de$\rightarrow$en to avoid using the WMT 2021 development sets to prompt
for the WMT 2016 test (“sampling from the future”).

Chowdhery et al. (2022) show bleu differences between clean and original test
sets, and provide some evidence that differences are not due to memorization,
but it still isn’t clear how much overlap actually inflates a model’s score.
We directly quantify the effect of train-test overlap on decision making by
comparing 5-shot PaLM to Google Translate (GT)111111We chose Google Translate
for comparison because it is non-trivial to build a SOTA baseline for older
WMT scenarios. Through personal communication, we understand that Google
Translate has no overlap with WMT test sets. on our two sets with substantial
overlap, testing under original, clean and $\neg$clean (including only
overlapping examples) scenarios. bleu and bleurt scores for the two systems
and three test sets are shown in Table 10.

We can see that directly comparing original and clean results for a single
system conflates differences from overlap with those from the increased
difficulty of the clean subset. For example, for de$\rightarrow$en bleu,
comparing PaLM’s original and clean scores gives an overlap gap of 2.6-bleu,
in line with the gaps reported by Chowdhery et al. (2022). However, the non-
overlapping GT system also has lower scores on the clean set, indicating that
it may simply be more difficult.121212The difference in difficulty between
Clean and $\neg$Clean for systems without overlap is not easily explained. A
common difficulty indicator is sentence length, but average lengths, as
measured by number of sacrebleu tokens per sentence, are similar between Clean
and $\neg$Clean for both de$\rightarrow$en (23.8 versus 23.0) and
fr$\rightarrow$en (21.1 versus 22.7). It’s more useful to see that the
original test indicated a 1.5-bleu difference between the two systems, while
the clean test indicates a 2.0-bleu difference, meaning PaLM benefited from
overlap by 0.5 bleu in this comparison. The fully overlapping $\neg$clean
further distorts the difference between the two systems: the true (clean)
delta of 2.0 bleu shrinks to only 0.4. Trends for fr$\rightarrow$en are
similar: though PaLM and GT are very close according to the original test set,
the clean set reveals a delta of 0.8 bleu. Interestingly, bleurt may be less
sensitive to overlap, with the original-versus-clean deltas hovering around 0
for fr$\rightarrow$en regardless of the test subset, and de$\rightarrow$en
showing that PaLM benefits from an overlap bonus of only 0.3 bleurt.

In summary, overlap between the target side of the test data and the large LM
training data can have an impact on both bleu and bleurt scores, altering the
delta between two systems where one benefits from overlap and another does not
by up to 0.7 bleu or 0.3 bleurt for a 20-30%-overlap scenario. However, we
should emphasize that the scoring differences due to overlap are small
overall, and certainly much smaller than one might expect if one looked only
at the difference between original and clean scores.

###  6.4 Example Translations

Example translations comparing PaLM and SOTA systems for
German$\rightarrow$English and English$\rightarrow$Chinese are given in Table
11 and Table 12, respectively. We compared the translations of both systems
and chose examples that are short, but include the most frequent patterns that
we observed also in longer translations. In general, PaLM’s translations are
less literal when compared to supervised NMT systems. Even though this is one
of the strengths of PaLM, it occasionally misses some important information in
the source or hallucinates facts not present in the source sentence. The
supervised models on the other hand are faithful to the source; this reduces
the risk of omission and addition errors, but occasionally leads to
translations that are not natural in the target language (e.g. translating
street names or using the wrong time format). The mqm results reaffirm these
findings—compared to SOTA, PaLM tends to have a higher number of "Accuracy/*"
errors (especially "Accuracy/Omission") with "Major" severity, and it also
tends to have a slightly lower or comparable number of "Style/Awkward" errors.

Source |  Wegen eines ersten offiziellen Corona-Verdachtsfalls im Land hat Nordkorea aus Furcht vor einer Katastrophe eine Großstadt komplett abgeriegelt.
---|---
Reference |  Due to a first official suspected case of coronavirus in the country, North Korea has completely shut down a major city out of fear of a disaster.
SOTA |  Due to a first official suspected case of coronavirus in the country, North Korea has completely sealed off a major city for fear of a catastrophe.
PaLM |  North Korea has completely sealed off a city of 300,000 people because of a first suspected case of coronavirus in the country, fearing a catastrophe.
(a) PaLM generated two errors:  (i) Makes the assumption that the major city
has 300,000 citizens,  (ii) Forgot to translate the word official. SOTA
generates a perfect translation. Note: this example also exemplifies PaLM’s
less literal translation style.

Source |  Für mindestens drei Menschen endet die Tour mit dem Motorrad tödlich.
---|---
Reference |  For at least three people, their bike ride ended in death.
SOTA |  The motorcycle tour ends fatally for at least three people.
PaLM |  At least three people die in motorcycle accidents.
(b) The source mentions a single accident happening on a bike tour. PaLM
refers to multiple accidents happening independently.

Source |  Ein Zeuge hörte gegen 3.40 Uhr Geräusche in der Talstraße und lief in Richtung des Imbisses.
---|---
Reference |  One witness heard noises on Talstraße around 3:40 am and ran in the direction of the snack stand.
SOTA |  A witness heard noises in the valley road at around 3.40 a.m. and ran towards the snack bar.
PaLM |  A witness heard noises in Talstraße at around 3:40 a.m. and ran towards the snack bar.
(c) SOTA generates an overly-literal translation, resulting in copying the
street name (Talstrasse) and using the wrong time format.

Table 11: Example translations from newstest2021 German$\rightarrow$English.
PaLM translations are generated with the high-end prompt pool. These are
typical of error patterns observed in the translation output. We also observed
the same pattern when using WMT-dev as the prompt pool. In general, SOTA is
more faithful to the source while PaLM generates less literal translations
that occasionally miss some information from the source.

Source |  French World Cup winner Dembele, who has struggled for game time at the Camp Nou, was recently linked with a move to PSG in a swap deal with Neymar.
---|---
Reference |  在诺坎普球场冲锋陷阵的法国世界杯冠军得主 Dembele 最近通过与 Neymar 交换转投 PSG。
SOTA |  法国世界杯冠军登贝莱在诺坎普一直在为比赛时间而挣扎，最近他与内马尔交换转会巴黎圣日尔曼。
PaLM |  法国世界杯冠军德容，在诺坎普的出场时间一直不多，最近被传与内马尔进行交换加钱转会到PSG。
(a) PaLM produces two errors:  (i) translates a wrong player’s name;  (ii)
adds extra information that the player received a raise in the swap deal. SOTA
produces a perfect translation, but is much more literal than PaLM.

Source |  …in the wake of September 11, ASIO was given power to compulsorily question people for up to seven days in relation to terrorism offences.
---|---
Reference |  …在 911 事件之后，澳安全情报局有权对牵涉恐怖主义行为的人员进行为期最高 7 天的强制性询问。
SOTA |  …在9月11日之后，安全情报组织被授权对与恐怖主义罪行有关的人进行长达7天的强制性讯问。
PaLM |  …澳大利亚安全情报局在9·11恐怖袭击之后获得了强制询问人员的权力，可以在7天内就恐怖主义罪行进行询问。
(b) The source phrase “September 11” is translated literally by SOTA into a
date, whereas PaLM produces a more appropriate translation by describing it as
a terrorist attack.

Table 12: Example translations from newstest2021 English$\rightarrow$Chinese.
PaLM translations are generated with the WMT-dev prompt pool. We find SOTA to
generate more literal translations than PaLM, but PaLM suffers from more
omissions and mistranslations.

##  7 Conclusion

We perform a careful assessment of the sentence-level MT capabilities of PaLM,
a 540B-parameter decoder-only language model trained on a heavily English-
centric multilingual corpus, with no explicitly-included parallel text. We
compare PaLM to SOTA and a current off the shelf (COTS) MT system for three
high-resource languages—German, Chinese, and French—into and out of English,
using the latest test sets from WMT.

Comparing $k\text{NN}$ and random strategies for selecting 5-shot translation
examples to instantiate fixed prompt templates, we find that $k\text{NN}$’s
potential advantage in identifying examples relevant to the current source
sentence is outweighed by its susceptibility to corpus noise. Choosing
examples randomly from small, high-quality pools works well, and performance
appears to be independent of the domain and translation style of the pool,
suggesting that example quality is the most important factor. Further evidence
for this hypothesis comes from a preliminary experiment showing good
performance when using only a single heuristically-chosen prompt for all
inputs.

Using both the bleurt metric and mqm human evaluations, we show that PaLM’s
performance, while very impressive for an apparently unsupervised system,
still significantly lags that of competition-grade SOTA systems on recent WMT
test sets, and to a lesser extent the performance of COTS systems as well.
This contrasts with some of the findings of Chowdhery et al. (2022). As in
that work, we find that performance into English is somewhat better than the
reverse direction. We establish that our results on WMT 2021 test sets are
unaffected by train/test overlap, and calibrate the effect of overlap on our
WMT 2014 French setting by comparing to an overlap-free COTS system. Finally,
we perform extensive analysis on the characteristics of PaLM’s MT output,
notably finding that in all languages we tested it tends to be creative and
fluent but prone to omissions and other accuracy errors; broadly speaking, it
matches the fluency but lags the accuracy of conventional NMT.

In future work we look forward to testing PaLM on document-level translation
tasks, unleashing its formidable ability to leverage long contexts. We would
also like to explore prompt tuning methods that are more sophisticated than
the hard-prompt setting we adopted for this paper, particularly to see if
these might offer a way to tighten up PaLM’s MT accuracy without destroying
its impressive ability to generate highly-fluent text.

## Limitations

As we use only a small number of language pairs, it is not clear how general
our conclusions are; in particular, they pertain only to languages that are
well represented in PaLM’s training corpus, and only to translation into and
out of English. Our restriction to independent sentence-level translations may
have caused us to underestimate PaLM’s true capabilities, since some of the
accuracy problems we observed might be considered less severe in the context
of whole-document translation where less literal translations are more
typical. Our exploration of prompting barely scratches the surface of the many
methods that have been proposed for adapting LLMs to particular tasks, and we
may have missed a technique that produces higher-quality translations than we
observed. Finally, the human evaluation we rely on to provide our most
accurate results is necessarily subjective, and if we were to have carried out
the evaluation with different raters and a different methodology, our
conclusions might well have been different.

## Ethical Considerations

Working with large language models comes with many ethical concerns that are
discussed in detail in Brown et al. (2020) and Chowdhery et al. (2022). There,
MT is often one task of many, while we focus on the question of proper example
selection for few-shot prompting of MT, which adds a few specific concerns.
Our conclusion that prompt quality is important could lead one to build a
system with prompts drawn from a small set of trusted sources; indeed, our
high-end set is one such example of this. In such a scenario, this small
source will have an outsized impact on the output of the translation system,
and one must be careful to manage issues of attribution and intellectual
property. Furthermore, an editorial choice defining high-quality language can
potentially reduce quality for groups and topics not typically discussed in
this style Gururangan et al. (2022). Finally, by highlighting the power of
few-shot examples, one might be tempted to turn example selection over to the
users of a system. There, special steps must be taken to avoid exposing users
to biased or toxic outputs, which may be triggered by unconstrained prompting
Gehman et al. (2020); Costa-jussà et al. (2022).

## Acknowledgements

## References

  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Bach et al. (2022) Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. 2022.  Promptsource: An integrated development environment and repository for natural language prompts.  _arXiv preprint arXiv:2202.01279_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2020) Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020.  Rethinking embedding coupling in pre-trained language models.
  * Costa-jussà et al. (2022) Marta R. Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Javier Ferrando, and Carlos Escolano. 2022.  Toxicity in multilingual machine translation at scale.
  * Deutsch et al. (2021) Daniel Deutsch, Rotem Dror, and Dan Roth. 2021.  A statistical analysis of summarization evaluation metrics using resampling methods.  _Transactions of the Association for Computational Linguistics_ , 9:1132–1146.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Freitag et al. (2019) Markus Freitag, Isaac Caswell, and Scott Roy. 2019.  APE at scale and its implications on MT evaluation biases.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 34–44, Florence, Italy. Association for Computational Linguistics.
  * Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830.
  * Garcia and Firat (2022) Xavier Garcia and Orhan Firat. 2022.  Using natural language prompts for machine translation.  _arXiv preprint arXiv:2202.11822_.
  * Garg et al. (2022) Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022.  What can transformers learn in-context? a case study of simple function classes.  _arXiv preprint arXiv:2208.01066_.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020.  Accelerating large-scale inference with anisotropic vector quantization.  In _International Conference on Machine Learning_.
  * Gururangan et al. (2022) Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022.  Whose language counts as high quality? measuring language ideologies in text data selection.
  * Hambardzumyan et al. (2021) Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021.  WARP: Word-level Adversarial ReProgramming.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4921–4933, Online. Association for Computational Linguistics.
  * Hou et al. (2022) Yutai Hou, Hongyuan Dong, Xinghao Wang, Bohan Li, and Wanxiang Che. 2022.  Metaprompting: Learning to learn better prompts.  _arXiv preprint arXiv:2209.11486_.
  * Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021\.  Nearest neighbor machine translation.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 478–494, Online. Association for Computational Linguistics.
  * Kumar and Talukdar (2021) Sawan Kumar and Partha Talukdar. 2021.  Reordering examples helps during priming-based few-shot learning.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4507–4518.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2022a) Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2022a.  Probing via prompting.  _arXiv preprint arXiv:2207.01736_.
  * Li et al. (2022b) Junyi Li, Tianyi Tang, Jian-Yun Nie, Ji-Rong Wen, and Xin Zhao. 2022b.  Learning to transfer prompts for text generation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3506–3518, Seattle, United States. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2022a) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _arXiv preprint arXiv:2205.05638_.
  * Liu et al. (2022b) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lommel et al. (2014) Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014.  Multidimensional Quality Metrics (MQM) : A Framework for Declaring and Describing Translation Quality Metrics.  _Tradumàtica_ , pages 0455–463.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_.
  * Patel et al. (2022) Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. 2022.  Bidirectional language models are also few-shot learners.  _arXiv preprint arXiv:2209.14500_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–7.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235.
  * Singh et al. (2022) Chandan Singh, John X Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao. 2022\.  Explaining patterns in data with language models via interpretable autoprompting.  _arXiv preprint arXiv:2210.01848_.
  * Strobelt et al. (2022) Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022.  Interactive and visual prompt engineering for ad-hoc task adaptation with large language models.  _IEEE transactions on visualization and computer graphics_.
  * Tran et al. (2021) Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021.  Facebook AI’s WMT21 news translation task submission.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 205–215, Online. Association for Computational Linguistics.
  * Valvoda et al. (2022) Josef Valvoda, Yimai Fang, and David Vandyke. 2022.  Prompting for a conversation: How to control a dialog model?  _arXiv preprint arXiv:2209.11068_.
  * Wang et al. (2021) Longyue Wang, Mu Li, Fangxu Liu, Shuming Shi, Zhaopeng Tu, Xing Wang, Shuangzhi Wu, Jiali Zeng, and Wen Zhang. 2021.  Tencent translation system for the WMT21 news translation task.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 216–224, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zeng et al. (2021) Xianfeng Zeng, Yijin Liu, Ernan Li, Qiu Ran, Fandong Meng, Peng Li, Jinan Xu, and Jie Zhou. 2021.  WeChat neural machine translation systems for WMT21.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 243–254, Online. Association for Computational Linguistics.

## Appendices

##  Appendix A High-end pool

Table 13 describes the high-end pool. All listed articles were manually
downloaded in June–August 2022, and semi-automatically divided into bilingual
paragraphs. Our high-end pool consists of all paragraphs from all articles.

LP | paras | words | URL
---|---|---|---
en $\leftrightarrow$ de | 4 | 255 | www.deutschland.de/en/news/new-supercomputer-in-operation
4 | 208 | www.deutschland.de/en/news/patents-germany-ranks-second
11 | 609 | www.deutschland.de/en/news/syrian-swimmer-yusra-mardini-provides-message-of-hope-at-olympics
24 | 1787 | www.zeit.de/kultur/2019-12/schoenheit-fotografie-aesthetik-rankin-mitch-epstein-roger-ballen-english
28 | 2817 | www.zeit.de/kultur/2020-07/
|  | desinformation-peter-pomerantsev-social-media-regulation-democracy/komplettansicht
60 | 2961 | www.zeit.de/politik/ausland/2020-11/
|  |  | polarization-us-elections-democrats-republicans-donald-trump-family-division-english
| 21 | 2757 | www.zeit.de/politik/deutschland/2015-11/helmut-schmidt-obituary-english/komplettansicht
en $\leftrightarrow$ zh | 30 | 1323 | cn.nytimes.com/asia-pacific/20220509/taiwan-china-covid/dual
31 | 1317 | cn.nytimes.com/china/20220427/brownface-barrack-okarma-1968-hong-kong/dual
6 | 780 | cn.nytimes.com/china/20220401/china-cheng-lei-australia/dual
13 | 609 | cn.nytimes.com/china/20220421/china-eastern-crash-report/dual
23 | 1520 | cn.nytimes.com/china/20220412/china-russia-propaganda/dual
22 | 1373 | cn.nytimes.com/business/20220621/china-housing-real-estate-economy/dual
13 | 478 | cn.nytimes.com/china/20220415/shanghais-food-crisis-prompts-residents-in-beijing-to-stockpile-supplies/dual
26 | 1202 | cn.nytimes.com/obits/20220418/peng-ming-min-dead
6 | 843 | https://cn.nytimes.com/world/20220330/solomon-islands-china/dual
en $\leftrightarrow$ fr | 6 | 846 | france-amerique.com/a-france-of-many-colors
10 | 1177 | france-amerique.com/alice-guy-cinema-forgotten-pioneer
10 | 1237 | france-amerique.com/americanization-is-back-did-it-ever-go-away
8 | 666 | france-amerique.com/a-propos-a-hard-hitting-french-american-podcast
3 | 457 | france-amerique.com/camille-laurens-a-womans-life
8 | 970 | france-amerique.com/football-and-soccer
6 | 377 | france-amerique.com/france-united-states-naval-battle-and-diplomatic-crisis
7 | 615 | france-amerique.com/jeanne-damas-all-the-women-in-her-city
6 | 631 | france-amerique.com/guedelon-building-a-castle-by-hand
11 | 811 | france-amerique.com/raphael-francois-culinary-director
12 | 874 | france-amerique.com/thierry-mugler-provocateur
7 | 934 | france-amerique.com/winds-of-change-over-democracy
4 | 255 | www.deutschland.de/en/news/new-supercomputer-in-operation
Table 13: Sizes and provenance for articles in the high-end prompt pool. The
_words_ column contains the number of English words (whitespace-separated
character sequences) in each article.

##  Appendix B Variability of Random Runs

Table 14 shows the automatic scores for random runs for the
German$\rightarrow$English language pair. It can be observed that the range of
scores is quite small, less than 0.5 bleurt points for all language
directions. For both directions, the use of dev, as opposed to full, for the
random pool reduces the observed range in bleurt by at least 0.1.

|  | bleurt | bleu
---|---|---|---
LP | Pool | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 1 | Run 2 | Run 3 | Run 4 | Run 5
en $\rightarrow$ de | full | 71.9 | 71.9 | 71.6 | 71.8 | 71.9 | 32.4 | 32.8 | 32.1 | 32.9 | 32.9
dev | 74.7 | 74.7 | 74.7 | 74.9 | 74.8 | 32.7 | 32.6 | 32.6 | 32.6 | 32.8
de $\rightarrow$ en | full | 74.8 | 75.0 | 74.8 | 74.5 | 74.7 | 38.4 | 38.5 | 38.2 | 38.0 | 38.3
dev | 75.9 | 75.9 | 76.0 | 75.7 | 75.9 | 38.0 | 38.0 | 38.0 | 38.3 | 38.2
Table 14: Results for random runs for the German$\rightarrow$English
translation direction.

##  Appendix C Detailed mqm Scores

Table 15 presents mqm scores for PaLM WMT-dev random and SOTA systems in the
four language pairs evaluated, along with the breakdown of the scores into
their Accuracy and Fluency components. Table 16 presents detailed mqm error
counts for PaLM WMT-dev random and SOTA systems in en$\rightarrow$de and
de$\rightarrow$en.

| PaLM |  | SOTA
---|---|---|---
|  mqm $\downarrow$ | Accuracy$\downarrow$ | Fluency$\downarrow$ |  |  mqm $\downarrow$ | Accuracy$\downarrow$ | Fluency$\downarrow$
en $\rightarrow$ de | 1.58 | 1.12 | 0.46 |  | 1.18 | 0.81 | 0.37
en $\rightarrow$ zh | 3.24 | 2.69 | 0.52 |  | 2.47 | 1.96 | 0.48
de $\rightarrow$ en | 1.92 | 1.43 | 0.48 |  | 1.31 | 0.88 | 0.43
zh $\rightarrow$ en | - | - | - |  | - | - | -
Table 15: mqm scores for PaLM WMT-dev random and SOTA systems, split into Accuracy and Fluency. Accuracy scores include "Accuracy/*," "Terminology/*," and "Non-translation!" error categories. Fluency scores include "Fluency/*," "Style/*," and "Locale/*" categories. The "Other" and "Source error" categories are not included in Accuracy or Fluency scores. | en $\rightarrow$ de |  | de $\rightarrow$ en
---|---|---|---
| PaLM | SOTA |  | PaLM | SOTA
| Major | minor | Major | minor |  | Major | minor | Major | minor
Non-translation! | 0 | 0 | 0 | 0 |  | 2 | 0 | 0 | 0
Acc/Mistrans. | 103 | 89 | 79 | 67 |  | 73 | 41 | 61 | 49
Acc/Omission | 26 | 6 | 7 | 3 |  | 51 | 33 | 19 | 11
Acc/Addition | 1 | 6 | 3 | 1 |  | 10 | 2 | 0 | 3
Acc/Untranslated | 12 | 4 | 14 | 0 |  | 6 | 7 | 5 | 8
Ter/Inappr | 0 | 7 | 0 | 7 |  | 17 | 21 | 12 | 15
Ter/Incons | 0 | 4 | 0 | 4 |  | 1 | 5 | 1 | 7
Fl/Grammar | 0 | 133 | 0 | 100 |  | 18 | 41 | 5 | 38
Fl/Register | 0 | 2 | 0 | 3 |  | 0 | 0 | 0 | 0
Fl/Inconsistency | 0 | 2 | 0 | 5 |  | 0 | 2 | 0 | 2
Fl/Punctuation | 0 | 260 | 0 | 31 |  | 1 | 38 | 2 | 29
Fl/Spelling | 0 | 12 | 0 | 16 |  | 0 | 16 | 0 | 17
Fl/Encoding | 0 | 0 | 0 | 0 |  | 0 | 0 | 0 | 2
St/Awkward | 0 | 166 | 0 | 144 |  | 13 | 73 | 16 | 81
Locale/Date | 0 | 0 | 0 | 0 |  | 0 | 1 | 0 | 5
Locale/Name | 0 | 0 | 0 | 0 |  | 2 | 8 | 2 | 5
Locale/Time | 0 | 0 | 0 | 0 |  | 0 | 5 | 0 | 5
Source Error | 0 | 0 | 0 | 0 |  | 1 | 0 | 0 | 1
Other | 0 | 0 | 1 | 0 |  | 0 | 3 | 0 | 3
Total Errors | 142 | 673 | 102 | 362 |  | 189 | 296 | 123 | 281
Table 16: mqm error counts for PaLM WMT-dev random and SOTA systems for
en$\rightarrow$de and de$\rightarrow$en. Abbreviations are as follows: “Acc”:
Accuracy, “Fl”: Fluency, “St”: Style, “Ter”: Terminology, “Inappr”:
Inappropriate for context, “Incons”: Inconsistent.

##  Appendix D Significance numbers

We calculate pairwise significance numbers based on PERM-BOTH pair-wise
significance testing Deutsch et al. (2021). Results can be seen in Table 17.

|  | SOTA | GTrans. | WMT-dev | high-end | WMT-full
---|---|---|---|---|---|---
|  | random | random | random | kNN
de$\to$en | MQM | 1.31 | 1.71 | 1.92 | 1.89 | 2.38 | 3.03
SOTA | - | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Google Trans. | - | - | 0.073 | 0.124 | 0.0 | 0.0
WMT-dev random | - | - | - | 0.588 | 0.001 | 0.0
high-end random | - | - | - | - | 0.001 | 0.0
WMT-full random | - | - | - | - | - | 0.001
en$\to$de | MQM | 1.18 | 1.59 | 1.58 | 1.67 | 1.90 | 1.93
SOTA | - | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Google Trans. | - | - | 0.512 | 0.225 | 0.003 | 0.003
WMT-dev random | - | - | - | 0.175 | 0.001 | 0.0
high-end random | - | - | - | - | 0.021 | 0.01
WMT-full random | - | - | - | - | - | 0.372
zh$\to$en | MQM | 3.03 | 3.21 | 3.59 | 3.97 | 3.99 | 4.12
SOTA | - | 0.078 | 0.0 | 0.0 | 0.0 | 0.0
Google Trans. | - | - | 0.01 | 0.0 | 0.0 | 0.0
WMT-dev random | - | - | - | 0.002 | 0.003 | 0.001
high-end random | - | - | - | - | 0.441 | 0.22
WMT-full random | - | - | - | - | - | 0.241
en$\to$zh | MQM | 2.47 | 3.23 | 3.24 | 3.70 | 4.35 | 5.06
SOTA | - | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Google Trans. | - | - | 0.488 | 0.004 | 0.0 | 0.0
WMT-dev random | - | - | - | 0.002 | 0.0 | 0.0
high-end random | - | - | - | - | 0.0 | 0.0
WMT-full random | - | - | - | - | - | 0.0
Table 17: p-values based on PERM-BOTH pair-wise significance testing Deutsch
et al. (2021). We highlight all numbers with p<0.05.