# Automatic Correction of Human Translations

Jessy Lin♢ Geza Kovacs♣ Aditya Shastry♣
Joern Wuebker♣ John DeNero♢♣
♢ University of California, Berkeley ♣ Lilt
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce translation error correction (TEC), the task of automatically
correcting human-generated translations. Imperfections in machine translations
(MT) have long motivated systems for improving translations post-hoc with
automatic post-editing. In contrast, little attention has been devoted to the
problem of automatically correcting human translations, despite the intuition
that humans make distinct errors that machines would be well-suited to assist
with, from typos to inconsistencies in translation conventions. To investigate
this, we build and release the Aced corpus with three TEC datasets111Dataset
available at: https://github.com/lilt/tec. We show that human errors in TEC
exhibit a more diverse range of errors and far fewer translation fluency
errors than the MT errors in automatic post-editing datasets, suggesting the
need for dedicated TEC models that are specialized to correct human errors. We
show that pre-training instead on synthetic errors based on human errors
improves TEC F-score by as much as 5.1 points. We conducted a human-in-the-
loop user study with nine professional translation editors and found that the
assistance of our TEC system led them to produce significantly higher quality
revised translations.

##  1 Introduction

Error Type | Example Text
---|---
Monolingual: typos |  $s$: Do your feet roll inwards when running?
$t$: K ppen deine Füße beim Laufen nach innen?
$t^{\prime}$: K ppen deine Füße beim Laufen nach innen?
Monolingual: grammar |  $s$: Own tough winter runs in the …
$t$: Bei harten Winterläufe sorgt der …
$t^{\prime}$: Bei harten Winterläufe  sorgt der …
Monolingual: fluency |  $s$: The traffic emerges from the VPN server and …
$t$: Der  wird vom VPN-Server ausgegeben und …
$t^{\prime}$: Der  wird vom VPN-Server ausgegeben und …
Bilingual |  $s$: Quad Core XEON E3-1501M, 2.9GHz
$t$: Quad Core XEON 2,9 GHz
$t^{\prime}$: Quad Core XEON , 2,9 GHz
Preferential |  $s$: VersaMax I / O auxiliary spring clamp style
$t$: VersaMax Zusatz-E / A
$t^{\prime}$: VersaMax Zusatz-E / A
Table 1: Error taxonomy for the Aced corpus, with examples from the dataset.

Despite recent progress in machine translation (MT), a tremendous amount of
translated content in the world is still written by humans (DePalma, 2021).
Humans are often assumed to produce trusted, high-quality translations. In
reality, they do make errors, including spelling, grammar, and translation
errors (Hansen, 2009). This paper introduces the task of translation error
correction (TEC). Given a source sentence $s$ and a human-generated
translation $t$, the goal of TEC is to produce an improved translation
$t^{\prime}$ by correcting all errors in $t$.

“Translation correction” has long been studied in the MT community through the
task of automatic post-editing (APE), which aims to correct errors in machine-
generated translations (Simard et al., 2007). TEC is structurally identical to
APE. However, it requires modeling a different data distribution: errors made
by humans, which differ from those made by MT systems (Freitag et al., 2021).
We characterize the error distribution in TEC by building, analyzing, and
releasing the Aced corpus, a collection of three TEC datasets from varying
domains, with a total of 35,261 English–German translations produced and
corrected by professional translators in the natural course of their work.
While APE is dominated by the fluency errors that are characteristic of MT
systems (74% of sentences), our TEC corpus exhibits a broader distribution of
errors that human translators are prone to make.

Using this error analysis, we propose an approach for TEC that pre-trains on
synthetic corruptions more similar to errors made by humans, outperforming
models that were developed for the related tasks of MT, grammatical error
correction, and APE on all Aced datasets.

The task of TEC is often currently performed by humans, e.g. translators hired
to review and edit translations (“reviewers”). Can a TEC system help reviewers
edit faster, or produce higher quality final translations than they would have
without assistance? We ran a human-in-the-loop user study with nine
professional translators using our best-performing TEC model. We found that
the reviews produced when assisted with a TEC system were rated as higher
quality than those produced without, and produced with less manual effort.
Qualitatively, users commented that trust and consistency of the suggestions
were critical. They speculated future automated assistance could be helpful
for onboarding to new content, spotting technical errors, and improving their
own awareness of errors to catch.

Looking forward, a natural question arises of whether the research community
should focus on learning to revise _model_ outputs (APE) or _human_ outputs
(TEC). With recent improvements in MT, it has been increasingly difficult for
APE models to improve model output that is already high quality (Chollampatt
et al., 2020). On the other hand, we should expect that humans will continue
to make errors. TEC models will continue to provide benefit as a way of
assisting humans, whether for professional translators or everyday language
learners. TEC is synergistic with continuing advancements in MT: improved MT
will lead to improved error correction for human-generated translations. While
APE pits models against models, TEC is an opportunity to combine the best of
humans and models _because_ humans and models make different errors.

In sum, this paper revisits the notion of “translation correction” conceived
narrowly as the MT-centered task of APE, with an empirical investigation of
translation error correction (TEC), the task of learning to correct human
translations. Our contributions are:

  1. 1.

We release Aced, the first corpus for TEC containing three datasets of human
translations and revisions generated naturally from a commercial translation
workflow.

  2. 2.

We analyze the kinds of errors humans make in Aced, finding that while APE is
dominated by correcting translation _fluency_ , TEC focuses on correcting a
broader range of errors that appear in translation.

  3. 3.

We propose a pre-training approach for TEC that outperforms approaches
developed for similar tasks such as APE. Together, our results suggest the
need for distinct approaches to correct human translation errors.

  4. 4.

We perform a human-in-the-loop user study, finding that professional
translators produce higher quality translations when assisted by a TEC model.

##  2 The $\spadesuit$ Aced Corpus for TEC

Dataset | Domain | # sentences | % edited | # edits | Edit distance
---|---|---|---|---|---
train | dev | test | (mean) | (mean)
Asics | Marketing |  01395 |  0525 |  0616 | 29 % | 1.6 | 7.5
Emerson | Technical |  04287 | 1255 | 1662 | 20 % | 1.5 | 5.8
DigitalOcean (DO) | Technical | 11773 | 6104 | 7644 |  08 % | 1.5 | 7.1
Table 2: Corpus statistics for each dataset in the Aced corpus, including edit
statistics on % of sentences that edited, and for edited sentences, the
average number of edits and average edit distance.

Given a source language sentence $s$ and a human-generated translation $t$,
the goal of TEC is to produce a corrected target language sentence
$t^{\prime}$.

We introduce the Aced corpus, a set of three TEC datasets: Asics, Emerson, and
DigitalOcean (DO), each consisting of English–German sentence triples ($s$,
$t$, $t^{\prime}$) from varying domains.

Aced is a _real-world benchmark_ , containing naturalistic data from a task
humans perform, rather than manually annotated data. All translations were
created by professional translators working with Lilt, a localization services
provider. All translators have at least 5 years of professional translation
experience and experience working with the customer and domain. Each document
was translated from scratch (i.e. not post-edited) by a human translator using
an interactive neural MT system. Each translated document was then reviewed by
a reviewer, who Lilt selects as one of the more senior translators. As a
result, the examples in our corpus exhibit real errors that translators make,
and the corrected translations are publication quality.

Secondly, Aced is _diverse_ , with the three datasets from varying domains
exhibiting different error distributions and difficulty for initial work on
the TEC task. Information for each dataset is shown in Table 2. The Asics
dataset consists of marketing content with product names and descriptions for
an activewear company. Emerson consists of industrial product names for a
manufacturing company. DO consists of software engineering tutorials. The
various content types pose different challenges for translators and thus for
TEC systems, which we discuss in Section 2.1 and Section 2.2.

Duplicate sentences with the same source $s$ were removed. A portion of
sentences were rewritten by the reviewer rather than being edited (a relative
edit distance of more than 25% and a minimum of two edited words). We replace
$t$ in these sentences with $t^{\prime}$ in the corpus, so that training and
evaluation focuses on local edits rather than re-translations. The train, dev,
and test splits were constructed by splitting along document boundaries.

###  2.1 How do TEC and APE errors differ?

To understand how the human errors in the TEC task differ from model errors in
APE, we compare the types of errors in Aced with 100 randomly sampled errors
in the WMT 2021 APE shared task dev set 222https://www.statmt.org/wmt21/ape-
task.html, which we then annotate with error types. We define an error
taxonomy that classifies each edit as one of three types: (1) Monolingual
edits are identifiable from only the target-side text. We divide these further
into subcategories that highlight different capabilities needed to correct
edits: _typos_ (including spelling, punctuation, spacing, orthographic
issues), _grammar_ , and _fluency_ (awkward phrasing, word choice, or non-
native-sounding disfluencies); (2) Bilingual edits concern mismatches between
the source and target text, e.g. over- or under-translation, mis-translations;
(3) Preferential edits correct text that is inconsistent with the preferences
of the customer, as described in extralinguistic project requirements (e.g.
terminology or stylistic preferences). Examples of each error type are shown
in Table 1. Our error taxonomy closely mirrors those of previous analyses of
human translation errors (Specia and Shah, 2014; Yuan and Sharoff, 2020; Gupta
et al., 2021), and we confirm their findings that human translation errors
differ from MT errors. However, while previous work focuses on error
_detection_ and quality estimation, TEC is concerned with error _correction_.
Our error types are intended to isolate the capabilities that models need to
learn to correct edits (e.g., target-side language models can learn to correct
monolingual errors, but cannot do well on bilingual edits).

We annotate and release error labels for all test sentences in Asics to enable
its use as a diagnostic set for per-type evaluation of models. On the larger
Emerson and DO datasets, we randomly sample 50 errors to annotate for this
analysis. Error types were annotated by a professional German translator. Each
segment can have multiple error types. In Table 3, we report the percentage of
sentences with at least one error of each type. 74% of sentences in APE
exhibit a fluency error, in contrast to up to 22% of sentences in Aced, while
other types like monolingual grammar, bilingual, and preferential errors are
notably underrepresented in APE. We also note that all sentences in the APE
shared task are edited, while a key feature of TEC is identifying when a
sentence does _not_ need to be edited. The error distributions suggest that
different modeling techniques may shine: while APE challenges models to
correct disfluent translations characteristic of MT systems, our task is
designed to focus on identifying and correcting the typos, mismatches, and
grammatical errors more commonly exhibited by humans. Guided by this
observation, we describe an approach in Section 3 that pre-trains on synthetic
edits that are more representative of this error distribution.

###  2.2 How difficult is it to learn to edit?

| TEC | APE
---|---|---
| Asics | Emerson | DO | WMT ’21
Monolingual |  |  |  |
typos | 13 | 16 | 22 | 16
grammar | 41 |  04 |  02 |  06
fluency | 22 |  00 | 20 | 74
Bilingual | 22 | 70 | 32 |  05
Preferential |  07 | 24 | 40 |  06
Table 3: Percentages of erroneous sentences that contain at least one error of
each type for Asics, Emerson, DO, and the dev set of the WMT 2021 APE shared
task. As a task, APE exhibits many more fluency errors than TEC.

To quantify how difficult it may be to learn the correct edits in Aced, we
report statistics on _edit overlap_ : what proportion of edits that we expect
models to perform (e.g. adding a comma) appear exactly in the training set? We
use the errant toolkit to identify discrete edits Bryant et al. (2017). Each
edit is represented as a tuple (original span, replacement span), e.g. (“auf”,
“an”) to replace “auf” with “an.” Edit statistics are reported in Table 4: in
Asics and DO, $\scriptstyle\mathtt{\sim}$20% of the total number of edits in
dev and test appear in the training set, while Emerson has
$\scriptstyle\mathtt{\sim}$60% of dev and test edits appear in the training
set.

While the edit overlap rate provides a relative sense of scale for precision
and recall numbers, it does not provide an upper bound on recall. It is
possible to learn edits that do not exactly appear in the training set. For
example, capitalizing product names (“winterized” $\rightarrow$ “WINTERIZED”)
is a learnable pattern that would appear as many distinct edits. Additionally,
some errors can be corrected without fine-tuning because they are generic
typo, grammatical, fluency, or bilingual errors. Conversely, it is also
possible that it is wrong to make an edit that appears in the training set,
depending on the surrounding sentential context.

|  | Asics | Emerson | DO
---|---|---|---|---
Train | Total Edits | 606 | 1436 | 1212
Unique Edits | 418 |  0486 |  0940
Dev | Total Edits | 246 |  0381 | 1004
% in train |  014 |  0063 |  0021
Test | Total Edits | 287 |  0364 |  0766
% in train |  023 |  0060 |  0021
Table 4: Edit and edit overlap statistics for each Aced dataset: total number
of edits, unique edits in each train split, and percentage of total edits in
each dev and test split that appear in the train split.

##  3 Approaches to TEC

Figure 1: Overview of model architectures, pre-training data, and fine-tuning
data for each approach to TEC. Transformer encoders and decoders are depicted
as yellow and blue rectangles, respectively. The GEC and TEC models are pre-
trained with synthetic corruptions of $t^{\prime}$
($t^{\prime}_{\text{corrupted}}$), as detailed in the description of the TEC
and GEC models. APE uses MT-generated translations of $s$ ($t_{\text{MT}}$) as
synthetic data. BERT-APE is a state-of-the-art pre-trained APE model made
available by Correia and Martins (2019).

We propose a TEC model and compare it to several models designed for related
tasks to determine whether they are also effective for TEC. An overview of the
differences between the models is shown in Figure 1. All models use the
Transformer neural architecture (Vaswani et al., 2017) that generates the
target sequence $t^{\prime}$ from left to right. All are pre-trained on 36M
sentences from the WMT18 translation task333http://www.statmt.org/wmt18 and
fine-tuned on Aced, unless indicated otherwise. All pre-training and fine-
tuning data is pre-processed by normalizing punctuation with the Moses toolkit
Koehn et al. (2007).

All models have 6 encoder and decoder layers, model dimension of 256, feed-
forward dimension of 512, and 8 attention heads. We use a joint English–German
vocabulary with 33k byte pair encoding subwords (Sennrich et al., 2016).
During pre-training, we set the dropout to 0.1 and use the Adam optimizer
(Kingma and Ba, 2015) with a learning rate of $0.0002$. During fine-tuning, we
decrease the learning rate to $0.0001$ and reset the Adam momentum parameters.
We select the best fine-tuning checkpoint with edit-level F0.5 score on our
dev set. We use greedy inference.

###  3.1 Dual-Source Encoder-Decoder Model

We first describe the dual-source encoder-decoder we use for the APE and TEC
models. Formally, the original Transformer architecture (Vaswani et al., 2017)
takes a sequence of $J$ source tokens $\mathbf{s}_{1\dots J}$ and predicts a
sequence of $I^{\prime}$ target tokens $\mathbf{t^{\prime}}_{1\dots
I^{\prime}}$. We adapt the architecture to additionally encode the original
translation $t$, a sequence of $I$ tokens, $\mathbf{t}_{1\dots I}$. We
independently project $\mathbf{t}$ into the embedding space, add an offset
vector $\mathbf{o}$, and then concatenate the embedding with the embedding of
the source $\mathbf{s}$ to form the encoder input. To allow the dual-source
model to copy tokens from the original translation $t$, we implement the copy-
mechanism proposed by Zhao et al. (2019), which augments the model with an
additional encoder-decoder attention layer. An expanded description of the
model can be found in Appendix Section A.

###  3.2 Synthetic Data Generation

For the TEC and GEC model, we generate synthetic triples ($s$, $t$,
$t^{\prime}$) for pre-training. We generate a synthetic $t$ by corrupting the
German side of the translation data into $t^{\prime}_{\text{corrupted}}$. For
each sentence, we sample the probability of corruption
$p_{c}\sim\mathcal{N}(\mu=0.01,\sigma=0.04)$ clipped at $0$. On each character
and word in that sentence, with probability $p_{c}$, we randomly select one of
the following perturbations to apply at that position: insertion, deletion,
transposition, repetition.

###  3.3 TEC Models

| Asics | Emerson | DO
---|---|---|---
Model | Prec. | Rec. | F0.5 | Prec. | Rec. | F0.5 | Prec. | Rec. | F0.5
MT | 3.3 | 31.4 | 4.0 | 16.2 | 78.6 | 19.2 | 1.2 | 41.9 | 1.5
GEC | 52.8 | 6.6 | 22.0 | 78.0 | 53.3 | 71.4 | 20.5 | 2.0 | 7.1
APE | 51.2 | 7.3 | 23.3 | 78.1 | 54.4 | 71.8 | 14.4 | 1.7 | 5.8
BERT-APE | 6.8 | 10.8 | 7.3 | 32.0 | 57.8 | 35.1 | 2.3 | 3.8 | 2.5
TEC | 57.4 | 9.4 | 28.4 | 82.1 | 57.2 | 75.5 | 21.7 | 2.0 | 7.2
Table 5: Main results on Aced. Our fine-tuned TEC model outperforms on F0.5.
The fine-tuned MT model scores highest on recall because it makes many edits,
but at the cost of unacceptably low precision.

The five approaches we compare are:

##### TEC (this work)

We implement the dual-source encoder-decoder model that encodes two inputs
($s$, $t$) and outputs $t^{\prime}$, as described in Section 3.1, and then
pre-train on synthetic data from the procedure in Section 3.2. We then fine-
tune on Aced.

##### MT

We train an English-German neural machine translation model (with the standard
architecture described previously) and fine-tune it on ($s$, $t^{\prime}$)
Aced pairs, ignoring the original translation $t$.

##### GEC

We evaluate a encoder-decoder (monolingual) GEC model that takes an incorrect
German sentence $t$ as input and outputs a corrected $t^{\prime}$. We use the
same copy mechanism to attend to $t$ as our TEC model. To pre-train, we
perturb $t^{\prime}$ using the procedure described in Section 3.2, throwing
away the source side to obtain
$(t,t^{\prime})=(t^{\prime}_{\text{corrupted}},t^{\prime})$ pairs. We then
fine-tune on the Aced corpus, ignoring $s$.

##### APE

We implement a dual-source encoder-decoder model that is identical to our TEC
model. Following common practice in APE (Junczys-Dowmunt and Grundkiewicz,
2016; Negri et al., 2018), we pre-train on synthetic “post-editing” triples
($s$, $t$, $t^{\prime}$) where $t=t_{\text{MT}}$ is generated by translating
$s$ with an MT system. We split the training dataset into two parts, train an
MT model on each half, and use each model to translate the other half of the
dataset not seen during training. We then fine-tune on Aced.

##### BERT-APE

We also evaluate whether a state-of-the-art APE model can be directly applied
to our task. We evaluate the BERT-based encoder-decoder of Correia and Martins
(2019), on which the WMT 2019 shared task winner was based (Lopes et al.,
2019). Following Correia and Martins (2019), we fine-tuned on 23K
English–German SMT triplets from the WMT18 shared
task444http://www.statmt.org/wmt18/ape-task.html, with 12K train triplets from
2016 and 11k from 2017. We reproduce their results on the APE shared task test
sets, and continue fine-tuning this model on Aced. Following their paper, the
inputs are pre-processed by tokenizing and joining the two inputs with a
separator to form ($s$ [SEP] $t$, $t^{\prime}$) pairs.

##  4 Results & Discussion

The primary metric for TEC is MaxMatch scores (M2) (Dahlmeier and Ng, 2012)
computed with the errant toolkit (Bryant et al., 2017). M2 is a standard
metric for GEC that aligns $t$ and $t^{\prime}$ to extract discrete “edits.”
We choose to follow the GEC evaluation practice of up-weighting precision by
comparing F0.5, since the original translation is mostly correct: it is better
to suggest few correct edits than potentially introduce new errors.

Table 5 shows that TEC achieves the best overall F0.5 score on all datasets,
from $+0.1$ (on DO) up to $+5.1$ (on Asics) above the next-best model. Fine-
tuning on actual human corrections provides substantial gains; results without
fine-tuning can be found in Appendix Section B.

Both the MT model (which ignores $t$) and the GEC model (which ignores $s$)
underperform TEC. The MT model’s high edit recall can be attributed to the
fact that it proposes many edits, greatly trading off precision. Without
conditioning on $t$, direct MT translations of the source diverge from the
reference. The GEC model obtains high precision but underperforms on recall.
Conditioning on $s$ not only makes it possible to propose bilingual edits, but
also provides additional information to correct monolingual edits, as we show
in Section 4.1.

##### Can APE models be directly adapted for TEC?

Since TEC is structurally identical to APE, a natural question is whether
models that are trained on the APE objective can be directly adapted for TEC.
The APE and TEC models differ only in pre-training, but the performance
difference between them is substantial, indicating that the more GEC-like data
synthesis procedure is a better fit for TEC. Even more, the BERT-APE model,
which is first fine-tuned to achieve state-of-the-art on APE before fine-
tuning on Aced, achieves a particularly low F0.5 score because it makes too
many edits (low precision). Although future work may find insights in APE, our
results emphasize that models that excel at correcting machine errors cannot
be assumed to work well on human translations.

###  4.1 Fluency & Per-category Error Analysis

|  | Sentence-level Accuracy (%)
---|---|---
|  | Overall | Unedited | Mono. | Mono. | Mono. | Bilingual | Preferential
|  | Typos | Grammar | Fluency
Model | GLEU | /616 | /435 | /16 | /78 | /41 | /41 | /14
No-edit | 87.85 | 70.62 | (100) | - | - | - | - | -
MT | 44.79 | 31.66 | 40.00 |  00.00 | 11.54 | 2.44 | 24.39 | 14.29
GEC | 88.46 | 71.10 | 97.01 | 31.25 | 14.10 | 0.00 |  00.00 |  00.00
APE | 88.39 | 71.43 | 97.47 | 12.50 | 16.67 | 2.44 |  00.00 |  00.00
BERT-APE | 82.35 | 39.77 | 52.18 | 6.25 | 19.23 | 0.00 |  04.88 |  00.00
TEC (this work) | 88.81 | 71.92 | 97.01 | 25.00 | 17.95 | 0.00 |  07.32 |  00.00
Table 6: Additional analysis of n-gram overlap (via GLEU) and exact-match
sentence accuracy over all test sentences (Overall) and per error category for
the Asics dataset. Number of sentences with an error of each type are
indicated in gray, with some sentences containing errors from multiple
categories. For each error category, we report the percentage of those
sentences with at least one error of that type that a system predicted
exactly.

We perform a more in-depth comparison using alternative metrics on Asics,
which includes annotated error labels as a diagnostic tool. First, to
understand how much models are editing, we look at _n-gram overlap_ with the
GLEU metric (Napoles et al., 2015), a variant of BLEU used in GEC evaluation
to measure the fluency of holistic rewrites (Sakaguchi et al., 2016). Next, we
compare _sentence-level accuracy_ , which measures exact match with
$t^{\prime}$. We compute overall sentence-level accuracy, which includes
unedited sentences (which some models may incorrectly edit). We also report
accuracy per error type over (edited) sentences annotated with that error
type. These metrics need to be interpreted carefully: a no-edit baseline
achieves a GLEU score of $87.85$ (since original translations are mostly close
in edit distance to the final) and sentence-level accuracy of $70.62\%$ (the %
of unedited sentences), outperforming models like MT and BERT-APE that make
too many incorrect edits.

Our TEC model achieves the best score overall on both alternative metrics over
all sentences, but various models outperform on specific error types. The full
results are shown in Table 6. Examples of system outputs for different error
types can be found in Appendix Section C.

Notably, APE models lose the most accuracy relative to our model on
monolingual typo edits. This may be because neural MT decoders much less
frequently introduce target-side errors that would be similar to typos
(compared to the frequency of fluency errors). Still, the observation that
different models do well at different errors suggests that future work can
improve on TEC by leveraging the strengths of different models, e.g. using MT
models to propose alternative translations.

##  5 User Study: Assisting Professional Translators with TEC

Our automatic evaluation shows how our TEC model can outperform other baseline
systems, but we are ultimately interested in whether any TEC system is indeed
useful in practice. Presently, TEC is done manually by humans. To investigate
whether TEC systems can already be useful to humans—improving the quality,
speed, or ease of human review—we performed a human-in-the-loop user study
with our TEC model.

###  5.1 Methodology

We recruited 9 professional translators to serve as reviewers. None of them
had prior experience with Asics content. They were allowed to read and
reference the sentences in the Asics training set to familiarize themselves
with the content and preferred terminology. Then, they were each assigned to
review 74 sentences from the test set of Asics. Of the 74 sentences, our TEC
system predicted a suggested edit for 57 sentences, and for the remaining 17
sentences our TEC system did not predict any edits.

We opt for a within-subjects design to control for speed and experience
differences between reviewers. For each reviewer, the 74 sentences were
randomized such that half were in the “assisted condition” showing the TEC
suggestion if available for the sentence, and the other half were in the
“unassisted condition” where no TEC suggestion was shown.

Figure 2: The interface used to show suggestions to reviewers in our user
study.

The reviewing interface is shown in Figure 2. If the sentence has suggestions
available, the reviewer is asked to first accept or reject each of the
suggestions. Then, they are asked to make edits to the text until they are
satisfied with the translation. They then click a button to confirm their
translation and move to the next sentence.

During the review process, we track:

  1. 1.

Whether the TEC suggestion, if shown, was accepted or declined

  2. 2.

Total time spent reviewing each sentence

  3. 3.

Number of edit operations (insertions and deletions) the user made

  4. 4.

Levenshtein edit distance from the original text to the final text

| Review Time | Inserts | Levenshtein
---|---|---|---
Suggestions | (ms/char) | \+ Deletes | Distance
Hidden | 361 | 0.0625 | 0.0347
Shown (Overall) | 367 | 0 | 0.0185
Accepted | 328 | 0 | 0.0176
Declined | 841 | 0.0625 | 0.0426

Table 7: Length-normalized medians from our user study for review times,
number of characters the user inserted and deleted, and final Levenshtein edit
distances from the original to the final translations. Data is split based on
whether the suggestion was hidden or shown, and “Suggestion Shown” is further
broken down according to whether the user accepted or declined the suggestion.

Figure 3: Box plot showing quality rankings of segments reviewed with
suggestions hidden vs shown. Lower is better. Notches indicate median quality
rankings. Bars indicate the upper fence ($3^{rd}$ quartile + IQR*1.5).

Finally, to evaluate whether TEC has an effect on quality, we asked a 10th
translator to compare the quality of the reviewed sentences by ranking the 9
reviewed translations, with ties allowed. This translator was the translator
who had reviewed the reference translations in the corpus, as the explicit
goal is to ensure consistency with conventions in the training documents.

###  5.2 Results

In the user study, 79% of TEC suggestions were accepted. For the purpose of
analyzing effects of the TEC suggestions on time spent and translation
quality, we will focus on only the 255 sentences (across 9 reviewers) where a
TEC suggestion exists. Results are shown in Table 7. For all statistical
significance tests, we use the Mann-Whitney (MW) U-test for testing
statistical significance as all quantities are neither normally distributed
nor log-normal.

####  5.2.1 Effects of Suggestions on Time Spent During the Review Process

We first analyze how TEC suggestions influence the time spent reviewing a
sentence. We compare the time durations normalized by the length of the
sentence that needed to be reviewed (the length-normalized review time), as
longer sentences require more time to be read and reviewed.

There is no significant difference in length-normalized review time when the
suggestion is hidden vs. shown (MW $U=31654,p=0.460$). When suggestions are
shown, the length-normalized review time is significantly less on sentences
where reviewers accepted the suggestion, compared to sentences where they
declined (MW $U=3555,p<0.0005$).

A potential explanation for these results is that when reviewers are shown
incorrect TEC suggestions, they are distracted and slowed down, providing some
evidence that precision should indeed be emphasized in automatic evaluations
of TEC.

####  5.2.2 Effects of Suggestions on Edits Made During the Review Process

We also analyze the effects of suggestions on the editing effort, as measured
by the number of characters the reviewer had to insert and delete, as well as
how different the final reviewed sentences were from the original.

When suggestions are shown vs. hidden, there is a significant reduction in the
number of insertions+deletions (MW $U=41348.5,p<0.0001$). There is also a
significant reduction when a shown suggestion is accepted vs. declined (MW
$U=4007.5,p<0.005$). There is no significant difference in the Levenshtein
distance from the original translation to the final translation, between when
a suggestion is shown vs. hidden (MW $U=33750.0,p=0.611$), or between when a
shown suggestion is accepted vs. declined (MW $U=5485.0,p=0.783$).

Thus, the TEC system suggestions help to significantly reduce the amount of
manual typing that the user must perform.

####  5.2.3 Effects of Suggestions on Reviewed Sentence Quality

To assess the effects of TEC assistance on quality, we used the quality
rankings produced by the independent reviewer. Quality rankings were not
normally distributed, so we use the Mann-Whitney U-test for testing
statistical significance. A box plot of the quality rankings is shown in
Figure 3. The median quality ranking when the suggestion is shown is 1, vs. 2
when the suggestion is hidden. The quality ranking is significantly lower
(meaning quality is higher) when the suggestion is shown, vs. hidden (MW
$U=28738.0,p<0.01$).

This suggests that showing TEC suggestions may be helping reviewers correct
errors they may not have otherwise noticed, or help nudge them towards desired
corrections.

###  5.3 Qualitative Findings

We also conducted a post-study survey for reviewers to report qualitative
feedback. To understand common themes in the responses, we present all themes
that at least two reviewers mention in their commentary.

####  5.3.1 The Role of Reliability and Trust

Five reviewers commented that _reliability is critical_ : it was difficult to
trust the system when they noticed some suggestions were incorrect, or the
system did not reliably make an edit when applicable (e.g. always hyphenating
when appropriate):

> “Because I wasn’t sure I can trust the suggestions (because I saw several
> incorrect ones) so it took me longer to think/check whether the suggestion
> is right. And I have to read the entire sentence again anyway to check for
> other errors the suggestion didn’t catch…would only work if I knew 100% that
> the suggestions are always right”

These comments are in concordance with our quantitative findings. Perhaps
unlike other assistive applications, it is not enough to only have high
precision: if reviewers cannot trust that the system has caught most or all
errors, they will not save time as they still have to read the entire sentence
carefully. Conversely, a high-recall, low-precision system is not only
distracting, but also leads reviewers to be suspicious of whether suggestions
are correct in general. In general, future TEC systems must manage this
balance of precision and recall for user trust.

####  5.3.2 Use Cases for TEC

Many reviewers highlighted scenarios where trustworthy TEC systems could be
particularly useful.

Two reviewers said TEC is helpful for corrections and typos, similar to the
use cases for GEC in the wild (Omelianchuk et al., 2021):

> “If the tool would manage to reliably show missing punctuation marks, or
> numbers, or that the translation contains different numbers than the source,
> that would be helpful and save time.”; “recurring mistakes”

On the other hand, three reviewers mentioned that they hoped such a system
would make more substantial corrections in order to save a non-negligible
amount of time, although of course these edits may come at the expense of
precision:

> “There were not many suggestions, and they only offered small
> improvements…Not clear whether I would save time or not.”

Three reviewers commented that a TEC system could be a memory aid or
substitute for researching client-specific requirements, which is often an
intensive part of the production translation process. One reviewer pointed out
it could be particularly useful as an _instructive_ tool for translators who
are new to a client:

> “if I am new to an account and don’t yet know whether this client wants
> hyphens or not (always an issue with German). So usually I have to research…
> (or guess), but if the QA suggestions knew this client’s preference and
> would tell me, that would save me time.”

Finally, three reviewers commented that it could be useful as an attention-
directing tool by making them aware of what errors they might look out for,
especially in repetitive content where it may be easy to miss details:

> “makes you more sensitive for spotting similar errors”; “makes you aware of
> what kind of errors to look for in upcoming segments”; “maybe it helps with
> [repetitive sentences] that you would otherwise just quickly glance at.”

##  6 Conclusion & Future Work

We introduced the task of translation error correction (TEC) and released the
Aced corpus to study automatic correction of human translations, consisting of
three TEC datasets across varying domains. In our analysis of TEC data, we
showed how the errors that humans make differ from those made by MT systems,
suggesting that this task warrants different approaches from those previously
studied in the task of automatic post-editing. We confirm this empirically by
proposing a synthetic data generation procedure that more closely matches the
distribution of human translation errors and showing that our TEC model, pre-
trained on this data, consistently outperforms models developed for APE, as
well as those for MT and GEC. Finally, we showed how our TEC system is helpful
to real humans, assisting professional reviewers and leading them to produce
higher quality reviewed translations.

Future work may improve on our TEC system by investigating how to leverage the
strengths of recent MT systems (e.g. for initializing systems or proposing
edits) or developing more sophisticated synthetic data generation techniques
(e.g. using the source sentence or linguistic knowledge). Beyond our
benchmark, it would be interesting to apply TEC systems to other settings in
which human translation errors appear, e.g., to correct translations written
by language learners, denoise MT training sets, or clean up MT evaluation
sets.

From the perspective of human-AI interaction, TEC presents a real-world use
case and testbed to study how to assist experts with modern NLP systems,
hinting at the opportunity to combine the best of humans and machines.

## Acknowledgments

We thank Sai Gouravajhala, Yunsu Kim, Eric Wallace, and the other members of
the Lilt research team and Berkeley NLP group for helpful discussion and
feedback. We thank Morgan Raymond and Spence Green for their support in
releasing the dataset. Finally, we are grateful to the professional
translators who annotated the dataset and participated in the user study.

## References

  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Chollampatt et al. (2020) Shamil Chollampatt, Raymond Hendy Susanto, Liling Tan, and Ewa Szymanska. 2020.  Can automatic post-editing improve NMT? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2736–2746, Online. Association for Computational Linguistics.
  * Correia and Martins (2019) Gonçalo M. Correia and André F. T. Martins. 2019.  A simple and effective approach to automatic post-editing with transfer learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy. Association for Computational Linguistics.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572, Montréal, Canada. Association for Computational Linguistics.
  * DePalma (2021) Donald A. DePalma. 2021.  The language sector in eight charts.  Accessed: 2022-05-01.
  * Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.
  * Gupta et al. (2021) Prabhakar Gupta, Ridha Juneja, Anil Kumar Nelakanti, and Tamojit Chatterjee. 2021\.  Detecting over/under-translation errors for determining adequacy in human translations.  _ArXiv_ , abs/2104.00267.
  * Hansen (2009) Gyde Hansen. 2009.  A classification of errors in translation and revision.  In _CIUTI-Forum: Enhancing Translation Quality: Ways, Means, Methods._
  * Junczys-Dowmunt and Grundkiewicz (2016) Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2016.  Log-linear combinations of monolingual and bilingual neural machine translation models for automatic post-editing.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 751–758, Berlin, Germany. Association for Computational Linguistics.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018\.  Approaching neural grammatical error correction as a low-resource machine translation task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 595–606, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Lopes et al. (2019) António V. Lopes, M. Amin Farajian, Gonçalo M. Correia, Jonay Trénous, and André F. T. Martins. 2019.  Unbabel’s submission to the WMT2019 APE shared task: BERT-based encoder-decoder for automatic post-editing.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 118–123, Florence, Italy. Association for Computational Linguistics.
  * Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015.  Ground truth for grammatical error correction metrics.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 588–593, Beijing, China. Association for Computational Linguistics.
  * Negri et al. (2018) Matteo Negri, Marco Turchi, Rajen Chatterjee, and Nicola Bertoldi. 2018.  ESCAPE: a large-scale synthetic corpus for automatic post-editing.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Omelianchuk et al. (2021) Kostiantyn Omelianchuk, Vipul Raheja, and Oleksandr Skurzhanskyi. 2021.  Text Simplification by Tagging.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 11–25, Online. Association for Computational Linguistics.
  * Sakaguchi et al. (2016) Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016.  Reassessing the goals of grammatical error correction: Fluency instead of grammaticality.  _Transactions of the Association for Computational Linguistics_ , 4:169–182.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Simard et al. (2007) Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Kuhn. 2007.  Rule-based translation with statistical phrase-based post-editing.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 203–206, Prague, Czech Republic. Association for Computational Linguistics.
  * Specia and Shah (2014) Lucia Specia and Kashif Shah. 2014.  Predicting human translation quality.  In _Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track_ , pages 288–300, Vancouver, Canada. Association for Machine Translation in the Americas.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Yuan and Sharoff (2020) Yu Yuan and Serge Sharoff. 2020.  Sentence level human translation quality estimation with attention-based neural networks.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1858–1865, Marseille, France. European Language Resources Association.
  * Zenkel et al. (2019) Thomas Zenkel, Joern Wuebker, and John DeNero. 2019.  Adding interpretable attention to neural translation models improves word alignment.  _arXiv preprint arXiv:1901.11359_.
  * Zhao et al. (2019) Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019.  Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 156–165, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A Transformer Architecture Background and Model Description

###  A.1 Transformer Architecture

The neural models implemented in this work are based on the self-attentional
Transformer architecture (Vaswani et al., 2017). Formally, given a sequence of
source tokens (encoded as one-hot vectors) $\mathbf{s}_{1\dots
J}=(s_{1},\dots,s_{J})$, $s_{j}\in\mathbf{V}$, the goal is to predict a
sequence of target tokens $\mathbf{t^{\prime}}_{1\dots
I^{\prime}}=(t^{\prime}_{1},\dots,t^{\prime}_{I^{\prime}})$,
$t^{\prime}_{i}\in\mathbf{V}$, that is a translation of the source sequence,
where $\mathbf{V}$ is the vocabulary. The model has two main components, the
_encoder_ and the _decoder_. The encoder transforms the source sequence
$\mathbf{s}_{1\dots J}$ into a sequence of hidden states by first mapping each
individual token into a continuous embedding space, adding a positional
embedding and then processing it through a sequence of self-attention and
feed-forward layers:

$\displaystyle\mathbf{x}_{1\dots J}$
$\displaystyle=\mathbf{E}\mathbf{s}_{1\dots J}+\mathbf{p}_{1\dots J}$ (1)
$\displaystyle\mathbf{h}_{1\dots J}^{\text{enc}}$
$\displaystyle=\text{encoder}(\mathbf{x}_{1\dots J}),$ (2)

where $x_{j}\in\mathbf{V},j\in(1,\dots,J)$, $\mathbf{E}$ is the embedding
matrix for vocabulary $\mathbf{E}$ and $\mathbf{p}_{1\dots J}$ is the sequence
of positional embeddings described in Sec. 3.5 of (Vaswani et al., 2017). At a
given time step $i$, the decoder defines a probability distribution $P_{i}$
over all vocabulary items in $\mathbf{V}$:

$\displaystyle\mathbf{y^{\prime}}_{1\dots i-1}$
$\displaystyle=\mathbf{E}\mathbf{t^{\prime}}_{1\dots i-1}+\mathbf{p}_{1\dots
i-1}$ (3) $\displaystyle\mathbf{h}_{i}^{\text{dec}}$
$\displaystyle=\text{decoder}(\mathbf{y^{\prime}}_{1\dots
i-1},\mathbf{h}_{1\dots J}^{\text{enc}})$ (4) $\displaystyle
P_{i}(t^{\prime}_{i})$
$\displaystyle=\text{softmax}(\mathbf{h}_{i}^{\text{dec}}\mathbf{E}^{\top})$
(5)

where we assume a single shared vocabulary $\mathbf{V}$ and embedding matrix
$\mathbf{E}$. At training time we optimize the cross-entropy loss

$\displaystyle\mathcal{L}_{\text{CE}}(P)=-\sum_{i}\log(P_{i}(t^{\prime}_{i})).$
(6)

###  A.2 Dual-Source Encoder-Decoder Model

Given an additional input sequence $\mathbf{t}_{1\dots
I}=(t_{1},\dots,t_{I})$. the dual-source model used for the APE and TEC models
is implemented by independently projecting $\mathbf{t}_{1\dots I}$ into the
embedding space, adding an offset vector $\mathbf{o}$ and concatenating the
embedding sequences. Equations 2 and 4 are rewritten as

$\displaystyle\mathbf{y}_{1\dots I}$
$\displaystyle=\mathbf{E}\mathbf{t}_{1\dots I}+\mathbf{p}_{1\dots
I}+\mathbf{o}$ (7) $\displaystyle\mathbf{h}_{1\dots(J+I)}^{\text{enc}}$
$\displaystyle=\text{encoder}([\mathbf{x}_{1\dots J};\mathbf{y}_{1\dots I}])$
(8) $\displaystyle\mathbf{h}_{i}^{\text{dec}}$
$\displaystyle=\text{decoder}(\mathbf{y^{\prime}}_{1\dots
i-1},\mathbf{h}_{1\dots(J+I)}^{\text{enc}}),$ (9)

where $\mathbf{o}$ is a single learned vector that is broadcast to all
positions $i\in(1,\dots,I)$ and $[\cdot;\cdot]$ denotes the concatenate
operation.

###  A.3 Copy-Attention Mechanism

The new output probability distribution for the next target token
$P_{i}(t_{i}^{\prime})$ is a weighted sum of the probability of generating and
the probability of copying token $t_{i}^{\prime}$:

$\displaystyle\hat{P}_{i}(t^{\prime}_{i})=(1-\alpha_{i}^{\text{copy}})P_{i}(t^{\prime}_{i})+\alpha_{i}^{\text{copy}}P_{i}^{\text{copy}}(t^{\prime}_{i}),$
(10)

where the copy probabilities are calculated from the attention matrix of an
additional encoder-decoder attention layer that is added on top of the final
decoder layer, $\mathbf{A}_{i}$:

$\displaystyle
P_{i}^{\text{copy}}(t^{\prime}_{i})=\text{softmax}(\mathbf{A}_{i})$ (11)

The copy probability weight $\alpha_{i}^{\text{copy}}$ is determined with the
attention context vector $\mathbf{c}_{i}$, computed as a weighted sum of the
attention values (i.e. linearly transformed encoder states) where the weights
are defined by $\mathbf{A}_{i}$:

$\displaystyle\alpha_{i}^{\text{copy}}=\text{sigmoid}(\mathbf{W}^{\top}\mathbf{c}_{i}).$
(12)

This copy-attention layer applies a source-side mask so that it only attends
to the positions $(J+1,\dots,J+I)$ that correspond to the second input
sequence $\mathbf{t}_{1\dots I}$, and its implementation follows Zenkel et al.
(2019). In particular, it uses a single attention head, no skip connection,
and contains a separate output layer that predicts the target word based on
its context vector with probability distribution
$P_{i}^{\text{align}}(\cdot)$. At training time both output layers are
optimized jointly by defining the overall loss $\mathcal{L}$ as the weighted
sum of both cross-entropy losses:

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{CE}}(\hat{P})+\lambda\mathcal{L}_{\text{CE}}(P^{\text{align}})$
(13)

$\lambda$ is set to $0.05$ in all experiments. We further apply source-word
dropout (Junczys-Dowmunt et al., 2018), setting the full embedding vector for
words in $t$ to $1/p_{\text{src}}$ with probability $p_{\text{src}}=0.05$.

##  Appendix B Results without finetuning

See Table 8.

| Asics | Emerson | DO
---|---|---|---
Model | Prec. | Rec. | F0.5 | Prec. | Rec. | F0.5 | Prec. | Rec. | F0.5
MT | 0.8 | 12.2 | 1.0 | 1.6 | 16.9 | 2.0 | 0.5 | 23.4 | 0.6
GEC | 5.9 | 1.4 | 3.6 | 0.7 | 0.6 | 0.7 | 0.3 | 1.2 | 0.3
APE | 2.7 | 7.3 | 3.1 | 3.6 | 3.3 | 3.5 | 0.4 | 5.0 | 0.5
BERT-APE | 0.5 | 5.9 | 0.6 | 0.1 | 2.2 | 0.1 | 0.3 | 9.2 | 0.4
TEC | 41.7 | 1.7 | 7.5 | 12.2 | 4.2 | 8.8 | 5.3 | 2.5 | 4.3
Table 8: Results without finetuning.

##  Appendix C Examples of System Output

See Table 9, Table 10, Table 11.

Type: | Monolingual: technical
---|---
$s$: |  Do your feet roll inwards when running?
$t$: |  K ppen deine Füße beim Laufen nach innen ?
Reference $t^{\prime}$: |  K ppen deine Füße beim Laufen nach innen ?
Model | Predicted $t^{\prime}$
MT |  deine Füße beim Laufen nach innen ?
APE |  Correctly predicts $t^{\prime}$
BERT-APE |  No change to $t$
GEC |  Correctly predicts $t^{\prime}$
TEC (ours) |  Correctly predicts $t^{\prime}$
Table 9: A monolingual technical error the APE, GEC and TEC models edit correctly. Type: | Monolingual: technical
---|---
$s$: |  Run further than you thought possible in the GEL-NIMBUS ™ 21 LS running shoe for men by ASICS - packed full of plush cushioning and special technologies to respond to your natural stride.
$t$: |  Mit  dem ASICS GEL-NIMBUS ™ 21 LS Laufschuh für Herren mit weicher Dämpfung und speziellen Technologien , durch die sich der Schuh an deine natürlichen Schritte anpasst , läufst du weiter , als du es selbst je für möglich gehalten hast .
Reference $t^{\prime}$: |  Mit dem ASICS GEL-NIMBUS ™ 21 LS Laufschuh für Herren mit weicher Dämpfung und speziellen Technologien , durch die sich der Schuh an deine natürlichen Schritte anpasst , läufst du weiter , als du es selbst je für möglich gehalten hast .
Model | Predicted $t^{\prime}$
MT |  GEL-NIMBUS ™ 21 LS Laufschuh für Herren  mit weicher Dämpfung und speziellen Technologien ,
APE |  No change to $t$
BERT-APE |  Correctly predicts $t^{\prime}$
GEC |  Correctly predicts $t^{\prime}$
TEC (ours) |  Correctly predicts $t^{\prime}$
Table 10: A monolingual technical error the BERT-APE, GEC and TEC models edit correctly. Type: | Bilingual
---|---
$s$: |  The DUOMAX ™ midsole offers smooth overpronation control by combining two different density materials to reduce the risk of flat feet and bunions.
$t$: |  Die DUOMAX ™ -Mittelsohle bietet mühelos Halt bei Überpronation , indem zwei unterschiedliche Dichtematerialien kombiniert werden , um das Risiko von  und  zu verringern .
Reference $t^{\prime}$: |  Die DUOMAX ™ -Mittelsohle bietet mühelos Halt bei Überpronation , indem zwei unterschiedliche Dichtematerialien kombiniert werden , um das Risiko von  und  zu verringern .
Model |
MT |  Die DUOMAX ™ -Mittelsohle bietet  Überpronation , indem  zwei unterschiedliche Dichtematerialien kombiniert , um das Risiko von flachen Füßen und  zu verringern .
APE |  No change to $t$
BERT-APE |  Die DUOMAX ™ -Mittelsohle bietet mühelos Halt bei Überpronation , indem zwei unterschiedliche Dichtematerialien kombiniert werden , um das Risiko von flachen Füßen und  zu verringern .
GEC |  No change to $t$
TEC (ours) |  No change to $t$
Table 11: A bilingual error all models fail to edit correctly.

##  Appendix D Full User Study Results

|  | Suggestion
---
Hidden
| Suggestion
---
Shown
| Suggestion Shown
---
and Accepted
| Suggestion Shown
---
and Declined
Review Time (median) | 34.0475 sec | 33.103 sec | 26.606 sec | 50.524 sec
Review Time (length-norm, median) | 361 ms/char | 367 ms/char | 328 ms/char | 841 ms/char
Inserts (median) | 1.5 chars | 0 chars | 0 chars | 4 chars
Inserts (length-norm, median) | 0.0248 | 0 | 0 | 0.0345
Deletes (median) | 2 chars | 0 chars | 0 chars | 4 chars
Deletes (length-norm, median) | 0.0294 | 0 | 0 | 0.0375
Inserts+Deletes (median) | 4 chars | 0 chars | 0 chars | 9 chars
Inserts+Deletes (length-norm, median) | 0.0625 | 0 | 0 | 0.0625
Levenshtein Dist (median) | 2 chars | 1 char | 1 char | 6 chars
Levenshtein Dist (length-norm, median) | 0.0347 | 0.0185 | 0.0176 | 0.0426
Table 12: All data from our user study about review times, number of
characters the user inserted and deleted, and final Levenshtein distances from
the original. Data shown are medians (raw and length-normalized) across the
segments, based on whether the suggestion was hidden or shown. “Suggestion
Shown” is further broken down according to whether the user accepted or
declined the suggestion.

See Table 12.


# Role of filler lanthanide ions on lattice dynamics of phosphide
skutterudites RFe4P12 (R = La, Ce, and Pr) from first principles

Rafaela F. S. Penacchio  Institute of Physics, University of São Paulo, São
Paulo, SP, Brazil  Nicholas Burns  Department of Physics, University of
Guelph, Guelph, ON, Canada  Maurício B. Estradiote  Institute of Physics,
University of São Paulo, São Paulo, SP, Brazil  Milton S. Torikachvili
Department of Physics, San Diego State University, San Diego, CA, USA  Stefan
W. Kycia  Department of Physics, University of Guelph, Guelph, ON, Canada
Sérgio L. Morelhão<EMAIL_ADDRESS>Institute of Physics, University of São
Paulo, São Paulo, SP, Brazil

###### Abstract

Phosphide skutterudites primarily show promise for thermoelectric applications
due to their chemical stability at high temperatures and relatively low cost.
Ion doping and band gap engineering have been used to enhance their typically
poor thermoelectric performance, opening avenues for practical applications.
Herein, we report a comparative lattice dynamics study on the impact of filler
and temperature on the structural and vibrational properties of RFe4P12 (R =
La, Ce, and Pr) skutterudites. Calculations are performed within the quasi-
harmonic approximation, and the results are critically compared against
experimental data and other ab initio calculations. We found gaps between the
heat-carrying acoustic and optical modes, a-o gaps, of approximately 4, -2,
and 0.01 cm-1 for La, Ce, and Pr compounds, respectively. These results
suggest a filler-induced reduction in the a-o gap is attributed to the
softening of the optical modes instead of the conventionally considered upward
shift of acoustic modes proposed in the rattling scenario. The distinct
softening of the optical modes is rationalized by the stiffening of chemical
bonds between the filler and host lattice.

Thermoelectric materials are crucial for sustainable energy solutions as they
convert heat into electricity.Funahashi (2021) Their efficiency is measured by
the figure of merit $ZT$, inversely proportional to the lattice thermal
conductivity $\kappa$. However, many high-$ZT$ materials are either too
expensive or have limited stability at high temperatures.Popuri  _et al._
(2016) In this context, filled skutterudites (SKDs) R1-δT4X12 stand out due to
their chemical stability and unusual electrical and thermal transport
properties, achieving $ZT$ values as high as 1.8 at 823 K.Rogl  _et al._
(2015) Despite having $ZT\sim 10^{-2}$ at room temperature,Watcharapasorn  _et
al._ (2000a) phosphide SKDs (X = P) are particularly appealing as phosphorous
is highly abundant and relatively low-cost.Quinn and Bos (2023) Furthermore,
doping and band gap engineering can significantly improve their thermoelectric
performance, with a recent study predicting $ZT=0.82$ at 800 K for Nd-doped
CeFe4P12.Limbu  _et al._ (2022)

SKDs have a cubic structure (space group $Im\overline{3}$), with the host
lattice T4X12 forming a large icosahedral cage. The progressive filling of
this cage with electropositive elements R significantly reduces the lattice
thermal conductivity. For example, $\kappa$ drops over one order of magnitude
when filling the Co4Sb12 structure with Nd and by a factor of 6-7 when the
complete filling is achieved in the Ce1-δFe4Sb12 structure.Kuznetsov,
Kuznetsova, and Rowe (2003); Koza  _et al._ (2010) This suppression of
$\kappa$ was initially rationalized as the filler ion R rattling independently
in the cage, acting as individual, non-dispersive, low-energy oscillators and
giving rise to additional scattering channels for heat-carrying acoustic
phonons.Keppens  _et al._ (1998) Presently, it is well understood that the
vibrational properties of most filled SKDs subsystems arise from correlated
filler-cage dynamics, particularly true for antimonide SKDs (T = Sb).Koza  _et
al._ (2008, 2015); de Abrantes  _et al._ (2022)

Although phosphide SKDs RFe4P12 have been extensively studied due to
interesting phenomena such as superconductivity, antiferromagnetism, valence
fluctuation, and filler atom dependent metal-insulator
transitions,Torikachvili  _et al._ (1987) the role of filler ions in the
vibrational dynamics remains controversial. Experimental evidence supporting
the rattling scenario is largely based upon large atomic displacements
(B-factors) obtained from diffraction data,Grandjean  _et al._ (1984);
Jeitschko and Braun (1977); Keller  _et al._ (2001) Einstein temperatures
obtained from EXASF studies,Cao  _et al._ (2004) and frequencies from infrared
and optical reflectance spectroscopy.Dordevic  _et al._ (1999) Conversely,
other results support a more complex filler-cage interaction such as the
absence of second-order phonons in Raman spectroscopy measurements,Kojima  _et
al._ (2007); Ogita  _et al._ (2007) no site-dependence with the spin-lattice
relaxation rate in La-NMR and P-NMR/Sb-NQR measurements,Nakai  _et al._ (2007)
and relative atomic displacements from multiwavelength dynamical diffraction
data.Valério  _et al._ (2020) On top of this, most current first-principles
modeling on these materials has been performed at fixed temperatures, mainly
focusing on the 0 K equilibrium structure. Such modeling leaves room for more
comprehensive and systematic computational studies of the effect of
temperature on the properties of these materials.Hasegawa  _et al._ (2008); Xu
and Yi (2007); Hachemaoui  _et al._ (2010); Ameri  _et al._ (2014);
Abdelakader  _et al._ (2024) In this work, a comparative lattice-dynamics
study is carried out on the impact of both filler atom and temperature on the
structural and vibrational properties of RFe4P12 (R = La, Ce, and Pr) SKDs.
Among the main results are the distinct gaps between the acoustic- and
optical-phonon modes, the a-o gap, depending on the filler, as well as the
thermal expansion coefficients of the compounds, which were experimentally
verified in powder samples from 80 K to room temperature.

Density functional theory (DFT) calculations were performed using the Quantum
Espresso distributionGiannozzi  _et al._ (2009, 2017) at the generalized
gradient approximation (GGA) level. Modeling structural, vibrational, and
electronic properties were based on the Perdew-Burke-Ernzerhof (PBE) exchange-
correlation functional,Perdew, Burke, and Ernzerhof (1996) together with the
standard solid-state pseudopotentials (SSSP) optimized for precisionPrandini
_et al._ (2018). A kinetic energy cutoff of 1360 eV was sufficient in
producing convergence in the total energy up to 2.5 meV/atom for a given
k-point sampling. A $6\times 6\times 6$ $\Gamma$-centered Monkhorst-Pack mesh
was applied for structural calculations, in addition to a Gaussian smearing
width of 0.1 eV for La- and Pr-filled SKDs.

Lattice-dynamics calculations were carried out using Phonopy within the
finite-displacement supercell approach.Togo (2023) A $2\times 2\times 2$
supercell (containing 136 atoms) sampling only the $\Gamma$-point was
employed. During post-processing, thermodynamic properties were calculated by
sampling a $100\times 100\times 100$ $\Gamma$-centered $q$ mesh. The
temperature dependence of lattice constants, bulk moduli, and specific heat
were calculated by the quasi-harmonic approximation (QHA) as implemented in
Phonopy. For the QHA calculations, additional finite-displacement calculations
were performed on unit cells at approximately $\pm$2.5% of the equilibrium
volume in steps of 0.5%. After computing the thermal expansion, further
calculations were performed on structures with volumes corresponding to 4, 80,
180, 320, and 500 K.

X-ray scattering experiments were carried out at the Brockhouse Diffraction
Sector beamlines of the Canadian Light Source.Gomez, Dina, and Kycia (2018)
Resonant diffraction in single-crystals, near the Fe absorption edge,Penacchio
_et al._ (2023) at the wiggler low energy (WLE) beamline;Leontowich  _et al._
(2021) and total scattering as a function of temperature in powder samples,
with 55 keV X-rays, at the wiggler high energy (WHE) beamline. Specifics about
the total scattering data acquisition and processing are detailed in recent
work.Burns  _et al._ (2023) All RFe4P12 samples were originally obtained as
high-quality single-crystals synthesized by the tin-flux method.Torikachvili
_et al._ (1987) Temperature values were calibrated through a NIST nickel
powder sample, while the lattice constants were determined by Rietveld
analysis using the GSAS-II software package.Toby and Von Dreele (2013) In
single-crystals, the lattice constants were obtained by the azimuthal scan
method.Avanci  _et al._ (2004); Freitas  _et al._ (2007); de Menezes  _et al._
(2010); Valério  _et al._ (2020)

Figure 1:  DFT results (solid lines) as a function of temperature $T$ in LaFe4P12 (La), CeFe4P12 (Ce), and PrFe4P12 (Pr) SKDs. (a) Bulk modulus $B$. Additionally, few experimental values (symbols) available in the literature are also shown.Shirotani  _et al._ (2004); Hayashi  _et al._ (2010). (b) Lattice specific heat $C$, given in terms of the ratio $C/T^{3}$. (c) Relative lattice constant variation $\Delta a/a$ together with experimental values from X-ray scattering in powder (open symbols) and single-crystal (closed symbols) samples. Table 1: Lattice constants $a$, Einstein $\theta_{E}$ and Debye $\theta_{D}$ temperatures, as obtained from DFT calculations. Experimental values of $a$ are from X-ray scattering in powder and single-crystal (s.-c.) samples, while $\theta_{E}$ and $\theta_{D}$ are from literature. Uncertainties are presented in parentheses. | DFT | Experimental
---|---|---
|  $a$ (Å) |  |  |  $a$ (Å) ($\sim$ 300 K) |  |
SKD | 0 K | 296 K | $\theta_{E}({\rm K})$ | $\theta_{D}({\rm K})$ | powder Jeitschko and Braun (1977) | s.-c. | $\theta_{E}({\rm K})$ | $\theta_{D}({\rm K})$
LaFe4P12 | 7.8489 | 7.8587 | 145 | 662 | 7.8316(5) | 7.8338(14) | 147(1) Matsuhira  _et al._ (2009); Mizumaki  _et al._ (2011) | 670 Matsuhira  _et al._ (2009)
CeFe4P12 | 7.8095 | 7.8189 | 135 | 694 | 7.7920(10) | 7.7914(3) | 148(5) Cao  _et al._ (2004)/142(1) Mizumaki  _et al._ (2011) | —
PrFe4P12 | 7.8053 | 7.8149 | 134 | 678 | 7.8149(9) | 7.8116(7) | 137(1) Mizumaki  _et al._ (2011) | —

Main results from ab initio calculations as a function of temperature $T$ and
filler R are presented in Fig. 1. The bulk moduli $B$, Fig. 1(a), decreases by
about 5% between 0 and 500 K for all three SKDs. At 300 K, the calculated $B$
values for La-, Ce- and Pr-filled SKDs are 148, 153, and 152 GPa,
respectively. The former is in good agreement with the experimentally measured
value of 150 GPa,Hayashi  _et al._ (2010) while the value obtained for the Ce-
SKD deviates by about 5% from experiment.Shirotani  _et al._ (2004) For the Pr
compound, the experimental bulk modulus of 211 GPa at 77 K, not shown in Fig.
1(a), exceeds the calculated value by approximately 35%.Nakanishi  _et al._
(2001) Comparatively, calculations within local density approximation (LDA)
deviate from experimental values by 3 to 18%.

Calculated values of lattice specific heat $C$ are presented in Fig. 1(b).
Absence of the superconducting transition in La-SKD at $4.7$ K and magnetic
ordering in Pr-SKD below 6.2 K is a result of the limited capability of
standard DFT calculations at the GGA level in describing the strong electron
correlations involved in these transitions.Cohen, Mori-Sánchez, and Yang
(2012); Pavarini (2021) Peak positions in Fig. 1(b) were used to estimate the
Einstein temperatures $\theta_{E}$ of La-, Ce-, and Pr-filled SKDs, resulting
in 145, 135, and 134 K, respectively. These values are all within 10% of
experimental data,Matsuhira  _et al._ (2009); Cao  _et al._ (2004); Mizumaki
_et al._ (2011) as shown in Table 1. The $C(T)$ curve below 7.5 K is well-fit
by the Debye $T^{3}$ law, yielding Debye temperatures $\theta_{D}$ of 662,
694, and 678 K, respectively. For the La-SKD, the obtained result agrees well
with the experimentally measured value of 670 K available in the
literature;Matsuhira  _et al._ (2009) there is a lack of experimental data for
the other two SKDs. It should be noted that the Debye temperatures are
behaving differently than the expected trend $\theta_{D}\propto M^{-1/3}$ with
molar mass $M$.Poirier (1989) The La-, Pr-, and Ce-SKDs exhibit $\theta_{D}$
values in ascending order, suggesting a corresponding increase in chemical
bond strength and sound velocities.

Theoretical and experimental relative variations of lattice constants as a
function of $T$ are shown in Fig. 1(c). Table 1 reports the absolute values of
the lattice constants at 0 K and 296 K (room temperature), alongside with
additional theoretical and experimental values. At 296 K, the lattice
constants of La- and Ce-SKDs are 0.35% larger than the experimental
values,Jeitschko and Braun (1977) as expected for GGA functionals.Haas  _et
al._ (2009) Conversely, the lattice constant of Pr-SKD is in perfect agreement
with the literature values. The calculated values, in Table 1 (column 3), are
much closer to the experimentally measured values than other previous
calculations at LDA and LSDA (local spin density approximation) levels.Benalia
_et al._ (2008); Hachemaoui  _et al._ (2010) Both LDA and LSDA underestimated
0 K lattice constants by more than 1%. The QHA calculations effectively
reproduce the thermal expansion of La- and Ce-SKDs in the temperature range of
80 to 300 K, as seen in Fig. 1(c). Lattice constants from single crystal
samples at room temperature are also presented in Table 1 (column 7) for the
sake of comparison. Despite the $\Delta a/a(T)$ curves for Ce- and Pr-SKDs
being nearly coincident, more accurate experimental data points are needed to
confirm the latter. Nonetheless, at room temperature, the QHA calculations
reproduce the absolute values of the lattice constants of these filled SKDs
reasonably well.

Figure 2:  Vibrational properties obtained from ab initio lattice-dynamics
calculations in La-, Ce-, and Pr-filled SKDs. (a) Partial (colored lines with
shaded areas) and total (dashed lines) phonon DOS at T = 320 K, as indicated
on each panel. The a-o gap (arrows) at around 110 cm-1 is also indicated. (b)
Temperature dependence of the low-energy phonon branches around the a-o gap in
the range 4 K (purple) to 500 K (orange). (c) A-o gap as a function of
temperature.

The phonon density of states (DOS) at 320 K in Fig. 2(a) is representative,
within QHA, of the evaluated temperature range. Increasing temperature only
results in a downward shift of both acoustic and optical modes for all three
SKDs, as shown in Fig. 2(b); for additional information please refer to the
Supplementary Material (SM). In general, vibrational modes associated with the
fillers are located in the low-frequency range below 100 cm-1, contributing
predominantly to three acoustic branches, consistent with the scenario
reported for heavy-filler antimonide skutterudites.Koza  _et al._ (2010)
Regardless of the filler R, the DOS for P and Fe atoms are widely distributed
in the range of phonon frequencies above 100 cm-1, showing an overlap across
the entire spectrum due to the strong covalent bond between these atoms.
However, in La-SKD, P and Fe vibrations are almost totally decoupled from each
other at high frequencies, above 380 cm-1 as presented in Fig. 2(a) (top
panel). This result can be correlated to the smaller Debye temperature
obtained for this compound, Table 1 (column 5).

Although the filler R predominantly contributes to the acoustic modes, the
vibrational dynamics of the host lattice Fe4P12 depend upon the filler type,
as demonstrated in Fig. 2(a). For instance, the DOS at high frequencies is
associated with cage vibrational modes yet still undergoes significant changes
as a function of the filler, mainly for La with respect to Ce and Pr. Another
distinct filler-dependent feature is the softening of optical modes that
directly affects the gap between acoustic and optical modes, as highlighted in
Fig. 2(b) along the $\Gamma-N$ path. This softening is particularly pronounced
in the Ce-SKD, leading to a negative indirect a-o gap, where the lower energy
optical mode is less energetic than the higher energy acoustic mode. In
contrast, the weaker softening in La-SKD implies a positive gap, while the Pr-
SKD falls between the two, exhibiting a null gap. As a function of
temperature, the a-o gaps are nearly constant, which is observed in Fig. 2(c).
Null or negative gaps favor phonon scattering from heat-carrying acoustic to
optical branches, leading to a reduction in lattice thermal conductivity. This
result suggests that the Ce-filled SKD is more promising for thermoelectric
applications than isotypic La- or Pr-SKDs, as previously highlighted when
considering the transport properties of these materials Sato  _et al._ (2000);
Dordevic  _et al._ (1999); Watcharapasorn  _et al._ (2000a). For example, the
Ce-SKD is a semiconductor and its Seebeck coefficient has been observed to
increase with temperature, almost linearly from 58 $\mu$V/K at room
temperature to 128 $\mu$V/K at near 900 K.Watcharapasorn  _et al._ (2000b)

Distinct a-o gaps, as shown in Fig. 2(b,c), are caused by changes in force
constants of the filler-cage. La exhibits only the 3+ valency without
hybridization of $p-f$ orbitals, while Ce can manifest a mix of 3+ and 4+
valences, each associated with different chemical bonding.Grosvenor, Cavell,
and Mar (2006); Rayjada  _et al._ (2010) Similarly, Pr also presents mixed 3+
and 4+ valences with $p-f$ hybridization in both instances.Yamasaki  _et al._
(2005); Matsunami  _et al._ (2008) Further details on the electronic density
of states are provided in SM. It is worth mentioning that the hybridization of
$4f$-orbitals with the conduction band is also suggested to be responsible for
the unusual temperature dependence of the transport properties of the Ce-
SKD.Sato  _et al._ (2000)

Figure 3:  Atomic root-mean-square displacements (rms disp.) as a function of
temperature in filled SKDS: LaFe4P12 (circles), CeFe4P12 (triangles), and
PrFe4P12 (squares). Isotropic values for fillers (left panel) and host lattice
atoms, Fe (right top panel) and P (right bottom panel).

Lastly, Fig. 3 shows the calculated isotropic root-mean-square (rms)
displacement as a function of temperature. The values obtained for the filler
R ($=$ La, Ce, and Pr) at room temperature agree well with reported
experimental data from Rietveld refinement within a tolerance of 7%.Jeitschko
and Braun (1977); Grandjean  _et al._ (1984); Keller  _et al._ (2001) This
close correspondence was also verified for the rms displacements of Fe and P
atoms in La- and Ce-filled SKDs. Conversely, in the Pr-SKD, the calculated
values for Fe and P atoms exceed the experimental data by up to 28%. According
to the calculations presented here, Fe atoms always have the smaller rms
displacement regardless of filler and temperature. Moreover, cage P atoms have
thermal motion larger than the fillers at low temperatures below 140 K for the
La-SKD, and 86 K for both Ce- and Pr-SKDs. In other words, QHA calculations
predict that, at room temperature, the thermal motion of the filler ions has a
larger amplitude than that of the cage atoms, in agreement with experimental
data.

In summary, our comparative lattice-dynamics study of RFe4P12 (R = La, Ce, and
Pr) SKDs has effectively reproduced the thermal expansion of La and Ce
compounds. Also, it has predicted values for the bulk modulus at room
temperature, as well as the Debye and Einstein temperatures, with reasonable
accuracy. By considering the phonon frequencies and thermal displacements
within the quasi-harmonic approximation, we find out that the vibrational
dynamics of the host lattice are mainly affected by the chemical properties of
the filler ions. The filler-induced stiffening of chemical bonds, as evidenced
by an increase in Debye temperature, results in a downshift shift of the
energy of the optical modes with a consequent reduction in the a-o gap, which
contains the heat-carrying acoustic phonons. Such a trend is the opposite
situation to the expected rattling scenario, where phonon scattering is
favored by an increase in the energy of acoustic modes rather than the
softening of optical modes. Furthermore, the a-o gaps are considered to be
nearly constant as a function of temperature for all three compounds evaluated
here, suggesting that the reduction in lattice thermal conductivity through
phonon scattering is more pronounced in the Ce-filled SKD, which exhibits the
smallest gap. In terms of material design, this work provides fruitful
insights into strategies for improving the thermoelectric performance of
phosphide skutterudites.

## Supplementary Material

See the supplementary material for the temperature dependence of phonon
dispersion curves, electronic density of states at 320 K, experimental results
from X-ray total scattering in powder samples, and dynamical diffraction in
single crystals.

This work was supported by the Fundação de Amparo à Pesquisa do Estado de São
Paulo - FAPESP and by the National Council for Scientific and Technological
Development - CNPq.

###### Acknowledgements.

We acknowledge the Canadian Light Source, a National Research Facility of the
University of Saskatchewan, which is supported by the Canada Foundation for
Innovation (CFI), the Natural Sciences and Engineering Research Council
(NSERC), the National Research Council (NRC), the Canadian Institutes of
Health Research (CIHR) and the Government of Saskatchewan.

## Conflict of Interest

The authors have no conflicts to disclose.

## Data Availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Funahashi (2021) R. Funahashi,  _Thermoelectric energy conversion: theories and mechanisms, materials, devices, and applications_ (Woodhead Publishing, 2021).
  * Popuri  _et al._ (2016) S. R. Popuri, M. Pollet, R. Decourt, F. D. Morrison, N. S. Bennett, and J.-W. G. Bos, “Large thermoelectric power factors and impact of texturing on the thermal conductivity in polycrystalline SnSe,” Journal of Materials Chemistry C 4, 1685–1691 (2016).
  * Rogl  _et al._ (2015) G. Rogl, A. Grytsiv, K. Yubuta, S. Puchegger, E. Bauer, C. Raju, R. Mallik, and P. Rogl, “In-doped multifilled $n$-type skutterudites with $ZT=1.8$,” Acta Materialia 95, 201–211 (2015).
  * Watcharapasorn  _et al._ (2000a) A. Watcharapasorn, R. C. Demattei, R. S. Feigelson, T. Caillat, A. Borshchevsky, G. J. Snyder, and J.-P. Fleurial, “Thermoelectric properties of some phosphide skutterudite materials,” Japanese Journal of Applied Physics 39, 14 (2000a).
  * Quinn and Bos (2023) R. J. Quinn and J.-W. G. Bos, “Recent progress in phosphide materials for thermoelectric conversion,” Journal of Materials Chemistry A 11, 8453–8469 (2023).
  * Limbu  _et al._ (2022) N. Limbu, M. Ram, H. Joshi, A. Saxena, and A. Shankar, “Electronic and thermoelectric properties of Nd-doped Ce-filled skutterudites,” Physical Chemistry Chemical Physics 24, 4533–4546 (2022).
  * Kuznetsov, Kuznetsova, and Rowe (2003) V. Kuznetsov, L. Kuznetsova, and D. Rowe, “Effect of partial void filling on the transport properties of NdxCo4Sb12 skutterudites,” Journal of Physics: Condensed Matter 15, 5035 (2003).
  * Koza  _et al._ (2010) M. M. Koza, L. Capogna, A. Leithe-Jasper, H. Rosner, W. Schnelle, H. Mutka, M. R. Johnson, C. Ritter, and Y. Grin, “Vibrational dynamics of filled skutterudites ${M}_{1-x}{\text{Fe}}_{4}{\text{Sb}}_{12}$ ($M=\text{Ca}$, Sr, Ba, and Yb),” Phys. Rev. B 81, 174302 (2010).
  * Keppens  _et al._ (1998) V. Keppens, D. Mandrus, B. C. Sales, B. Chakoumakos, P. Dai, R. Coldea, M. Maple, D. Gajewski, E. Freeman, and S. Bennington, “Localized vibrational modes in metallic solids,” Nature 395, 876–878 (1998).
  * Koza  _et al._ (2008) M. M. Koza, M. R. Johnson, R. Viennois, H. Mutka, L. Girard, and D. Ravot, “Breakdown of phonon glass paradigm in La-and Ce-filled Fe4Sb12 skutterudites,” Nature materials 7, 805–810 (2008).
  * Koza  _et al._ (2015) M. M. Koza, M. Boehm, E. Sischka, W. Schnelle, H. Mutka, and A. Leithe-Jasper, “Low-energy phonon dispersion in LaFe4Sb12,” Physical Review B 91, 014305 (2015).
  * de Abrantes  _et al._ (2022) J. G. de Abrantes, M. R. Cantarino, W. R. da Silva Neto, V. V. Freire, A. G. Figueiredo, T. M. Germano, B. Mounssef, E. M. Bittar, A. Leithe-Jasper, and F. A. Garcia, “Vibrational and structural properties of the $R{\mathrm{Fe}}_{4}{\mathrm{Sb}}_{12}$ $(R=\mathrm{Na},\;\mathrm{K},\;\mathrm{Ca},\;\mathrm{Sr},\;\mathrm{Ba})$ filled skutterudites,” Phys. Rev. Mater. 6, 085403 (2022).
  * Torikachvili  _et al._ (1987) M. Torikachvili, J. Chen, Y. Dalichaouch, R. Guertin, M. McElfresh, C. Rossel, M. Maple, and G. Meisner, “Low-temperature properties of rare-earth and actinide iron phosphide compounds MFe4P12 (M= La, Pr, Nd, and Th),” Physical Review B 36, 8660 (1987).
  * Grandjean  _et al._ (1984) F. Grandjean, A. Gerard, D. Braung, and W. Jeitschko, “Some physical properties of LaFe4P12 type compounds,” Journal of Physics and Chemistry of Solids 45, 877–886 (1984).
  * Jeitschko and Braun (1977) W. Jeitschko and D. Braun, “LaFe4P12 with filled CoAs3-type structure and isotypic lanthanoid–transition metal polyphosphides,” Acta Cryst. B 33, 3401–3406 (1977).
  * Keller  _et al._ (2001) L. Keller, P. Fischer, T. Herrmannsdörfer, A. Dönni, H. Sugawara, T. Matsuda, K. Abe, Y. Aoki, and H. Sato, “Structural and magnetic properties of RFe4P12 (R= Pr, Nd) studied by neutron diffraction,” Journal of alloys and compounds 323, 516–519 (2001).
  * Cao  _et al._ (2004) D. Cao, F. Bridges, P. Chesler, S. Bushart, E. Bauer, and M. Maple, “Evidence for rattling behavior of the filler atom (L) in the filled skutterudites LT4X12 (L= Ce, Eu, Yb; T= Fe, Ru; X= P, Sb) from EXAFS studies,” Phys. Rev. B 70, 094109 (2004).
  * Dordevic  _et al._ (1999) S. Dordevic, N. Dilley, E. Bauer, D. Basov, M. Maple, and L. Degiorgi, “Optical properties of MFe4P12 filled skutterudites,” Physical Review B 60, 11321 (1999).
  * Kojima  _et al._ (2007) R. Kojima, N. Ogita, T. Hasegawa, Y. Takasu, M. Udagawa, H. Sugawara, and H. Sato, “Raman scattering investigation of Pr-based filled skutterudites,” in  _2007 26th International Conference on Thermoelectrics_ (IEEE, 2007) pp. 200–202.
  * Ogita  _et al._ (2007) N. Ogita, R. Kojima, T. Hasegawa, Y. Takasu, M. Udagawa, T. Kondo, N. Takeda, T. Ikeno, K. Ishikawa, H. Sugawara, _et al._ , “Raman scattering study of filled skutterudite compounds,” in  _Journal of Physics: Conference Series_, Vol. 92 (IOP Publishing, 2007) p. 012123.
  * Nakai  _et al._ (2007) Y. Nakai, K. Ishida, K. Magishi, H. Sugawara, D. Kikuchi, and H. Sato, “Rattling phonons in the filled skutterudite LaT4X12 T = Fe, Ru, Os; X = P, Sb) studied with La-NMR, P-NMR/Sb-NQR,” Journal of Magnetism and Magnetic Materials 310, 255–257 (2007), proceedings of the 17th International Conference on Magnetism.
  * Valério  _et al._ (2020) A. Valério, R. F. Penacchio, M. B. Estradiote, M. R. Cantarino, F. A. Garcia, S. L. Morelhão, N. Rafter, S. W. Kycia, G. A. Calligaris, and C. M. Remédios, “Phonon scattering mechanism in thermoelectric materials revised via resonant X-ray dynamical diffraction,” MRS Communications 10, 265–271 (2020).
  * Hasegawa  _et al._ (2008) T. Hasegawa, Y. Takasu, N. Ogita, and M. Udagawa, “First-principles calculation of lattice dynamics of skutterudite compounds,” Journal of the Physical Society of Japan 77, 248–250 (2008).
  * Xu and Yi (2007) B. Xu and L. Yi, “Optical properties of the filled skutterudite LaFe4P12,” Physica B: Condensed Matter 390, 147–150 (2007).
  * Hachemaoui  _et al._ (2010) M. Hachemaoui, R. Khenata, A. Bouhemadou, S. Bin-Omran, A. H. Reshak, F. Semari, and D. Rached, “Prediction study of the structural and elastic properties for the cubic skutterudites LaFe4A12 (A = P, As and Sb) under pressure effect,” Solid State Communications 150, 1869–1873 (2010).
  * Ameri  _et al._ (2014) M. Ameri, B. Abdelmounaim, M. Sebane, R. Khenata, D. Varshney, B. Bouhafs, Y. Al-Douri, K. Boudia, and I. Ameri, “First-principles investigation on structural, elastic, electronic and thermodynamic properties of filled skutterudite PrFe4P12 compound for thermoelectric applications,” Molecular Simulation 40, 1236–1243 (2014).
  * Abdelakader  _et al._ (2024) A. Abdelakader, B. Ahmed, M. Noureddine, B. Mokhtar, Z. Abdelhalim, M. Omar, B. Djillali, A. Yahia, and Y. Al-Douri, “Theoretical investigations of electronic, thermodynamic and thermoelectric properties of filled skutterudites ThFe4P12 and CeFe4P12 using DFT calculations,” Solid State Communications 380, 115435 (2024).
  * Giannozzi  _et al._ (2009) P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. D. Corso, S. de Gironcoli, S. Fabris, G. Fratesi, R. Gebauer, U. Gerstmann, C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari, F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto, C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov, P. Umari, and R. M. Wentzcovitch, “QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials,” Journal of Physics: Condensed Matter 21, 395502 (2009).
  * Giannozzi  _et al._ (2017) P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. B. Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, N. Colonna, I. Carnimeo, A. D. Corso, S. de Gironcoli, P. Delugas, R. A. DiStasio, A. Ferretti, A. Floris, G. Fratesi, G. Fugallo, R. Gebauer, U. Gerstmann, F. Giustino, T. Gorni, J. Jia, M. Kawamura, H.-Y. Ko, A. Kokalj, E. Küçükbenli, M. Lazzeri, M. Marsili, N. Marzari, F. Mauri, N. L. Nguyen, H.-V. Nguyen, A. O. de-la Roza, L. Paulatto, S. Poncé, D. Rocca, R. Sabatini, B. Santra, M. Schlipf, A. P. Seitsonen, A. Smogunov, I. Timrov, T. Thonhauser, P. Umari, N. Vast, X. Wu, and S. Baroni, “Advanced capabilities for materials modelling with Quantum ESPRESSO,” Journal of Physics: Condensed Matter 29, 465901 (2017).
  * Perdew, Burke, and Ernzerhof (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, “Generalized gradient approximation made simple,” Phys. Rev. Lett. 77, 3865–3868 (1996).
  * Prandini  _et al._ (2018) G. Prandini, A. Marrazzo, I. E. Castelli, N. Mounet, and N. Marzari, “Precision and efficiency in solid-state pseudopotential calculations,” npj Computational Materials 4, 72 (2018).
  * Togo (2023) A. Togo, “First-principles phonon calculations with Phonopy and Phono3py,” Journal of the Physical Society of Japan 92, 012001 (2023).
  * Gomez, Dina, and Kycia (2018) A. Gomez, G. Dina, and S. Kycia, “The high-energy X-ray diffraction and scattering beamline at the Canadian Light Source,” Review of Scientific Instruments 89, 063301 (2018).
  * Penacchio  _et al._ (2023) R. F. S. Penacchio, M. B. Estradiote, C. M. R. Remédios, G. A. Calligaris, M. S. Torikachvili, S. W. Kycia, and S. L. Morelhão, “Introduction to Python Dynamic Diffraction Toolkit (PyDDT): structural refinement of single crystals via X-ray phase measurements,” Journal of Applied Crystallography 56, 1574–1584 (2023).
  * Leontowich  _et al._ (2021) A. F. Leontowich, A. Gomez, B. Diaz Moreno, D. Muir, D. Spasyuk, G. King, J. W. Reid, C.-Y. Kim, and S. Kycia, “The lower energy diffraction and scattering side-bounce beamline for materials science at the canadian light source,” Journal of Synchrotron Radiation 28, 961–969 (2021).
  * Burns  _et al._ (2023) N. Burns, A. Rahemtulla, S. Annett, B. Moreno, and S. Kycia, “An inclined detector geometry for improved X-ray total scattering measurements,” Journal of Applied Crystallography 56, 510–518 (2023).
  * Toby and Von Dreele (2013) B. H. Toby and R. B. Von Dreele, “GSAS-II: the genesis of a modern open-source all purpose crystallography software package,” Journal of Applied Crystallography 46, 544–549 (2013).
  * Avanci  _et al._ (2004) L. H. Avanci, A. A. Quivy, S. L. Morelhão, and C. M. R. Remédios, “On the high accuracy lattice-parameters determination by $n$-beam diffraction: Theory and application to the InAs quantum dots grown over GaAs(001) substrate system,”  (2004), arXiv:cond-mat/0411508 [cond-mat.mtrl-sci] .
  * Freitas  _et al._ (2007) R. O. Freitas, T. E. Lamas, A. A. Quivy, and S. L. Morelhão, “Synchrotron X-ray Renninger scanning for studying strain in InAs/GaAs quantum dot system,” Phys. Status Solidi A 204, 2548–2554 (2007).
  * de Menezes  _et al._ (2010) A. S. de Menezes, A. O. dos Santos, J. M. A. Almeida, J. R. R. Bortoleto, M. A. Cotta, S. L. Morelhão, and L. P. Cardoso, “Direct observation of tetragonal distortion in epitaxial structures through secondary peak split in a synchrotron radiation Renninger scan,” Cryst. Growth Des. 10, 3436–3441 (2010).
  * Shirotani  _et al._ (2004) I. Shirotani, T. Noro, J. Hayashi, C. Sekine, R. Giri, and T. Kikegawa, “X-ray study with synchrotron radiation for P- and Sb-based skutterudite compounds at high pressures,” Journal of Physics: Condensed Matter 16, 7853 (2004).
  * Hayashi  _et al._ (2010) J. Hayashi, K. Akahira, K. Matsui, H. Ando, Y. Sugiuchi, K. Takeda, C. Sekine, I. Shirotani, and T. Yagi, “Bulk moduli of superconducting skutterudites YT4P12 (T = Fe, Ru and Os),” Journal of Physics: Conference Series 215, 012142 (2010).
  * Matsuhira  _et al._ (2009) K. Matsuhira, C. Sekine, M. Wakeshima, Y. Hinatsu, T. Namiki, K. Takeda, I. Shirotani, H. Sugawara, D. Kikuchi, and H. Sato, “Systematic study of lattice specific heat of filled skutterudites,” J. Phys. Soc. Jpn 78, 124601 (2009).
  * Mizumaki  _et al._ (2011) M. Mizumaki, S. Tsutsui, T. Uruga, H. Tanida, D. Kikuchi, H. Sugawara, and H. Sato, “Rare earth dependence of einstein temperatures in filled skutterudite compounds REFe4P12 (RE= La, Ce, Pr, Nd, and Sm),” J. Phys. Soc. Jpn 80, 074603 (2011).
  * Nakanishi  _et al._ (2001) Y. Nakanishi, T. Simizu, M. Yoshizawa, T. Matsuda, H. Sugawara, and H. Sato, “Elastic constants of ${\mathrm{PrFe}}_{4}{\mathrm{P}}_{12}$ in magnetic fields,” Phys. Rev. B 63, 184429 (2001).
  * Cohen, Mori-Sánchez, and Yang (2012) A. J. Cohen, P. Mori-Sánchez, and W. Yang, “Challenges for density functional theory,” Chemical reviews 112, 289–320 (2012).
  * Pavarini (2021) E. Pavarini, “Solving the strong-correlation problem in materials,” La Rivista del Nuovo Cimento 44, 597–640 (2021).
  * Poirier (1989) J. P. Poirier, “Lindemann law and the melting temperature of perovskites,” Physics of the earth and planetary interiors 54, 364–369 (1989).
  * Haas  _et al._ (2009) P. Haas, F. Tran, P. Blaha, K. Schwarz, and R. Laskowski, “Insight into the performance of GGA functionals for solid-state calculations,” Phys. Rev. B 80, 195109 (2009).
  * Benalia  _et al._ (2008) S. Benalia, M. Ameri, D. Rached, R. Khenata, M. Rabah, and A. Bouhemadou, “First-principle calculations of elastic and electronic properties of the filled skutterudite CeFe4P12,” Computational materials science 43, 1022–1026 (2008).
  * Sato  _et al._ (2000) H. Sato, Y. Abe, H. Okada, T. D. Matsuda, K. Abe, H. Sugawara, and Y. Aoki, “Anomalous transport properties of $R{\mathrm{Fe}}_{4}{\mathrm{P}}_{12}$ (R= La, Ce, Pr, and Nd),” Phys. Rev. B 62, 15125–15130 (2000).
  * Watcharapasorn  _et al._ (2000b) A. Watcharapasorn, R. C. Demattei, R. S. Feigelson, T. Caillat, A. Borshchevsky, G. J. Snyder, and J.-P. Fleurial, “Thermoelectric properties of some phosphide skutterudite materials,” Jpn. J. Appl. Phys. 39, 14–18 (2000b).
  * Grosvenor, Cavell, and Mar (2006) A. P. Grosvenor, R. G. Cavell, and A. Mar, “X-ray photoelectron spectroscopy study of rare-earth filled skutterudites LaFe4P12 and CeFe4P12,” Chemistry of materials 18, 1650–1657 (2006).
  * Rayjada  _et al._ (2010) P. Rayjada, A. Chainani, M. Matsunami, M. Taguchi, S. Tsuda, T. Yokoya, S. Shin, H. Sugawara, and H. Sato, “Kondo scaling of the pseudogap in CeOs4Sb12 and CeFe4P12,” Journal of Physics: Condensed Matter 22, 095502 (2010).
  * Yamasaki  _et al._ (2005) A. Yamasaki, S. Imada, A. Sekiyama, M. Tsunekawa, C. Dallera, L. Braicovich, T.-L. Lee, H. Sugawara, H. Sato, R. Settai, _et al._ , “Bulk-sensitive photoemission spectroscopy for heavy-fermion Pr compounds using hard X-ray,” Journal of the Physical Society of Japan 74, 2045–2051 (2005).
  * Matsunami  _et al._ (2008) M. Matsunami, K. Horiba, M. Taguchi, K. Yamamoto, A. Chainani, Y. Takata, Y. Senba, H. Ohashi, M. Yabashi, K. Tamasaku, _et al._ , “Electronic structure of semiconducting CeFe4P12: Strong hybridization and relevance of single-impurity Anderson model,” Physical Review B 77, 165126 (2008).