# Predicting Human Translation Difficulty with Neural Machine Translation

Zheng Wei Lim, Ekaterina Vylomova, Charles Kemp,  Trevor Cohn
The University of Melbourne
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Now at Google DeepMind

###### Abstract

Human translators linger on some words and phrases more than others, and
predicting this variation is a step towards explaining the underlying
cognitive processes. Using data from the CRITT Translation Process Research
Database, we evaluate the extent to which surprisal and attentional features
derived from a Neural Machine Translation (NMT) model account for reading and
production times of human translators. We find that surprisal and attention
are complementary predictors of translation difficulty, and that surprisal
derived from a NMT model is the single most successful predictor of production
duration. Our analyses draw on data from hundreds of translators operating
across 13 language pairs, and represent the most comprehensive investigation
of human translation difficulty to date.

##  1 Introduction

During the Nuremberg trials, a Soviet interpreter paused and lost her train of
thought when faced with the challenge of translating the phrase “Trojan Horse
politics”, and the presiding judge had to stop the session Matasov (2017).
Translation difficulty rarely has such extreme consequences, but the process
of translating any text requires a human translator to handle words and
phrases that vary in difficulty. Translation difficulty can be operationalized
in various ways Sun (2015), and one approach considers texts to be difficult
if they trigger translation errors Vanroy et al. (2019). Here, however, we
focus on difficulty in cognitive processing, and consider a word or phrase to
be difficult if it requires extended processing time. Figure 1 illustrates how
this notion of translation difficulty varies at the level of individual words.
Across this sample, words like “societies” and “population” are consistently
linked with longer production times than words like “result” and “tend”.

Processing times have been extensively studied by psycholinguists, but the
majority of this work is carried out in a monolingual setting. Within the
literature on translation, analysis of cognitive processing is most prominent
within a small but growing area known as Translation Process Research (TPR).
Researchers in this area aim to characterize the cognitive processes and
strategies that support human translation, and do so by analyzing eye
movements and keystrokes collected from translators Carl (2012). Here we build
on this tradition and focus on three variables from the CRITT TPR-DB database
Carl et al. (2016b): source reading time (TrtS), target reading time (TrtT),
and translation duration (Dur). Our analyses are relatively large in scale by
the standards of previous work in this area, and we draw on data from 312
translators working across 13 language pairs.

A central goal of our work is to bring translation process research into
contact with modern work on Neural Machine Translation (NMT). Recent work in
NLP has led to dramatic improvements in the multilingual abilities of NMT
models Kudugunta et al. (2019); Aharoni et al. (2019), and these models can
support tests of existing psycholinguistic theories and inspire new theories.
Our work demonstrates the promise of NMT models for TPR research by testing
whether surprisal and attentional features derived from an NMT model are
predictive of human translation difficulty. Two of these predictors are shown
in the right panels of Figure 1, and both are correlated with translation
duration for the example sentence shown.

In what follows we introduce the surprisal and attentional features that we
consider then evaluate the extent to which they yield improvements over
baseline models of translation difficulty. We find that surprisal is a strong
predictor of difficulty, which supports and extends previous psycholinguistic
findings that surprisal predicts both monolingual processing Levy (2008);
Wilcox et al. (2023) and translation processing Teich et al. (2020); Wei
(2022); Carl (2021a). The attentional features we evaluate predict difficulty
less well, but provide supplementary predictive power when combined with
surprisal.

Figure 1: Translation surprisal and cross-attention entropy are predictive of
average word translation duration, which is the time taken (in ms, log scale)
to produce an aligned segment on the target side, divided by the number of
source words aligned with this segment. Error bars show 95% confidence
intervals over 47 translations in five target languages: Chinese, Japanese,
Hindi, German and Spanish. ‘*’ marks significant Pearson correlations
($p<.05$) of predictors with translation duration (leftmost panel).

##  2 Related work

An extensive body of monolingual work has demonstrated that the pace of human
reading is based on next-word predictability — more contextually surprising
words incur higher cognitive costs and are slower to process Hale (2001); Levy
(2008). The phenomenon is observed across a wide range of language usage,
including reading Monsalve et al. (2012); Smith and Levy (2013); Meister et
al. (2021); Shain et al. (2022); Wilcox et al. (2021, 2023), listening and
comprehension Russo et al. (2020, 2022); Kumar et al. ; Yu et al. (2023),
speech Jurafsky (2003); Levy and Jaeger (2006); Cohen-Priva and Jurafsky
(2008); Demberg et al. (2012); Malisz et al. (2018); Dammalapati et al.
(2019); Pimentel et al. (2021), typing Chen et al. (2021) and code-switching
Calvillo et al. (2020). Surprisal has also been proposed as a predictor of
translation difficulty, but existing evaluations of this proposal are
relatively limited Teich et al. (2020); Carl (2021a); Wei (2022); Deilen et
al. (2023).

Along with surprisal, contextual entropy has also been proposed as a predictor
of translation difficulty Teich et al. (2020); Carl (2021a); Wei (2022).
Entropy is the expected surprisal of a translation distribution, and indicates
the effort in resolving the uncertainty over all possible translation choices.
However, the results of Wei (2022) and Carl (2021a) suggest entropy is a
weaker predictor of translation duration relative to surprisal. Entropy also
shows limited evidence in predicting monolingual reading difficulty, despite
being hypothesized to indicate a reader’s anticipation of processing effort
Hale (2003); Linzen and Jaeger (2016); Lowder et al. (2018); Wilcox et al.
(2023).

Previous studies have also explored whether attentional weights in language
models can account for monolingual processing time Ryu and Lewis (2021); Oh
and Schuler (2022). Other NMT studies contend that the contextualisation of
difficult tokens occurs in encoder self-attention Tang et al. (2019); Yin et
al. (2021). NMT models also exhibit higher cross-attention entropy with
increasingly difficult tokens Dabre and Fujita (2019); Zhang and Feng (2021);
Lu et al. (2021). Further, NMT’s non-compositional processing of figurative
expressions leads to paraphrasing, as opposed to literal translation Dankers
et al. (2022).

Building on previous work, we focus on surprisal rather than entropy, and
provide a comprehensive evaluation of the extent to which surprisal predicts
translation difficulty. While we do not expect NMT difficulty to align
perfectly with human difficulty, we propose several attentional features based
on previous literature and test whether they contribute predictive power that
goes beyond surprisal alone.111Code will be made available upon publication.

##  3 Surprisal: An information-theoretic account of translation processing

In contrast with traditional monolingual surprisal, translation surprisal is
based on a context that includes a complete sequence in a source language (SL)
and a sequence of previously translated words in a target language (TL).
Target words with high translation surprisal are hypothesized to require
extended cognitive processing because they are relatively unpredictable in
context.

Relative to prior work, we aim to provide a more comprehensive and rigorous
evaluation of the role of surprisal in translation processing. We consider
predictors of translation difficulty based on both monolingual and translation
surprisal, which we estimate using a large language model and a neural
translation model respectively.

###  3.1 Monolingual surprisal

The monolingual surprisal of a word or segment can be estimated from an
autoregressive language model (LM). Let $\bm{w}=[w_{1},...,w_{m}]$ be a
complete sequence of tokens (e.g., a full sentence), and let $\bm{w}_{\bm{i}}$
denote a segment of $\bm{w}$ where $\bm{i}\subseteq\\{1,...,m\\}$ denotes the
token indices. The surprisal of $\bm{w}_{\bm{i}}$ is the sum of negative log-
probabilities of each token $w_{i}$ conditioned on the preceding context
$\bm{w}_{<i}$:

$s_{\mathrm{lm}}(\bm{w}_{\bm{i}})=\sum\limits_{i\in\bm{i}}-\log
p_{\mathrm{lm}}(w_{i}|\bm{w}_{<i}),$ (1)

where $p_{\mathrm{lm}}$ denotes the LM distribution. Note that this definition
of surprisal applies to both words and segments, which consist of a sequence
of tokens defined under a model tokenization scheme.222$\bm{w}_{\bm{i}}$ is
also defined in a way that allows difficulty prediction of non-contiguous
segments.

Given the well-established link between $s_{\mathrm{lm}}$ and reading time
Monsalve et al. (2012); Smith and Levy (2013); Shain et al. (2022); Wilcox et
al. (2021), we test whether $s_{\mathrm{lm}}$ predicts reading times for
source and target texts when participants are engaged in the act of
translation. Previous work also establishes links between surprisal and
language production in tasks involving speech Dammalapati et al. (2021), oral
reading Klebanov et al. (2023), code-switching Calvillo et al. (2020) and
typing Chen et al. (2021). Building on this literature, we also test whether
monolingual surprisal predicts the time participants take to type their
translations.

###  3.2 Translation surprisal

The surprisal of a translation can be obtained from the distribution of a
neural machine translation model, $p_{\mathrm{mt}}$, conditioned on a source
sequence and previously translated tokens. Let $\bm{x}$ and $\bm{y}$ be a pair
of parallel sequences, and let $\bm{y}_{\bm{j}}$ be a segment of $\bm{y}$ with
token indices $\bm{j}$ and $\bm{y}_{<j}$ be the preceding context of each
token $y_{j}$. The translation surprisal of segment $\bm{y}_{\bm{j}}$ is
defined as

$s_{\mathrm{mt}}(\bm{y}_{\bm{j}})=\sum\limits_{j\in\bm{j}}-\log
p_{\mathrm{mt}}(y_{j}|\bm{x},\bm{y}_{<j}).$ (2)

We will compare translation surprisal and monolingual surprisal as predictors
of target-side reading time and production duration. We expect translation
surprisal to be the more successful predictor because this measure
incorporates context on both the source and target sides, and because NMT
models are trained specifically for translation. Translation surprisal,
however, cannot be used to predict source-side reading time because the model
must encode the entire source sequence before providing an output
distribution.

##  4 Predicting translation difficulty using NMT attention

State-of-the-art NMT models rely on the transformer architecture and
incorporate three kinds of attention: encoder self-attention, cross-attention
and decoder self-attention.333Cross-attention is sometimes known as encoder-
decoder attention. We consider all three sets of attention weights as
potential predictors of translation difficulty. By some accounts, reading,
transferring and writing are three distinct stages in the human translation
process Shreve et al. (1993); Macizo and Bajo (2004, 2006); Shreve and Lacruz
(2017), and we propose that the three attentional modules roughly align with
these three stages.

Let $\bm{x}=[x_{1},...,x_{m}]$ and $\bm{y}=[y_{1},...,y_{n}]$ denote parallel
source and target sequences, with $\bm{m}=\\{1,...,m\\}$ and
$\bm{n}=\\{1,...,n\\}$ as their token indices. Let $\bm{u}$ and $\bm{v}$
denote segments of $\bm{x}$ and $\bm{y}$ respectively such that
$\bm{u}=\bm{x}_{\bm{i}}$ and $\bm{v}=\bm{y}_{\bm{j}}$, where the indices
$\bm{i}\subseteq\bm{m}$ and $\bm{j}\subseteq\bm{n}$. Note that $\bm{x}$ and
$\bm{y}$ do not include special tokens (e.g., the end-of-sequence tag eos)
added to the sequences under the NMT’s tokenization scheme. We additionally
define $\bm{\mkern 1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern
1.5mu}=\bm{x}_{\bm{\mkern 1.5mu\overline{\mkern-1.5mui\mkern-1.5mu}\mkern
1.5mu}}$ and $\bm{\mkern 1.5mu\overline{\mkern-1.5muv\mkern-1.5mu}\mkern
1.5mu}=\bm{y}_{\bm{\mkern 1.5mu\overline{\mkern-1.5muj\mkern-1.5mu}\mkern
1.5mu}}$ as the contexts where the corresponding segments are excluded, i.e.,
$\bm{\mkern 1.5mu\overline{\mkern-1.5mui\mkern-1.5mu}\mkern
1.5mu}=\bm{m}\setminus\bm{i}$ and $\bm{\mkern
1.5mu\overline{\mkern-1.5muj\mkern-1.5mu}\mkern
1.5mu}=\\{1,...,\max(\bm{j})\\}\setminus\bm{j}$. In contrast to $\bm{\mkern
1.5mu\overline{\mkern-1.5mui\mkern-1.5mu}\mkern 1.5mu}$, $\bm{\mkern
1.5mu\overline{\mkern-1.5muj\mkern-1.5mu}\mkern 1.5mu}$ only includes token
indices from the context preceding $\bm{v}$. We define $\bm{\mkern
1.5mu\overline{\mkern-1.5muj\mkern-1.5mu}\mkern 1.5mu}$ in this way because
the decoder typically does not have access to future tokens when generating
translations.

We consider two kinds of attentional features. The first captures the total
attentional flow from segment $\bm{w}$ to segment $\bm{z}$:

$\mathrm{flow}(A,\bm{w},\bm{z})=\sum\limits_{l\in\bm{l}}\sum\limits_{k\in\bm{k}}a_{kl},\\\
$ (3)

where $A$ is an attention matrix, $\bm{k}$ and $\bm{l}$ are token indices of
$\bm{w}$ and $\bm{z}$, and $a_{kl}$ is the attention from the token at index
$k$ to the token at index $l$.

The second kind of attentional feature sums the entropies of all attentional
distributions from segment $\bm{w}$ to segment $\bm{z}$:

$\displaystyle
H(A,\bm{w},\bm{z})=\sum\limits_{k\in\bm{k}}\sum\limits_{l\in\bm{l}}-\hat{a}_{kl}\log\hat{a}_{kl}$
(4)
$\displaystyle\hat{a}_{kl}=\frac{a_{kl}}{\lVert\bm{a}_{k\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char
41\relax}}\bm{l}}\rVert},$

where again $\bm{k}$ and $\bm{l}$ are the indices of $\bm{w}$ and $\bm{z}$. We
renormalize $a_{kl}$ along
$\bm{a}_{k\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char
41\relax}}\bm{l}}$, the attention vector at $k$ with target indices
$\bm{l}$.444$\ell_{1}$ normalization This step allows us to filter out
attention to special tokens (e.g. eos) when considering the entropy of a
distribution.

Using Equations 3 and 4, we define in the following 6 attentional features as
candidate predictors of source-side reading time and 5 features for target-
side reading time and production duration.

###  4.1 Predicting source text difficulty

Encoder attention. We propose four features extracted from encoder attention
that are inspired by Dankers et al. (2022), who find that when producing a
paraphrase rather than a literal translation of a figurative expression, an
NMT encoder tends to allocate more attention from the phrase to itself, while
reducing attention directed to and received from the context. This result is
relevant to our goal of predicting translation difficulty because paraphrasing
is known to be more effortful than literal translation Balling et al. (2014);
Schaeffer and Carl (2014); Rojo (2015); Carl and Schaeffer (2017).

Given an encoder self-attention matrix $A^{e}$, we use Equation 3 to define
features that capture the total attentional flow from $\bm{u}$ to $\bm{u}$, to
its context $\bm{\mkern 1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern
1.5mu}$, and to the eos tag:

$\displaystyle f^{\mathrm{e}}_{u,u}$
$\displaystyle=\mathrm{flow}(A^{\mathrm{e}},\bm{u},\bm{u})$ $\displaystyle
f^{\mathrm{e}}_{u,\mkern 1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern
1.5mu}$ $\displaystyle=\mathrm{flow}(A^{\mathrm{e}},\bm{u},\bm{\mkern
1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern 1.5mu})$ $\displaystyle
f^{\mathrm{e}}_{u,\texttt{eos}}$
$\displaystyle=\mathrm{flow}(A^{\mathrm{e}},\bm{u},\texttt{eos}).$

In line with Dankers et al. (2022), we hypothesize that harder-to-translate
segments direct more attention to themselves and less to their contexts. To
examine if the model also reduces attention flow from context to $\bm{u}$, we
further define $f^{\mathrm{e}}_{\mkern
1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern 1.5mu,u}$:

$f^{\mathrm{e}}_{\mkern 1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern
1.5mu,u}=\mathrm{flow}(A^{\mathrm{e}},\bm{\mkern
1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern 1.5mu},\bm{u}).$

The NMT encoder relies on attention to relevant context to disambiguate input
meanings and to resolve anaphoric pronouns Tang et al. (2019); Yin et al.
(2021). These words may take longer to read, as more time is required to
determine pronoun referents and word senses. Ambiguous words also contribute
to low attentional entropy Tang et al. (2019). To characterize this feature of
$\bm{u}$, we compute $H^{\mathrm{e}}_{u,x}$, the overall attentional entropy
based on Equation 4:

$H^{\mathrm{e}}_{u,x}=H(A^{\mathrm{e}},\bm{u},\bm{x}).\\\ $

We hypothesize that low $H^{\mathrm{e}}_{u,x}$ predicts longer reading time of
$\bm{u}$.

Cross-attention. Cross-attention allows information to pass from encoder to
decoder and establishes rough alignments between input and output tokens in an
NMT model Alkhouli et al. (2018); Li et al. (2019). However, it is unclear
from previous work whether more attention weight received from the target
sequence contributes to harder or easier translation. Tu et al. (2016) and Mi
et al. (2016) show that increased attention received by part of the source
text is related to over-translation, a phenomenon in which the model focuses
too much on some parts of the input and neglects others when generating a
translation. In contrast, Dankers et al. (2022) demonstrate that paraphrasing
a figurative expression instead of literal translation reduces attention to
corresponding source tokens.

On one hand, source tokens that receive more attention and stronger alignments
are deemed more important by the model; on the other, the same phenomenon
corresponds to literal translation, which is easier than paraphrasing.
Nevertheless, both studies show that cross-attention flow to $\bm{u}$ is
pertinent to our goal of characterizing translation difficulty. Given a cross-
attention matrix $A^{\mathrm{c}}$, we define the attentional flow from
$\bm{y}$ to $\bm{u}$ as:

$f^{\mathrm{c}}_{y,u}=\mathrm{flow}(A^{\mathrm{c}},\bm{y},\bm{u}),$

and test $f^{\mathrm{c}}_{y,u}$ as a predictor of source reading time.

###  4.2 Predicting target text difficulty

Cross-attention. When paraphrasing non-literal nouns, NMT also tends to shift
cross-attention weight away from these nouns, focusing instead on surrounding
tokens and source eos Dankers et al. (2022). We therefore hypothesize that the
translation difficulty associated with target segment $\bm{v}$ may be
predicted by the total attention directed from $\bm{v}$ to source eos:

$f^{\mathrm{c}}_{v,\texttt{eos}}=\mathrm{flow}(A^{\mathrm{c}},\bm{v},\texttt{eos}).$

Prior work attributes cross-attention uncertainty to lack of confidence in
translation outputs and less informative source inputs Dabre and Fujita
(2019); Zhang and Feng (2021); Lu et al. (2021). Following these studies, we
hypothesize that higher uncertainty in cross-attention indicates less
confident alignment with source tokens and therefore predicts increased
translation difficulty. We measure alignment uncertainty between $\bm{v}$ and
the source sequence $\bm{x}$ as:

$H^{\mathrm{c}}_{v,x}=H(A^{\mathrm{c}},\bm{v},\bm{x}).$

Decoder attention. While previous work on NMT models has considered encoder
and cross-attention in depth, decoder self-attention has received less
investigation. Yang et al. (2020) demonstrates that the role of decoder self-
attention is to ensure translation fluency. Relative to encoder attention and
cross attention, decoder attention aligns less well with human annotations,
and contributes less to improving NMT performance when regularized with human
annotations Yin et al. (2021).

For completeness, however, we consider three decoder attentional features that
parallel those introduced in Section 4.1 for encoder self-attention. Despite
their similarity, these attentional features are evaluated against different
behavioral measures. The encoder features are treated as candidate predictors
of source reading time, and the decoder features are used to predict target
reading and production time. If $A^{\mathrm{d}}$ is the decoder attention
matrix, the following features capture attention flow from $\bm{v}$ to itself
and to the preceding context, as well as self-attention entropy:

$\displaystyle f^{\mathrm{d}}_{v,v}$
$\displaystyle=\mathrm{flow}(A^{\mathrm{d}},\bm{v},\bm{v})$ $\displaystyle
f^{\mathrm{d}}_{v,\mkern 1.5mu\overline{\mkern-1.5muv\mkern-1.5mu}\mkern
1.5mu}$ $\displaystyle=\mathrm{flow}(A^{\mathrm{d}},\bm{v},\bm{\mkern
1.5mu\overline{\mkern-1.5muv\mkern-1.5mu}\mkern 1.5mu})$ $\displaystyle
H^{\mathrm{d}}_{v,\overleftarrow{v}}$
$\displaystyle=H(A^{\mathrm{d}},\bm{v},\bm{\overleftarrow{v}}).$

As mentioned earlier, a decoder does not attend to future tokens, including
target-side eos. Decoder features analogous to
$f^{\mathrm{e}}_{u,\texttt{eos}}$ and $f^{\mathrm{e}}_{\mkern
1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern 1.5mu,u}$ are not possible for
this reason. Attentional entropy $H^{\mathrm{d}}_{v,\overleftarrow{v}}$ is
computed over $\bm{\overleftarrow{v}}$, which includes all target-side tokens
up to the rightmost token in $\bm{v}$.

| Study | Token | Segment
---|---|---|---
|  | TrtS | TrtT | Dur | TrtS | TrtT | Dur
en $\rightarrow$ da |  ACS08 Sjørup (2013), BD13, BD08 Dragsted (2010) | 5305 | 5320 | 6176 | 4121.0 | 4203.0 | 4779.0
en $\rightarrow$ de |  SG12 Nitzke (2019) | 3691 | 3956 | 4534 | 2991.0 | 3243.0 | 3589.0
en $\rightarrow$ es |  BML12 Mesa-Lao (2014) | 4067 | 4020 | 8280 | 3555.0 | 3330.0 | 6072.0
en $\rightarrow$ hi |  NJ12 Carl et al. (2016b) | 4717 | 4828 | 5205 | 2917.0 | 2851.0 | 2933.0
en $\rightarrow$ ja |  ENJA15 Carl et al. (2016a) | 6806 | 8329 | 2168 | 4263 | 4299 | 2130
en $\rightarrow$ nl |  ENDU20 Vanroy (2021) | 0 | 0 | 7814 | 0 | 0 | 6318
en $\rightarrow$ pt |  JLG10 Alves and Gonçalves (2013) | 0 | 0 | 2443 | 0 | 0 | 2217
en $\rightarrow$ zh |  RUC17, STC17 Carl and Báez (2019), CREATIVE Vieira et al. (2023) | 8949 | 7934 | 3876 | 6097 | 5922 | 3925
da $\rightarrow$ en |  LWB09 Jensen et al. (2009) | 3844 | 4177 | 5327 | 3445 | 3493 | 4315
fr $\rightarrow$ pl |  DG01 Płońska (2016) | 0 | 0 | 17041 | 0 | 0 | 13283
pt $\rightarrow$ en | JLG10 | 0 | 0 | 2053 | 0 | 0 | 1876
pt $\rightarrow$ zh |  MS13 Schmaltz et al. (2016) | 1011 | 830 | 203 | 781 | 755 | 237
zh $\rightarrow$ pt | MS13 | 1210 | 1237 | 1509 | 1101 | 1027 | 1209
Table 1:  Data drawn from studies in CRITT-TPRDB with the number of valid
samples ($\geq$ 20ms) per segmentation level.

##  5 Methods

###  5.1 Experimental setup

Data. Our measures of translation difficulty are derived from the CRITT
Translation Process Research Database (TPR-DB) Carl et al. (2016b). We focus
on three behavioral measures of translation difficulty: source text reading
time (TrtS), target text reading time (TrtT) and translation production
duration (Dur). Both reading time measures are based on eye-tracking data, and
the translation production measure is based on keylogging data. The data set
is organized in terms of words, segments, and sentences, and we carry out
separate analyses at the word and segment levels.

In CRITT TPR-DB, word and segment boundaries and alignments are provided by
human annotators. For consistency, we remove alignments of words and segments
that cross sentence boundaries. Following Carl (2021b), we filter values of
TrtS, TrtT and Dur lower than 20ms. The remaining values are log scaled. We
analyze data from 17 public studies available from the public database.555The
translation studies selected from the TPR database exclude data sets where
many alignments cross sentence boundaries, or that contain too many errors
(e.g., missing values and inconsistent sentence segmentations) across tables.
These studies represent 13 different language pairs, and each study includes
data from an average of 18 human translators. The studies included along with
the size of each one are summarized in Table 1. For cross-validation, we
divide the samples into 10 folds. To ensure that all predictions are evaluated
using previously unseen sentences, we randomly sample test data at the
sentence level, which means that the source sentences in train and test
partitions do not overlap.

LM and NMT models. We use mGPT Shliazhko et al. (2022), a multilingual
language model, to estimate monolingual surprisal ($s_{\mathrm{lm}}$).666mGPT
checkpoint To compute translation surprisal ($s_{\mathrm{mt}}$), we use
NLLB-200’s 600M variant, which is distilled from a much larger 54.5B Mixture-
of-Experts model Costa-jussà et al. (2022).777NLLB-200 checkpoint We compute
attentional features for each of 16 heads across 12 layers, then average
across heads and layers to create the final set of attentional features for
our analyses.

Normalization. Sections 3 and 4 describe feature definitions that are sums
over a sequence of tokens, which makes it crucial to control for segment
length when predicting reading time and production duration. All surprisal
values are therefore normalized by the lengths of the input segments,
$\bm{w}_{\bm{i}}$ and $\bm{y}_{\bm{j}}$. To normalize attentional features, we
first calculate dummy feature values by replacing $a_{lk}$ in Equation 3 and 4
with uniform attention values (i.e., $a_{lk}=1/|k|$ where $|k|$ is the length
of the attention vector). A normalized attentional feature is defined as the
ratio of the raw feature value (defined in Section 4) to its dummy value.

Control features. Although we are most interested in surprisal and attentional
features as predictors of translation difficulty, other simple features may
also predict difficulty. In particular, longer segments, low-frequency
segments, and segments towards the beginning of a sentence might be
systematically more difficult than shorter segments, high-frequency segments,
and segments towards the end of a sentence. We therefore include segment
length, average unigram frequency888https://pypi.org/project/wordfreq/ (log
scaled) and average position quantile as control features in all models, where
both averages are computed over all tokens belonging to a segment.

Linear models. Following previous studies, we use linear models to evaluate
the predictive power of both surprisal and attentional features. To predict
translation difficulty for all languages, we use a mixed model that includes
language pair and participant id as random effects. In addition, we fit
individual linear regression models for the four language pairs (en
$\rightarrow$ da, en $\rightarrow$ de, en $\rightarrow$ hi, and en
$\rightarrow$ zh) for which we have most data.

###  5.2 Log-likelihood delta as an estimation of predictive power

Following previous psycholinguistic studies Goodkind and Bicknell (2018);
Kuribayashi et al. (2021); Wilcox et al. (2020, 2023); De Varda and Marelli
(2023) we evaluate our models using log-likelihood of each sample in the held-
out data. To assess the predictive power of a feature, we train a model with
the feature of interest in addition to all control features, and compare
against a baseline model which includes only the control features. The
contribution of the predictor feature is then measured as the difference in
log-likelihood of the held-out test data ($\Delta\mathrm{llh}$) between the
two models. A positive $\Delta\mathrm{llh}$ indicates added predictive power
from the feature relative to the baseline model, whereas
$\Delta\mathrm{llh}\leq 0$ means that we have no evidence for the effect of
the feature on reading and production times.999 Wilcox et al. (2023) point out
that $\Delta\mathrm{llh}$ may be $\leq 0$ because of overfitting, or because
the relationship between the predictor feature and the target variable is not
adequately captured by the model class used (in our case, linear models). Like
Wilcox et al. (2023), we test if $\Delta\mathrm{llh}>0$ is significant across
held-out samples with a paired permutation test.101010Our test uses 1000
random permutations.

##  6 Results

###  6.1 Surprisal and attentional features

Figure 2 shows $\Delta\mathrm{llh}$ of surprisal predictors and attentional
features at both word and segment levels. Data points shown in red indicate
predictors that are statistically significant (p < .05) relative to the
baseline model. When fitted on all language pairs, $s_{\mathrm{lm}}$ is a
significant predictor of source text reading time and target production
duration, but not target reading time. On the target side, $s_{\mathrm{mt}}$
is the best overall predictor of difficulty. Target reading time has fewer
significant predictors than does target production duration, and therefore
appears to be harder to predict. From here on, we restrict our analyses to
features that are significant at at least one of the two segmentation levels.

Figure 2: $\Delta\mathrm{llh}$ plotted against control features. 95%
confidence intervals are estimated based on the 10 cross-validation folds.

Figure 3 shows analogous results for four individual language pairs. Surprisal
features $s_{\mathrm{lm}}$ and $s_{\mathrm{mt}}$ remain strong predictors for
reading time and target production duration in general. Among the attentional
features, $f^{\mathrm{e}}_{u,\mkern
1.5mu\overline{\mkern-1.5muu\mkern-1.5mu}\mkern 1.5mu}$ and
$H^{\mathrm{e}}_{u,x}$ most consistently predict source reading time across
language pairs and segmentation levels. However, the predictions of target
reading time and duration by attentional features are less consistent —
despite predicting en $\rightarrow$ da target difficulty, most attentional
features fail to contribute in other language pairs. One possible reason is
that these features overfit to small samples of individual language pairs,
compared to a mixed model that is fitted on a much larger data set including
all language pairs.

Figure 3: $\Delta\mathrm{llh}$ by language pairs with features that are
significant for at least one of the word and segment levels in Figure 2.

###  6.2 Attention is supplementary to $s_{\mathrm{lm}}$ and $s_{\mathrm{mt}}$

Our results so far confirm that both $s_{\mathrm{lm}}$ and $s_{\mathrm{mt}}$
individually predict translation difficulty, whereas attentional features on
their own are less consistent. We next ask if the attentional features that
proved significant in Section 6.1 provide supplementary predictive power when
combined with $s_{\mathrm{lm}}$ and $s_{\mathrm{mt}}$. To predict source
reading time, we train models that include control features, $s_{\mathrm{lm}}$
and one attentional feature. For target difficulty, the models are trained
with control features, $s_{\mathrm{mt}}$ and an attentional feature. We then
calculate two variants of $\Delta\mathrm{llh}$ for these models; the first
compares against the baseline model, and the second compares against a model
that is trained on control features and either $s_{\mathrm{lm}}$ or
$s_{\mathrm{mt}}$.

We repeat the same significance tests as before, and the results are shown in
Figure 4. For the entire data set (top row of Figure 4), models with the
addition of individual attentional features predict translation difficulty
better than those trained with surprisal and control features only (except
$H^{\mathrm{c}}_{v,x}$). Again, however, these results are weaker for
individual language pairs.

Figure 4: $\Delta\mathrm{llh}$ with one feature in addition to
$s_{\mathrm{lm}}$/$s_{\mathrm{mt}}$ and control features in training. Light
and dark colored stars indicate $\Delta\mathrm{llh}$ to be significantly above
zero when compared against control features and control features with
surprisal respectively. For example, $f^{\mathrm{c}}_{v,\texttt{eos}}$
significantly contributes in addition to $s_{\mathrm{mt}}$ in predicting
target duration at both word and segment levels, but $H^{\mathrm{c}}_{v,x}$
does not contribute predictive power beyond $s_{\mathrm{mt}}$.

(a) Source reading time

(b) Target reading time

(c) Target production duration

Figure 5: Predictor coefficients for linear models that include all three
control features (length, frequency, position) along with one additional
predictor (either surprisal or an attentional feature). Confidence intervals
are plotted but are hard to see because they are so narrow.

###  6.3 Predictor coefficients

Thus far we have only demonstrated the predictive power of surprisal and
attentional features. To enable conclusions about the nature of the
relationship between individual features and translation difficulty, Figure 5
shows average mixed model coefficients over data folds. The coefficients
plotted support conclusions about the direction and effect size of the
relationship between each predictor and translation difficulty, but the bar
heights may not reflect predictive power, which has been indicated previously
by $\Delta\mathrm{llh}$.111111We tested for collinearity by calculating
variance inflation factors (VIF) for each set of features. The highest VIF is
2.2, which is below the thresholds (2.5/3) recommended by Szmrecsanyi (2006)
and Zuur et al. (2010). In general, segments are more difficult when they are
longer and occur earlier in the sequence. Rare words in general take longer to
read and produce, as our baseline models consistently converge to negative
coefficients for frequency (not shown in the figure). However, with the
addition of surprisal and attentional features, the frequency effect for rare
source words is reversed, whereas rare targets still require more attention
and take longer to produce. As expected, increases in $s_{\mathrm{lm}}$ and
$s_{\mathrm{mt}}$ are associated with increased reading time and production
duration.

On the source side, the coefficents related to encoder self-attention indicate
that harder-to-translate source texts direct less attention to context and
more to eos, which reduces their entropy. Difficult source words are also
singled out as important by having more incoming cross-attention from the
target sequence.

On the target side, harder translations tend to show slight increases in
cross-attention to source eos, and show more diffuse attention across the
source sequence.121212Mean coefficients of $f^{\mathrm{c}}_{v,\texttt{eos}}$
for token and segment are .001 and .002 respectively. Our results thus support
Dankers et al. (2022)’s claim that paraphrases show increased attention to eos
and take longer to produce than literal translations.

Figures 5(b) and 5(c) also suggest that harder translations have more
informative decoder attention, and direct more attention to themselves and the
context. These results imply reduced attention to bos, the initial token of a
translation sequence that conveys the target language to the NLLB model.

##  7 Discussion

Section 6.1 showed that monolingual surprisal predicts source reading time,
but that translation surprisal is a more consistent predictor of target
reading time and production duration. On its own, NMT attention also predicts
translation difficulty to some degree, but the most accurate predictions are
achieved by combining surprisal and attentional features.

###  7.1 Psycholinguistic implications

Our results support previous findings that surprisal predicts translation
difficulty Wei (2022); Teich et al. (2020); Carl (2021a). Surprisal has
several justifications as a cognitive difficulty metric Levy (2013); Futrell
and Hahn (2022), and one approach interprets surprisal as a measure of a shift
in cognitive resource allocation. On this account, higher translation
surprisal indicates that more effort is needed to shift cognitive resources to
the word ultimately selected Wei (2022).

Teich et al. (2020) suggests that translators aim for translations that are
both faithful to the source ($p_{\mathrm{mt}}(t|s)$ is high) and fluent
($p_{\mathrm{lm}}(t)$ is high). These goals do not always align, and capture
two different translation strategies — literal translation optimizes MT
probability at the expense of LM probability, whereas figurative translation
prioritizes the latter. Although increases in $s_{\textrm{lm}}$ and
$s_{\textrm{mt}}$ both predict increased target difficulty, our data reveals
that these predictors have a weak but significant negative correlation
($p<.001$) at both token ($\rho=-.053$) and segment ($\rho=-.079$) levels. We
therefore find quantitative support for a trade-off between fidelity and
fluency Müller et al. (2020).

###  7.2 Translatability by parts of speech

(a) Source word reading time and predictors

(b) Average translation duration and predictors

Figure 6: Values are grouped by part-of-speech tags based on the multiLing
corpus, and are sorted in order of difficulty. Compared to function wor ds,
open-class words take longer to read and translate. Pearson correlations with
translation difficulty (leftmost panel) are shown at the top of the predictor
panels, and ‘*’ indicates statistical significance ($p<.001$).

To gain more insight into the aspects of translation difficulty captured by
surprisal and attention, we analyzed human difficulty and model predictions
for different parts of speech. All results that follow are based on a subset
of studies that use the same source texts, multiLing, a small sample of news
articles and sociological texts in English.131313Studies included from
multiLing corpus are RUC17, ENJA15, NJ12, STC17, SG12, ENDU20 and BML12. We
break down the difficulty of these English words by their part-of-speech (POS)
tags, which are available in the corpus.141414POS tags are predictions of NLTK
tagger converted to universal POS tags.

Figure 6(a) shows reading time, $s_{\mathrm{lm}}$ and attentional features of
words grouped by their POS tags. Compared to function words, open-class words,
such as proper nouns, nouns and adjectives, are the most difficult to
translate and have higher surprisal. These words also direct more attention to
eos and less to the context, and attract more cross-attention from the
translated sequence.

For target difficulty, the distinction between open-class and function words
is also evident in Figure 6(b). For each source word, translation duration is
defined as the duration of the target segment aligned with the source word
divided by the number of alignments between the target segment and the source
sentence. Translations of coordinating conjunctions and punctuation stand out
as among the easiest by humans, but are surprising for the LM and difficult
for NMT. One possible reason is that conjunctions can be cross-linguistically
ambiguous Li et al. (2014); Gromann and Declerck (2014); Novák and Novák
(2022). For example, English “but” and “and” have been shown to affect NMT
fluency Popović and Castilho (2019); Popović (2019). For punctuation, He et
al. (2019) demonstrate that the importance of these tokens in NMT can vary by
language pairs; for example, translation to Japanese often relies on
punctuation to demarcate coherent groups, which is useful for syntactic
reordering.

##  8 Conclusion

Our results support the prevailing view that current NLP models, including LM
and NMT align partially with human language usage and are predictive of
language processing complexity. We evaluated surprisal and NMT attention as
predictors of human translation difficulty, and found that both factors
predict reading times and production duration. Previous work provides some
evidence that surprisal and NMT attention capture important aspects of
translation difficulty, and our work strengthens this conclusion by estimating
surprisal based on state-of-the-art models and analyzing data based on 13
language pairs and hundreds of human translators.

Although the attentional features we consider are empirically successful and
grounded in prior literature, they are not without limitations. These features
are relatively simple and combining attention weights in more sophisticated
ways may allow stronger predictions of human translation difficulty. A more
theoretically-motivated approach that builds on recent studies of the
interpretability of attention distributions Vashishth et al. (2019); Zhang et
al. (2021); Madsen et al. (2022) is worth exploring to develop more fine-
grained predictors of translation processing.

To work with as much data as possible, we focused primarily on analyses that
combine data from all 13 language pairs, but analyzing translation challenges
in individual language pairs is a high priority for future work. A possible
next step is an analysis exploring whether the predictors considered here are
sensitive to constructions in specific languages that are known sources of
processing difficulty Campbell (1999); Vanroy (2021). Factors such as
surprisal and attentional flow are appealing in part because their generality
makes them broadly applicable across languages, but understanding the
idiosyncratic ways in which each pair of languages poses translation
challenges is equally important.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3874–3884.
  * Alkhouli et al. (2018) Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018.  On the alignment problem in multi-head attention-based neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 177–185.
  * Alves and Gonçalves (2013) Fabio Alves and José Luiz Gonçalves. 2013.  Investigating the conceptual-procedural distinction in the translation process: A relevance-theoretic analysis of micro and macro translation units.  _Target. International Journal of Translation Studies_ , 25(1):107–124.
  * Balling et al. (2014) Laura Winther Balling, Kristian Tangsgaard Hvelplund, and Annette C Sjørup. 2014\.  Evidence of parallel processing during translation.  _Meta_ , 59(2):234–259.
  * Calvillo et al. (2020) Jesús Calvillo, Le Fang, Jeremy Cole, and David Reitter. 2020.  Surprisal predicts code-switching in Chinese-English bilingual text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4029–4039.
  * Campbell (1999) Stuart Campbell. 1999.  A cognitive approach to source text difficulty in translation.  _Target. International Journal of Translation Studies_ , 11(1):33–63.
  * Carl (2012) Michael Carl. 2012.  The CRITT TPR-DB 1.0: A database for empirical human translation process research.  In _Workshop on Post-Editing Technology and Practice_.
  * Carl (2021a) Michael Carl. 2021a.  Information and entropy measures of rendered literal translation.  _Explorations in empirical translation process research_ , pages 113–140.
  * Carl (2021b) Michael Carl. 2021b.  Translation norms, translation behavior, and continuous vector space models.  In _Explorations in Empirical Translation Process Research_ , pages 357–388. Springer.
  * Carl et al. (2016a) Michael Carl, Akiko Aizawa, and Masaru Yamada. 2016a.  English-to-Japanese translation vs. dictation vs. post-editing: Comparing translation modes in a multilingual setting.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 4024–4031.
  * Carl and Báez (2019) Michael Carl and M Cristina Toledo Báez. 2019.  Machine translation errors and the translation process: A study across different languages.  _Journal of Specialised Translation_ , 31:107–132.
  * Carl et al. (2016b) Michael Carl, Moritz Schaeffer, and Srinivas Bangalore. 2016b.  The CRITT translation process research database.  In _New directions in empirical translation process research_ , pages 13–54. Springer.
  * Carl and Schaeffer (2017) Michael Carl and Moritz Jonas Schaeffer. 2017.  Why translation is difficult: A corpus-based study of non-literality in post-editing and from-scratch translation.  _HERMES-Journal of Language and Communication in Business_ , (56):43–57.
  * Chen et al. (2021) Robert Chen, Roger Levy, and Tiwalayo Eisape. 2021.  On factors influencing typing time: Insights from a viral online typing game.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 43.
  * Cohen-Priva and Jurafsky (2008) Uriel Cohen-Priva and Dan Jurafsky. 2008.  Phone information content influences phone duration.  In _Conference on prosody and language processing_.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Dabre and Fujita (2019) Raj Dabre and Atsushi Fujita. 2019.  Recurrent stacking of layers for compact neural machine translation models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6292–6299.
  * Dammalapati et al. (2019) Samvit Dammalapati, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2019.  Expectation and locality effects in the prediction of disfluent fillers and repairs in English speech.  In _Proceedings of the 2019 conference of the North American Chapter of the association for computational linguistics: Student research workshop_ , pages 103–109.
  * Dammalapati et al. (2021) Samvit Dammalapati, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2021.  Effects of duration, locality, and surprisal in speech disfluency prediction in English spontaneous speech.  In _Proceedings of the Society for Computation in Linguistics 2021_ , pages 91–101.
  * Dankers et al. (2022) Verna Dankers, Christopher Lucas, and Ivan Titov. 2022.  Can transformer be too compositional? Analysing idiom processing in neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3608–3626.
  * De Varda and Marelli (2023) Andrea De Varda and Marco Marelli. 2023.  Scaling in cognitive modelling: A multilingual approach to human reading times.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 139–149.
  * Deilen et al. (2023) Silvana Deilen, Ekaterina Lapshinova-Koltunski, and Michael Carl. 2023.  Cognitive aspects of compound translation: Insights into the relation between implicitation and semantic complexity from a translation process perspective.  _Available at SSRN 4375757_.
  * Demberg et al. (2012) Vera Demberg, Asad Sayeed, Philip Gorinski, and Nikolaos Engonopoulos. 2012.  Syntactic surprisal affects spoken word duration in conversational contexts.  In _Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning_ , pages 356–367.
  * Dragsted (2010) Barbara Dragsted. 2010.  Coordination of reading and writing processes in translation: An eye on uncharted territory.  In _Translation and Cognition_ , pages 41–62. John Benjamins Publishing Company.
  * Futrell and Hahn (2022) Richard Futrell and Michael Hahn. 2022.  Information theory as a bridge between language function and language form.  _Frontiers in Communication_ , 7:657725.
  * Goodkind and Bicknell (2018) Adam Goodkind and Klinton Bicknell. 2018.  Predictive power of word surprisal for reading times is a linear function of language model quality.  In _Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018)_ , pages 10–18.
  * Gromann and Declerck (2014) Dagmar Gromann and Thierry Declerck. 2014.  A cross-lingual correcting and completive method for multilingual ontology labels.  _Towards the multilingual semantic web: Principles, methods and applications_ , pages 227–242.
  * Hale (2001) John Hale. 2001.  A probabilistic Earley parser as a psycholinguistic model.  In _Second meeting of the north american chapter of the association for computational linguistics_.
  * Hale (2003) John Hale. 2003.  The information conveyed by words in sentences.  _Journal of psycholinguistic research_ , 32:101–123.
  * He et al. (2019) Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019\.  Towards understanding neural machine translation with word importance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 953–962.
  * Jensen et al. (2009) Kristian Tangsgaard Hvelplund Jensen, Annette C Sjørup, and Laura Winther Balling. 2009.  Effects of l1 syntax on l2 translation.  _Copenhagen Studies in Language_ , 38:319–336.
  * Jurafsky (2003) Dan Jurafsky. 2003.  Probabilistic modeling in psycholinguistics: Linguistic comprehension and production.  _Probabilistic linguistics_ , 21:1–30.
  * Klebanov et al. (2023) Beata Beigman Klebanov, Michael Suhan, Zuowei Wang, and Tenaha O’Reilly. 2023\.  A dynamic model of lexical experience for tracking of oral reading fluency.  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , pages 567–575.
  * Kudugunta et al. (2019) Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019.  Investigating multilingual NMT representations at scale.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1565–1575.
  * (35) Manoj Kumar, Ariel Goldstein, Sebastian Michelmann, Jeffrey M Zacks, Uri Hasson, and Kenneth A Norman.  Bayesian surprise predicts human event segmentation in story listening.
  * Kuribayashi et al. (2021) Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021.  Lower perplexity is not always human-like.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5203–5217.
  * Levy (2008) Roger Levy. 2008.  Expectation-based syntactic comprehension.  _Cognition_ , 106(3):1126–1177.
  * Levy (2013) Roger Levy. 2013.  Memory and surprisal in human sentence comprehension.  In R. P. G. van Gompel, editor, _Sentence processing_ , pages 78–114. Psychology Press Hove, UK.
  * Levy and Jaeger (2006) Roger Levy and T Florian Jaeger. 2006.  Speakers optimize information density through syntactic reduction.  In _Proceedings of the 19th International Conference on Neural Information Processing Systems_ , pages 849–856.
  * Li et al. (2014) Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014.  Cross-lingual discourse relation analysis: A corpus study and a semi-supervised classification system.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 577–587.
  * Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019.  On the word alignment from neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1293–1303.
  * Linzen and Jaeger (2016) Tal Linzen and T Florian Jaeger. 2016.  Uncertainty and expectation in sentence processing: Evidence from subcategorization distributions.  _Cognitive science_ , 40(6):1382–1411.
  * Lowder et al. (2018) Matthew W Lowder, Wonil Choi, Fernanda Ferreira, and John M Henderson. 2018.  Lexical predictability during natural reading: Effects of surprisal and entropy reduction.  _Cognitive Science_ , 42(4):1166–1183.
  * Lu et al. (2021) Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. 2021.  Attention calibration for transformer in neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1288–1298.
  * Macizo and Bajo (2004) Pedro Macizo and M Teresa Bajo. 2004.  When translation makes the difference: Sentence processing in reading and translation.  _Psicológica_ , 25(2):181–205.
  * Macizo and Bajo (2006) Pedro Macizo and M Teresa Bajo. 2006.  Reading for repetition and reading for translation: Do they involve the same processes?  _Cognition_ , 99(1):1–34.
  * Madsen et al. (2022) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022.  Post-hoc interpretability for neural nlp: A survey.  _ACM Computing Surveys_ , 55(8):1–42.
  * Malisz et al. (2018) Zofia Malisz, Erika Brandt, Bernd Möbius, Yoon Mi Oh, and Bistra Andreeva. 2018\.  Dimensions of segmental variability: Interaction of prosody and surprisal in six languages.  _Frontiers in Communication_ , 3:25.
  * Matasov (2017) Roman Matasov. 2017.  Nuremberg: the trial of six million words.  https://aiic.org/document/995/.
  * Meister et al. (2021) Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021.  Revisiting the uniform information density hypothesis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 963–980.
  * Mesa-Lao (2014) Bartolomé Mesa-Lao. 2014.  Gaze behaviour on source texts: An exploratory study comparing translation and post-editing.  In _Post-editing of machine translation: Processes and applications_ , pages 219–245. Cambridge Scholars Publishing.
  * Mi et al. (2016) Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016.  Coverage embedding models for neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 955–960.
  * Monsalve et al. (2012) Irene Fernandez Monsalve, Stefan L Frank, and Gabriella Vigliocco. 2012.  Lexical surprisal as a general predictor of reading time.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 398–408.
  * Müller et al. (2020) Mathias Müller, Annette Rios, and Rico Sennrich. 2020.  Domain robustness in neural machine translation.  In _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 151–164, Virtual. Association for Machine Translation in the Americas.
  * Nitzke (2019) Jean Nitzke. 2019.  _Problem solving activities in post-editing and translation from scratch: A multi-method study_.  Language Science Press.
  * Novák and Novák (2022) Attila Novák and Borbála Novák. 2022.  Cross-lingual transfer of knowledge in distributional language models: Experiments in Hungarian.  _Acta Linguistica Academica_ , 69(4):405–449.
  * Oh and Schuler (2022) Byung-Doh Oh and William Schuler. 2022.  Entropy-and distance-based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9324–9334.
  * Pimentel et al. (2021) Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021.  A surprisal–duration trade-off across and within the world’s languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 949–962.
  * Płońska (2016) Dagmara Płońska. 2016.  Problems of literality in french-polish translations of a newspaper article.  _New directions in empirical translation process research: exploring the CRITT TPR-DB_ , pages 279–291.
  * Popović (2019) Maja Popović. 2019.  Evaluating conjunction disambiguation on English-to-German and French-to-German WMT 2019 translation hypotheses.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 464–469.
  * Popović and Castilho (2019) Maja Popović and Sheila Castilho. 2019.  Are ambiguous conjunctions problematic for machine translation?  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , pages 959–966.
  * Rojo (2015) Ana Rojo. 2015.  Translation meets cognitive science: The imprint of translation on cognitive processing.  _Multilingua_ , 34(6):721–746.
  * Russo et al. (2022) Andrea G Russo, Maria De Martino, Annibale Elia, Francesco Di Salle, and Fabrizio Esposito. 2022.  Negative correlation between word-level surprisal and intersubject neural synchronization during narrative listening.  _Cortex_ , 155:132–149.
  * Russo et al. (2020) Andrea G Russo, Maria De Martino, Azzurra Mancuso, Giorgio Iaconetta, Renzo Manara, Annibale Elia, Alessandro Laudanna, Francesco Di Salle, and Fabrizio Esposito. 2020.  Semantics-weighted lexical surprisal modeling of naturalistic functional MRI time-series during spoken narrative listening.  _Neuroimage_ , 222:117281.
  * Ryu and Lewis (2021) Soo Hyun Ryu and Richard L Lewis. 2021.  Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 61–71.
  * Schaeffer and Carl (2014) Moritz Schaeffer and Michael Carl. 2014.  Measuring the cognitive effort of literal translation processes.  In _Proceedings of the EACL 2014 Workshop on Humans and Computer-assisted Translation_ , pages 29–37.
  * Schmaltz et al. (2016) Márcia Schmaltz, Igor AL da Silva, Adriana Pagano, Fabio Alves, Ana Luísa V Leal, Derek F Wong, Lidia S Chao, and Paulo Quaresma. 2016.  Cohesive relations in text comprehension and production: An exploratory study comparing translation and post-editing.  _New Directions in Empirical Translation Process Research: Exploring the CRITT TPR-DB_ , pages 239–263.
  * Shain et al. (2022) Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger P Levy. 2022\.  Large-scale evidence for logarithmic effects of word predictability on reading time.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mGPT: Few-Shot Learners Go Multilingual.
  * Shreve and Lacruz (2017) Gregory M Shreve and Isabel Lacruz. 2017.  Aspects of a cognitive model of translation.  _The Handbook of Translation and Cognition_ , pages 127–143.
  * Shreve et al. (1993) Gregory M Shreve, Christina Schäffner, Joseph H Danks, and Jennifer Griffin. 1993.  Is there a special kind of “reading” for translation? An empirical investigation of reading in the translation process.  _Target_ , 5(1):21–41.
  * Sjørup (2013) Annette Camilla Sjørup. 2013.  _Cognitive effort in metaphor translation: An eye-tracking and key-logging study_.  Frederiksberg: Copenhagen Business School (CBS).
  * Smith and Levy (2013) Nathaniel J Smith and Roger Levy. 2013.  The effect of word predictability on reading time is logarithmic.  _Cognition_ , 128(3):302–319.
  * Sun (2015) Sanjun Sun. 2015.  Measuring translation difficulty: Theoretical and methodological considerations.  _Across languages and cultures_ , 16(1):29–54.
  * Szmrecsanyi (2006) Benedikt Szmrecsanyi. 2006.  _Morphosyntactic persistence in spoken English: A corpus study at the intersection of variationist sociolinguistics, psycholinguistics, and discourse analysis_.  Mouton de Gruyter.
  * Tang et al. (2019) Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019.  Encoders help you disambiguate word senses in neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1429–1435.
  * Teich et al. (2020) Elke Teich, José Martínez Martínez, and Alina Karakanta. 2020.  Translation, information theory and cognition.  _The Routledge Handbook of Translation and Cognition_ , pages 9781315178127–24.
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling coverage for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 76–85.
  * Vanroy (2021) Bram Vanroy. 2021.  _Syntactic difficulties in translation_.  Ph.D. thesis, Ghent University.
  * Vanroy et al. (2019) Bram Vanroy, Orphée De Clercq, and Lieve Macken. 2019.  Correlating process and product data to get an insight into translation difficulty.  _Perspectives: Studies in Translation Theory and Practice_ , 27(6):924–941.
  * Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019\.  Attention interpretability across NLP tasks.  _arXiv preprint arXiv:1909.11218_.
  * Vieira et al. (2023) Lucas Nunes Vieira, Natalie Zelenka, Roy Youdale, Xiaochun Zhang, and Michael Carl. 2023.  Translating science fiction in a CAT tool: Machine translation and segmentation settings.  _Translation & Interpreting_, 15(1):216–235.
  * Wei (2022) Yuxiang Wei. 2022.  Entropy as a measurement of cognitive load in translation.  In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Workshop 1: Empirical Translation Process Research)_ , pages 75–86.
  * Wilcox et al. (2023) Ethan Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023\.  Testing the predictions of surprisal theory in 11 languages.  _arXiv e-prints_ , pages arXiv–2307.
  * Wilcox et al. (2021) Ethan Wilcox, Pranali Vani, and Roger Levy. 2021.  A targeted assessment of incremental processing in neural language models and humans.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 939–952.
  * Wilcox et al. (2020) Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020\.  On the predictive power of neural language models for human real-time comprehension behavior.
  * Yang et al. (2020) Yilin Yang, Longyue Wang, Shuming Shi, Prasad Tadepalli, Stefan Lee, and Zhaopeng Tu. 2020.  On the sub-layer functionalities of transformer decoder.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4799–4811.
  * Yin et al. (2021) Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André FT Martins, and Graham Neubig. 2021.  Do context-aware translation models pay the right attention?  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 788–801.
  * Yu et al. (2023) Chi-Lin Yu, Rachel Eggleston, Kehui Zhang, Nia Nickerson, Xin Sun, Rebecca A Marks, Xiaosu Hu, Jonathan R Brennan, Henry Wellman, and Ioulia Kovelman. 2023\.  Neural processing of children’s theory of mind in a naturalistic story-listening paradigm.
  * Zhang and Feng (2021) Shaolei Zhang and Yang Feng. 2021.  Modeling concentrated cross-attention for neural machine translation with Gaussian mixture model.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1401–1411.
  * Zhang et al. (2021) Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. 2021.  A survey on neural network interpretability.  _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 5(5):726–742.
  * Zuur et al. (2010) Alain F Zuur, Elena N Ieno, and Chris S Elphick. 2010.  A protocol for data exploration to avoid common statistical problems.  _Methods in ecology and evolution_ , 1(1):3–14.