# On Compositional Generalization of Neural Machine Translation

Yafu Li♠♡ , Yongjing Yin♠♡ , Yulong Chen♠♡ , Yue Zhang♡♢
♠ Zhejiang University
♡ School of Engineering, Westlake University
♢ Institute of Advanced Technology, Westlake Institute for Advanced Study
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Modern neural machine translation (NMT) models have achieved competitive
performance in standard benchmarks such as WMT. However, there still exist
significant issues such as robustness, domain generalization, etc. In this
paper, we study NMT models from the perspective of compositional
generalization by building a benchmark dataset, CoGnition, consisting of 216k
clean and consistent sentence pairs. We quantitatively analyze effects of
various factors using compound translation error rate, then demonstrate that
the NMT model fails badly on compositional generalization, although it
performs remarkably well under traditional metrics.

##  1 Introduction

Neural machine translation (NMT) has shown competitive performance on
benchmark datasets such as IWSLT and WMT Vaswani et al. (2017); Edunov et al.
(2018a); Liu et al. (2020a), and even achieves parity with professional human
translation under certain evaluation settings Hassan et al. (2018). However,
the performance can be relatively low in out-of-domain and low-resource
conditions. In addition, NMT systems show poor robustness and vulnerability to
input perturbations Belinkov and Bisk (2018a); Cheng et al. (2019). One
example is shown in Table 1, where simple substitution of a word yields
translation with completely different semantics. Many of these issues origin
from the fact that NMT models are trained end-to-end over large parallel data,
where new test sentences can be sparse.

Input | Translation
---|---
Taylor breaks his promise | 泰勒信守诺言
(Taylor keeps his promise)
James breaks his promise | 詹姆斯违反诺言
(James breaks his promise)
Table 1:  Translation samples obtained from one popular web translation engine on January 19, 2021.  Dataset | Type | Source | Target
---|---|---|---
SCAN | Atoms | jump, twice | JUMP JUMP
Compounds | jump twice
CFQ | Atoms | Who [predicate] [entity], directed, Elysium | SELECT DISTINCT ?x0 WHERE {
Compounds | Who directed Elysium | ?x0 a ns:people.person .
|  | ?x0 ns:film.director.film m.0gwm_wy}
CoGnition | Atoms | the, doctor, he liked | 他喜欢的医生病了
Compounds | the doctor he liked
Sentences | The doctor he liked was sick
Table 2:  Examples of SCAN, CFQ, and our CoGnition datasets.

Disregarding out-of-vocabulary words, a main cause of sparsity is semantic
composition: given a limited vocabulary, the number of possible compositions
grows exponentially with respect to the composite length. The ability to
understand and produce a potentially infinite number of novel combinations of
known components, namely compositional generalization Chomsky ; Montague ;
Lake and Baroni (2018); Keysers et al. (2020), has been demonstrated deficient
in many machine learning (ML) methods Johnson et al. (2017a); Lake and Baroni
(2018); Bastings et al. (2018); Loula et al. (2018); Russin et al. (2019a).

In this paper, we study compositional generalization in the context of machine
translation. For example, if “red cars” and “blue balls” are seen in training,
a competent algorithm is expected to translate “red balls” correctly, even if
the phrase has not been seen in training data. Intuitively, the challenge
increases as the composite length grows. Recently, several studies have taken
steps towards this specific problem. They either use a few dedicated samples
(i.e., 8 test sentences) for evaluation Lake and Baroni (2018); Li et al.
(2019b); Chen et al. (2020), or make simple modifications in sampled source
sentences such as removing or adding adverbs, and concatenating two sentences
Raunak et al. (2019); Fadaee and Monz (2020a). Such experimental data is
limited in size, scope and specificity, and the forms of composition are
coarse-grained and non-systematic. As a result, no qualitative conclusions
have been drawn on the prevalence and characteristics of this problem in
modern NMT.

We make a first large-scale general domain investigation, constructing the
CoGnition dataset (Compositional Generalization Machine Translation Dataset),
a clean and consistent parallel dataset in English-Chinese, along with a
synthetic test set to quantify and analyze the compositional generalization of
NMT models. In particular, we define frequent syntactic constituents as
compounds, and basic semantic components in constituents as atoms. In addition
to the standard training, validation and test sets, the CoGnition dataset
contains a compositional generalization test set, which contains novel
compounds in each sentence, so that both the generalization error rate can be
evaluated, and its influence on BLEU Papineni et al. (2002) can be quantified.
Our compositional generalization test set consists of 2,160 novel compounds,
with up to 5 atoms and 7 words. In this way, generalization ability can be
evaluated based on compound translation error rate.

Empirical results show that the dominant Transformer Vaswani et al. (2017) NMT
model faces challenges in translating novel compounds, despite its competitive
performance under traditional evaluation metrics such as BLEU. In addition, we
observe that various factors exert salient effects on model’s ability of
compositional generalization, such as compound frequency, compound length,
atom co-occurrence, linguistic patterns, and context complexity. The CoGnition
dataset along with the automatic evaluation tool are realesed on
https://github.com/yafuly/CoGnition.

##  2 Related Work

#### Analysis of NMT.

Our work is related to research analyzing NMT from various perspectives. There
has been much linguistic analysis of NMT representations Shi et al. (2016);
Belinkov et al. (2017); Bisazza and Tump (2018), interpretability Ding et al.
(2017); He et al. (2019); Voita et al. (2019a), and attention weights Voita et
al. (2019b); Michel et al. (2019). Robustness is also an important research
direction. Work has shown that NMT models are prone to be negatively affected
by both synthetic and natural noise Belinkov and Bisk (2018b); Cheng et al.
(2018); Ebrahimi et al. (2018). For better exploration of robust NMT, Michel
and Neubig (2018) propose an MTNT dataset containing several types of noise.
Wang et al. (2020) provide in-depth analyses of inference miscalibration of
NMT resulting from the discrepancy between training and inference. Our work is
in line but we discuss robustness from the perspective of compositional
generalization.

In this respect, Lake and Baroni (2018) propose a simple experiment to analyze
compositionality in MT, followed by Chen et al. (2020) and Li et al. (2019b).
Specifically, they introduce a novel word “dax”, and their training data
contains a single pattern of sentence pairs (e.g. “I am daxy”, “je suis
daxiste”) while the test set contains different patterns. However, their work
is limited in that there are only 8 sentences in the test set. Raunak et al.
(2019) observe a performance drop on a dataset of concatenated source
sentences. Fadaee and Monz (2020b) modify source sentences by removing
adverbs, substituting numbers, inserting words that tend to keep syntax
correct (e.g. “very”), and changing the gender, and find unexpected changes in
the translation. In contrast to these studies, we quantitatively measure
compositionality of NMT under compound translation error rate.

Translation involves various challenges such as low-frequency words, polysemy
and compositional complexity. In this work, we focus on how the NMT model
generalizes to complex compositions in a controllable setting and minimize the
effects of the other factors.

#### Compositional Generalization.

Neural networks have been shown sample-inefficient, requiring large-scale
training data, which suggests that they may lack compositionality Lake and
Baroni (2018). Lake and Baroni (2018) introduce the SCAN dataset to help study
compositional generalization of neural networks, which has received increasing
interests Russin et al. (2019b); Dessì and Baroni (2019); Li et al. (2019c);
Lake (2019); Andreas (2020); Gordon et al. (2020). Various benchmarks have
been proposed including in the area of visual reasoning Johnson et al.
(2017b); Hudson and Manning (2019), mathematics Saxton et al. (2019), and
semantic parsing (CFQ) Keysers et al. (2020). However, no benchmark has been
dedicated to machine translation in practice. We fill this gap by introducing
a dataset with 216,000 instances and an average sentence length of 9.7 tokens.

Figure 1:  Summary of dataset construction.

##  3 Problem Definition

Following Keysers et al. (2020), compositional generalization is defined as
the capacity to systematically generalize to novel combinations of components
which are learned sufficiently during training. Key elements to measure
compositional generalization include atoms and compounds. Specifically, atoms
are primitive elements in the train set whereas compounds are obtained by
composing these atoms. The research question is whether neural models perform
well on unseen compounds. Take Table 2 for example, in the SCAN dataset, the
atoms are simple commands such as “jump” and the composite command “jump
twice” is a compound. In the CFQ, the compounds are questions such as “Who
directed Elysium”, and the atoms correspond to the primitive elements in the
questions such as the predicate “directed”, the question patterns “Who
[predicate] [entity]” and the entities “Elysium”.

In theory, compounds in MT can be defined as phrases, sentences or even
document. In practice, however, we want to control the number of atoms in a
novel compound for quantitative evaluation. In addition, it can be highly
difficult to construct a large-scale dataset where novel compounds are
sentences of practical sizes (the number of synthesized sentences increases
exponentially with their length) while ensuring their grammatical correctness.
Therefore, we constrain compounds to syntactic constituents, and define atoms
as basic semantic components in constituents according to syntactic and
semantic rules for forming constituents Partee (1995). As a result, we
randomly assign multiple sentential contexts for investigating each novel
compound. Table 2 shows a contrast between our dataset and existing datasets
for compositional generalization in semantics.

Mistakes caused by weakness in computational generalization can be easily
found in state-of-the-art NMT models. In particular, we train a Transformer-
based model Vaswani et al. (2017) on WMT17 En-Zh Dataset
111http://www.statmt.org/wmt17/. One sentence in the standard test set, “but
the problem is , with the arrival of durant , thompson ’s appearance rate will
surely decline , which is bound to affect his play”, is translated into “但问题是,
随着杜兰特的到来, 汤普森的外表肯定会下降, 这一定会影响到他的表演” (English: but the problem is , with the
arrival of durant , thompson ’s will surely look worse , which is bound to
affect his play). The novel compound “appearance rate” is composed of two
atoms (i.e., “appearance” and “rate”), both with a high frequency of more than
27,000 times in the training set. However, the sentence semantics is
completely distorted due to the failure of semantic composition, which is
possibly influenced by the context word “play”. More importantly, as the
overall translation highly overlaps with the reference, the model achieves a
high score in similarity-based metrics such as BLEU, demonstrating that fatal
translation errors can be overlooked under traditional evaluation metrics.

Pattern # | Composition | Example
---|---|---
Pattern 1.1 | DET+N | all the sudden the waiter screamed in pain .
Pattern 1.2 | DET+ADJ+N | one day another lazy lawyer snapped and broke every window in the car .
Pattern 1.3 | DET+N+MOD |  each doctor he liked was talking to a friend on the phone .
Pattern 1.4 | DET+ADJ+N+MOD |  every smart lawyer at the store decided to go back next week .
Pattern 2.1 | V+DET+N | she said she liked the building !
Pattern 2.2 | V+DET+ADJ+N | he soon met the special girl named taylor .
Pattern 2.3 | V+DET+N+MOD | she took the child he liked out to enjoy the snow .
Pattern 2.4 | V+DET+ADJ+N+MOD | when taylor saw the dirty car he liked , he was amazed .
Pattern 3.1 | P+DET+N | taylor felt really awful about the bee .
Pattern 3.2 | P+DET+ADJ+N |  inside the small apartment were some of my old toys .
Pattern 3.3 | P+DET+N+MOD | taylor forgot about the chair on the floor !
Pattern 3.4 | P+DET+ADJ+N+MOD | he jumped from the bench towards the large airplane on the floor .
Table 3:  Compound patterns in the CG test set. Compounds are in bold and
shown in sentence context.

##  4 Dataset

Figure 1 gives an overview of our data construction process. We first source
monolingual data (Section 4.1), and then build parallel data based by
translation (Section 4.2). Then we synthesize a test set of novel compounds
(Section 4.3), and offer an automatic evaluation method (Section 4.4).

###  4.1 Monolingual Data Source

Our goal is to focus on compositional generalization and minimize the
influence of additional factors such as polysemy (Berard et al., 2019),
misalignment (Munteanu and Marcu, 2005), and stylistic problems (Hovy et al.,
2020). The dataset should ideally have following characteristics. First, the
vocabulary size should be small and contain only words of high-frequency in
order to avoid problems caused by rare words. In other words, variety of
composition should come from combining different frequent words instead of
word diversity, as suggested in Keysers et al. (2020). Metaphorical words,
which can increase the translation difficulty, should be excluded. Second,
source sentences should not be too long or have complex syntactic structures.
As a result, a sentence can be translated literally, directly, and without
rhetoric. Third, the corpus size should be large enough for training an NMT
model sufficiently.

Widely-adopted corpora such as parallel data released on WMT and
IWSLT222https://wit3.fbk.eu/ have large vocabularies and also contain noisy
sentences and rich morphology Li et al. (2019a), which do not fully meet our
goal. We choose Story Cloze Test and ROCStories Corpora (Mostafazadeh et al.,
2016, 2017) as our data source. The dataset is created for commonsense story
understanding and generation, and consists of $101903$ 5-sentence stories.
These stories are rather simple in items of vocabulary and syntax, but still
contain rich phrases. In addition, the topic is constrained to daily life.

Since the vocabulary size of $42,458$ is large, we select the top $2,000$
frequent words as our vocabulary and extract sentences where the words are
exclusively from the restricted vocab. Moreover, sentences that are longer
than 20 words are removed. In this way, we finally obtain $216,246$ sentences
for parallel data construction. More detailed statistics including comparison
to WMT and IWSLT data are shown in Appendix B.

###  4.2 Parallel Data Construction

We take an MT post-editing method to construct parallel data, first using a
public translation engine to obtain model-generated translations, and then
requesting expert translators to post-edit them. The following aspects are
highlighted:

  * •

Ensure the fluency of translations.

  * •

Ensure word-level matching between translated sentences and source sentences.
Typically, every word should be correctly translated, without omission for
legibility.

Finally, we obtain a parallel dataset of $216,246$ sentences in CoGnition, and
randomly split it into three subsets: $196,246$ sentence pairs for training,
$10,000$ sentence pairs for validation, and $10,000$ sentence pairs as the
random test set. In addition to the above split, we additionally make a
compositional generalization test set, which is described in the next section.

Type | Candidates
---|---
DET | the, every, any, another, each
N | car, dog, girl, doctor, boyfriend, apartment, child, sandwich
chair, farm, building, hat, waiter, airplane, lawyer, peanut, farmer, clown,
bee
ADJ | small, large, red, special, quiet, empty, dirty, lazy, smart, fake, silly
MOD | he liked, at the store, on the floor
V | took, told, found, asked, saw, left, gave, lost, liked
woke, stopped, invited, met, caught, heard, hated, watched, visited, chose
P | to, for, on, with, from, about, before, like, around
inside, without, behind, under, near, towards, except, toward
Table 4:  Atoms used in constructing compounds, sorted by frequency in the
training set.

###  4.3 Compositional Generalization Test Set

We manually construct a special test set dedicated for evaluation of
compositional generalization, by synthesizing new source sentences based on
novel compounds and known contexts.

#### Designing Compound Patterns

We use Berkeley Parser to obtain constituent trees Kitaev and Klein (2018). In
CoGnition, noun phrases (NP), verb phrases (VP) and positional phrases (PP)
are three most frequent constituents, accounting for $85.1\%$ of all
constituents, and thus we construct compounds based on them. According to
syntactic and semantic rules Partee (1995), we choose basic semantic
components as our atoms including determiners (DET), nouns (N), verbs (V),
prepositions (P), adjectives (ADJ), and postpositive modifiers (MOD).
Specifically, postpositive modifiers include prepositional phrases and
relative clauses, and can contain multiple words. We consider them as a single
atom due to their semantic inseparability. In this way, we generate 4 compound
patterns for NP, VP, and PP, respectively, which are listed in Table 3 with
corresponding examples.

#### Making Novel Compounds

We use Stanza Qi et al. (2020) to obtain POS tagging for each word in training
sentences. We construct novel compounds by first selecting atom candidates
with relatively consistent translation in the training set. The frequency of
candidate atoms covers a wide range from $34$ to $73518$. We list full set of
atom candidates in Table 4. For constructing compounds, we enumerate all
possible combinations of atoms according to the patterns in Table 3, and then
remove those that are ungrammatical or likely to cause ethic issues, obtaining
2,160 compounds finally. We do not deliberately make all compounds unseen, yet
only $0.93\%$ of them appear in the training data.

#### Synthesizing Source Sentences

We embed the compounds in specific context to form complete source sentences.
Concretely, we first apply Berkeley Parser on the training sentences to obtain
sentence templates, where certain constituents are replaced by placeholders
according to their constituent types, e.g., “NP-placeholder spent a lot of
time to set up a wedding .”. Then we select 5 sentence templates for each
constructed compound accordingly, so that every compound can be evaluated
under 5 different contexts. To distinguish from VP and PP, we put NP compounds
only in sentences with the placeholder outside VP and PP.

#### Making Reference

To maintain statistical consistency, target translations of synthetic
sentences are also obtained using the same MT post-edit approach. In addition
to the annotation principles listed in 4.2, we set several additional rules:

  * •

Filter sentences with ethical issues and replace them with other synthetic
ones.

  * •

Ensure the accuracy of compound translation.

Finally, we obtain a compositional generalization test set (CG test set) of
$10,800$ parallel sentences. The final dataset statistics is shown in table 5.

Split | # Samples
---|---
Training set | 196,246
Validation set | 10,000
Random test set | 10,000
CG test set | 10,800
Table 5:  Statistics of CoGnition Dataset.

###  4.4 Automatic Evaluation

We mainly adopt human evaluation for the experiments of this paper (Section 5)
for ensuring reliability of findings. Despite its accuracy, human evaluation
can be expensive. To facilitate fast evaluation in future research, we
introduce an automatic evaluation approach to quantify a model’s
generalization ability on our CG test set.

In particular, we manually construct a dictionary for all the atoms based on
the training set (See Appendix C). The prerequisite of correctly translating
one compound is that all of the atom translations should be contained.
Besides, in most cases the translation of nouns should be placed after that of
other atoms. Based on this, we design a heuristic algorithm to determine
whether compounds are translated correctly. With the human annotation as
ground truth, our automatic evaluation tool achieves a precision of $94.80\%$
and a recall of $87.05\%$, demonstrating it can serve as an approximate
alternative to human evaluation.

##  5 Experiments

We conduct experiments on CoGnition dataset and perform human evaluation on
the model results.

###  5.1 Settings

We tokenize the English side using Moses tokenizer and do not apply byte pair
encoding (BPE) Sennrich et al. (2016) due to the small vocabulary (i.e.,
2000). The Chinese sentences are segmented by jieba
segmenter333https://github.com/fxsjy/jieba. We employ BPE with 3,000 merge
operations, generating a vocabulary of 5,500 subwords.

We focus on Transformer Vaswani et al. (2017) because of its state-of-the-art
performance on machine translation Edunov et al. (2018b); Takase and Kiyono
(2021); Raffel et al. (2020); Zhu et al. (2020); Liu et al. (2020b) and better
performance on existing compositional generalization dataset Daniel et al.
(2019). We implement our model using BASE configuration provided by Fairseq
Ott et al. (2019). The model consists of a 6-layer encoder and a 6-layer
decoder with the hidden size 512. We tie input and output embeddings on the
target side. The model parameters are optimized by Adam Kingma and Ba (2015),
with $\beta_{1}=0.1$, $\beta_{2}=0.98$ and $\epsilon=10^{-9}$. The model is
trained for 100,000 steps and we choose the best checkpoint on validation set
for evaluation.

We report character-level BLEU scores using SacreBLEU Post (2018) to measure
the overall translation performance. In addition, we request expert
translators to annotate the correctness of compound translation. Translators
are asked to only focus on examining whether the compound itself is translated
correctly or not, disregarding errors in context. Specifically, a compound is
correct only if its translation contains semantic meaning of all atoms and is
fluent in human language. Since each of the 2,160 compounds is provided with 5
contexts, we can compute the translation error-rate for each compound.

###  5.2 Main Results

Table 6 shows the results. Besides the CG test set, we also list results on
three of its subsets, which only contain NP, VP or PP compounds respectively.
The model achieves a 69.58 BLEU score on the random test set, which partly
indicates distributional consistency and quality of the dataset. In
comparison, the performance on the CG test set drops dramatically by more than
20 BLEU points. Given that the only difference between synthetic sentences and
training sentences is the unseen compounds (i.e., contexts are seen in
training), the decrease of 20 BLEU points indicates that unseen compounds pose
a significant challenge, which is however easy to be overlooked in traditional
evaluation metrics. For example, the model mis-translates “alas , he became
sick from eating all of the peanut butter on the ball” into
“唉，他因为吃掉了球场上所有的花生酱而生病了” (English: alas , he became sick from eating all of the
peanut butter on the field). With a minor mistake on the compound “on the
ball”, the model achieves a sentence-level BLEU of 61.4, despite that the full
sentence meaning is largely affected. In other words, the BLEU score of 69.58
can be misleading since novel compounds can be rare in the random test set.
Such mistakes in generalizing new compounds can severely hinder overall
performance of translation engines in practice, as shown earlier in Table 1.
Also, we calculate BLEU for the original training sentences that provide
contexts for the CG test set (row 3). The model achieves 99.74 BLEU, further
demonstrating that the performance degradation is mainly caused by the unseen
compounds.

Instance-wise, 27.31% compounds are translated incorrectly. However, when
aggregating all 5 contexts, $61.62\%$ compounds suffer at least one incorrect
translation. This suggests that a well-trained NMT model is not robust in
translating compounds, though all atoms within them are highly frequent in the
training set. We also observe that the error rate of PP compounds, $37.72\%$,
is much higher than the other two, $21.94\%$ and $22.25\%$, which we will
discuss in detail in the following section.

Test Set | Error Rate | BLEU
---|---|---
Instance | Aggregate
Random-test | - | - | 69.58
Train | - | - | 99.74
CG-test | 27.31% | 61.62% | 48.66
CG-test/NP | 21.94% | 54.03% | 51.29
CG-test/VP | 22.25% | 55.56% | 47.55
CG-test/PP | 37.72% | 75.28% | 47.14
Table 6:  BLEU score and compound translation error rate on the random test
set and the CG test set.

##  6 Analysis

We conduct experiments to explore in what situations the model is error-prone
by considering compound frequency, compound length, compound structure, atom
frequency, atom co-occurrence, and the complexity of external context.

###  6.1 Compound Frequency

Intuitively, compounds with higher frequencies in the training set are easier
to infer. We classify compounds according to their frequency levels, including
many-shots (frequency higher than 10), few-shots (frequency from 1 to 10) and
zero-shot, and show the error rate for each bucket in Figure 2. The model
translates all the many-shots compounds correctly. For few-shot compounds,
translation error rate increases to $5.00\%$, but is still much lower than
zero-shot compounds with an error rate of $27.53\%$. The result suggests the
model is good at memorizing correspondence between sentence segments. However,
the model deteriorates severely when test samples are unseen in the training
set, which further confirms model’s weakness in compositional generalization
Lake and Baroni (2018).

Figure 2:  Effect of compound frequency on compound translation error rate.

###  6.2 Compound Length

Figure 3:  Effect of compound length on compound translation error rate.

As shown in Figure 3, the error rate grows with the increase of compound
length (i.e., the number of atoms in a compound). Only 4.50% of the shortest
compounds are translated incorrectly, each of which consists of a determiner
and a noun. The error rate increases to 13.72% when the compound length grows
to 3 atoms (e.g., “the smart lawyer”). The longest compounds contain a
determiner, a noun, an adjective, a modifier and a preposition or verb in each
of them, e.g., “taking every special chair he liked”. The error rate increases
to 36.63%, demonstrating that it is more difficult to generalize in longer
compounds, which contain richer semantic information. We conjecture that if
the range of compound is further expanded, the error rate will be much higher.

###  6.3 Atom Frequency

Figure 4:  Effect of atom frequency on compound translation error rate.

We empirically divide compounds into multiple groups according to the minimum
frequency of their atoms, where each group consists of similar numbers of
compounds. The intuition is that the atom with low frequency might be
difficult to translate and therefore hinders the whole compound translation.
We fix the compound length to 3 in order to reduce effects of compound length.

As shown in Figure 4, the error rate has no strong correlation with the atom
frequency. This can be because all atoms in our corpus are simple and
relatively frequent and thus it is easy for the NMT model to memorize the
semantics of most atoms. Therefore, simply increasing atom frequency does not
enhance model’s generalization ability of novel compounds. We observe similar
patterns for compounds of other lengths (Appendix A).

###  6.4 Atom Co-occurrence

Figure 5:  Effect of atom co-occurrence on compound translation error rate.

Although the NMT model may never see a compound, there can exist many local
segments where atoms co-occur. For example, in the unseen compound “the smart
lawyer”, “smart” and “lawyer” may occur within some training sentences.
Intuitively, the compounds of which atoms co-occur more frequently may be
translated better. We calculate pointwise mutual information (PMI) and compare
error rates of compounds with positive or negative mean PMI scores (MPMI):

${\rm MPMI}(C)=\frac{1}{M}\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}{\rm
PMI}(a_{i},a_{j}),$ (1)

where $a_{i}$ is the $i$-th atom in the compound $C$, $N$ is the compound
length, $M$ is the number of possible combinations of two atoms, and PMI score
is computed as:

$PMI(x,y)=log\frac{p(a_{i},a_{j})}{p(a_{i})p(a_{j})},$ (2)

where the probabilities $p(a_{i})$ and $p(a_{i},a_{j})$ are obtained by
dividing the number of n-grams in which one word or both words occur by the
total number of n-grams444We use 5-gram here.

We divide compounds into 4 groups by their length and compare error rates
within each group. As shown in Figure 5, across all groups, the error rates
with positive mean PMI scores are lower than those with negative ones,
verifying our hypotheses.

Figure 6:  Compound translation error rates of different patterns.

###  6.5 Linguistic Factors

Figure 6 shows the error rates of all compound patterns in Table 3. The MOD
atom exerts salient influence on translation error rate. The error rate of
compounds with MOD is 19.78% higher than those without on average. In
contrast, adding ADJ into compounds only increases error rate by 2.66%. The
major difficulty caused by MOD is word reordering. One can translate “the
small dog” monotonically without adjusting word order. However, compounds like
“the dog he liked” require the model to recognize “he liked” as MOD and put
its translation before that of “the dog” in Chinese. We find many cases where
the model translates such compounds without reordering or breaking the
connection between nouns and modifiers.

Across these groups, we can see that the error rate of NP (Pattern 1.*) is
generally lower than that of VP (Pattern 2.*) and PP (Pattern 3.*). Such
phenomenon is more obvious for the patterns without MOD. The reason is that
compounds in Pattern 1.* are generally shorter and contain less semantic and
syntactic information. However, the error rates of Pattern 2.3 and 2.4 are
lower than other patterns with MOD (i.e., Pattern 1.3, 1.4, 3.3 and 3.4),
indicating the model performs better in “V+DET(+ADJ)+NN+MOD”. This can be
because under certain situations the MOD can be useful for correctly
translating verbs, which are more commonly seen in the training set, e.g.,
“found the chair on the floor”.

We also observe that compounds of PP (Pattern 3.*) are more difficult to
translate compared with VP (Pattern 2.*), although both types of compounds
share the same compound length. In the training set, verbs typically have
consistent translations, whereas the meanings of prepositions vary with
contexts. Therefore prepositional compounds are more difficult to translate as
more context information is required to ground their meanings.

###  6.6 Effect of External Context

Figure 7:  Effect of external context on compound translation error rate.

Due to the nature of NMT, the semantic representation of each compound is
context-aware. Intuitively, translation of compounds is also influenced by
external context, which is sentential in our case but can also be document-
level in practice. We investigate effects of context lengths and sentence
comprehension difficulty. In particular, the context length is calculated by
subtracting the sentence length by the number of words in the compound.
Comprehension difficulty of the training sentences which provide contexts, is
quantified by the dependency distance Liu (2008): ${\rm
MMD}(x)=\frac{1}{N-1}\sum_{i}^{N}D_{i}$, where $N$ is the number of words in
the sentence and $D_{i}$ is the dependency distance of the $i$-th syntactic
link of the sentence.

The results are shown in Figure 7. The translation error rate increases stably
with the context length as well as the dependency distance. These observations
demonstrate that the generalization for novel compounds correlates strongly
with context complexity. Sentences with higher dependency distances are harder
for model to comprehend during training. Given that our test sentences are
restricted to 20 words, compositional generalization can be more challenging
in practice where average sentence lengths can be much longer.

##  7 Conclusion

We proposed a dedicated parallel dataset for measuring compositional
generalization of NMT and quantitatively analyzed a Transformer-based NMT
model manually. Results show that the model exhibits poor performance on novel
compound translation, which demonstrates that the NMT model suffers from
fragile compositionality, and it can be easily overlooked under transitional
metrics. To the best of our knowledge, we are the first one to propose a
practical benchmark for compositionality of NMT, which can be a testbed for
models tailored for this specific problem.

##  8 Ethics Consideration

As mentioned, we collected our data from Story Cloze Test and ROCStories
Corpora that all are public to academic use, and they contain no sensitive
information (Mostafazadeh et al., 2016, 2017). The legal advisor of our
institute confirms that the sources of our data are freely accessible online
without copyright constraint to academic use. Our data construction involves
manual annotation. Annotators were asked to post-edit machine translation and
filter out samples that may cause ethic issues, which do not involve any
personal sensitive information.

We hired 4 annotators who have degrees in English Linguistics or Applied
Linguistics. Before formal annotation, annotators were asked to annotate $100$
samples randomly extracted from the dataset, and based on average annotation
time we set a fair salary (i.e., 32 dollars per hour) for them. During their
training annotation process, they were paid as well.

## Acknowledgment

Yue Zhang is the corresponding author. We thank all reviewers for their
insightful comments. This work is supported by National Natural Science
Foundation of China (NSFC) under grant No.61976180 and a grant from Lan-bridge
Information Technology Co., Ltd. We thank colleagues from Lan-bridge for
examining data and evaluating results. Major contributors include Xianchao
Zhu, Guohui Chen, Jing Yang, Jing Li, Feng Chen, Jun Deng and Jiaxiang Xiang.

## References

  * Andreas (2020) Jacob Andreas. 2020.  Good-enough compositional data augmentation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7556–7566, Online. Association for Computational Linguistics.
  * Bastings et al. (2018) Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018\.  Jump to better conclusions: SCAN both left and right.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 47–55, Brussels, Belgium. Association for Computational Linguistics.
  * Belinkov and Bisk (2018a) Yonatan Belinkov and Yonatan Bisk. 2018a.  Synthetic and natural noise both break neural machine translation.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Belinkov and Bisk (2018b) Yonatan Belinkov and Yonatan Bisk. 2018b.  Synthetic and natural noise both break neural machine translation.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Belinkov et al. (2017) Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017.  Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1–10, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Berard et al. (2019) Alexandre Berard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, and Vassilina Nikoulina. 2019.  Machine translation of restaurant reviews: New corpus for domain adaptation and robustness.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 168–176, Hong Kong. Association for Computational Linguistics.
  * Bisazza and Tump (2018) Arianna Bisazza and Clara Tump. 2018.  The lazy encoder: A fine-grained analysis of the role of morphology in neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2871–2876, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2020) Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020.  Compositional generalization via neural-symbolic stack machines.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cheng et al. (2019) Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019.  Robust neural machine translation with doubly adversarial inputs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4324–4333, Florence, Italy. Association for Computational Linguistics.
  * Cheng et al. (2018) Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018.  Towards robust neural machine translation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1756–1766, Melbourne, Australia. Association for Computational Linguistics.
  * (11) Noam Chomsky.  _Syntactic Structures_.  Mouton, The Hague.
  * Daniel et al. (2019) Jeanne E. Daniel, Willie Brink, Ryan Eloff, and Charles Copley. 2019.  Towards automating healthcare question answering in a noisy multilingual low-resource setting.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 948–953, Florence, Italy. Association for Computational Linguistics.
  * Dessì and Baroni (2019) Roberto Dessì and Marco Baroni. 2019.  CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networks.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3919–3923, Florence, Italy. Association for Computational Linguistics.
  * Ding et al. (2017) Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017.  Visualizing and understanding neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1150–1159, Vancouver, Canada. Association for Computational Linguistics.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018.  On adversarial examples for character-level neural machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 653–663, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Edunov et al. (2018a) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018a.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Edunov et al. (2018b) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018b.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Fadaee and Monz (2020a) Marzieh Fadaee and Christof Monz. 2020a.  The unreasonable volatility of neural machine translation models.  In _Proceedings of the Fourth Workshop on Neural Generation and Translation_ , pages 88–96, Online. Association for Computational Linguistics.
  * Fadaee and Monz (2020b) Marzieh Fadaee and Christof Monz. 2020b.  The unreasonable volatility of neural machine translation models.  In _Proceedings of the Fourth Workshop on Neural Generation and Translation_ , pages 88–96, Online. Association for Computational Linguistics.
  * Gordon et al. (2020) Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. 2020.  Permutation equivariant models for compositional generalization in language.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018.  Achieving human parity on automatic chinese to english news translation.  _CoRR_ , abs/1803.05567.
  * He et al. (2019) Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019\.  Towards understanding neural machine translation with word importance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 953–962, Hong Kong, China. Association for Computational Linguistics.
  * Hovy et al. (2020) Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari. 2020.  Can you translate that into man? commercial machine translation systems include stylistic biases.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, WA, USA. Association for Computational Linguistics_.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 6700–6709. Computer Vision Foundation / IEEE.
  * Johnson et al. (2017a) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017a.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 1988–1997. IEEE Computer Society.
  * Johnson et al. (2017b) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017b.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 1988–1997. IEEE Computer Society.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020.  Measuring compositional generalization: A comprehensive method on realistic data.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency parsing with a self-attentive encoder.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
  * Lake (2019) Brenden M. Lake. 2019.  Compositional generalization through meta sequence-to-sequence learning.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 9788–9798.
  * Lake and Baroni (2018) Brenden M. Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2879–2888. PMLR.
  * Li et al. (2019a) Bei Li, Yinqiao Li, Chen Xu, Ye Lin, Jiqiang Liu, Hui Liu, Ziyang Wang, Yuhao Zhang, Nuo Xu, Zeyang Wang, Kai Feng, Hexuan Chen, Tengbo Liu, Yanyang Li, Qiang Wang, Tong Xiao, and Jingbo Zhu. 2019a.  The NiuTrans machine translation systems for WMT19.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 257–266, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2019b) Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. 2019b.  Compositional generalization for primitive substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4293–4302, Hong Kong, China. Association for Computational Linguistics.
  * Li et al. (2019c) Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. 2019c.  Compositional generalization for primitive substitutions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4293–4302, Hong Kong, China. Association for Computational Linguistics.
  * Liu (2008) Haitao Liu. 2008.  Dependency distance as a metric of language comprehension difficulty.  _Journal of Cognitive Science_ , 9(2):159–191.
  * Liu et al. (2020a) Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020a.  Very deep transformers for neural machine translation.  _CoRR_ , abs/2008.07772.
  * Liu et al. (2020b) Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020b.  Very deep transformers for neural machine translation.  _CoRR_ , abs/2008.07772.
  * Loula et al. (2018) João Loula, Marco Baroni, and Brenden Lake. 2018.  Rearranging the familiar: Testing compositional generalization in recurrent networks.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 108–114, Brussels, Belgium. Association for Computational Linguistics.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one? In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 14014–14024.
  * Michel and Neubig (2018) Paul Michel and Graham Neubig. 2018.  MTNT: A testbed for machine translation of noisy text.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 543–553, Brussels, Belgium. Association for Computational Linguistics.
  * (41) Richard Montague.  Universal grammar.  In Richmond H. Thomason, editor, _Formal Philosophy: Selected Papers of Richard Montague_ , 222–247. Yale University Press, New Haven, London.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849, San Diego, California. Association for Computational Linguistics.
  * Mostafazadeh et al. (2017) Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017.  LSDSem 2017 shared task: The story cloze test.  In _Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics_ , pages 46–51, Valencia, Spain. Association for Computational Linguistics.
  * Munteanu and Marcu (2005) Dragos Stefan Munteanu and Daniel Marcu. 2005.  Improving machine translation performance by exploiting non-parallel corpora.  _Computational Linguistics_ , 31(4):477–504.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Partee (1995) Barbara Partee. 1995.  Lexical semantics and compositionality.  _An invitation to cognitive science: Language_ , 1:311–360.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Raunak et al. (2019) Vikas Raunak, Vaibhav Kumar, and Florian Metze. 2019.  On compositionality in neural machine translation.  _CoRR_ , abs/1911.01497.
  * Russin et al. (2019a) Jake Russin, Jason Jo, Randall C. O’Reilly, and Yoshua Bengio. 2019a.  Compositional generalization in a deep seq2seq model by separating syntax and semantics.  _CoRR_ , abs/1904.09708.
  * Russin et al. (2019b) Jake Russin, Jason Jo, Randall C. O’Reilly, and Yoshua Bengio. 2019b.  Compositional generalization in a deep seq2seq model by separating syntax and semantics.  _CoRR_ , abs/1904.09708.
  * Saxton et al. (2019) David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019.  Analysing mathematical reasoning abilities of neural models.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shi et al. (2016) Xing Shi, Inkit Padhi, and Kevin Knight. 2016.  Does string-based neural MT learn source syntax? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1526–1534, Austin, Texas. Association for Computational Linguistics.
  * Takase and Kiyono (2021) Sho Takase and Shun Kiyono. 2021.  Lessons on parameter sharing across layers in transformers.  _CoRR_ , abs/2104.06022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Voita et al. (2019a) Elena Voita, Rico Sennrich, and Ivan Titov. 2019a.  The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4396–4406, Hong Kong, China. Association for Computational Linguistics.
  * Voita et al. (2019b) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019b.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2020) Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020.  On the inference calibration of neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3070–3079, Online. Association for Computational Linguistics.
  * Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020.  Incorporating BERT into neural machine translation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A Atom Frequency

For compounds of other lengths, we also compute their error rates with respect
to minimum atom frequency. As shown in Figure 8, 9 and 10, the error rate does
not correlates with atom frequency across all compound lengths.

Figure 8:  Effect of atom frequency with compound length fixed to 2.  Figure
9:  Effect of atom frequency with compound length fixed to 4.  Figure 10:
Effect of atom frequency with compound length fixed to 5.

##  Appendix B Data Statistics

Table 7 and Table 8 lists statistics of several monolingual data sources,
compared with the data source (ROC-Filter) used in constructing the CoGnition
dataset. We can see that our dataset has both shorter sentences and vocabulary
made up of more frequent words.

Data Property | Vocab | #Tokens | #Sents
---|---|---|---
WMT17 En-Zh | 1,201,752 | 518,286,577 | 20,616,495
IWSLT17 En-Zh | 70,950 | 4,715,201 | 231,266
ROC-Original | 42,458 | 5,283,521 | 532,093
ROC-Filter | 2,000 | 2,096,524 | 216,246
Table 7:  Statistics of data sources: vocabulary size, number of tokens and number of sentences.  Data Property | Avg Len | Avg Freq | Min Freq
---|---|---|---
WMT17 En-Zh | 25.1 | 431.3 | 1
IWSLT17 En-Zh | 20.4 | 66.5 | 1
ROC-Original | 9.3 | 124.4 | 1
ROC-Filter | 9.7 | 1048.3 | 35
Table 8:  Statistics of data sources: average sentence length, average token
frequency and minimum token frequency.

##  Appendix C Lexicon

Part of the lexicon for automatic evaluation is shown in Table 9.

Atom | Lexical Translation
---|---
dog | 狗/犬
doctor | 医生
sandwich | 三明治
hat | 帽
waiter | 服务员
lawyer | 律师
peanut | 花生
farmer | 农夫/农场主/农贸市场/农民
small | 小
red | 红
dirty | 脏
lazy | 懒
smart | 聪明/明智/智能
the | -
every | 每/所有
any | 任何
another | 另/又/再/还/别
each | 每
he liked | 他喜欢的
at the store | 店里/商店
Table 9:  Lexicon for automatic evaluation.


# Gaussian Blur and Relative Edge Response

Austn C. Bergstrom, David Conran, David W. Messinger
Chester F. Carlson Center for Imaging Science
Rochester Institute of Technology
Rochester, NY
<EMAIL_ADDRESS>
###### Abstract

It is often convenient to use Gaussian blur in studying image quality or in
data augmentation pipelines for training convoluional neural networks. Because
of their convenience, Guassians are sometimes used as first order
approximations of optical point spread functions. Here, we derive and evaluate
closed form relationships between Gaussian blur parameters and relative edge
response, finding good agreement with measured results. Additionally, we
evaluate the extent to which Gaussian approximations of optical point spread
functions can be used to predict relative edge response, finding that Gaussian
relationships provide a reasonable approximation in limited circumstances but
not across a wide range of optical parameters.

_K_ eywords Relative edge response $\cdot$ Gaussian blur $\cdot$ image quality

##  1 Introduction

Relative edge response (RER) represents a convenient image quality metric
summarizing the sharpness of an image by quantifying the spatial derivative of
an image in the direction normal to an edge. Of note, RER is one of the three
parameters used in the General Image Quality Equation used by the remote
sensing community to quantify the utility of overhead images [1].

Because of the Gaussian distribution’s mathematical simplicity, its ubiquity
in image processing libraries [2, 3, 4], and its qualitative similarity to
optical point spread functions, it is often convenient to use Gaussian kernels
to blur images. For instance, a range of studies on the relationship between
image quality and deep learning have used Gaussian blur to manipulate image
sharpness and spatial frequency content [5, 6, 7, 8, 9].

Given the convenience and ubiquity of Gaussian blur kernels in studying image
quality, we derive and evaluate closed form functions that approximate the
relationship between Gaussian blur and RER. Additionally, we evaluate the
extent to which we can approximate optical point spread functions with
Gaussians for the purpose of quickly mapping system point spread functions to
RER using the Gaussian RER relationships that we have derived. We observe that
simple Gaussian approximations do not directly predict the RER that results
from convolving an ideal edge with a realistic system point spread function.

Specifically, we make the following contributions:

  * •

We derive simple, closed form relationships between RER and Gaussian blur.

  * •

We verify our RER relationships using synthetic edge images blurred with
Gaussian kernels.

  * •

We show that Gaussian approximations of optical point spread functions yield
images with RER that differs from the RER produced by the optical point spread
functions themselves.

##  2 Derivations

###  2.1 First order approximation

Figure 1: Illustration of convolution and the effect of PSF width on relative
edge response (RER). To first order, RER is given by the slope of the image at
the edge location.

We begin by noting that we can approximate an imager as a linear shift
invariant (LSI) system, allowing us to model the action of the system as a
convolution of an input scene with the system’s point spread function, where
the convolution operation in given by

$g\left(x\right)=f\left(x\right)*h\left(x\right)=\int_{-\infty}^{\infty}h\left(\alpha\right)f\left(x-\alpha\right)d\alpha=\int_{-\infty}^{\infty}h\left(x-\alpha\right)f\left(\alpha\right)d\alpha.$
(1)

To derive the relationship between Gaussian blur and RER, we consider a
1-dimensional image $g\left(x\right)$ formed by convolution of the edge object
$f\left(x\right)$ with the the system point spread function $h\left(x\right)$.
The edge object is described by the unit step function

$f\left(x\right)=\text{STEP}\left(x\right)=\begin{cases}0&\text{ where }x<0\\\
\frac{1}{2}&\text{ where }x=0\\\ 1&\text{ where }x>0\end{cases},$ (2)

and we approximate our point spread function $h\left(x\right)$ as the
normalized Gaussian

$h\left(x\right)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-x^{2}}{2\sigma^{2}}\right).$
(3)

Applying the definition of a convolution, we see that we can express our
1-dimensional image $g\left(x\right)$ with the integral

$g\left(x\right)=\int_{-\infty}^{\infty}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)\text{STEP}\left(x-\alpha\right)d\alpha.$
(4)

Exploiting the properties of a step function and reversing the limits of
integration for convenience, which is allowable since relative edge response
is sign independent, we reach the expression for our edge image

$g\left(x\right)=\int_{-\infty}^{x}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha$
(5)

RER describes the sharpness of an image based on the slope of its edge
response function; RER is a first order approximation of the spatial
derivative of an image at the edge location [10]. In our 1D example,
therefore, we have

$\text{RER}\approx\frac{d}{dx}\int_{-\infty}^{x}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha\bigg{\rvert}_{x=0}=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-x^{2}}{2\sigma^{2}}\right)\bigg{\rvert}_{x=0}=\frac{1}{\sigma\sqrt{2\pi}},$
(6)

which is illustrated by Fig. 1.

###  2.2 Refined approximation

While RER is an approximation of the derivative of an image at the location of
an edge, it is by definition a discrete approximation of this slope, measured
by interpolating the edge spread function at $\pm\frac{1}{2}$ pixel [10].
Without accounting for the discrete sampling inherent in measurement, RER
could approach infinity for sufficiently narrow point spread functions. To
account for this sampling, we start wth edge image

$g\left(x\right)=\int_{-\infty}^{x}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha$
(7)

formed by a system with a Gaussian PSF of standard deviation $\sigma$.
Applying the defintion of RER that accounts for discrete measurement, we have

$\textup{RER}=g\left(0.5\right)-g\left(-0.5\right),$ (8)

which is much less than $\frac{1}{\sigma\sqrt{2\pi}}$ as $\sigma\rightarrow
0$. To account for the effects of discrete sampling at $x=\pm 0.5$, we note
that there is a convenient closed form relationship for
$g\left(0.5\right)-g\left(0.5\right)$. If we re-write $g\left(x\right)$ as the
sum of two integrals, with

$g\left(x\right)=\int_{-\infty}^{0}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha+\int_{0}^{x}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha,$
(9)

we can discard first term since it is constant and falls out in subtraction,
finding that

$\textup{RER}=\int_{0}^{\frac{1}{2}}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha-\int_{0}^{-\frac{1}{2}}\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-\alpha^{2}}{2\sigma^{2}}\right)d\alpha.$
(10)

In this form, we can see that RER is given by the difference of two scaled
error functions, where the error function $\textup{erf}\left(x\right)$ is
given by

$\textup{erf}\left(x\right)=\frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}dt.$
(11)

Using the change of variables

$t=f\left(\alpha\right)=\frac{1}{\sqrt{2}\sigma}\alpha,$ (12)

we arrive at the expression

$\textup{RER}=\frac{1}{\sigma\sqrt{2\pi}}\left(\int_{0}^{\frac{1}{2\sqrt{2}\sigma}}e^{-\alpha^{2}}\sqrt{2}\sigma
d\alpha-\int_{0}^{\frac{-1}{2\sqrt{2}\sigma}}e^{-\alpha^{2}}\sqrt{2}\sigma
d\alpha\right),$ (13)

which simplifies to

$\textup{RER}=\frac{1}{2}\left(\textup{erf}\left(\frac{1}{2\sqrt{2}\sigma}\right)-\textup{erf}\left(\frac{-1}{2\sqrt{2}\sigma}\right)\right)=\textup{erf}\left(\frac{1}{2\sqrt{2}\sigma}\right)$
(14)

###  2.3 Extension to multiple blur stages

In studying image quality, we are likely to be concerned with estimating the
RER of images that have two distinct blur contributions, first by their system
PSF and second in post-processing. For a real image $g\left(x\right)$,
therefore, we have

$g\left(x\right)=f\left(x\right)*h_{0}\left(x\right)*h_{1}\left(x\right)$ (15)

where $f$ is the object imaged, $h_{0}$ is the system PSF, and $h_{1}$ is the
Gaussian blur kernel applied in post-processing. If we approximate the optical
psf $h_{0}$ as a Gaussian of standard deviation $\sigma_{0}$, then

$h_{0}\left(x\right)=\frac{1}{\sigma_{0}\sqrt{2\pi}}\exp\left(\frac{-x^{2}}{2\sigma_{0}^{2}}\right),$
(16)
$h_{1}\left(x\right)=\frac{1}{\sigma_{1}\sqrt{2\pi}}\exp\left(\frac{-x^{2}}{2\sigma_{1}^{2}}\right),$
(17)

and

$g\left(x\right)=f\left(x\right)*h_{effective}\left(x\right),$ (18)

where

$h_{effective}\left(x\right)=h_{0}\left(x\right)*h_{1}\left(x\right).$ (19)

Here, we can apply the filter theorem (discussed in detail in [11]) and note
that the Fourier transform of our image
$\mathfrak{F}\left\\{g\left(x\right)\right\\}=G\left(\xi\right)$ is given by

$G\left(\xi\right)=\mathfrak{F}\left\\{f\left(x\right)\right\\}\cdot\mathfrak{F}\left\\{h_{effective}\left(x\right)\right\\}=F\left(\xi\right)\cdot
H_{effective}\left(\xi\right),$ (20)

where $H_{effective}$ represents the effective optical transfer function and
is given by

$H_{effective}\left(\xi\right)=H_{0}\left(\xi\right)\cdot
H_{1}\left(\xi\right).$ (21)

Using the Fourier properties of a Gaussian, we have

$H_{0}\left(\xi\right)=\exp\left({2\pi^{2}\sigma_{0}^{2}\xi^{2}}\right),$ (22)
$H_{1}\left(\xi\right)=\exp\left({2\pi^{2}\sigma_{1}^{2}\xi^{2}}\right),$ (23)

and

$H_{effective}\left(\xi\right)=\exp\left(2\pi^{2}\left(\sigma_{0}^{2}+\sigma_{1}^{2}\right)\xi^{2}\right).$
(24)

Having found the effective transfer function, we can find the effective point
spread function by taking the inverse Fourier transform of the effective
transfer function according to

$h_{effective}\left(x\right)=\mathfrak{F}^{-1}\left\\{H_{0}\left(\xi\right)\cdot
H_{1}\left(\xi\right)\right\\}.$ (25)

Here,

$h_{effective}\left(x\right)=\mathfrak{F^{-1}}\left\\{\exp\left(2\pi^{2}\left(\sigma_{0}^{2}+\sigma_{1}^{2}\right)\xi^{2}\right)\right\\}$
$=\frac{1}{\sqrt{2\pi\left(\sigma_{0}^{2}+\sigma_{1}^{2}\right)}}\exp\left(\frac{x^{2}}{2\left(\sigma_{0}^{2}+\sigma_{1}^{2}\right)}\right)$
(26)
$=\frac{1}{\sqrt{2\pi}\sigma_{effective}}\exp\left(\frac{x^{2}}{2\sigma_{effective}^{2}}\right),$

where $\sigma_{effective}=\sqrt{\sigma_{0}^{2}+\sigma_{1}^{2}}$.

Accordingly, for two stages of Gaussian blur, we can model the effective point
spread function as a Gaussian of standard deviation $\sigma_{effective}$,
where

$\sigma_{effective}=\sqrt{\sigma_{0}^{2}+\sigma_{1}^{2}}.$ (27)

##  3 Method

(a) $\sigma=1$

(b) $\sigma=2$

(c) $\sigma=4$

(d) $\sigma=8$

Figure 2: Synthetic edge images with varied Gaussian blur.

To evaluate the RER models derived above, we generated synthetic edge image
chips (Fig. 2) and measured RER using the slanted edge method outlined in
[10], with our code available at [12]. Specifically, we generated ideal
slanted edges by defining the location of a near-vertical in an $xy$-plane
onto which to superposed our pixel grid. We set pixels to the left of the edge
equal to our dark value and pixels on the right side of the edge to our light
value, and we assigned values to the border pixels according to the fraction
of each on the light and dark side of the edge. Next, applied varying levels
of Gaussian blur to our ideal edge image to generate edge images of varying
RER. Last, we down-sampled a subset of our edge images using integer pixel
binning. Table 1 shows the blur parameters used for image chips without
downsampling, and Tab. 2 displays the parameters used for image chips that
were down-sampled after blurring.

Table 1: Parameters used in generating edge images with one and two-stage Gaussian blur and no down sampling First stage blur | 0.1 - 3 pixels
---|---
Second stage blur | 0 - 3 pixels
Combined blur (two stage images) | 0.1 - 4.25 pixels
Table 2: Parameters used in generating edge images with two-stage Gaussian blur and integer down-sampling First stage blur | 0.75 - 6 pixels
---|---
Second stage blur | 0 - 6 pixels
Combined blur (before down sampling) | 0.75 - 8.5 (pixels)
down sampling ratios | 2, 3, 4, 5 (dimensionless)
Combined effective blur (after down sampling) | 0.15 - 4.2 pixels

We performed this down sampling in order to approximate the process of
applying optical blur in the analog domain and then down-sampling with a focal
plane array. We used integer pixel binning in order to avoid the effects of
pixel interpolation. For images with down-sampling applied, blur is linearly
scaled by the down-sampling ratio.

Finally, we assessed the extent to which Gaussian approximations of optical
PSFs yielded equivalent RER values when used to blur synthetic edge chips. Do
do so, we simulated system point spread functions using the code developed and
described by Conran in [13]. Conran’s model incorporates the optical system
parameters shown in Tab. 3, with our simulations encompassing the ranges
shown.

Table 3: System parameters used in optical PSF simulation f-number | 20 (dimensionless)
---|---
aperture fill factor | 0.8 (dimensionless)
pixel pitch | 8 $\mu m$
wavelength | 0.8 $\mu m$
wavefront error | 0.025 - 0.135 $\mu m$
smear | 0.05 - 0.15 pixel
rms jitter | 2.6e-5 - 5e-4 pixel
down sampling ratios | 1, 2 (dimensionless)

For each optical PSF generated, we found the nearest two dimensional Gaussian
using a non-linear least squares fitting routine. For each simulated optical
PSF and its Gaussian best fit sibling, we blurred an ideal synthetic edge and
measured resulting RER.

##  4 Results and Analysis

###  4.1 Gaussian Point Spread Functions

For synthetic edge images blurred with Gaussian kernels, we observed good
agreement between our ideal edge slope model in Eqn. 6 for $\sigma\gtrsim 1$
pixel, with predicted RER exploding as $\sigma\rightarrow 0$. Our model
incorporating discrete sampling in Eqn. 14 avoids the small $\sigma$
catastrophy of the first model but still does not fit the data particularly
well for $\sigma<1$. Figure 3 depicts these fits for modeled and measured RER
using both of these models for edges blurred once and for edges blurred in two
stages, where the combined standard deviation $\sigma_{effective}$ is
calculated according to Eqn. 27 for the edges blurred twice.

(a) $0.1\leq\sigma\leq 3$, single blur stage

(b) $0.1\leq\sigma_{effective}\leq 3$, two blur stages

(c) $0.1\leq\sigma\leq 2$, single blur stage

(d) $0.1\leq\sigma_{effective}\leq 2$, two blur stages

Figure 3: Relative edge response modeled and measured.

Two factors explain the relatively poor performance performance at small
$\sigma$ of our Eqn. 14 model. First, the model over-predicts RER for very
small $\sigma$ because it neglects the the transfer function of the pixels
themselves. Second, at very small $\sigma$, our blur kernels cease to be
Gaussian in character due to the discrete sampling inherent in kernel
generation. Figure 4 shows the 1-dimensional profiles of the blur kernels from
the Torchvision library that we used in generating our edge images. As
standard deviation $\sigma$ approaches 0, our blur kernels lose their Gaussian
character and approach discrete delta functions. This non-Gaussian character
of our blur kernels tends to drive RER up for small $\sigma$, leading our 14
model to under-predict RER for moderately small $\sigma$. Because of this
effect, our simplest model in Eqn. 6 yields the best prediction for RER when
$\sigma>0.5$ pixels and the images have not been down-sampled.

Figure 4: Gaussian kernel profiles from the Torchvision library. Here, we see
that because of discrete sampling, blur kernels lose their Gaussian character
for small $\sigma$

To work around the effects of discrete blur kernels, we next consider
synthetic edge images blurred at high resolution and then down-sampled. With
down-sampling after blurring, our final Gaussian effective standard deviation
is scaled by the same ratio as the image, enabling larger kernel standard
deviations and therefore kernels of a more Gaussian character before down-
sampling occurs. For our synthetic edge images blurred this way to better
approximate optical imaging, we observed that our simplest model (Eqn. 6)
still performs reasonably well for $\sigma>1$ pixel, above which we avoid the
problems inherent in the $1/\sigma$ relationship. We also see that our Eqn. 14
discrete sampling now systematically over predicts RER at all small $\sigma$,
as shown in Fig. 5.

Figure 5: RER as a function of Gaussian blur, measured and predicted by ideal
edge slope model (Eqn. 6) and discrete sampling model (Eqn. 14). These

We can understand this divergence between RER as measured and RER as predicted
by the discrete sampling model at low $\sigma$ by recognizing that the
transfer function of the pixels themselves significantly impacts RER at low
blur values. Importantly, this transfer function is _distinct_ from the
discrete sampling effects considered in 2.2 where we derived the discrete
sampling model. While the discrete sampling model accounts for the impact of
approximating the derivative of the edge spread function by measuring
discretely at $\pm 1/2$ pixel, the pixel transfer function significantly
changes the edge spread function by averaging the image signal across the
width of the pixel.

To estimate the impact of the pixel transfer function, we calculated the
combined transfer function

$H_{combined}\left(\xi;\sigma\right)=H_{Gauss}\left(\xi;\sigma\right)\cdot
H_{pixel}\left(\xi\right),$ (28)

for a range of $\sigma$ values. Because we have specified $\sigma$ in units of
pixels, we can conveniently treat our pixels as being unit width, yielding the
pixel transfer function

$H_{pixel}\left(\xi\right)=\mathfrak{F}\left\\{\textup{RECT}\left(x\right)\right\\}=\textup{SINC}\left(\xi\right),$
(29)

where $\textup{RECT}\left(x\right)$ and $\textup{SINC}\left(\xi\right)$ are a
Fourier transform pair, expressed by

$\textup{RECT}\left(\xi\right)\equiv\begin{cases}1&\textup{ when
}-\frac{1}{2}<\xi<\frac{1}{2}\\\ \frac{1}{2}&\textup{ when
}\xi=\pm\frac{1}{2}\\\ 0&\textup{otherwise}\\\ \end{cases}$ (30)

and

$\textup{sinc}\left(x\right)\equiv\frac{\textup{sin}\left(\pi x\right)}{\pi
x}$ (31)

respectively. The transfer function of a Gaussian is a second Gaussian with
standard deviation inversely proportional to the first Gaussian’s standard
deviation [11], leading to transfer function

$H_{Gauss}=\mathfrak{F}\left\\{\frac{1}{\sigma\sqrt{2\pi}}\exp\left(\frac{-x^{2}}{2\sigma^{2}}\right)\right\\}=\exp\left({2\pi^{2}\sigma^{2}\xi^{2}}\right)$
(32)

for a Gaussian PSF of standard deviation $\sigma$. For a Gaussian PSF of
standard deviation $\sigma$, therefore, we have a combined transfer function

$H_{combined}\left(\xi;\sigma\right)=\exp\left({2\pi^{2}\sigma^{2}\xi^{2}}\right)*\textup{SINC}\left(\xi\right).$
(33)

(a) $\sigma=0.1$, $\sigma_{f}=0.354$

(b) $\sigma=0.25$, $\sigma_{f}=0.404$

(c) $\sigma=0.5$, $\sigma_{f}=0.583$

(d) $\sigma=1$, $\sigma_{f}=1.042$

Figure 6: Combined transfer functions for varying Gaussian PSF widths and unit
width pixels. Note that in all cases $\sigma$ refers to the standard deviation
of a normalized Gaussian PSF, whereas it is inversely proportional to the
width of the Gaussian transfer function (see Eqn. 32). Here, we see that the
pixel transfer function begins to significantly change the combined transfer
function for $\sigma<0.5$ pixels.

Figure 6 illustrates the interactions between the two terms in this transfer
function. For wide Gaussian PSFs with large $\sigma$, we have narrow Gaussian
transfer functions, in which case the pixel transfer function has minimal
impact. Conversely, for narrow Gaussian PSFs with small $\sigma$, we have wide
Gaussian transfer functions, in which case the pixel transfer function has a
significant impact. We fit a Gaussian of the form
$H_{f}\left(\xi\right)=\exp\left({2\pi^{2}\sigma_{f}^{2}\xi^{2}}\right)$ to
each combined transfer function $H_{combined}$ and observed that for small
$\sigma$, the difference between the original PSF blur parameter and the best
fit Gaussian blur parameter $\sigma_{f}$ varied significantly due to the
impact of the pixel transfer function. Figure 7 shows the difference between
original $\sigma$ and $\sigma_{f}$ for varied $\sigma$, with the residuals
following an approximately Lorentzian pattern, where the Lorentzian [14] is
given by

$P\left(x\right)=\frac{1}{\pi}\frac{b}{\left(x-m\right)^{2}+b^{2}}.$ (34)

Figure 7: Residuals between original Gaussian $\sigma$ and the best fit
$\sigma_{f}$ for the combined transfer functions given by 33

Applying the fit parameters shown in Fig. 7 to Eqn. 34, we are able to
estimate a combined Gaussian that accounts for the the Gaussian PSF as well as
the pixel transfer function. Adding this correction to the original blur
parameter $\sigma$, where

$\sigma_{corrected}=\sigma+\frac{1}{\pi}\frac{b}{\left(\sigma-m\right)^{2}+b^{2}},$
(35)

and using the corrected blur value in the RER model given by Eqn. 14, we
observe excellent agreement between modeled and measured RER across a wide
range of blur parameter $\sigma$, as shown in Fig. 8.

(a) With and without blur correction

(b) With blur correction, small $\sigma$

Figure 8: RER as a function of Gaussian blur, measured and predicted by
discrete sampling model (Eqn. 14) with and without pixel transfer function
correction (Eqn. 35) (left) The fully corrected model performs well down to
roughly $\sigma\geq 0.25$.

Given the good agreement between modeled and measured RER, we can conclude
that the model given by Eqn. 14 can accurately predict RER as a function of
Gaussian blur down to very low $\sigma$ if we apply a correction to account
for the pixel transfer function. We highlight that the correction itself is
found without reference to RER but is the result of finding the best fit
Gaussian for the combined transfer function that results when a Gaussian PSF
is convolved with a unit width pixel RECT function; our RER model is never fit
to RER results, suggesting that the derivation from fist principles is sound.

###  4.2 Gaussian Approximations of Optical Point Spread Functions

Having established that we can predict the RER of a system with a purely
Gaussian PSF, we next examined whether Gaussian approximations of simulated
optical PSFs could be used to accurately predict RER. Table 3 shows the
parameters used for our optical simulation. We note that our simulated PSFs
correspond to a $Q=2$ system before down-sampling, where $Q$ is the optical
quality factor given by

$Q=\frac{\lambda F}{p},$ (36)

for wavelength $\lambda$, f-number $F$, and pixel pitch $p$. 111Conceptually,
$Q$ is the ratio of the Airy radius to pixel pitch; it is also the ratio of
detector cutoff frequency to optical cutoff frequency for a sensor with a 100%
fill factor. After down sampling, our pixel pitch effectively doubles, which
causes the drop in $Q$. Figure 9 shows the RER that results from simulated
optical PSFs and their best fit Gaussian approximations.

Figure 9: RER for images blurred with simulated optical PSFs and their best
fit Gaussian approximations. Simulated PSFs are from a combination of $Q=1$
and $Q=2$ systems with WFE, jitter, and smear values varying within the ranges
shown in 3.

From these results, we observe that the specific shape of optical kernels
matters; the RER of an image blurred by one of our simulated optical kernels
differs from the RER of an image blurred by its best-fit Gaussian kernel. We
note that we used other optical simulation parameters and observed similar
differences (not shown here) between the RER of images blurred with our
optical kernels and their least squares Gaussian siblings.

##  5 Conclusion

Here, we have derived closed form approximations for the relationship between
Gaussian blur and relative edge response, and we have shown the limitations of
these approximations. For images blurred at their final resolution, we find
that our models do a reasonably good job of predicting RER down to blur
standard deviations of approximately $0.5$ pixels, with our simplest
relationship given by

$\text{RER}\approx\frac{1}{\sigma\sqrt{2\pi}}$ (37)

performing better than our refined approximation when $\sigma>0.5$ but
diverging rapidly as $\sigma\rightarrow 0$.

For images blurred at high resolution and then down-sampled, which better
approximates the process of optical blur and sampling by a focal plane, we
find that our refined approximation,

$\text{RER}\approx\textup{erf}\left(\frac{1}{2\sqrt{2}\sigma}\right),$ (38)

yields good RER predictions for blur values greater than roughly 0.75 pixels,
above which the Gaussian transfer function dominates the pixel transfer
function. At smaller blur values, we can account for the pixel transfer
function with the correction given by Eqn. 35. Additionally, when Gaussian
blur is applied in two stages, we demonstrated that we can find the combined
Gaussian standard deviation by combining the two standard deviations in
quadrature,

$\sigma_{combined}=\sqrt{\sigma_{0}^{2}+\sigma_{1}^{2}}.$ (39)

Finally, we found that blurring with the least squares Gaussian approximation
of an optical PSF does not yield the same RER as blurring with the optical PSF
itself.

## References

  * [1] L. Harrington, D. Blanchard, J. Salacain, S. Smith, and P. Amanik, “General image quality equation: Giqe version 5,” Nat. Geospatial-Intell. Agency, Fort Belvoir, VA, USA, Tech. Rep , 2015.
  * [2] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” Advances in Neural Information Processing Systems 32 , pp. 8024–8035, Curran Associates, Inc., 2019.
  * [3] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods 17, pp. 261–272, 2020.
  * [4] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: Large-scale machine learning on heterogeneous systems,” 2015.  Software available from tensorflow.org.
  * [5] S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX) , pp. 1–6, 2016.
  * [6] S. Dodge and L. Karam, “A study and comparison of human and deep learning recognition performance under visual distortions,” 2017 26th International Conference on Computer Communications and Networks, ICCCN 2017 , 9 2017.
  * [7] R. Geirhos, C. R. M. Temme, J. Rauber, H. H. Schütt, M. Bethge, and F. A. Wichmann, “Generalisation in humans and deep neural networks,” CoRR abs/1808.08750, 2018.
  * [8] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” 12 2019.
  * [9] A. C. Bergstrom and D. W. Messinger, “Understanding the relationship between image quality and convolutional neural network performance,” Pattern Recognition and Tracking XXXIII 12101, pp. 10–24, 2022.
  * [10] P. D. Burns et al., “Slanted-edge mtf for digital camera and scanner analysis,” Is and Ts Pics Conference , pp. 135–138, 2000.
  * [11] R. L. E. Jr, Fourier methods in imaging, John Wiley & Sons, 2010.
  * [12] A. Bergstrom, “relative-edge-response,” 10 2022.  Available at https://github.com/acb08/relative-edge-response.
  * [13] D. Conran, “High fidelity psf/mtf simulations for remote sensing imagers - grss-ieee.”  Available at https://www.grss-ieee.org/events/high-fidelity-psf-mtf-simulations-for-remote-sensing-imagers/.
  * [14] E. W. Weisstein, “Lorentzian function. From MathWorld—A Wolfram Web Resource.”  Last visited on 12/15/2022.